The project and the contents of this report are entirely the original work of the authors except as listed in the REFERENCE section and as follows

This document describes the design of the interactive Audio Manipulation Processor (iAMP) project for Electrical Engineering 552. The iAMP allows a user to locate a monophonic audio source in the two-dimensional plane of a four-speaker audio system using a graphical user interface. The iAMP system consists of four speakers, ideally placed in the four corners of a room, a VGA monitor and joystick, an audio amplifier capable of amplifying the four channels, and a digital system of our design with four analog RCA output jacks and one analog RCA input jack. An audio source, such as a compact disc player or computer, is connected to the audio input. The four audio outputs feed through the audio amplifier to each of the four speakers.

The iAMP in its entirety is too large of a design to fit in the supplied Altera EPF10K20 device. Hence, a scaled-down design without RAM was implemented.

In the end, the iAMP design was too large to fit onto the EPF10K20 device on the Altera UP1 board. It was necessary to scale down the design by removing the RAM driver and removing the RAM-related parts of the ADC-RAM-audiopath interface entity.

The video output operates properly, on the whole, as does the joystick input. The DAC driver sends valid data to the DAC. As well, the handshaking between the RAM, ADC, and audio control path works correctly. The audio manipulation code successfully handles the task of splitting an input signal among the audio outputs of the system.

The interactive Audio Manipulation Processor system block diagram is as follows:

The input signal is produced from any analog audio source, i.e. CD player, computer, radio, turntable, etc. The input signal is connected to the analog input of an Analog-to-Digital Converter (ADC), which converts the analog signal into a stream of two's-complement 24-bit digital samples, of which only the 8 most significant bits are used. The digital samples of the audio source are then multiplied by a gain between 1 and 0 for each of the four speaker output streams depending on the X and Y locations currently selected by the user. These samples are then sent to Digital-to-Analog Converters (DACs) that convert the digital signals into analog signals, that are then amplified and produced audibly at the speakers. The effect of this process is to create sound directionality using the differences in intensity between the outputs of the four speakers. For a discussion of how humans perceive sound, see Appendix A of the Resource Requirements document.

The user interface consists of a joystick and a VGA monitor. The center of the screen features an icon denoting it as the center of the square bounded by the four speakers. A sprite cursor appears onscreen, controlled by the joystick. Pushing the joystick will cause the cursor to move accordingly, which adjusts the X and Y location of the output signal. Moving the cursor to the top-right corner of the screen, for example, will pan the sound to the front-right speaker of the room. In a similar fashion, moving the cursor to the center of the screen will result in an equal sound level from each speaker, thereby placing the location of the sound at the center of the room. This will require hardware instructions to redraw the screen regularly through the 5-pin VGA connector as well as a joystick driver that polls the joystick periodically for input.

The operation of the audio is split into three parts: the analog-to-digital converter interface, the audio processor and the digital-to-analog converter interface.

The analog-to-digital interface receives sample data from the Crystal CS5360 ADC and places it into RAM. The data captured is 24-bit two’s-complement integers. We reduce it to 8 bits, and pass the value to the RAM interface, which places the sample on the sample stack. The Audio Manipulator sequentially asks for eight samples from the RAM interface, providing a 5-bit delay value and setting the "Sample_Req" signal high. The Ram_interface multiplies the delay by a coefficient (i.e.64) and retrieves sample S(0 - 64*delay), where sample S(0) is the current sample. All eight samples are placed into a register bank for pipelining. (As is described below, the RAM feature was removed due to the contraints on the available space on the EPF10K20 FPGA on the UP1 board. However, the audio system is not aware of this and processes data as described here)

All eight samples are then multiplied by a gain value and placed into a second register bank. The four 'incident' samples and the four 'echo' samples are summed with each other, producing a resultant four samples, one for each speaker. These are then passed to the DAC which produces a corresponding audio signal.

A block diagram describes the audio path below:

The basic sound algorithm is as follows:

The echo delays and the gain coefficients are calculated based on the current X and Y location of the cursor. Gain is calculated by the inverse square of the distance between the location designated by the cursor and the speaker -- near the speaker, the gain will be high, and away from the speaker, the sound is much lower. (Since we calculate our gain as a value between 0 and 1, it would more accurately be called a loss coefficient.) The distance square calculations are complex and required a large amount of space on the FPGA, and as such we found we could not implement them without using up an overly significant amount of space on the UP1. As a result, we designed a simplified first-order equation using a 16x16 location grid:

(Note that the values are divided by 16, so the values are between 15/16 and 1/16. The top left corner is the corner nearest the speaker.)

Gain = 15 - (MAX(X,Y) - 1)

Echo delay similarly requires a large amount of calculation. Our calculations are based on a radial distance from the speaker. Assuming that the speakers are set in the corners of the room (assumed square) and that the sound comes from the location that the cursor marks, our delay is the 'echo' produced when the sound waves reach that speaker and 'reflect' back towards the listener. If the sound is near one speaker, the sound will reflect off that corner sooner than the opposite corner. This also required a significant amount of calculation (pathygoras' rule, in particular.) As a result, we found through least-squares method the following linearization that most accurately matches the time delay:

(These are in time-delay units. These are multiplied by a value (such as 64 or 256) to get an appropriate time delay.)

Delay = X + Y -2.

These tables are similar for all four speakers. The gain coefficients for the echoed signals were simply half of the incident gain for the same speaker.

Interface with top-level VGA entity

Timing and count signals

The internal structure of the video entity is shown below.

Constructing the onscreen image

The onscreen image will be somewhat like that shown in the following figure.

Objects are represented by blocks of colour. The image is constructed by colouring pixels in order of priority. The joystick/sound source cursor has the highest priority – the colour of its icon will always take priority over any other part of the onscreen image. The background, the "you are here" marker, and the speaker markers are of the lowest priority. Pixels will only take on the colours of these elements when there is no cursor on top of them.

All the necessary information in creating a ram driver came from the RAM data sheet [14]. The data buses, both internal and external, are bi-directional. The bidirectionality of the buses were implemented using an entity obtained from Altera [11]. Through the use of a state machine the control of the timing of the driver interface became much simpler.

Each state of the machine corresponds to the matching point on the timing diagram of when the different signals become valid. Information passed to and from the driver is more reliable due to the handshaking implemented into the design. See the RAM test bench section for more details.

The following text is adapted from Stephen’s final report for EE 582 (reference [10]).

Due to delays in getting a working mouse driver and due to the large code size of the mouse driver in progress, we chose to forgo use of a mouse as an input device and selected a joystick interface instead.

The Atari joystick uses a female DB9 connector, the pinout of which is shown below (from reference [9]). The diagram shows the connector when held facing you.

Chip used = EPF10K20RC240-4 on UP1

logic cell usage = 100%

EAB usage = 0

speed ratings = 25.175Mhz

The following table summarizes the chip resources consumed by each part of the design. Most of these tests are based on isolated unit tests of each component, since many components had not yet been integrated in the main entity. Table entries describe the resource usage of two or more integrated components are described following the table. Elements listed after the "totals" row were not counted toward the totals, since they overlap with the elements above the "totals" row. They are listed only to give an idea of how much space is devoted to subcomponents of the design.

Note that the entire iAMP system could not fit into the space available on the EPF10K20.

The complete functionality of the iAMP could not be realized on the 10K20, so we elected to remove the ram driver entirely and scale down the ram_interface component. The new ram_interface2 component does not actually interface with RAM, but rather passes the most recent samples to the audio manipulator regardless of whether the audio manipulator wants a delayed sample or not. In this way, the audio manipulator code did not have to be revised. The following table shows the logic cell usage breakdown for the final build that was implemented on the 10K20.

Due to all the features we initially wanted to implement many "experiments" were conducted to reduce the size of all vhdl components. Actually in most cases the actual vhdl module itself was cut. Many features such as RAM, and echo had to be removed. In the case of the remaining modules there size was reduced tremendously by having the sampling size decreased to 8 from 24 bits. This allowed us to include features we had to have such as the ADC, 2 DAC and graphical volume control. I could see many projects in themselves of the modules we cut away. The initial idea of having mouse control, now joystick control, would be a great deal of work to reduce the size needed in this project. Attempts were made to decrease the size enough but in the end it was not included even though this had been a component in which many hours of design, debug and testing have gone.

We decided to compare this complicated state-machine based technique versus a more direct method. In fact, we chose to try using the "*" operator and letting MAXPLUS II perform the design for us. The results were what we expected, i.e. much larger but much faster. The multiplier took 56% of the logic gates, and the maximum delay between input and output is 89 nanoseconds, much quicker than our sequential multiplier.

While the "*" operator method results in times almost 20 times faster, the multiplier uses up over half the space, unacceptable for our design. As a result, we find that our multiplier design is acceptable for our audio processor.

(the sample clock period is 1/44100, or 22.7 microseconds.)

ADDENDUM: We realized that we did not need such a complex multiplication, since this circuit was meant for 'mixing,' not realizing that the audio mixing process is in fact a summation! A multiplier is used (multiplying a sample by a 4-bit gain value, then right-shifting by 4 bits to divide by 16. This was created out of adders, resulting in a small size.)

Initially, we were going to use 24-bit samples with full echo (using all eight samples). Once the entire audio_manipulator.vhd entity was compiled, we found that it used up 1314 Logic Cells, which is roughly 200 Cells more than the 10K20 has available! A close examination of the circuit revealed that most of the bulk was made up of 24-bit registers. Thus, to reduce the size of our design, two options were available. The first option was to reduce the sample length to 16 or 8 bits, although the sound quality would decrease. The second option was to remove the four 'echo' signals, thus removing half the registers and eliminating the need for the summers and some of the controller logic. The following chart shows the size of each of these options with and without echo, for different sample lengths.

As we can see, the without-echo version is almost 40% smaller than the equivalent with-echo version. Each 8 bit jump is a 32% gain for the with-echo case, and an 18% gain for the without-echo gain. Regardless, this portion is significant in all versions.

The testing of the video code was very straightforward. After compiling the code, the generated SOF file was downloaded to the UP1 board and a VGA monitor was connected to the board. The code was debugged until the video on the monitor appeared as desired. It was noted that the same code would produce slightly different results on different monitors, that is, the image would be shifted with respect to the edges of the screen. On one of the three monitors used, red, green, and blue artifacts that were less than a pixel in size appeared onscreen. Neither of the other two monitors did this.

Due to the simplicity of the joystick code, no test waveforms were made. Rather, the code was immediately tested on with the hardware by observing the motion of the onscreen cursor in response to the joystick inputs.

The audio coordinate converter (file AudioCoords.vhd) takes the VGA display coordinates of the onscreen cursor, which occupies a 640 by 480 pixel space, and converts them to the smaller coordinate space that is required by the audio manipulator. This space is 15 by 15 squares.

The algorithm basically chops the 640 by 480 pixel space into a set of 15x15 blocks, and determines which block the cursor is located in, then outputs the new coordinate values to the audio manipulator. 480 divided by 15 is 32, which, being an integer, is a very easy number to work with. However, 640/15=42.667. Nonintegral numbers are not easy to deal with in a digital system.

version 1: lookup table comparisons

Our first approach was to use huge 15-condition if-else statements to sample down the X-Y coordinates of the sound source. If the Y coordinate fell between 0 and (32-1), the sound source was assigned a new Y coordinate of 1. If the Y coodinate fell between 32 and (64-1), the source was assigned a new Y value of 2, and so on. It was slightly more difficult to deal with the X coordinate. What we did was to come up with a series of multiples of 42.667 and placed the rounded-down copies of these multiples into the lookup table. The first fruits of our labour are shown in the code listing audiocoords.vhd.first_version, which is on the following page.

This approach consumed 71 logic cells.

version 2: counters and comparators

The previous method was inelegant, so we chose to try a different technique. This time, we chose to use counters that incremented in steps of 32 (for the Y coordinate) or steps of 42.667 (for the X coordinate). A variable would hold the previous value of the counter, and the coordinate value would be compared to the new and old counter values in order to determine what the downsampled coordinate should be.

The astute reader will notice that we could not really count increments of 42.667. Rather, we implemented logic that would set the counter value to the closest whole number to the ideal value.

The results of this approach are shown in the listing audiocoords.vhd.second_version. This time, 75 logic cells were used. This was not very surprising, considering the logic that had to be implemented to make the X counter work.

version 3: a combination of methods

The counter method seemed very clean when used for the Y coordinate, but not for the X coordinate. What if we tried to use the table lookup method for the X coordinate but continued to use the counter method for the Y coordinate? Listing 3, our final version of the code, implements this idea. Indeed, this method seemed to be optimal as only 67 logic cells were consumed.

Index to Test Waveforms

Note: No waveforms are available for the joystick and video components since these were tested directly on the UP1 board. Also, no tests for the ram_port, adc_port, and dac_port components were possible since a bug in MAX+plus II causes "inout" signals" to be converted into two separate signals – an "in" signal and an "out" signal. The MAX+plus II simulator considers these two split signals to be completely independent, and hence cannot be used to reliably simulate the above named components.

ac_gain_apply.scf

This waveform shows two clocks - the system clock (~12.5 MHz) and the sample clock (normally 44.1 KHz but for display purposes such time length is unnecessary.) When sample_clock goes high, reg2_enable will rise and fall, then sample_select will increment, up to the set limit (in this case, three.) When sample_clock goes low, the system returns to its original state.

ac_get_samples.scf

This waveform shows two clocks - the system clock (~12.5 MHz) and the sample clock (normally 44.1 KHz but for display purposes such time length is unnecessary.) When sample_clock goes high, sample_req with go high and a delay will be selected (reg0_load is the MUX control codes on the outputs of echo_delay.) when data_ready goes high, implying that the data is ready, reg0_enable will momentarily rise. The reg0_load will increment after data_ready returns to a low value.

adc_port.scf

The period of clock is 40 ns, the m_clock is 160 ns, the s_clock is 40 us, and the lr_clock is 1.92 ms. S_data is clocked in MSB first for 24 consecutive bits. We are only using one input of the ADC, the left channel, therefore we use all data inputted while the lr_clock is high. After the data is collected we have 1.92 ms before the next rising edge of the lr_clock. This is plenty of time for the small handshaking used to verify the output of adc_port.vhd.

audio_controller.scf

This simulation shows a number of different things. Since this contains echo_delay, incident_gain, ac_gain_apply, and ac_get_samples, we see the interaction of each of these signals. When sample_clock goes high, sample_req goes high and four samples will be received by the audio path (controlled with reg0_enable and reg0_gain.) we now have a delay associated with each sample, which are marked down on the signal diagram.

audio_manipulator.scf

This is the entire audio path. Samples are requested using the sample_req signal and acknowledged with the data_ready signal (for each clock pulse, the input sample is identical). After three sample_clocks, output starts coming out. Since the X and Y locations are in the center (8,8) each sample has been retrieved at delay 14, and all output samples are exactly one-half of the input (this is on purpose.) We have an incrementing input sample, showing the pipelining occuring through the audio path. As each input signal rises by 2222(base 16), our outputs rise by one half. This is just a demonstration of the audio path in operation.

dac_port.scf

The period of clock is 40 ns, the m_clock is 160 ns, the s_clock is 40 us, and the lr_clock is 1.92 ms. S_data is clocked out MSB first for 24 consecutive bits. We are using both outputs of the DAC therefore all data inputted before the lr_clock changes state. Data is collected and we have 160 us before the changing edge of the lr_clock. This is plenty of time for the small handshaking used to verify the input of dac_port.vhd.

echo_delay.scf

This takes in X and Y values and produces four outputs, T00, T01, T10, and T11, all five-bit unsigned integers. We can see here that it is, more or less, combinational logic.

Gain_mult.scf

This is a multiplier: Z = A*B/16. Here, A is 1225689, and B varies between 0 and 15. The output varies accordingly.

incident_gain.scf

This takes in X and Y values and produces four outputs, A00, A01, A10, and A11, all five-bit unsigned integers. We can see here that it is, more or less, combinational logic.

AudioCoords

The UI components define the onscreen location of the sound source in terms of X and Y coordinates in a 640x480-pixel grid. However, the audio manipulator can only handle the sound source location in terms of a 15x15 grid, with a coordinate of (7,7) being in the center of the speaker array and with a possible distance, (0,0) being at the front left speaker, (15,15) being at the back right speaker, etc. The AudioCoords component maps the screen coordinates to the 15x15 grid components. For more detail on how this was achieved, refer to the section," Optimizing Code Size of Audio Coordinate Converter".

Consider the Y coordinates. 480/15 = 32. Therefore, a pixel Y coordinate of 0 to 31 is mapped to 1 on the audio grid, and pixel Y coordinate of 32 to 63 is mapped to 2 on the audio grid. Things are more complicated for the X coordinates, since 640/15 is not an integer. The mapping scheme for the X coordinates is given in the code file AudioCoords.vhd.

The overall design hierarchy of the project is described in the following figure. Arrows indicate the direction of signals between components.

An alternative representation of this design hierarchy is given by the following tree diagram of VHDL entities. An entity that is pointed to by an arrow is a component of the entity that the arrow originates from.

ac_get_samples.vhd

Part of audio_controller.vhd. This is a state-machine for sequentially retrieving samples from the audio sample controller. Simulated, no known bugs.

ac_gain_apply.vhd

Part of audio_controller.vhd. This is a state-machine for sequentially multiplying each sample by its respective gain coefficient.

adc_port.vhd

Interface to ADC. compiled – no errors

and2.vhd

A 2-input AND gate. Simulated, no known bugs.

AudioCoords.vhd

Takes the VGA coordinates of the sound source, which is a 640x480-point space, and converts them to a smaller set of coordinates in a 15x15-point space. Simulated, no known bugs.

audio_controller.vhd

This is the controller for the audio path contained in audio_manipulator.vhd. It calculates gain and delay values for each speaker based on X and Y coordinates and sequentially retrieves samples from the audio sample controller and multiplies each sample by its respective gain coefficient. Simulated, no known bugs.

audio_manipulator.vhd

This is the audio path. Consisting of the parallel audio path and the controller, this retrieves samples from the audio sample controller and applies gain and delay, outputting four samples to the DAC controller. Simulated, no known bugs.

ColourChooser.vhd

Determines the colour of the pixel at the current X-Y coordinates of the VGA scan. Simulated with no known errors.

constants.vhd

Contains constants common to various parts of the design. Simulated, no known errors.

count_xy.vhd

Provides two counts that are used to time the horizontal and vertical sync signals to the VGA monitor. Taken from reference [7]. Simulated, no known errors.

dac_interface.vhd

Takes the four 8-bit samples from the audio path and passes them to the DACs. Not compiled.

dac_port.vhd

Interface to DAC. Compiled, no errors

echo_delay.vhd

Calculates delay information based on X and Y (the delays are 5-bit unsigned integers, which are then multiplied by the audio sample controller by 128 to calculate how many sample back it must retrieve the sample from.) Simulated, no known bugs.

gain_mult.vhd

Multiplies a sample by the gain (4-bit integer, 1-15) and divides by 16. Simulated, no known bugs.

GenerateSlowClocks

Creates clock waveforms slower than the 25.175MHz system clock. Simulated, no known bugs.

Incident_gain.vhd

Calculated gain information based on X and Y coordinates. Simulated, no known bugs.

joystick.vhd

The joystick driver software, used as a user input device in lieu of the mouse driver. Simulated and tested, no known errors.

mymux.vhd

4-input MUX of variable width. Simulated, no known bugs.

myregister.vhd

n-bit register. Simulated, no known bugs.

myselector2.vhd

ram_interface.vhd, ram_interface2.vhd

ram_interface was the state machine responsible for performing the handshaking and sample-passing between the ADC, audio manipulator, and RAM. After we determined that the RAM code was using too much space, we produced ram_interface2.vhd, which is actually nothing more than a direct pipe between the ADC and the audio manipulator. ram_interface.vhd: simulated, no known errors; ram_interface2.vhd: compiled, no known bugs

ram_port.vhd

Interface to external SRAM; compiled – no errors

syncgen.vhd

Takes the counter signals from count_xy in order to generate the synchronization signals for the VGA monitor. Taken from reference [6]. Simulated, no known errors.

ui.vhd

The top-level user interface entity that ties the mouse/joystick interface with the graphics and audio components. Simulated, no known errors.

video.vhd

The top-level video interface entity. Simulated and tested, no known errors.

Overview of RAM Queue

The entire purpose of having RAM in the system is to provide a memory of past audio samples from the ADC. This memory would allow the system to handle effects such as echo, reverberation, and binaural phase difference.

Overview of Handshaking Model

A summary of the functions of these signals is given below:

The RAM Interface Test Benches

Waveform TB1 depicts the results of the test bench. Areas of note are labeled on the waveform, and these labeled portions are explained below. The TB2 series of waveforms provide a more detailed view of the information in waveform TB1.

A

(Recall that only the most significant 8 bits of the 24-bit audio sample are used by the iAMP)

B

C

Here we see the second write sequence. Since address 0x00000 is now full, the ram interface chooses the next address, 0x00001, to store the next sample in. Indeed, the most significant byte of the latest sample is pushed into the RAM.

D

We see here the result of running past the maximum allowed RAM address. The address pointer simply loops back to the starting address and the first sample is then overwritten.

A

B

This is another case where the circular nature of the RAM queue comes into play. For a delay of zero, address 0x00000 would be read from. However, the delay is 1, so we read 0x00003, the location 2 addresses back from the head of the queue.

C

This example is very simple. We read from address 0x00003 – 2 – 1.

D

Here we see a delay whose size is just short of the total number of addresses. The effect is that the sample read is the same address that the next sample will be written to.

E

We have come full circle – once again the delay is zero.