Method and Apparatus for Maintaining Audio Sample Correlation

The present invention pertains to a method and apparatus for maintaining audio sample correlation, and more particularly, to a method and apparatus for maintaining input and output audio sample correlation for echo cancellation.

Referring to FIG. 1, a general block diagram of a speaker phone system is shown. User A has a speaker 21a and microphone 23a while User B has his/her own speaker 21b and microphone 23b. User A and User B are coupled together via a transmission medium 20 such as the Internet or telephone system (e.g., Plain Old Telephone Service or POTS). When an input signal is received at microphone 23a (e.g., when User A speaks), there is a possibility that the sound being emitted by speaker 21a will also be received as an input signal at microphone 23a thus causing an echo at User B. To control this echo, an echo cancellation component (EC) 25a receives the input signal from microphone 23a and the signal being output to speaker 21a. Accordingly, echo cancellation component 25a cancels the signal output by speaker 21a from the input signal at microphone 23a (i.e., eliminates the echo that would be perceived by User B if the input signal did not go through this processing). A similar echo cancellation component 25b is provided for User B.

Referring to FIG. 2, a known system for implementing a speaker phone with echo cancellation is shown. The system includes a coder/decoder (codec) 15 coupled to an audio output device (e.g., speaker 16) and an audio input device (e.g., microphone 17). A central processing unit (CPU) 11 is also provided coupled to one or more buses 10. A component including a device driver 13 is coupled to bus 10 and performs a variety of functions to facilitate the sending and receiving of data buffers to and from codec 15. A memory device 19 such as Random Access Memory (RAM) is coupled to bus 10 and is used to store data that is sent to and from codec 15 by device driver 13. During the operation of the system in FIG. 2, incoming data from transmission medium 8 is received by transceiver 9 and placed in buffer RD. The protocol information from the incoming data is "stripped" away and the data is placed in an output buffer OUT.

The output buffer (OUT) for speaker 16 flows to codec 15 and to speaker 16. If the system of FIG. 2 is used in a speaker phone application, there is a substantial possibility that sound emitted from speaker 16 will be received as input to microphone 17. Correlated input/output buffers (e.g., for time t.sub.1) are created at codec 15 and sent back to memory 19. Referring back to FIG. 2, echo cancellation component 12 operates on correlated samples which includes an audio input sample and the audio output sample which was playing at the time the input sample was taken. These samples are typically provided in correlated buffers stored in memory 19. For example, an input buffer of 256 samples taken at time t.sub.1 (IB.sub.1) can be stored adjacent to a correlated output buffer of 256 samples output at time t.sub.1 (OB.sub.1) in memory 19. Likewise, additional input and output buffers can be stored for times t.sub.2 to t.sub.n. The echo cancellation component 12 accesses the correlated input/output buffers of memory 19 and generates the appropriate output data to be sent to the end user which is placed in buffer SD. The proper protocol information is added to buffer SD and the outgoing data is sent by transceiver 9 over transmission medium 8. In the system of FIG. 2, there are several quantities of data flowing to and from codec 15. Thus, at least three buffers of data (OUT, IB.sub.1, and OB.sub.1) flow between codec 15 and device driver 13. If the codec resides in a modem coupled to bus 10 via a serial channel, the amount of data flowing to and from codec 15 may exceed the available bandwidth available across the serial channel, thus degrading performance.

According to the apparatus and method of the present invention, input/output sample correlation is achieved by first preparing an output buffer which includes a plurality of output samples and appending a first correlation tag to this output buffer that identifies one of the output samples in the output buffer. The output samples are then output from the system while input samples are input to the system. As the input samples are input they are placed in an input buffer and a second correlation tag is appended to the input buffer which identifies the input sample that is input at the time the output sample identified by the first correlation tag is output. The input sample identified by the second correlation tag is correlated with the output sample identified by the first correlation tag, thus providing input/output sample correlation which can be effectively used in echo cancellation operations or the like. By using correlation tags in the input and output buffers, fewer buffers are needed, which improves bandwidth.

FIG. 1 is a generalized block diagram of a speaker phone system using echo cancellation.

FIG. 2 is a block diagram of a speaker phone system known in the art that employs echo cancellation.

FIG. 3 is an exemplary block diagram of a speaker phone system constructed and operated according to the present invention.

FIG. 4 shows an output buffer including a correlation tag according to the present invention.

FIG. 5 shows the selection of values for correlation tags performed according to the present invention.

FIG. 6 shows an exemplary correlation of samples between output buffers and input buffers.

FIG. 7 shows an exemplary construction of a correlated buffer according to an embodiment of the present invention.

Referring to FIG. 3, a block diagram of a speaker phone system constructed and operated according to the present invention is shown. Similar to the system in FIG. 2, a CPU 31 is provided coupled to one or more buses 33 (e.g., running according to the Peripheral Component Interconnect specification, Version 2.1, 1995 (PCI Special Interest Group, Portland, Oreg.)). A memory device 34 is coupled to bus 33 as well as a device driver 35 and a serial controller 36. The serial controller 36, in turn is coupled to a modulator/demodulator (modem) 38 via a serial line 37 (e.g., operated according to the RS-232 standard; Electronics Industries Association). The modem 38 is coupled to a speaker 41 and a microphone 42 via a codec 40. The components within block User A are coupled to User B via a transmission medium 45 such as a POTS telephone system or the Internet.

Data from User B is received from the transmission medium 45 at a transceiver 55 (which can be a part of modem 38) and placed in buffer RD. The Contents of buffer RD are sent across serial line 37, and after any protocol information is stripped, this incoming data is placed into an output buffer 51 having a length of 256 bytes, for example, in device driver 35. According to the present invention, a first correlation tag 52 is appended (e.g., prepended) to output buffer 51 in device driver 35. Referring to FIG. 4, an output buffer 51 with a correlation tag 52 is shown. The correlation tag 52 can be a sixteen bit data structure comprising two eight-bit bytes which includes a sequence number 52a and a sample index 52b. The sequence number 52a refers to output buffer 51, and is incremented for each output buffer sent to speaker 41. The sample index 52b refers to a particular output sample (e.g., an eight-bit audio sample) in the output buffer identified by the sequence number 52a. The value selected for the sample index 52b is somewhat arbitrary, and can be easily set to zero which would refer to the first sample in the output buffer 51 (identified at the left by the numeral "0").

A copy of the output buffer 51 with the correlation tag 52 is stored locally (e.g., in memory device 34). The output buffer 51 and correlation tag 52 are sent to modem 38 via serial controller 36. In this example it is assumed that output buffer 51 shown in modem 38 has a sequence number of 2 and a sample index number of 0. The codec 40 receives the output buffer 51 and correlation tag 52. At the time that codec 40 is converting the output buffer 51 to output signals for speaker 41, codec 40 is also receiving input data from microphone 42. The codec 40 creates an input buffer 53 with a second correlation tag 54 prepended to it having the same structure as the output buffer 51 and correlation tag 52, respectively. As each input sample comes into codec 40, each input sample is placed into a consecutive location in the input buffer 53. For example, the input samples are placed into locations 0-255 in a consecutive manner. At the point that the output sample referred to by the sample index 52b of the correlation tag 52 is output (e.g., sent to speaker 41), the sequence number 52a of the output buffer in which this identified output sample resides is copied into the sequence number field 54a of the correlation tag 54 of the input buffer 53 currently being filled. The input sample that is input concurrently with the aforementioned identified output sample is placed into the input buffer 53 and the location (e.g., a value between 0 and 255 in this example) is placed in the sample index field 54b of the correlation tag 54 of this input buffer 53. The input buffer 54 is subsequently sent to device driver 35 via modem 38 and serial controller 36.

To further illustrate this procedure, reference is made to FIG. 5, where an input buffer 53 and output buffer 51 are shown with their corresponding correlation tags 54 and 52. The sequence number field 52a of the correlation tag 52 refers to the output buffer 51, itself, and has a value in this example of 2. The sample index field 52b refers to the identified output sample of the output buffer 51 and has a value of 0 in this example. When the identified output sample is sent to speaker 41, the value in the sequence number field 52a is placed into the sequence number field 54a of the input buffer 54 currently being filled by codec 40. The input sample that is input at the time the identified output sample is sent to speaker 41 is placed into the appropriate location in the input buffer 53 (in this example location 78). The location value for this input sample is placed in the sample index field 54b of the correlation tag 54 for the input buffer 54 currently being filled.

The input buffer 53 is eventually received at device driver 35 where it is stored either locally or in memory device 34 or the like. The device driver 35 acts upon the input buffers received from codec 40 (e.g., input buffer 53) and copies of the output buffers that were sent to codec 40 (e.g., output buffer 51). Referring to FIG. 6, a collection of output buffers 51'a-d with corresponding correlation tags 52'a-d and input buffers 53'a-d with corresponding correlation tags 54'a-d is shown. The device driver 35 looks at the sequence number field of each correlation tag 54'a-d for the input buffers 53'a-d. This indicates to device driver 35 that an input sample in a particular buffer will match up with an output sample in a buffer bearing the same sequence number field entry. The device driver 35 then looks to the sample index field for the particular input buffer and the corresponding output buffer to find the correlated input sample and output sample. In the example shown in FIG. 6, the sequence number field for correlation tag 54'b is the same as the value in the sequence number field for correlation tag 52'b. Accordingly, a correlated input/output sample pair exists in input buffer 53'b and output buffer 51'b. The location of the correlated input/output sample pair is indicated by the sample index fields in correlation tags 54'b and 52'b. In this example, the output sample in location 0 of output buffer 2 (51'b) is correlated with the input sample in location 78 of input buffer 2 (53'b).

Once a correlation has been made between the input and output samples, a correlated buffer is created for echo cancellation. Such a correlated buffer can be created locally in the device driver or in memory device 34. The appropriate manner for storing the correlated input and output samples is dictated by the operation of echo cancellation component 49 of device driver 35 (of FIG. 3) and component 12 of device driver 13 (of FIG. 1). An example for such storage would be that correlated input/output samples would be stored in an input buffer of 256 samples taken at time t.sub.1 (IB.sub.1) adjacent to a correlated output buffer of 256 samples output at time t.sub.1 (OB.sub.1) in device driver 35. Referring to FIG. 7, an example is shown of how the correlated buffer is constructed. The sample data from buffer 53'b, sample 0 through sample 255 are simply copied into buffer IB.sub.1. Using the correlation established earlier by the correlation tags, sample data from buffer 51'b is copied into buffer OB.sub.1. Because sample 0 of buffer 51'b correlates with sample 78 of buffer 53'b, sample 0 of buffer 51'b is copied into location 78 of buffer OB.sub.1. Each successive sample in buffer 51'b is copied into the next successive location in buffer OB.sub.1, until location 255 has been filled. Because the input samples are taken in at the microphone 42 (FIG. 3) at the same rate as output samples are sent to speaker 41 (FIG. 3), sample data from the previous output buffer 51'a can be used to fill in buffer OB.sub.1, not copied from buffer 51'b. In this example, locations 0 through 77 of buffer OB.sub.1, are filled with data copied from samples 178 through 255 of buffer 51'a.

Once the input and output samples are stored in an appropriate adjacent fashion, the echo cancellation component 49 of device driver 35 acts upon the correlated input/output samples to negate the detrimental effects of output signals from speaker 41 being input at microphone 42. In this example, echo cancellation component 49 sends modified input samples to buffer SD at modem 38. After protocol information has been added, these modified input samples are sent over transmission medium 45 by transceiver 55 to User B where they are eventually output.

The method and apparatus of the present invention provides for a saving of the total input/output bandwidth of the communication between serial controller 36 and modem 38 over serial channel 37. In particular, a correlated input/output buffer need not be sent back from codec 40 in order to present correlated input/output samples to echo cancellation component 49 of the device driver. Instead, only a slight increase in the input bandwidth and output bandwidth is seen to compensate for the correlation tag prepended to the output and input buffers. The present system is also robust in that if any buffer is lost or damaged, the correlation is re-acquired with every good input buffer received from codec 40.

One skilled in the art will appreciate that numerous modifications within the scope of the present invention can be made to the apparatus and method of the present invention described above. For example, the present invention can be used to provide echo cancellation in a speaker phone application over a transmission medium such as a telephone system (e.g., POTS) or the Internet system. Also, the present invention can be used to provide echo cancellation in a video conferencing application.

Although several embodiments are specifically illustrated and described herein, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.