Audio classifier for half duplex communication

[0002] The present invention relates generally to audio communication and in particular the present invention relates to capturing audio data transmissions.

[0003] In many digital communication systems, audio captured at a remote location is delivered to a local location in either a continuous stream of data, or in bursts of data packets. When a continuous stream is delivered, it contains all audio captured at the remote location. When bursts of data packets are delivered, the packets typically contain only speech or music deemed important by the remote endpoint. Thus, the packets containing silence are typically not delivered. These audio packets can arrive at the local location at unpredictable intervals, or may even be dropped, due to unreliable network behavior or audio system behavior caused by heavy loading. These unpredictable delivery patterns make it extremely difficult to design Half-Duplex Open Audio functionality into such systems.

[0004] Traditional half-duplex hands-free audio systems assume that a continuous stream of remote audio is delivered, and that the contents of remote audio can be analyzed using a voice activity detector (VAD) to make meaningful speech/noise classifications on the received audio data. Because remote locations adhering to new protocols attempt to conserve network bandwidth by dropping rather than transmitting unnecessary audio packets, the assumption of continuous data does not hold true on today's digital systems. Thus, the local half-duplex communication algorithms do not get a chance to analyze the content of all the audio captured at the remote location. Half-duplex communication algorithms operating under these conditions either rely on remote speech/noise classifications when determining whether the remote audio should be played at the local site or, play all audio received, under the assumption that all packets received from the remote site contain meaningful audio.

[0005] For the reasons stated above, and for other reasons stated below which will become apparent to those skilled in the art upon reading and understanding the present specification, there is a need in the art for a communication system which allows half-duplex communication in systems receiving either continuous data or packet-based data.

[0006] In one embodiment, a communication receiving device comprising a density measurement device is coupled to receive an input audio signal and provide an output indicating if the received input audio signal contains speech signals based upon a density of the input audio signal. A voice activity detector is coupled to receive the input audio signal and provide an output indicating if the received input audio signal contains speech signals based upon energy levels of the input audio signal. A parser device is coupled to receive the input audio signal and provide an output indicating if the received input audio signal contains speech signals based upon data provided with the input audio signal. A classifier device is coupled to the density measurement device, voice activity detector, and parser device for classifying the received input audio signal.

[0007] In another embodiment, a half duplex switching device comprising an input connection for receiving an input audio signal, and classification module are coupled to the input connection. The classification module provides an output which indicates a classification of the input signal based upon a density of the input signal, an energy level of the input signal, and classification data provided with the input audio signal. A switching device is coupled to the classification module. The switching device determines if the received input audio signal contains speech signals based upon the output of the classification module.

[0008] FIG. 1 is a communication system of one embodiment of the present invention;

[0009] FIG. 2 illustrates one embodiment of a local transmitter/receiver unit according to the present invention;

[0010] FIG. 3 provides an illustration of an example audio sample energy contour;

[0011] FIG. 4 illustrates one embodiment of a histogram of audio arrival; and

[0012] FIG. 5 is a flow chart of one embodiment of an audio density operation.

[0013] In the following detailed description of the preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific preferred embodiments in which the inventions may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, mechanical and electrical changes may be made without departing from the spirit and scope of the present inventions. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

[0014] As stated above, communication systems can transmit audio data as either continuous packets or as packets of audio which have silence removed (bandwidth-conservation). Additionally communication links can transmit all packets received, or may inadvertently lose some packets. Traditionally, half duplex (HDX) schemes that expect a continuous audio stream do not function properly if they receive audio data discontinuously, in bursts. Thus, newer half-duplex communication systems that rely on a remote location classification do not function properly with the old data streaming systems. This incompatibility is because the remote speech/noise classifications are absent. Further in systems that assume that all packets received from the remote site should be played, playback will always be active, preventing locally captured audio from ever being sent to the remote site. Additionally, instabilities in the network which cause unreliable packet delivery can lead to even more classification problems. This is true for both traditional and bandwidth-conservation systems.

[0015] The present invention takes into account not only the contents of individual audio packets, but also the delivery timing patterns for the packets and any remote classification information available.

[0016] Referring to FIG. 1, one embodiment of the present invention is illustrated. The communication system 100 includes a local transmitter/receiver (Tx/Rx) 106 which transmits audio data over a communication line(s) 108. One remote transmitter/receiver is also coupled to the communication line(s). The remote transmitter/receiver can provide either a continuous stream of data 102, or asynchronous data packets 104. It will be appreciated that only one of the remote transmitter/receiver devices illustrated is coupled to the system at once. Both remote transmitter/receiver devices have been illustrated to explain that different types of remote transmitter/receiver devices could be used with the present invention. The transmitter/receiver units can be provided in an audio only system, or in an audio/video system, such as a video conference system.

[0017] FIG. 2 illustrates one embodiment of the local transmitter/receiver unit 200. The invention comprises a packet classification parser 208, audio density measurement unit 206, a voice activity detector 204, and a packet classifier 210. A microphone 230 and speaker 226 are also included for providing local audio and playing received audio signals, respectively. A half-duplex switching device 220 controls communication over the communication line using either amplifier 232 and/or switch 234. A local voice activity detector 224 can also be provided, so that out-going signals are transmitted during a time in which the transmitter/receiver is not receiving voice signals.

[0018] The enhanced voice activity detector (VAD) 204 operates on bursty data streams as well as continuous data streams. During sparse audio delivery, when the audio density drops, the voice activity detector begins looking only for voiced speech packets. The term ‘audio density’ as used herein refers to audio arrival distribution. At these times, a noise floor is assumed to be approximately the minimum energy level received during speech segments. Thus, most packets will be classified as speech during periods of sparse delivery. As the audio density rises, the voice activity detector begins to look for long segments of noise and/or silence. When more noise or silence is detected in the stream, the voice activity detector determines which segments are meaningful to play and which should be dropped. The enhanced voice activity detector provides each packet's speech/noise classification to the packet classifier. The VAD in one embodiment performs both classification methods simultaneously and the packet classifier 210 determines which classification to use.

[0019] The dual, or enhanced, VAD is a hybrid of a sophisticated activity detector capable of detecting speech signals within a continuous stream of audio, and containing an energy parser that can make coarse discrimination between noise signals and non-noise signals. The VAD provides two classifications based upon both a ‘sophisticated’ method and a ‘simple’ method. The sophisticated activity detector can comprise any one of a generic class of voice or music activity detectors, and the method described herein is robust enough for speech applications. The sophisticated method uses a long-term energy level, while the simple method uses a minimum energy level. The decision to act on either of these classifications is the responsibility of the packet classifier, described below. The terms ‘simple’ and ‘minimum’ energy are used interchangeably herein.

[0020] It will be appreciated after studying the present disclosure that the classification methods can be replaced with more sophisticated classifications depending on the applications and system resources. Two audio energy moving averages (short-term, and long-term) are created by the VAD using the following equation, $\mathrm{Energy}=\frac{1}{N}\sum _{n=0}^{N-1}{\left(x\left(k-n\right)\right)}^2$

[0021] Where x( ) is a digital sample of audio data, k is the current time index, and N is the audio sample count determined by N=Window Duration (in seconds) multiplied by an Audio Sampling rate (in samples per second).

[0022] The first moving average computed is a short-term energy average which uses a window duration of about 0.030 seconds. The second moving average computed is a long-term energy average which uses a window duration of about 4 seconds. These two moving averages are similar in magnitude when the variations in the audio signal are small and deviate from one another when the variance of the signal energy increases. If the ratio of the short-term energy divided by the long-term energy is greater than about 2, then speech signals are considered present. This is representative of a 6 dB gain in the short-term energy over a floating background energy. This method is referred to herein as ST/LT (short-term/long-term) or sophisticated classification.

[0023] ST/LT provides an instantaneous classification of the received signals. Once this ratio drops below the value 2, the classifier declares the signal frames as non-speech. In this way it provides a very ‘raw’ classification of audio packets. More sophisticated methods can be added to this approach starting with zero crossings analysis and moving up in complexity to pitch detection and unvoiced speech discrimination methods. These additions can reduce classification errors during continuous audio reception, but are not sufficient when lost packets are occurring due to transport or remote endpoint characteristics. It will be appreciated that the present invention is not limited to the exact time and ratio values described. Using the present disclosure, other values for ST/LT can be developed without departing from the present invention. FIG. 3 provides an illustration of three example audio energy contours superimposed on a sample audio signal. The short term energy contour and the long term energy contour are illustrated.

[0024] Speech is classified by the VAD through a comparison of the short term energy to a short term energy minimum tracked over an approximately 24-second period. A minimum observed short-term energy is latched once per second, and the minimum value is maintained for the entire 24 second sliding window. Outliers, or extraneous data points, are discarded by a single pole smoothing filter. The process of acquiring an energy minimum is as follows:

[0025] 1SecLatched Minimum=1SecLatched Minimum, where 1SecLatched Minimum≦Short Term Energy

[0026] 1SecLatched Minimum=Short Term Energy, where 1SecLatched Minimum>Short Term Energy, and

[0027] 24SecSmoothed Minimum=24SecSmoothed Minimum * β+1SecLatched Minimum * (1−β)

[0028] Where β is chosen as a function of the short-term window duration. In one embodiment this variable is approximately 0.98. This process maintains a smoothed minimum energy over the last 24 seconds of audio.

[0029] If the current short-term energy divided by the short-term minimum is greater than about 2.8 (9 dB) then it is determined that the packet contains speech. Otherwise, the packet is considered non-speech. This method is referred to herein as a Minimum Energy classification, or ‘simple classifier’ classification. Like ST/LT, the simple classifier provides an instantaneous decision without any onset and decay considerations.

[0030] Reference is now made to the packet classification parser 208 of FIG. 2. In general, the packet classification parser extracts a remote speech/noise classification from each packet, if it is present. The packet classification parser also provides an output which indicates that the received packet is either SPEECH, SILENCE, or UNKNOWN (if no classification information exists in the packet).

[0031] The packet classification parser simply tallies the occurrences of Silent Packet information being provided from the remote endpoint. This is a somewhat minor task and is broken out herein as a separate process for modularity. Often, but not always, remote endpoints provide an indication that they have detected silence and will be stopping the transmission of audio until they detect the onset of new speech. This indication is usually contained in external packet header information. The parser tallies the number of times this information indicates Silence over a predetermined time, for example the last 12 seconds, excluding the current 0.500 seconds. This is referred to as a Silence Detection Sum (SD Sum) and is used by the packet classifier in conjunction with audio density characteristics to better determine the true classification, as described below.

[0032] Also, for each connection with a remote endpoint, any single observation of a Silence Classification is latched to assist in the general operation of the Audio Classifier. If the remote endpoint has transmitted a silence indication during the current connection then this indicator is set to TRUE. Otherwise, the indicator remains at a FALSE indication.

[0033] The audio density module 206 provides a measurement of received audio density, as explained in greater detail below. The audio density, or the amount of audio data received in a given period of time, is measured by monitoring when each audio packet is received and incorporating the receipt of the packet into a numerical value which indicates a level of continuousness of streaming. For example, a higher density figure indicates that streaming is more continuous, and a lower figure indicates that the streaming is more bursty. Both short-term and long-term density measurements can be taken, as explained above.

[0034] The short-term density measurement uses a short time window in which the ratio of the duration of audio received relative to the total window time is calculated as a percentage. The duration of audio received is equivalent to the playback time span of the audio packet. The resultant figure indicates the duty cycle of audio during the short window. The long-term density is measured in the same fashion, except that the fixed window is on the order of 10 times longer than the short window. The combination of these two values determines the audio density. The short-term density describes the distribution of delivery, while the long-term density describes the overall average density. Patterns of density behavior can be examined to determine whether any burstiness in the audio streaming may be caused by network problems, or by the remote transmitter/receiver dropping non-speech audio packets. Both the voice activity detector and the packet classifier use the audio density measurements to perform their tasks.

[0035] The audio density measurement provides a rough indication of the arrival characteristics of the audio packets. A histogram is provided for the audio playback ‘duration’ of all packets arriving over the past 12.5 seconds. FIG. 4 illustrates one embodiment of a histogram. The histogram is established by each packet's time-of-arrivals (TOA) into the system. The time resolution is about 0.500 seconds, thus creating 25 bins of 0.500 second duration. Packets arriving into the system are ‘stamped’ with a local system time (this is their TOA). Their audio playback ‘duration’ is summed into the appropriate bin in the histogram.

[0036] It is important to note that the histogram is a sliding 12.5 seconds window. New TOA bins are created on the right-hand side of the histogram as the system time progresses from ‘now’ into ‘infinity’, while bins are dropped on the left-hand side of the histogram as they become ‘older’ than ‘now minus 12.5 seconds’. Because this is being presented as an event-driven process and not a schedule-driven process, new packet arrivals do not occur at regular time intervals. They arrive into the system based on particular characteristics of the remote endpoint and communications link. This behavior makes the sliding window ‘jump’ and ‘pause’ as packets arrive at random TOAs.

[0037] One example arises when a packet arrives after 13 seconds of no packet arrivals. In this case a new 0.500 second bin for the new packet is ‘created’ and the bins for the previous 12 seconds of time are set to zero. All audio histograms older than the 12.5 seconds are thus dropped. The other extreme occurs when a packet arrives in less than 0.500 seconds after the last packet. In this case the previous packet TOA has already been used to create a new 0.500 second bin. The previous packets ‘duration’ has already been added to that bin. When the new packet arrives (for example 0.100 seconds later) its audio duration time is added to the previously created bin. In this way all 0.500 seconds audio bursts are summed into one bin, then the histogram moves to the next bin. This is segmented on 0.500 seconds boundaries of the system timer. This means that in the above example, if the second packet arrives 0.100 seconds after the previous packet, but the system timer has moved from 2.450 seconds to 2.550 seconds, then the second packet's playback duration is summed into a new bin.

[0038] After creation of the sliding window histogram, the audio density measurement updates its running statistics by interrogating (but not interpreting) the past 12 seconds of audio arrival. It excludes the current 0.500 seconds because this data is still being acquired, and passes measured values to the packet classifier for interpretation. The measurements are: $\mathrm{Density}=\frac{1}{D}\sum _{n=0}^{N-1}\mathrm{Bin}\left(n\right),$

[0039] where N is number of bins, D is the total bin duration (12 sec). $\mathrm{Standard}\mathrm{Deviation}=\sqrt{\frac{1}{N}\sum _{n=0}^{N-1}(\mathrm{Bin}\left(n\right)-{\left(\mathrm{Avg}\left(\mathrm{Bin}\left(n\right)\right)\right)}^2},$

[0040] where N is the number of bins (24), Bin(n) is the individual bin summation.

[0041] MaxGap=Maximum consecutive bins with zero sums * Bin Duration (0.500 seconds).

[0042] MaxBin=Maximum bin sum plus greatest adjacent bin sum.

[0043] SumGap=Number of gaps exceeding 0.250 seconds in the last 12 seconds.

[0044] A flow chart 300 of the audio density operation is illustrated in FIG. 5. After new data has been received at 302, the TOA is set as the current system time at 304. The TOA is rounded to the nearest 500 ms and the current TOA window index is set at 306. Audio data playback duration is added to the current TOA indexed time slot at 308. All bins are shifted later in time at 310. The previous 12 second window is averaged at 312, and the Max Bin, Max Gap, and Sum Gap of the 12-second window are calculated at 314. The standard deviation is then calculated at 316. Finally, the audio density is determined at 318 (sum of audio duration/window duration).

[0045] Referring again to FIG. 2, the packet classifier 210 determines whether a current audio packet is eligible for playback. This decision is made by taking into account whatever information is provided by the packet classification parser, the audio density measurement, and the enhanced voice activity detector. For example, if the audio stream is very bursty, all packets received are considered eligible for playback, unless the packet classification parser indicates that the incoming audio is noise or silence rather than speech. On the other hand, if the audio stream is continuous, then the voice activity detector's speech/noise decision is used to determine eligibility. Many other scenarios are possible, with the information from all sources accounted for, to make the best possible playback eligibility decision.

[0046] The audio classifier considers all the information at its disposal before making a final packet classification of the packet. It does not attempt to make HDX transition decisions, just raw classification decisions. Information channels made available to the classifier by the audio density, VAD and the packet classification parser are:

[0047] Audio Density (percentage);

[0048] Audio arrival distribution (Standard Deviation);

[0049] Sum of the audio silence gaps exceeding 250 ms (Sum Gap);

[0050] Max Silence gap (MaxGap);

[0051] Max Burst plus max adjacent (MaxBin);

[0052] Sum of Silence Detection classifications from remote endpoint (SD Sum);

[0053] Latched Silence Detection observed from this remote endpoint (TRUE/FALSE);

[0054] Sophisticated VAD Speech/non-Speech Classification (Speech/Non-Speech);

[0055] Simple VAD Speech/non-Speech Classification (Speech/Non-Speech);

[0056] The use of this information is mostly determined empirically with the basic rule for making a Speech/Non-Speech decision centering on the Audio Density and Silent Detection inputs.

[0057] In one embodiment, the audio classifier operates under the following fundamental Rules:

[0058] 1. If the Audio Density is >0.9 and the latched Silence Detection is FALSE, then the remote endpoint is a Full Duplex endpoint and the sophisticated VAD classification is used outright.

[0059] 2. If the Audio Density is ≦0.9 and the Latched Silence Detection is TRUE, then the packet's classification defaults to speech.

[0060] 3. If the Audio Density is <0.6 and Latched Silence Detection is FALSE, then the simple classification is used.

[0061] There are many other combinations that can result from the available information channels. These combinations are outlined in Table 1 and are used to remove ambiguities when the Audio Density range is between 0.9 and 0.6. The standard deviation (STD) over the past 12 seconds provides a confidence factor for the decision making process. For example, if the STD is large, then the stream is arriving in bursts. If the STD is low, then the audio is arriving steadily or not at all. This value, in conjunction with the Audio Density, suggests the stability of the stream.

[0062] The mere fact that packets arrived into the system is by itself an indication that they should be played on the loudspeakers. For lack of all other information, each packet classification will default to Speech. This is referred to herein as an Arrival classification and is used if there are no other means to classify the audio content.

[0063] The remaining input information is meaningful for detecting outliers. An example of an outlier is as follows: If the SD Sum or SumGap are large then there are too many fluctuations for this audio to represent meaningful speech. In a specific case, if the arriving packets each contain 0.120 seconds of audio data and SDSum over the past 12 seconds registers over 16 (i.e. SD packet arrivals average one every 0.750 seconds), then the remote endpoint is improperly transmitting audio. In this situation the simple classifier is used to sort the valid and invalid signals. Other possibilities are captured in Table 1.

[0064] Initialization of the Density, STD, and other statistics must be achieved before the values are considered for classifying packets. This is especially true when interacting with remote endpoints that are running Silence Detection algorithms. There will be large time slots where no audio will be received. During this time the audio density will drop, the STD will go to zero, and the SD Sum and SumGap will shrink. To properly reinitialize, the classifier will wait for 12 seconds for every method to establish meaningful data. During this time all packets will be declared as speech.

[0065] The final classification is shared with the HDX switching algorithm executed by the HDX switcher 220. This switcher can be any of a general type suitable for managing an HDX audio stream for echo suppression or HDX audio streaming. The classifications described above are raw, and considerations beyond this instantaneous classification may be needed for useful audio switching.

[0066] For example, after a classification transitions between speech and silence has occurred, the classifier should not turn off the audio until approximately 80-120 ms later. Likewise, once the signal has been classified as Speech for longer than 120 ms, it should remain (hang) in this classification for at least 80-180 ms. That is, during conversations there are often pauses contained in speech which should continue to be classified as speech. The half duplex device, therefore, is used to provide flexibility in the receiving device.

[0067] A half duplex switching device has been described which includes an input connection for receiving an input audio signal, and classification module coupled to the input connection. The classification module provides an output which indicates a classification of the input signal based upon a density of the input audio signal, an energy level of the input audio signal, and classification data provided with the input audio signal. A switching device has also been described which is coupled to the classification module. The switching device determines if the received input audio signal contains speech signals based upon the output of the classification module. As such, the communication receiving device can be used in both communication systems which provide continuous speech signals, and communication systems which remove silence and only provide speech signals. The modules of the present invention can be implemented in either hardware, software, or a combination of both. As such, the VAD, audio density module, packet classifier, parser, and HDX switch can be implemented in software executed by a processor. Further, the processor can be operating in response to instructions provided on a computer readable medium, such as a magnetic or optical disc.

[0068] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement which is calculated to achieve the same purpose may be substituted for the specific embodiment shown. This application is intended to cover any adaptations or variations of the present invention. Therefore, it is manifestly intended that this invention be limited only by the claims and the equivalents thereof.