Multi-source phoneme classification for noise-robust automatic speech recognition

[0001] This application claims priority to U.S. Provisional Patent Application No. 60/380,614 entitled “Noise-Robust Methods for High-Performance Automatic Speech Recognition” filed May 14, 2002, which is incorporated herein by reference for all purposes.

[0002] The present invention relates generally to signal processing, and more specifically, to a speech recognition system.

[0003] Current automatic speech recognition systems continued to show-difficulty in processing speech in multi-speaker and high noise environments despite the availability of substantially increased processing power. To achieve high-performance speech recognition in multi-speaker and high noise environments, a speech recognition system that uses the same cues for recognition and noise robustness that human beings do is needed. Such a system should be based on detailed neurobiological and psychoacoustic knowledge of human auditory function, accomplishing noise robustness via auditory stream separation and by using noise-robust phonetic cues.

[0004] The human approach to noise robustness is based on a high-resolution spectral analysis, followed by intelligent groupings of fine-grained sound features that can be ascribed to a common source. By contrast, conventional speech recognition systems achieve the groupings of fine-grained sound features by indiscriminately blurring them together in a 22-point mel-scale filterbank and a 10-20 millisecond frame. This approach works passably in quiet environments, but is the major limiting factor preventing conventional recognizers from achieving noise robustness—once the signal features have been blurred in with the other sounds, they can never be recovered.

[0005] A system and method are disclosed for processing an audio signal including separating the audio signal into a plurality of streams which group sounds from a same source prior to classification and analyzing each separate stream to determine phoneme-level classification. One or more words of the audio signal may then be outputted which may be utilized for further processing.

[0006] According to exemplary embodiments, a high-performance, noise-robust real-time speech recognition system based on biologically verifiable models of the human auditory pathway is provided. The noise-robust speech recognition system may comprise a performance target of 99.5% word accuracy in quiet conditions and robust performance (e.g., accuracy degradation of less than 1%) at a speech-to-noise ratio of 10 dB.

[0007] The high spectro-temporal resolution of the cochlea allows local regions of high signal-to-noise ratio to be isolated from the other sounds and to be intelligently combined for optimal overall performance. The cochlear resolution is fine enough to resolve individual harmonic components and glottal pulses, to track fast formant transitions during plosive bursts, and to detach timing differences on the order of a few microseconds for fine positioning of sound sources.

[0019] It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication links. It should be noted that the order of the steps of disclosed processes may be altered within the scope of the invention.

[0020] A detailed description of one or more exemplary embodiments of the invention is provided below along with accompanying figures that illustrate by way of example the principles of the invention. While the invention is described in connection with such embodiments, it should be understood that the invention is not limited to any embodiment. On the contrary, the scope of the invention is limited only by the appended claims, and the invention encompasses numerous alternatives, modifications, and equivalents. For the purpose of example, numerous specific details are set forth in the following description in order to provide a through understanding of the present invention. The present invention may be practiced according to the claims without some of all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention is not described in detail so that the present invention is not unnecessarily obscured.

[0021] According to exemplary embodiments, a high-performance, noise-robust real-time speech recognition system based on biologically verifiable models of the human auditory pathway is provided. The noise-robust speech recognition system may provides a performance target of 99.5% word accuracy in quiet conditions and robust performance (e.g., accuracy degradation of less than 1%) at a speech-to-noise ratio of 10 dB. A technical approach to the exemplary noise-robust speech recognition system may begin with a working speech recognition system based on established technology, and then carefully replacing and/or augmenting the exemplary noise-robust speech recognition system as needed to improve performance to the target levels, using detailed knowledge of neurobiology and psychoacoustics as a guide.

[0022] The exemplary noise-robust speech recognition system comprises key innovations in the technical approach, as compared to the standard approach, including: replacing a low-resolution Fast Fourier Transform with a high-resolution cochlear model, as necessary to support subsequent stream separation and phoneme identification steps; achieving noise robustness via high-resolution noise-threshold tracking; achieving noise robustness via auditory stream separation prior to speech-specific processing; achieving higher phoneme-level classification accuracy by providing higher-salience inputs to the classifier, including spectral envelope, transient detection, and pitch/voicing; achieving higher phoneme-level classification accuracy by normalizing for speaker characteristics prior to classification; achieving higher phoneme-level classification accuracy by training with a full (not reduced) phoneme target set; and achieving higher word-level accuracy by incorporating a model of syllabic stress.

[0023] A block diagram of an exemplary noise-robust speech recognizer is shown in FIG. 2(b), alongside block diagrams of the human auditory pathway (FIG. 2 (a)) and a conventional speech recognizer (FIG. 2 (c)) for reference. The exemplary noise-robust speech recognizer is designed to efficiently model known properties of the human auditory pathway, while building on and extending successful components of conventional recognizers.

[0024] In exemplary embodiments, the low-resolution Fast Fourier Transform is replaced with a high-resolution cochlear model, as necessary to support subsequent processing steps. Conventional recognizers typically compute a low-resolution spectrum (e.g., 64 spectral values, updated once every 10 milliseconds) based on the Fast Fourier Transform (FFT). By comparison, the human cochlea computes 3000 spectral values with 6 microsecond resolution. The noise-robust speech recognizer computes 600 spectral values, on a logarithmic frequency scale, with 6 microsecond resolution post-interpolation. The noise-robust speech recognizer is, in exemplary embodiments, designed to obey critical bandwidth curves, Fletcher-Munson loudness/compression curves, and matches latencies found in biological cochleas. In some embodiments, a minimum spectral resolution of 60 taps/octave (5 taps per musical semitone) is necessary to separate multiple speakers based on spectral pitch cues. Fine temporal resolution is necessary both for binaural inter-aural timing difference cues (e.g., separating sound sources based on position), for detecting glottal pulses as a temporal-pitch grouping cue in voiced sounds such as vowels, and for analyzing transients for higher phoneme classification performance on plosives.

[0025] In exemplary embodiments, noise robustness is achieved via high-resolution noise-threshold tracking. Conventional recognizers use bandpass filtering of cepstral coefficients (RASTA) or cepstral means subtraction to accomplish rudimentary noise robustness. By contrast, the human auditory system uses fine tracking of recent noise floor and adjustment of gain, in a feedback loop from the cochlea through the multipolar cells (MC), to the ventral nucleus of the trapezoid body (VNTB), and back to the outer hair cells of the cochlea through the olivocochlear bundle. In exemplary embodiments, the noise-robust speech recognizer uses noise-threshold tracking to adjust gain and set noise floor reference levels, so that the distributed but locally-concentrated energy in the speech signal can be detected in its regions of greatest signal-to-noise ratio.

[0026] Noise robustness may be achieved via auditory stream separation prior to speech-specific processing. Conventional recognizers completely fail when other nonstationary sounds are present in the environment. For example, other people talking in a background, impulsive noises such as explosions, and moving FM sounds such as sirens, are simply blurred into the low-resolution filterbank/cepstrum, thus corrupting the speech signal. The human auditory pathway computes many fine-grained sound features in the auditory brainstem, for feature grouping and separation into streams in a cortex. The noise-robust speech recognizer computes the same features (e.g., spectral energy, fine timing of spectral peaks, transient detection, spectral edges, binaural cues such as inter-aural time and level differences, etc.) and uses principles of auditory scene analysis (e.g., common onset, common FM, common AM, etc.) to group sounds that come from the same source. This allows sounds from multiple simultaneous sources to be separated from each other prior to recognition. This grouping process, operating on fine-grained, separable features, followed by spectral envelope detection within a grouped stream, replaces an “ignorance model” afforded by the mel-scale filterbank.

[0027] In exemplary embodiments, higher phoneme-level classification accuracy may be achieved by providing higher-salience inputs to the classifer, including spectral envelope, transient detection, and pitch/voicing. Existing recognizers use a cepstrum and its temporal derivatives, updated once every 10-20 ms, as a sole input to phoneme classification. This approach has led to some success because it provides a rudimentary invariance to pitch/voicing by throwing away pitch information, and extracting an estimate of spectral envelope. In contrast, the human auditory pathway uses multiple parallel pathways to extract spectral envelope, after stream separation.

[0028] The exemplary noise-robust speech recognizer extracts an accurate estimate of spectral envelope based on pitch/voicing cues, and on a much finer time-scale (e.g., 1 ms) so as to be able to track fast formant changes associated with consonant closures. These fact formant changes have been recently found to be vital cues for accurate detection of consonants in noise. Similarly, accurate determination of pitch/voicing is necessary for detecting voice-onset-time (e.g., a cue for distinguishing voice/unvoiced plosives such as P/B), for prosody and syllabic stress in English, and for word identification in languages such as Mandarin. Accurate spectral and temporal characterization of transients is necessary for identifying and distinguishing plosives. Taken together, these cues provide sufficient information to allow a phoneme classifier, according to one embodiment, to make the acoustic-phonetic distinctions necessary for high-performance recognition.

[0029] Higher phoneme-level classification accuracy may be achieved by normalizing for speaker characteristics prior to classification. Formant frequencies are dependent on vocal track dimensions, which are highly speaker-dependent. For example, men have lower formant frequencies, while women have intermediate formant frequencies, and children have the highest formant frequencies. Existing recognizers ignore this fact and attempt to create a single set of classification regions, with the result that existing recognizers are not robust to speaker variation. The exemplary noise-robust speech recognizer uses psychophysically-justified cues to build an adaptive expectation of speaker formant ranges, and normalizes prior to classification, to provide greater robustness to speaker variation.

[0030] Higher phoneme-level classification accuracy may be achieved by training with a full (not reduced) phoneme target set. In the late 1980's, when computing power was more limited, it was found that a performance improvement could be achieved by reducing a target phoneme set in a training phrase. For example, a stop closures (pcl, tcl, kcl, dcl, gcl) could be mapped to silence <.pau>, with no apparent degradation in performance but a reduction in training time. In another example, non-speech sounds such as inhalations and sniffs are often mapped to h#, (a symbol for pre-speech silence), when in fact these sounds are detectable audio events that humans clearly identify and then disregard. Part of the poor performance of existing recognizers can be ascribed to training with insufficient specificity in the target phoneme set. By using every available phoneme in the training set, the exemplary phoneme classifier may be configured to determine the right distinctions.

[0031] Higher word-level accuracy may be achieved by incorporating a model of syllabic stress. Current recognizers are based entirely on phoneme strings, with only a nod to syllabic stress (e.g., existence of stress-reduced phonemes such as axr for the unstressed “er” sound in “butter”, as compared to er for the stressed “er” sound in “bird”—where these classes are often folded together in a reduced phoneme set. The exemplary noise-robust speech recognizer is trained on data with syllabic stress identification, and is designed to use syllabic stress as an additional cue in a Viterbi search.

[0032] Exemplary embodiments of the present invention optimize initially for highest possible recognition performance and defer cost optimization until the performance targets have been met. The conceptual framework for existing recognizer architecture was formulated in the 1970's and 1980's, when computing power was extremely limited. At that time, there was not sufficient computing power to make progress with the high-resolution techniques utilized by exemplary embodiments of the present invention. However, many researchers saw the progress that was made by early bandwidth reduction in the computational pipeline, and concluded that throwing away information was a necessary step for recognition.

[0033] The exemplary noise-robust speech recognizer is configured to preserve and extract as much information as possible, to achieve the targets of high recognition performance and noise robustness. These computations are necessarily more expensive than the conventional approach. Premature optimization for cost has been a contributor to current performance limits in speech recognition. As a result, the exemplary noise-robust speech recognizer uses efficient algorithms and methods to achieve optimal performance, and exploits the existence of modern high-performance DSP technology and the power of Moore's Law to justify optimizing first for accuracy and noise robustness, then later optimizing for cost and computational complexity.

[0034] Exemplary methods based on exemplary embodiments of the present invention are provided below.

[0035] High Resolution Cochlear Model.

[0036] A first stage of processing comprises having adequate resolution to support stream separation and feature extraction for high-accuracy phoneme recognition. According to exemplary embodiments, individual harmonics and glottal pulses, as necessary, for stream separation, pitch detection, etc. are resolved.

[0037] Bi-aural Processing of Inter-Aural Time and Level Differences.

[0038] These representations are valuable for stream separation and sound source location, when, for example, stereo microphones are available. Stereo cochlear and ITD and ILD algorithms already exist for real-time desktop PC operation. FIG. 4 shows a response of binaural representations to a sound source positioned to a right of the stereo microphone pair.

[0039] Tone Tracking and Preliminary Stream Separation.

[0040] FIG. 5 illustrates the use of the high resolution cochlea model in a rudimentary proof of concept of stream separation: tracking a non-speech tone, separating it from speech, and recognizing the remaining speech. The exemplary recognizer is able to recognize the speech when, for example, a siren is removed, even though some thin sections of the speech spectrum are lost. Conversely, the FFT approach, without stream separation, is unable to recognize the speech.

[0041] Stream Separation Based on Polyphonic Pitch Extraction.

[0042] This exemplary method is capable of resolving pitch of multiple speakers simultaneously, and detecting multiple musical instruments simultaneously. This method may run in real-time on a computing device.

[0043] High Resolution Spectral Envelope.

[0044] A spectral envelope algorithm achieves pitch invariance in the context of feature grouping, without the indiscriminant blurring of the Mel-scale filterbank. This algorithm may run in real-time on a computing device.

[0045] Speech Recognizer based on integrating cochlear model into Neural-Network/Hidde-Markov-Model recognizer from Oregon Graduate Institute's Center for Spoken Language Understanding (OGI CSLU).

[0046] The exemplary recognizer is fully instrumented out for graphical inspection of cochlea, filterbank, cepstrum, context window, neural-network, diphone, and word output. The recognizer may also provide arbitrarily reconfigurable grammar and vocabulary, is capable of batch runs against TIMIT database, and may generate full statistics including a confusion matrix.

[0047] Comparison with Existing Technology

[0048] The OGI/CSLU system is used as a reference for the existing technology. Following are the primary differences between the existing technology and the exemplary embodiments of the present invention.

[0049]

Attribute	Existing Technology	Exemplary Embodiment
Spectral	FFT, 64 taps, linear	Cochlea model, 600 taps,
Computation	frequency scale, updated	logarithmic frequency scale,
	every 10 ms.	updated every 22
		microseconds.
Low-level	Spectral Energy.	Spectral Energy, fine timing,
features		broadband transients,
		spectral edges and notches.
Cues for stream	None	Inter-aural time and level
separation		difference, polyphonic pitch,
		tone tracking, common
		onset, AM, FM, etc.
Pitch	22-point Mel filterbank,	600-tap Spectral Envelope
invariance	updated every 10 ms.	updated every 1 ms.
Noise	RASTA or Cepstral	Full stream separation based
robustness	Mean Subtraction	on polyphonic pitch.
		Adaptive noise threshold.
Pitch/Voicing	None	Polyphonic pitch extraction
cue		based on propriety spectral
		pitch algorithm.
Speaker	None	Based on early estimate of
Normalization		speaker characteristics at
		detected beginning of
		utterance.
Phoneme	130 ms context of	150 ms context of spectral
classifier	cepstrum and delta-	envelope, pitch/voicing, and
inputs	cepstrum.	transient detection.
Phoneme	3-layer perception, with	3-layer perception, with di-
classifier	di-phone output targets,	phone output targets, trained
	trained on reduced	on full phoneme set.
	phoneme set.
Back-end	Viterbi search through	Viterbi search through
	Hidden Markov model	Hidden Markov model
	search space. Uses fixed	search space. Uses adaptive
	duration limits. No	duration limits with a
	syllabic stress model.	syllabic stress model.

[0050] FIG. 7 shows performance of classic speech recognition versus noise level. The Word Error Rate is presented on a logarithmic scale for better detail, and to show an area of improvement that is an area of focus. Current solutions are at ˜92% word accuracy, or 8% word error rate, in quiet conditions, and dramatically worse in the presence of noise. Even noise canceling techniques such as Blind Source Separation only marginally impact performance. By comparison, word-error-rate of humans is several orders of magnitude lower in quiet conditions, and humans are more robust in the presence of noise. The exemplary embodiment's performance target is at least an order of magnitude better than existing systems in quiet conditions, with much better robustness in noisy conditions.

[0051] Conventional speech recognition systems with 92% accuracy at the word level typically are performing at about 50-70% accuracy at the phoneme level. This indicates that a front-end and phoneme classifier is surprisingly inaccurate, but the Hidden Markov Model back-end is capable of using constraints of the vocabulary and grammar to determine the spoke words. To reach a target of <0.5% word error rate (99.5% accuracy), it may be necessary to improve phoneme recognition accuracy beyond current state-of-the-art of 75.8%.

[0052] An analysis of the errors made by the OGI/CSLU system on TIMIT phoneme recognition is shown in a confusion matrix in FIG. 8. Scores in FIG. 9 indicate that stop consonants are a least well-recognized category, with recognition accuracy mostly below 10%. Low performance on stop consonants validates the belief that performance will be improved by higher-resolution representations for fast transients and fast formant transitions, and by detection of pitch/voicing, for more accurate estimate of voice-onset time. A graphical summary of the score is given in FIG. 9.

[0053] Applications for exemplary embodiments of a noise-robust speech recognizer may include, for example automatic voice-dialing for cell-phones and telephones in general; automatic command and control of navigation systems, environmental controls, communications devices in automobiles; automatic command and control of home electronics appliances; interface to Personal Digital Assistants; and meeting transcription.

[0054] The indiscriminate spectral and temporal blurring of the mel-scale filterbank achieves pitch invariance in a low-cost way, at the expense of resolution for stop consonants and for the grouping processes necessary for noise robustness. To achieve human level performance on automatic speech recognition, it may be beneficial to extract and use as much information in an auditory signal as is possible. The present noise-robust speech recognizer exploits known signal processing of a human auditory pathway, efficiently mapped onto modern computational resources, to provide a speech recognition system that is capable of near-human performance.

[0055] Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. It should be noted that there are many alternative ways of implementing both the process and apparatus of the present invention. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.