Distributed speech recognition with back-end voice activity detection apparatus and method

[0001] This invention relates generally to speech recognition and more particularly to distributed speech recognition.

[0002] Speech recognition is known in the art. In general, speech audio input is digitized and then processed to facilitate identification of specific spoken words contained in the speech input. Pursuant to one approach, so-called features are extracted from the digitized speech and then compared against previously stored patterns to enable such recognition of the speech content. It is also known to parse or distribute the feature extraction and pattern matching activities over two different processing units. For example, The European Technical Standards Institute (ETSI) has proposed, in Standard ES 201 108, Ver. 1.1.2, April 2000, a distributed speech recognition system where a portable device such as a cellular handset executes the feature extraction function and transmits the resultant features to a fixed-end platform that then facilitates the pattern matching function.

[0003] It is also known that, in general, pattern matching can be more successfully accomplished when the input can be accurately characterized as being either speech or non-speech audio input. For example, when information is available to identify a given segment of audio input as being non-speech, that information can be used to beneficially influence the functionality of the pattern matching activity by, for example, eliminating pattern matching for that particular non-speech segment. Unfortunately, the benefits of voice activity detection are not ordinarily available in distributed speech recognition systems such as the ETSI standard noted above. Instead, the speech content features are transmitted sans any voice activity detection information to the remote pattern matching platform.

[0008] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are typically not depicted in order to facilitate a less obstructed view of these various embodiments of the present invention.

[0009] Generally speaking, pursuant to these various embodiments, a plurality of speech recognition features are processed to provide at least an approximation of precursor information that originally yielded the speech recognition features. This precursor information is then processed to detect portions that likely correspond to speech and to provide a corresponding identification thereof. Such voice detection information is then used to facilitate the recognition processing of the speech recognition features.

[0010] In one embodiment, the speech recognition features include Mel frequency cepstral coefficients. In one embodiment, the speech recognition features are processed by means of an inverse discrete cosine transformation to provide resultant values that are used to provide the approximation of the precursor information. If desired, these resultant values can themselves be processed by means of exponentiation to provide the precursor information. In yet another embodiment, the speech recognition features can be processed to ascertain signal to noise information, which information is used, alone or with other voice activity detection information, to aid the recognition processing of the speech recognition features.

[0011] So configured, the pattern matching activity in a distributed speech recognition system can benefit from voice activity detection information regardless of the fact that no such voice activity detection information was initially forwarded by the features-extraction front-end. This can result in improved recognition and/or reduced power and/or processing requirements.

[0012] Before presenting various embodiments pertaining to the invention, it will be helpful to first present an illustrative example of front-end feature extraction for a distributed speech recognition system. This example will then provide a helpful context for presenting some specific embodiments. Referring now to FIG. 1, audio input such as speech is digitized in an analog to digital converter 11 (optionally, the digitized speech can then pass through a DC-offset removal filter (not shown) as understood in the art). A framing unit 12 then parses the digitized speech into corresponding frames. The frame-size will ordinarily be dependent on the sampling frequency. For example, the ETSI distributed speech recognition standard referenced earlier accommodates three different sampling frequencies, these being 8, 11, and 16 KHz. A suitable frame-size for these three sampling frequencies would be 200, 256, and 400 samples, respectively. At this point, a log energy unit 13 computes the natural logarithm of the overall energy for each frame to provide a Log-E parameter, which comprises one of the speech recognition features that will eventually be provided to the back-end of the distributed speech recognition system.

[0013] The framed information is provided to a filter and fast Fourier transform (FFT) unit 14. In particular, a pre-emphasis filter emphasizes the higher frequency components of the speech content. These pre-emphasized frames are then windowed by a Hamming window of the same size as the frames. The windowed frames are then transformed into the frequency domain by a fast Fourier transform. The size of the FFT depends on the sampling frequency, viz. 256 for 8 and 11 KHz and 512 for 16 KHz. The FFT magnitudes in the frequency range between 64 Hz and F_s/2, where F_s is the sampling frequency, are then Mel-filtered 15.

[0014] Mel-filtering can be accomplished as follows. First, the frequency range above is warped into Mel-frequency scale using the expression: $\mathrm{Mel}\left(f\right)=2595.0*{\log }_{10}\left(1+\frac{f}{700.0}\right).$
The warped frequency range is then divided into 23 equal-sized, half-overlapping bands (a.k.a. channels or bins). For example, if F_s=8000 Hz, the frequency range between 64 Hz and 4000 Hz is warped into the Mel-frequency range between 98.6 and 2146.1 and divided into 23 bands with each band 170.6 wide and the center of each band 85.3 apart. The center of the first band is located at 98.6+85.3=183.9, and that of the last band is located at 2146.1−85.3=2060.8. These centers are then un-warped and rounded to the nearest FFT bin frequencies. In the linear frequency domain, the 23 bands are no longer equal-sized but the size of each band generally increases as the frequency increases. The FFT magnitudes falling inside each band are then combined using a triangular weighting window (with the weight at the center equal to 1.0 and at either end approaching 0.0).

[0015] The Mel-filter bank outputs are then subjected to a (natural) logarithm function 16. The 23 log values are then transformed by a discrete cosine transform (DCT) 17 to obtain the 13 Mel frequency cepstral coefficient values C₀ through C₁₂. In this embodiment, the values C₁₃ through C₂₂ are discarded, i.e., not computed, as they are not going to be transmitted or otherwise provided to the back-end pattern matching activity.

[0016] The Mel frequency cepstral coefficient parameters and the log-E parameter are then quantized and otherwise appropriately coded in a coder 18 and provided to a wireless transmitter of choice for transmission to a remote back-end pattern matching platform

[0017] The above feature extraction functionality can be readily provided in, for example, a wireless transceiver platform such as a cellular handset. So configured, it can be seen that audible speech as provided to the handset can have speech recognition features extracted therefrom for subsequent remote processing. In the example provided, the speech recognition features include Mel frequency cepstral coefficients and a log-E parameter. It should be understood that this particular example has been provided for purposes of illustration only, however, and to provide a useful basis for now providing a detailed description of some embodiments for carrying out the invention. There are numerous other speech recognition features that could be extracted, either in addition to those suggested or in lieu thereof The tenants of this invention are applicable to such alternative embodiments as well.

[0018] FIG. 2 provides a block diagram general overview of a back-end pattern matching platform suitable for use with the above described front-end feature extraction platform. An appropriate wireless receiver 21 receives the speech recognition feature information as transmitted by the front-end platform described above. A decoder 22 decodes the received information to specifically recover the speech recognition features described above. These features are provided to an inverse discrete cosine transform unit 23 that can be described by the following equation: $D_i=\frac{C_0}{23}+\frac{2}{23}\sum _{j=1}^{12}{C}_j\cos \left(\frac{\left(2i+1\right)j\pi }{2*23}\right);i=0,1,\dots ,22.$
(Notice that in the above equation the unavailable Cepstral Coefficients C₁₃ through C₂₂ are assumed to be zero.) The resulting D₁ values are then exponentiated in exponentiation unit 24 to obtain the filter bank outputs as follows:
F₁=exp(D₁); i=0, 1, . . . , 22.

[0019] The filter bank outputs F₀ through F₂₂ obtained as above are, of course, only an approximation of the original filter bank outputs computed at the front-end because of the earlier truncation operation (i.e., the dropping of the values C₁₃ through C₂₂) and the quantization of the Mel frequency cepstral values C₀ through C₁₂. These filter bank outputs represent an approximation of the precursor information that was used to develop the speech recognition features from which they are now derived.

[0020] This precursor information is provided to a voice activity detector 25 that serves to detect whether the precursor information likely includes at least a significant quantity of speech or not. A segmentation unit 26 then uses this information to provide one or more signals to a pattern matching unit 27 which signals identify which segments being provided to the pattern matching unit 27 by the decoder 22 likely include speech content.

[0021] So configured, the pattern matching unit 27 can process the speech recognition feature information accordingly, thereby likely increasing recognition quality and accuracy and/or reducing power/processing needs to achieve like results.

[0022] If desired, the voice activity detector 25 can also process the recovered precursor information to develop a signal representing the signal to noise ratio (SNR) as corresponds to the original audio input. Such information is also potentially usable by the pattern matching unit 27 for similar purposes as those already mentioned above.

[0023] Referring now to FIG. 3, a more detailed description of an exemplary voice activity detector 25 will be provided. The filter bank outputs F₀ through F₂₂ mentioned above may be regarded as average spectral magnitude estimates at the different frequency bands or channels for the current input frame. We will denote the filter bank output for the m^th frame and i^th channel by F(m,i), and when the specific channel is not important, we will denote the set of all filter bank outputs for the m^th frame by F(m). Using these values as input, a channel energy estimator 30 provides a smoothed estimate of the channel energies as follows.
E_ch(m,i)=max {E_mm,α_ch(m)E_ch(m−1,i)+(1−α_ch(m))(λ₁F(m,i))²}; i=0, 1, . . . , 22
where E_ch(m,i) is the smoothed channel energy estimate for the m^th frame and the i^th channel, E_min is the minimum allowable channel energy, {λ₁, i=0, 1, . . . , 22} are the correction factors to compensate for the effect of the pre-emphasis filter and the varying widths of the triangular weighting windows used in Mel-filtering, and α_ch(m) is the channel energy smoothing factor defined as: ${\alpha }_{\mathrm{ch}}\left(m\right)=\{\begin{array}{c}0;m\le 1\\ 0.45;m>1\end{array}$
which means that α_ch assumes a value of zero for the first frame (m=1) and a value of 0.45 for all subsequent frames. This allows the channel energy estimate to be initialized to the unfiltered channel energy of the first frame. In a preferred embodiment, E_min=5000, and the value of the correction factor λ₁ (for 8 kHz sampling frequency) is given by the i^th value in the 23-element table: {3.2811, 2.2510, 1.4051, 1.1038, 0.8867, 0.6487, 0.5482, 0.4163, 0.3234, 0.2820, 0.2505, 0.2036, 0.1680, 0.1397, 0.1179, 0.1080, 0.0931, 0.0763, 0.0674, 0.0636, 0.0546, 0.0478, 0.0046}. From the channel energy estimate, a peak to average ratio estimator 31 estimates the Peak-to-Average Ratio for the current frame m, denoted by P2A(m), as follows: $\mathrm{P2A}\left(m\right)=10{\log }_{10}\left(\frac{\max (E_{\mathrm{ch}}\left(m,i\right){}_{i=4}^{22})}{\left(1/23\right)\sum _{i=0}^{22}{E}_{\mathrm{ch}}\left(m,i\right)}\right).$

[0024] Similar to the channel energy estimated, the channel noise energy estimate (defined below), in a preferred embodiment, is initialized as follows:

[0025]

	if ((m ≦ INIT_FRAMES) OR (fupdate_flag == TRUE))
	{
	if (P2A(m) < PEAK_TO_AVE_THLD)
	{
E n ⁡ ( m , i ) = { E ch ⁡ ( m , i ) ; m = 1 , 0 ≤ i ≤ 22 ; 0.7 ⁢ E n ⁡ ( m - 1 , i ) + 0.3 ⁢ E ch ⁡ ( m , i ) ; 2 ≤ m ≤ INIT_FRAMES , 0 ≤ i ≤ 22 ;
	}
	else
	{
	En(m,i) = Emin; 0 ≦ i ≦ 22;
	}
	}

where E_n(m,i) is the smoothed noise energy estimate for the m^th frame and the i^th channel, INIT_FRAMES is the number of initial frames that are assumed to be noise-only frames, and fupdate_flag is a forced update flag defined below. In a preferred embodiment, INIT_FRAMES=10, and PEAK_TO_AVE_THLD=10.

[0026] The channel energy estimate and the channel noise energy estimate for the current frame m, and all the 23 channels are denoted by E_ch(m) and E_n(m) respectively. The channel energy estimate E_ch(m) and the channel noise energy estimate E_n(m) are used to estimate the quantized channel signal-to-noise ratio (SNR) indices at a channel SNR estimator 32 as: ${\sigma }_q\left(m,i\right)=\max \left(0,\min \left(89,\mathrm{round}\left(10{\log }_{10}\left(\frac{E_{\mathrm{ch}}\left(m,i\right)}{E_n\left(m,i\right)}\right)/0.375\right)\right)\right);0\le i\le 22$
where the values {σ_q(m, i), i=0, 1, . . . , 22} are constrained to be between 0 and 89 both inclusive.

[0027] From the channel SNR estimate σ_q(m) for the current frame, the voice metric V(m) for the current frame is computed at a voice metric calculator 33 as the sum: $V\left(m\right)=\sum _{i=0}^{22}v\left({\sigma }_q\left(i\right)\right)$
where v(k) is the k^th value of the 90-element voice metric table v defined as: $v=\left\{1,1,1,1,1,1,2,2,2,2,2,3,3,3,3,3,4,4,4,5,5,5,6,6,7,7,7,8,8,9,9,10,10,11,12,12,13,13,14,15,15,16,17,17,18,19,20,20,21,22,23,24,24,25,26,27,28,28,29,30,31,32,33,34,35,36,37,37,38,39,40,41,42,43,44,45,46,47,48,49,50,50,50,50,50,50,50,50,50,50\right\}.$

[0028] The channel energy estimate E_ch(m) for the current frame is also used as input to a spectral deviation estimator 34, which estimates the spectral deviation Δ_E(m) for the current frame as follows. First, the log energy spectrum is estimated as:
E_dB(m,i)=10 log₁₀(E_ch(m,i)); i=0, 1, . . . , 22.
Next, the spectral deviation Δ_E(m) is estimated as the sum of the absolute difference between the current log energy spectrum and an average long-term log energy spectrum denoted by Ē_dB(m), that is: ${\Delta }_E\left(m\right)=\sum _{i=0}^{22}E_{\mathrm{dB}}\left(m,i\right)-{\stackrel{\_}{E}}_{\mathrm{dB}}\left(m,i\right).$
The average long-term log energy spectrum is initialized as follows:

[0029]

	if ((m ≦ INIT_FRAMES) OR (fupdate_flag == TRUE))
	{
	ĒdB (m,i) = EdB (m,i); 0 ≦ i ≦ 22;
	}

The average long-term log energy spectrum is updated as follows: ${\stackrel{\_}{E}}_{\mathrm{dB}}\left(m+1,i\right)=\{\begin{array}{c}0.9{\stackrel{\_}{E}}_{\mathrm{dB}}\left(m,i\right)+0.1E_{\mathrm{dB}}\left(m,i\right);V\left(m\right)>\mathrm{SIG\_THLD}\left(m\right)\\ 0.7{\stackrel{\_}{E}}_{\mathrm{dB}}\left(m,i\right)+0.3E_{\mathrm{dB}}\left(m,i\right);V\left(m\right)\le \mathrm{SIG\_THLD}\left(m\right)\end{array}$
where the parameter SIG_THLD(M) depends on the quantized signal SNR described next.

[0030] A signal SNR estimator 35 estimates the speech signal SNR as follows. First, the total noise energy of the current frame E_tn(m) is computed as the sum of the channel noise energies, that is: $E_{\mathrm{tn}}\left(m\right)=\sum _{i=0}^{22}{E}_n\left(m,i\right).$
Next, the instantaneous total signal energy E_ts,inst(m) is computed as follows:

[0031]

	if (V(m) > SIG_THLD(m))
	{
E ts , inst ⁡ ( m ) = ∑ i = 0 22 ⁢ max ⁡ ( E ch ⁡ ( m , i ) , E n ⁡ ( m , i ) )
	}

The instantaneous total signal energy E_ts,inst(m) is updated as above only if the current frame is determined to be a signal frame by checking whether V(m)>SIG_THLD(m). It is also clear that the signal energy estimated above is really (signal+noise) energy in the strict sense. Initialization of E_ts,inst(m) is performed as follows.

[0032]

	if ((m≦INIT_FRAMES) OR (fupdate_flag == TRUE))
	{
	Ets,inst(m) = INIT_SIG_ENRG;
	}

where the value of INIT_SIG_ENRG=1.0E+09 in a preferred embodiment.

[0033] Once the total instantaneous signal energy and the total noise energy are computed, the instantaneous signal-to-noise ratio of the current frame denoted by SNR_inst(m) is computed as:
SNR_inst=max(0.0, 10 log₁₀(E_ts,inst(m)/E_tn(m))).
From the instantaneous SNR, the smoothed SNR is estimated as:

[0034]

	if ((m ≦ INIT_FRAMES) OR (fupdate_flag == TRUE))
	{
	SNR(m) = SNRinst(m);
	}
	else
	{
	if (V(m) > SIG_THLD(m))
	{
	SNR(m) = βSNR(m−1) + (1−β) SNRinst(m);
	β = min(β+0.003, HI_BETA);
	}
	else
	{
	β = max(β−0.003, LO_BETA);
	}
	}

In a preferred embodiment, the lower and upper limits of the smoothing factor β are respectively LO_BETA=0.950 and HI_BETA=0.998. Because we estimate the SNR as the ratio between (signal+noise) energy to signal energy, the lowest value of the estimated SNR is zero. The estimate is more accurate for higher values of SNR and it becomes less and less accurate as the SNR value decreases. The signal SNR is then quantized to 20 different values as:
SNR_q(m)=max(0,min(round(SNR(m)/1.5),19)).

[0035] The quantized signal SNR is used to determine different threshold values. For example, the signal threshold for the next frame SIG_THLD(m+1) is determined using SNR_q(m) as an index into the 20-element table {36, 43, 52, 62, 73, 86, 101, 117, 134, 153, 173, 194, 217, 242, 268, 295, 295, 295, 295, 295} in a preferred embodiment.

[0036] At this point, the voice metric V(m), the spectral deviation Δ_E(m), the peak-to-average ratio P2A(m), and the quantized signal SNR SNR_q(m) are input to an update decision determiner 36. The logic shown below in pseudo-code demonstrates how the noise estimate update decision can be made. It also demonstrates how a forced update decision is made (a forced update mechanism allows the voice activity detector to recover from wrong classification of background noise as speech whenever there is a sudden increase in background noise level).

[0037] First, the update threshold for the current frame UPDATE_THLD(m) is determined using SNR_q(m) as an index into a 20-element table given by {31, 32, 33, 34, 35, 36, 37, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38} in a preferred embodiment. The update decision determination process begins by clearing the update flag (update_flag) and the forced update flag (fupdate_flag). These flags are set if certain conditions are satisfied as illustrated in the pseudo-code below:

[0038]

update_flag = FALSE;
fupdate_flag = FALSE;
if ((m > INIT_FRAMES) AND (V(m) < UPDATE_THLD(m)) AND
(P2A(m) < PEAK_TO_AVE_THLD)
{
update_flag = TRUE;
update_cnt = 0;
}
else
{
if ((P2A(m) < PEAK_TO_AVE_THLD) AND
(ΔE(m) < DEV_THLD))
{
update_cnt = update_cnt + 1;
if (update_cnt ≧ UPDATE_CNT_THLD)
{
update_flag = TRUE;
fupdate_flag = TRUE;
}
}
}

In order to avoid long term “creeping” of the update counter (update_cnt) setting the forced update flag (fupdate_flag) falsely in the above pseudo-code, hysteresis logic is implemented as shown below:

[0039]

	if (update_cnt == last_update_cnt)
	{
	hyster_cnt = hyster_cnt + 1;
	}
	else
	{
	hyster_cnt = 0;
	last_update_cnt = update_cnt;
	}
	if (hyster_cnt > HYSTER_CNT_THLD)
	{
	update_cnt = 0;
	}

In a preferred embodiment, the values of (previously undefined) constants used above are as follows:

[0040] DEV_THLD=70

[0041] UPDATE_CNT_THLD=500

[0042] HYSTER_CNT_THLD=9

[0043] Whenever the above referenced update flag is set for a given frame, the channel noise estimate for the next frame is updated in a noise energy smoother 37 using:
E_n(m+1,i)=0.9E_n(m,i)+0.1E_ch(m,i)); i=0 1, . . . , 22.
The updated channel noise estimate is stored in noise energy estimate storage 38 for all future frames until the next update occurs. The output of the noise energy estimate storage 38 E_n(m) is used as an input to the channel SNR estimator 32 as described earlier.

[0044] Now the operation of a voice activity determiner 39 will be described. The voice metric V(m) and the quantized signal SNR value SNR_q(m) serve as inputs to the voice activity determiner 39. For the first INIT_FRAMES frames, the output of the voice activity determiner denoted by VAD_FLAG is set to FALSE since these frames are assumed to be noise-only frames. For the following frames, the voice activity determiner operates as follows. As a first step, the quantized SNR value is used to determine the voice metric threshold V_th, the hangover count threshold H_cnt, and the burst count threshold B_cnt for the current frame as:
V_th(m)=V_table[SNR_q(m)], H_cnt(m)=H_table[SNR_q(m)], B_cnt(m)=B_table[SNR_q(m)],
where SNR_q(m) is used as an index into the respective tables. In a preferred embodiment, these tables are defined by:
V_table={33, 35, 36, 37, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 58, 58, 58, 58, 58},
H_table={54, 52, 50, 48, 46, 44, 42, 40, 38, 36, 34, 32, 30, 28, 26, 24, 22, 20, 18, 16}, and
B_table={3, 3, 3, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6}.

[0045] Then, the voice activity of the current frame is determined by testing if the voice metric exceeds the voice metric threshold. If the output of this test is TRUE, then the current frame is declared “voice-active.” Otherwise, the hangover count variable (hangover_count) is tested to find out if it is greater than or equal to zero. If the output of this test is TRUE, then the current frame is also declared “voice-active.” If the outputs of both tests are FALSE, then the current frame is declared “voice-inactive.” The “hangover” mechanism is generally used to cover slowly decaying speech that might otherwise be classified as noise, and to bridge over small gaps or pauses in speech that may be degraded by aggressive voice activity detection. The hangover mechanism is activated if the number of consecutive “voice-active” frames (counted by the burst_count variable) is at least equal to B_cnt, the burst count threshold. To activate the mechanism, the number of hangover frames is set to H_cnt, the hangover count threshold. The pseudo-code for the voice activity determiner is shown below:

[0046]

	if (V(m) > Vth(m))
	{
	VAD_LOCAL = TRUE;
	burst_count = burst_count + 1;
	if (burst_count >= Bcnt(m))
	{
	hangover_count = Hcnt(m);
	}
	}
	else
	{
	VAD_LOCAL = FALSE:
	burst_count = 0;
	if (hangover_count >= 0)
	{
	hangover_count = hangover_count − 1;
	}
	}
	if ((VAD_LOCAL == TRUE) OR (hangover_count >= 0))
	{
	VAD_FLAG = TRUE;
	}
	else
	{
	VAD_FLAG = FALSE;
	}

The variable VAD_FLAG from 310 is output as the voice activity information for each frame. Optionally, the unquantized SNR value SNR(m) from 306 is also output.

[0047] So configured, an approximation of precursor information as derived from speech recognition features that have been provided by a remote front-end for a distributed speech recognition system can be developed and used to aid in identifying portions of the speech recognition feature information that itself corresponds to speech and non-speech. This information in turn is readily used, in accordance with well understood prior art technique, to inform the activities and functionality of a back-end pattern matching process to facilitate improved recognition and/or improved logistical operation. In short, the benefits of voice activity detection are provided to the back-end of a distributed voice recognition system notwithstanding that such information was not developed in the first instance at the front-end and then provided to the back-end. Instead, information as provided by the front-end is at least partially reverse processed to at least approximate information as it originally appeared at the front-end to then facilitate the voice activity detection activity.

[0048] Those skilled in the art will recognize that a wide variety of modifications, alterations, and combinations can be made with respect to the above described embodiments without departing from the spirit and scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept. In particular, it is again emphasized that the specific embodiments set forth are illustrative only; the essential concepts of the invention are applicable in a wide variety of distributed voice recognition systems and contexts and should not be viewed as being limited to only the embodiments set forth for purposes of illustration.