Device and method of channel effect compensation for telephone speech recognition

[0002] 1. Field of the invention

[0003] The present invention generally relates to a telephone speech recognition system. More particularly, the present invention relates to device and method of channel effect compensation for telephone speech recognition.

[0004] 2. Description of the Related Art

[0005] In a speech recognition application via a telephone network, speech signals are Inputted from the handset of a telephone and transmitted through a telephone line to a remote speech recognition system for recognition. Therein, the path speech signals pass includes the telephone handset and the telephone line, which are referred to as a “telephone channel or channel”. In terms of signal transmission, the characteristic of the telephone channel will affect the speech signals during transmission, referred as a “telephone channel effect or channel effect”. Mathematically, impulse response of the telephone channel is introduced with a convolved component into speech signals.

[0006] FIG. 1 is a diagram illustrating a typical telephone speech recognition system. As shown in the FIG. 1, a speech signal x(t) sent by the calling part becomes a telephone speech signal y(t) after passing through the telephone channel 1 comprising the telephone handset and the telephone line, and is inputted to the recognition system 10 for further processing. The recognition result R is generated by the recognition system 10. Here, assume the impulse response of the telephone channel 10 to be h(t), then the relationship between the speech signal x(t) and the telephone speech signal y(t) can be represented by:

[0007] y(t)=x(t){circle around (x)}h(t)  (1)

[0008] where symbol “{circle around (x)}” represents the convolution operator. Most importantly, the impulse response h(t) in the telephone channel 1 varies with the caller's handset and the transmits son path of speech signals in a telephone network (the transmission path determined by switching equipment). In other words, the same phone call (the same speech signal x(t)) will generate different telephone speech signals y(t) through different telephone channels (different impulse responses h(t). This environmental variation will affect the recognition ate of the recognition system 10. Therefore, compensation of telephone channel effect should be performed before undergoing telephone speech recognition to reduce such environmental variation.

[0009] The principle of typical telephone channel effect compensation will be briefly described in the following. Equation (1) represents the relationship between the speech signal x(t) and the telephone speech signal y(t) in time domain. If equation (1) is transformed to the spectral domain, then it can be represented by:

[0010] Y(f)=X(f)•|H(f)|²  (2)

[0011] where X(f) and Y(f) represent the power spectra of the speech signal x(t) and the telephone speech signal y(t), respectively, and H(f) represents the transfer function of the telephone channel 1.

[0012] The following logarithm spectral relation is obtained after processing the bilateral logarithms of equation (2):

[0013] log[Y(f)]=log[X(f)]+log└|H(f)|²¦  (3)

[0014] The following will be obtained when inverse Fourier transformation Is used for projecting equation (3) on a cepstral domain:

[0015] c_y(τ)=c_x(τ)+c_h(τ)  (4)

[0016] where c_x(τ), c_y(τ), and c_h(τ) are the respective cepstral vectors of x(t), y(t), and h(t).

[0017] From equations (3) and 4), in logarithmic spectral and cepstral domain, the influence of the telephone channel upon the speech signals in transmission can be described with a bias. Therefore, most of the current telephone channel effect compensation means are developed and based upon such a principle. The difference lies in the bias estimation method and bias elimination method.

[0018] FIG. 2 is a block diagram illustrating a conventional telephone speech recognition system. As shown in the figure, the telephone speech recognition system comprises a feature analysis section 100, a channel effect compensation section 102 and a recognizer 104 (comprising a speech recognition section 104afor speech recognition and acoustic models 104bfeature analysis section 100 first blocks the received telephone speech signal y(t) into frames, performs feature analysis on each telephone speech frame, and generates a corresponding feature vector o(t). in accordance with the description of the above equations (3) and (4), the feature vector o(t) may be a logarithmic spectral vector or a cepstral vector. Channel effect compensation section 102 subsequently performs compensation of the feature vector o(t), and the generated feature vector ô(t) is inputted to the recognizer 104. Speech recognition section 104aperforms the actual speech recognition according to the acoustic models 104band generates the desired recognition result R. The three most popular telephone channel effect compensation techniques are the following: the relative spectral technique (RASTA), the cepstral mean normalization (CMN), and the signal bias removal (SBR) . The first technique adopts a fixed filter type, whereas the last two techniques calculate the bias from feature vectors of a telephone speech signal. These conventional techniques will be briefly described in the following references, the content of which is expressly incorporated herein by reference.

[0019] (A) RASTA: Refer H. Hermansky, N. Morgan, “RASTA processing of speech” HEEE Trans. On Speech and Audio Processing, vol. 2, pp.578-589, 1994 for derails. The operation of RASTA makes use of filters Go eliminate low-frequency components contained in the logarithmic spectral vectors or cepstral vectors, that is, the bias introduced by the telephone channel, for the purpose of the channel effect compensation. According to aforementioned analysis, bandpass infinite impulse response (IIR) filters expressed by the following equation (5) can perform quite well. $\begin{array}{cc}H\left(z\right)=0\text{:}1\times \frac{1+z^{-1}-z^{-3}-2z^{-4}}{z^{-4}\left(1-0.98z^{-1}\right)}& \left(5\right)\end{array}$

[0020] The purposes of using a bandpass filter are twofold: firstly, for filtering out the bias by highpass filtering; and secondly, for smoothing the rapidly changing spectra by lowpass filtering. If only the telephone channel effect compensation is considered, only highpass filtering need be used. At this time, the transfer function of the highpass filter can be represented as follows: $\begin{array}{cc}H\left(z\right)=\frac{1-z^{-1}}{1-\left(1-\lambda \right)z^{-1}}& \left(6\right)\end{array}$

[0021] RASTA has demonstrated its advantage in that it can be easily realized without causing response time delay problems, however, its disadvantage is that the range of the frequency band of the filter is predetermined and cannot be adjusted with the inputted telephone speech signal. Therefore, some useful speech information may be also deleted when the bias introduced by the telephone channel effect is filtered out; the recognition result will then be affected. As a result, the recognition result of a telephone speech recognition system obtained with RASTA compensation method is less effective than those obtained by CMN and SBR compensation methods.

[0022] (B) CMN : Refer F. Liu, R. M. Stern, X. Huang and A. Acero, “Efficient cepstral normalization for robust speech recognition,” Proc. Of Human Language Technology, pp.69-74, 1993 for details. The operation of CMN is to estimate the bias representing the characteristic of the telephone channel and to eliminate the bias from the logarithmic spectral vectors or cepstral vectors of the telephone speech signal. In CMN, a bias is represented by the cepstral mean vector of telephone speech signals. Since the bias is estimated from telephone speech signals, the telephone channel characteristic can be acquired and a better compensation can be obtained. However, CMN is performed by assuming the cepstral mean vector of the speech signal before passing the telephone channel to be a zero vector. Experimental results have demonstrated that such an assumption is valid when the input speech signals are long enough. But, when the speech signals are rot long enough, the phonetic information or the speech signals will affect the estimation of the bias; thus, the compensation result is rot significant.

[0023] (C) SBR: Refer M. G. Rahim and B. H. Hwang, “Signal bias removal by maximum likelihood estimation for robust telephone speech recognition,” IEEE Trans. Speech and Audio Processing, vol. 4, pp. 19-30, 1996 for details. The SBR algorithm estimates the bias in an iterative manner based upon the maximum likelihood criterion. Similarly, the compensated logarithmic spectral vectors or cepstral vectors can be obtained by subtracting the estimated bias from the logarithmic spectral vectors or cepstral vectors of telephone speech signals in an iterative manner. In contrast to the CMN, the SBR algorithm can estimate the bias more accurately; however, the response time delay for recognition prolongs comparatIvely. Further, since the SBR algorithm is a technique based upon the maximum likelihood criterion, the estimation error of maximum likelihood may also affect the accuracy of the estimation of the bias when the telephone speech signals are not long enough.

[0024] The above-mentioned three telephone channel effect compensation techniques share a common drawback in that accurate results will not be achieved when the telephone speech signals are not long enough. Moreover, these techniques merely deal with the feature vectors without considering the connection with speech recognizers.

[0025] The object of the present invention is to provide device and method of channel effect compensation, to accurately estimate the bias representing the characteristic of the telephone channel.

[0026] According to the above object, the present invention provides device and method of channel effect compensation for telephone speech recognition. The channel effect compensation device comprises:

[0027] a compensatory neural network for receiving an input signal and compensating the input signal with a bias to generate an output signal, wherein the compensatory neural network provide a plurality of first parameters to determine the bias.

[0028] During a training process, a feedback section could be coupled to the compensatory neural network for adjusting the first parameters according to the error between the bias and a target function.

[0029] During the other training process, a recognizer could be coupled to the compensatory neural network, with a speech recognition section for classifying the output signal according to a plurality of second parameters in acoustic models to generate a recognition result and determine a recognition loss thereby; and an adjustment section coupled to the compensatory neural network and the recognizer for adjusting the first parameters and the second parameters according to the recognition loss determined by the recognition result and an adjustment means.

[0030] Also, a method of compensating an Input signal in a telephone speech recognition system, comprises the following steps of:

[0031] receiving the input signal by a compensatory neural network; determlnlng a bias in response to a plurality of first parameters provided by the compensatory neural network; compensating the input signal with the bias; and sending out the compensated input signal to be an output signal.

[0032] During a training process, the training process comprising the steps of: receiving a plurality of feature vectors of a training telephone speech signal by the compensatory neural network; generating a bias in response to the first parameters provided by the compensatory neural network, the bias representing the characteristic of the telephone channel; comparing the bias with a target function to generate an error; and adjusting the first parameters by an error back-propagation algorithm according to a minimum mean square error criteria.

[0033] Wherein, the first parameters are determined during a training process, the training process comprising the steps of: receiving a plurality of feature vectors of a training telephone speech signal by the compensatory neural network; generating a bias in response to the first parameters provided by the compensatory neural network, the bias representing the characteristic of the telephone channel; compensating the feature vectors with the bias to generate a plurality of compensated feature vectors; classifying the compensated feature vectors in response to a plurality of second parameters in acoustic models by a speech recognition section to generate a recognition result; determining a recognition loss in response to the recognition result; and adjusting the first and second parameters according to the recognition loss and an adjustment means.

[0034] FIG. 1 is a block diagram illustrating a telephone speech recognition system;

[0035] FIG. 2 is a block diagram illustrating a related telephone speech recognition system;

[0036] FIG. 3 is a block diagram illustrating a first embodiment of device and method of channel effect compensation of the telephone speech recognition system of the present invention;

[0037] FIG. 4 is a block diagram illustrating a second embodiment of device and method of channel effect compensation of the telephone speech recognition system of the present invention; and

[0038] FIG. 5 is a flowchart illustrating the operations between the recognizer and adjustment section of the second embodiment in accordance with the present invention.

[0039] Several embodiments of this invention will be described in detail with the accompanying drawings as follows.

[0040] First Embodiment

[0041] FIG. 3 is a block diagram illustrating the first embodiment of device and method of channel effect compensation of the telephone speech recognition system of the present invention, wherein the parts different than hose of the related system in FIG. 2 are indicated with a different numeral. As shown in the figure, the compensatory neural network 110 in the first embodiment replaces the channel effect compensation section 102 of the related system.

[0042] Compensatory neural network 110 is used for estimating the bias that represents the characteristic of the telephone channel and removing the bias from an input signal, such a bias being determined by the first parameters (for example, offsets of neurons and weights between neurons). A feedback section 111 is utilized during the training process to adjust the first parameters according to the error between the bias and a target function.

[0043] In this embodiment, the input layer and the output layer of the compensatory neural network 110 have D neurons, respectively, representing the D-dimensional feature vector o(t) of the telephone speech signal and the estimated bias b(t) at the corresponding time, and t=1, 2, . . . , a, where T represents the number of speech frames of the telephone speech signal. There may be one hidden layer or more hidden layers between the input layer and the output layer.

[0044] Assume at each time t, the error e(t) between an estimated bias b(t) and the actual bias B can be represented as:

[0045] e(t)=b(t)−B  (7)

[0046] According to minimum mean square error (EASE) criteria, the estimate of the actual bias B, {overscore (B)} is an average value of b(t), namely: $\begin{array}{cc}\stackrel{\_}{B}=\frac{1}{T}\sum _{t=1}^{T}b\left(t\right)& \left(8\right)\end{array}$

[0047] Thus, the compensated feature vector ô(t) can be represented as:

[0048] ó(t)=F(o(t);θ)=o(t)−{overscore (B)}  ( 9)

[0049] where function F(•) represents the compensatory function for the compensatory neural network 110, and θ is the parameter set of the compensatory neural network 110, comprising the offsets of neurons and weights between neurons in the compensatory neural network.

[0050] According to equations (7) and (8) , the smaller the error between B and {overscore (B)}, the more accurate the estimated bias. To achieve this objective, the error back-propagation algorithm is adopted to train the neural network by making use of minimum mean square error as criteria. Refer S. J. Lee, K. C. Kim, H. Yoon, and J. W. Cho, “Application of Fully Recurrent Neural Networks for Speech Recognition,” Proceeding of ICASSP91, pp. 77-80, 1991 for details of the aforementioned training process. In the training process of this embodiment, two requirements have yet to be met. First, the target function of the training process has to be the actual bias that represents the characteristic of the telephone channel. Though the actual bias couldn't be obtained, the target function could be set to be the average of feature vectors of all telephone speech signals uttered within a telephone call and would approach to the actual bias. Second, the feedback section 111 of FIG. 3 has to adjust the first parameters of the compensatory neural network 110 according to the back-propagation error at every predetermined time interval. According to the above-mentioned two requirements for training the neural network, the compensatory neural network 110 can accurately estimate the bias of the telephone speech signal.

[0051] Second Embodiment

[0052] In the first embodiment described above, the compensatory neural network and recognizer are separate devices; that is, compensation of telephone channel effect is performed independently by the compensatory neural network. However, the present invention demonstrates its advantage in that the compensatory neural network can be integrated with subsequent recognizer to be an integrated recognition framework; that is, adjusting the first parameters of the compensatory neural network and the second parameters of the recognizer through the same mechanism to improve the recognition rate. This embodiment will be described in further detail with the accompanying drawings as follows.

[0053] FIG. 4 is a block diagram illustrating the second embodiment of device and method of channel effect compensation of the telephone speech recognition system of the present invention, wherein the parts different than those of the first embodiment in FIG. 3 are indicated with a different numeral. As shown in the FIG. 4, in the training process, the second embodiment adjusts the related parameters (such as the offsets and weights) of the compensatory neural network 120 and the related parameters (such as mean vectors and covariance matrices of Gaussian distributions of HMMs) of the acoustic models 122bof the recognizer 22 through the adjustment section 124 based upon the recognition result R. In this embodiment, the compensatory neural network 120 may use the framework of the first embodiment; therefore, the description is omitted. The speech recognition section 122aperforms speech recognition according to the output feature vectors of the compensatory neural network 120 and thus generates a result R. The adjustment section 124 adopts minimum classification error (MCE) as the training criteria, and generalized probabilistic descent (GPD) (W. Chou, B. H. Juang, and C. H. Lee, “Segmental GPD Training of HMM-Based speech Recognizer,” Proceeding of ICASSP92, pp. I-473-I-476, 1992) as the training method.

[0054] In this embodiment, the parameter set Λ of the recognizer 122 and the parameter set θ of the compensatory neural network 120 together are regarded as the overall parameter set Ω={Λ′θ}. FIG. 5 is a flowchart representing the operation of various functions among the compensatory neural network 120, the recognizer 122 and MCE/PD adjustment section 124 of this embodiment, namely: the discriminant function g_iO;Ω), class misclassification measured (O), loss function l_i(O;Ω), and parameter adjustment formula, respectively. The various functions and formula will be described as follows.

[0055] First, assume there are M (M is a positive integer) words (or classes) to be recognized, marked respectively as C_i, i=1, 2, . . . , M. And g_i(O;Ω) (i=1, 2, . . . , M) is a set of discriminant functions defined by the parameter set Ω, where ${O=\left\{o\left(t\right)\right\}|}_{t=1}^T$

[0056] represents a sequence of feature vectors, such as cepstral vectors or logarithm spectral vectors, before passing through the compensatory neural network 120. The recognizer is performed according to the following decision rule: $\begin{array}{cc}C\left(O\right)=C_l\mathrm{if}{g}_l\left(O;\Omega \right)=\underset{j}{\max }{g}_j\left(O;\Omega \right)j=1,2,\dots ,M& \left(10\right)\end{array}$

[0057] where (•) is a function representing the integrated recognition framework.

[0058] A class misclassification measure d_i(O) is defined as: $\begin{array}{cc}\begin{array}{c}{d}_i\left(O\right)=-g_i\left(O;\Omega \right)+\log \left\{{\left[\frac{1}{M-1}\sum _{j,j\ne i}\exp \left[g_i\left(O;\Omega \right)\eta \right]\right]}^{1/\eta }\right\}\\ =-g_i\left(O;\Omega \right)+G_i\left(O;\Omega \right)\end{array}& \left(11\right)\end{array}$

[0059] where η is a positive number. Class misclassification measure d_i(O) is a function of the parameter set Λ of the recognizer 122 and the parameter set θ of the compensatory neural network 120. If a class misclassification measure d_i(O) is embedded in a smoothed zero-one function, then the corresponding loss function l_i(O;Ω) can be defined as:

[0060] l_i(O;Ω)=l(d_i(O))  (12)

[0061] where I(•) is a sigmoid function, generally represented as: $\begin{array}{cc}l\left(d\right)=\frac{1}{1+\exp \left(-\alpha d+\beta \right)}& \left(13\right)\end{array}$

[0062] where α and β are respective parameters of this function, β is normally set to 0, and d represents d_i(O). In terms of an unknown feature vector series O, the recognition loss can be represented as: $\begin{array}{cc}l\left(O;\Omega \right)=\sum _{i=l}^M{l}_i\left(O;\Omega \right)I\left(O\in C_i\right)& \left(14\right)\end{array}$

[0063] where I(OεC_i) is an indication function to indicate whether feature vector series O belongs to a particular class C_i.

[0064] The expected error L(O) for a type of recognition problems can be defined as: $\begin{array}{cc}L\left(O\right)=E_O\left\{1\left(O;\Omega \right)\right\}=\sum _{i=1}^M{\int }_{O\in \mathrm{Ci}}{l}_i\left(O;\Omega \right)p\left(O\right)dO& \left(15\right)\end{array}$

[0065] Currently, there are a number of minimization a algorithms for minimizing the expected error. In this embodiment, the minimization algorithm's implemented by means of generalized probabilistic descent to reduce the expected error L(O) by an iterative procedure, as depicted in W. Chou, B. H. Juang, and C. H. Lee, “Segmental GPD Training of HMM-Based speech Recognizer,” Proceeding of ICASSP92, pp. I-473-I-476, 1992. Furthermore, in this embodiment, both the parameter set Λ of recognizer 122 and the parameter set θ of the compensatory neural network 120 are adjusted through the same adjustment means. The adjustment means for the parameter set Ω is then:

[0066] Ω_n+1=Ω_n−ε_n∇1(O;Ω) |_Ω=Ω  (16)

[0067] where ε_nis a learning rate.

[0068] Thus, all parameters can be adjusted according to equation (16) for reducing the expected error. For example, the weights w of the compensatory neural network may be adjusted in this manner, that is, $\begin{array}{cc}{w_{n+1}=w_n-{ϵ}_n\frac{\partial l_i\left(O;\Omega \right)}{\partial w}|}_{\Omega ={\Omega }_n}& \left(17\right)\end{array}$

[0069] where $\begin{array}{c}\frac{\partial l_i\left(O;\Omega \right)}{\partial w}=\frac{\partial l\left(d_i\left(O\right)\right)}{\partial d_i}·\frac{\partial d_i\left(O\right)}{\partial w}\\ =\alpha l\left(d_i\left(O\right)\right)\left(1-l\left(d_i\left(O\right)\right)\right)\left[\frac{\partial G_i\left(O;\Omega \right)}{\partial w}-\frac{\partial g_i\left(O;\Omega \right)}{\partial w}\right]\end{array}$

[0070] In addition, the above discriminant function g_i(O;Ω) can be determined according to the framework of the recognizer. For example, when the hidden Markov model based recognizer is used, the discriminant function g_i(O;Ω) may be defined as the likelihood function; whereas if the dynamic time warping (DTW) is adopted, then the discriminant function g_i(O;Ω) is a distance function.

[0071] Experiments on a speaker-independent Mandarin polysyllabic word recognition task are performed to examine effectiveness of the mentioned embodiments. In experiments, the number of words Go be recognized is 1038. A telephone speech database collected from 362 speakers is used for experiments. 7674 utterances uttered by 292 speakers are used for training. 1892 utterances uttered by the other 70 speakers are used for testing. All telephone speech signals are sampled at a rate of 8KHz and pre-emphasized with a digital filter, 1-0.95⁻¹. It is then analyzed for each Hamming-windowed frame of 20 ms with 10 ms frame shift. The feature vector consists of 12 mel-cepstral coefficients, 12 delta mel-cepstral coefficients, the delta energy, and he delta-delta energy.

[0072] The HMM-based speech recognizer employed 138 sub-syllable models as basic recognition units, including 100 3-state right-context-dependent INITIAL models and 38 5-state context-independent FINAL models. The observation distribution for each state of the HMM is modeled by a multivariate Gaussian mixture distribution. The number of mixture components in each state varies from one to ten depending on tie amount of training data. For silence, a single-state model with ten mixtures is used.

[0073] The compensatory neural network is a recurrent neural network (RNN) with a three-layer architecture; namely, an input layer, a hidden layer and an output layer. As described In the first embodiment, when the telephone channel effect compensation is performed independently by he compensatory neural network, the setting of the target function is a very important factor in training stage. In this embodiment, the target function is sea to be the same for all speech signals recorded from a telephone call. For example, O₁^s, O₂^s, . . . , O₃₀^sare feature vector series of 30 utterances spoken by the speaker S within a telephone call. When the input of the compensatory neural network is anyone of these feature vector series, then the target function may be set as: $\begin{array}{cc}B=\frac{\sum _{i=1}^{30}\sum _{t=1}^{T_i}{o}_i^S\left(t\right)}{\sum _{i=1}^{30}{T}_i}& \left(18\right)\end{array}$

[0074] where T_irepresents the number of speech frame of the i^thutterance. Error-back-propagation, algorithm is carried out to adjust parameters of the compensatory neural network. And this algorithm is performed on “one-utterance one back-propagation” basis.

[0075] Table 1 is a list of the recognition rates of the baseline system (no channel effect compensation), three compensation systems of the related art (RASTA, CMN, SBR), the first embodiment and the second embodiment.

[0076] From Table 1, it can be seen that the recognition rate can be effectively improved with the use of the first embodiment or the second embodiment. However, the preferred recognition result will be obtained in the second embodiment.

[0077] Although the present invention has been described in its preferred embodiments, it is not intended to limit the invention to the precise embodiment disclosed herein. Those who are skilled in this technology can still make various alterations and modifications without departing from the scope and spirit of this invention. Therefore, the scope of the present invention shall be defined and protected by the following claims and their equivalents.