Speech Recognition Training

The present invention is directed to speech recognition training. More particularly, the present invention is directed to speech recognition training using Hidden Markov Models.

A popular approach to performing speech recognition is to use Hidden Markov Models (HMMs). An HMM is a probabilistic function of a Markov chain and can be defined as {S,X,II,A,B}, where S={s.sub.1, s.sub.2, . . . , s.sub.n } are the Markov chain states, X denotes the HMM output (observation) set, II is a vector of state initial probabilities, A=›a.sub.ij !.sub.n,n is a matrix of state transition probabilities (a.sub.ij =Pr{s.sub.j .vertline.s.sub.i }), and B(x)=diag {b.sub.j (x)} is a diagonal matrix of the output x.epsilon.X conditional probability densities in state s.sub.j. If X is discrete, B(x) is a matrix of probabilities (b.sub.j (x)=Pr {x.vertline.s.sub.j }). Without loss of generality, states are denoted by their indices (s.sub.i=i).

In order for a device to perform speech recognition, that device must first fit HMMs to experimental data which entails generating model parameters. This process is referred to as "training" the speech recognition device.

There are a number of well-known ways for building a Hidden Markov Model for speech recognition. For example, as set forth in L. Rabiner et al, "Fundamentals of Speech Recognitionp", Chapter 6, Section 6.15, a simple isolated word recognition model can be created by assigning each word in a vocabulary a separatic model, and estimating the model parameters (A, B, .pi.) that optimizes the likelihood of the training set observation vectors for that particular word. For each unknown word to be recognized, the system (a) carries out measurements to create an observation sequence X via feature analysis of the speech corresponding to the word; (b) calculates the likelihood for all possible word models; and (c) selects the word whose model likelihood is highest. Examples of the other speech recognition systems using Hidden Markov Models can be found in Rabiner et al. and in U.S. Pat. Nos. 4,587,670 to Levinson et al. (reissued as Re33,597) and 4,783,804 to Juang et al. which are incorporated by reference herein.

There are various known methods to perform training using HMMs by optimizing a certain criterion (e.g., a likelihood function, an a posteriori probability, an average discrimination measure, etc.). However, these known methods all have drawbacks. For example, known methods that use the Newton-Raphson algorithm or the Conjugate Gradient algorithm, both of which are disclosed in W. H. Press, et al., "Numerical Recipes in C", Cambridge University Press (1992), converge fast in a small vicinity of optimum, but are not robust. Therefore, if parameter values are not very close to optimum, they might no converge.

Further, a known method that uses the Baum-Welch algorithm in conjunction with the forward-backward algorithm (the "Baum-Welch" method) is disclosed in L. E. Baum et al., "A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains", Ann. Math. Statist, 41, pp. 164-171 (1970). Training using this method converges slowly and requires a large amount of memory. therefore, training using this method must be implemented on a powerful computer with a large amount of memory.

Various approaches are known that speed up the Baum-Welch method. For example, W. Turin, "Fitting Probabilistic Automata via the EM Algorithm", Commun. Statist.--Stochastic Models, 12, No. 3, (1996) pp, 405-424 discloses that the speed of the forward-backward algorithm can be increased if observation sequences have repeated patterns and, in particular, long stretches of repeated observations. S. Sivaprakasam et. al., "A Foward-Only Procedure for Estimating Hidden Markov Models", GLOBECOM (1995) discloses that in the case of discrete observations, a forward only algorithm can be used that is equivalent to the forward-backward algorithm. However, these known approaches require specialized situations (i.e., long stretches of repeated observations and discrete observations).

Based on the foregoing, there is a need for a speech recognition training method and apparatus for generalized situations that is robust and does not require a large amount of memory.

The present invention is a method and system for performing speech recognition training using Hidden Markov Models that satisfies the above needs and more. In one embodiment, the present invention first stores preprocessed speech signals that include a plurality of observations. Initial Hidden Markov Model (HMM) parameters are then assigned. Summations are then calculated using modified equations derived substantially from the following equations, wherein u.ltoreq.v.ltoreq.w:

and

The calculated summations are then used to perform HMM parameter reestimation. The present invention then determines whether the HMM parameters have converged. If they have, the HMM parameters are then stored. However, if the HMM parameters have not converged, the present invention again calculates summations, performs HMM parameter reestimation using the summations, and determines whether the parameters have converged. This process is repeated iteratively until the HMM parameters have converged.

FIG. 1 is flow diagram of a speech algorithm training apparatus in accordance with the present invention.

FIG. 2 is a flow chart of the steps performed by the prior art to perform summation equations.

FIG. 3 is a flowchart of the steps performed by one embodiment of the present invention to calculate summations.

FIG. 4 is a flowchart of the steps performed by another embodiment of the present invention to calculate summations.

FIG. 5 is a flowchart of the steps performed by another embodiment of the present invention to calculate summations.

The present invention is a method and apparatus for performing speech algorithm training. The present invention implements unique algorithms in order to generate HMM parameters and summations that are used in conjunction with the Baum-Welch algorithm. The use of the algorithms enables the speech recognition training of the present invention be more robust than the prior art and require less memory.

FIG. 1 is flow diagram of a speech algorithm training apparatus in accordance with the present invention. Because the present invention does not require a large amount of memory, it can be implemented on a relatively small computer compared to prior art speech algorithm training methods. In one embodiment, the present invention is implemented on a general purpose computer that includes a processor and a storage device. In another embodiment, the present invention is implemented on a computer with parallel processors.

Storage device 10 in FIG. 1 stores preprocessed speech samples, or training data, in the form of feature vectors. Preprocessing speech samples and converting them to vectors is well known and is disclosed in, for example, L. Rabiner et al., "Fundamentals of Speech Recognition", Prentice Hall, Englewood Cliffs, N.J. (1993).

For example, a sequence of measurements can be made on a speech input signal to define a test pattern. For speech signals, the feature measurements are usually the output of some type of spectral analysis technique, such as a filter bank analyzer, a linear predictive coding ("LPC") analysis, conversion to cepstral coefficients (or delta cepstral coefficients or delta energy, etc.), or a discrete Fourier transform ("DFT") analysis. Such acoustic feature analysis is well known in the art. For example, the two dominant methods of spectral analysis--namely filter-bank spectrum analysis and LPC analysis--are discussed at length in Chapter 3 of Rabiner et al. The output of the acoustic feature analysis is a time sequence of spectral feature vectors, or, as it is often referred to, a speech pattern.

At step 12, HMM initialization is performed in a known manner by assigning initial HMM parameters. One known method for performing HMM initialization uses the segmental K-means algorithm which is disclosed in L. Rabiner et al., "Fundamentals of Speech Recognition", Prentice Hall, Englewood Cliffs, N.J. (1993). The initial parameters are provided to step 14, described below, when the speech algorithm training apparatus is initialized.

At step 14 one of the modified forward-backward algorithms that will be described in detail below performs calculations based on training data stored in storage unit 10 and old HMM parameters. The old HMM parameters are input from either step 12 when the speech algorithm training apparatus is initialized, or from step 20, described below. The calculations are used in step 16 described below.

At step 16, model reestimation is performed using the Baum-Welch equations to improve the parameters received from step 14. Model reestimation using the Baum-Welch algorithm is an iterative procedure for maximum likelihood estimation of the parameters. The Baum-Welch algorithm finds the maximum iteratively. The Baum-Welch algorithm is well known and is disclosed in, for example, L. E. Baum et al., "A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains", Ann. Math. Statist, 41, pp. 164-171 (1970), herein incorporated by reference.

When performing model reestimation in step 16 using the following Baum-Welch equations are used: ##EQU1## is the estimated mean number of transitions from state i to state j and observing a sequence x.sub.l.sup.T =(x.sub.1, x.sub.2, . . . , x.sub.T), ##EQU2## is the probability of being in state j at the moment t and observing x.sub.l.sup.T,

is the probability of transferring from state i to state j at the moment t and observing x.sub.l.sup.T. Equations (1a)--(1f) are similar to the equations for estimating parameters of Markov chains disclosed in P. Billingsley, "Statistical Methods in Markov Chains," Ann. Math. Statist, vol. 32, pp. 12-40 (1961). The only difference is that actually observed transitions are replaced by the estimated mean number of transitions.

Equations (1a) and (1b) are used to fit a hidden Markov chain, while equation (1c) is used to fit state observation probability densities.

Step 14 in FIG. 1 provides the summation calculations that are needed in equations (1b), (1c), (1d) and (1e). The method of performing step 14 in accordance with the present invention requires less computing power and storage space than the prior art method of calculating the summations.

At step 20, model convergence is checked by using one of the well known criteria disclosed in, for example, W. H. Press, et al., "Numerical Recipes in C", Cambridge University Press (1992). One criteria is to determine how much the new model parameters from step 16 differ from the old ones. If the difference is greater than a predetermined value, step 14 is performed again. If the difference is less than the predetermined value, the model parameters are stored in storage unit 18. The HMM model parameters can then be used by a speech recognition device in a known way.

Probabilities of various events can be evaluated using the notion of matrix probability. It is convenient to introduce an observation x matrix probability density as

Using this notation the probability density of x.sub.l.sup.T can be expressed as ##EQU3## where 1 is a column vector of ones. If observations are discrete this formula represents the sequence probability. The matrix probability density (or probability in the discrete case) of the sequence is defined as ##EQU4## Then equation (3) can be written as

Therefore, probability density of any event in the .sigma.-field generated by the observation sequences can be computed similarly using the matrix probability of the event.

Let l<t.sub.1 <t.sub.2 <. . . <t.sub.k <T be a partition of the interval ›1,T!, then matrix probabilities defined by equation (4) can be written as:

Each matrix probability P(x.sub.u.sup.v) in this equation can be evaluated independently (possibly on parallel processors) using matrix forward or backward algorithms:

or, more generally,

It is also convenient to assume that P(x.sub.u.sup.v)=I is a unit matrix for u>v. Fast matrix exponentiation algorithm can be applied if x.sub.u =x.sub.u+1 =. . . =x.sub.v =x.

Using these matrix probabilities we can write

and

The present invention performs the function of step 14 in FIG. 1 by utilizing a matrix form of the Baum-Welch algorithm. The matrix form is derived from the Expectation Maximization ("EM") algorithm, which is disclosed in, for example, W. Turin, "Fitting Probabilistic Automata via the EM Algorithm", Commun. Statist.--Stochastic Models, 12, No. 3, (1996) pp. 405-424, herein incorporated by reference.

The EM algorithm can be modified to be used for multiple observations. For example, suppose that there are several observation sequences {x.sub.k }.sub.l.sup.T.sbsp.k, k=1,2, . . . , k. In this case the EM algorithm takes the form ##EQU5## Equation (7c) can be solved analytically for discrete and exponential family observation probability distributions.

It follows from equations (7a) (7b) and (7c) that the main difficulty in applying the Baum-Welch algorithm, which is a special case of the EM algorithm, is computing sums of the type ##EQU6## where .eta..sub.t are some weights. In order to develop an efficient algorithms for computing these sums .eta..sub.t B.sub.ij (t,x.sub.l.sup.T,.theta.) can be presented in the following matrix form

where

e.sub.i =(0,0, . . . ,0,1,0, . . . ,0) is a unit vector whose i-th coordinate is 1. Thus, efficient algorithms for computing the following sums need to be developed: ##EQU7##

The prior art for computing these sums is called the forward-backward algorithm, which is depicted in the flowchart set forth in FIG. 2. It consists of two parts; a forward part and a backward part. In the forward part, the forward vectors .alpha.(x.sub.l.sup.l) are initialized to .alpha.(x.sub.l.sup.0)=.pi. in step 30 and are evaluted in step 34 according to the equation

and saved in memory at step 36. In the backward part, the backward vectors are computed recursively in step 46 as .beta.(x.sub.r+1.sup.r)=1, and

then .beta..sub.ij (t, x.sub.l.sup.T, .sup..theta..sub.p) are computed according to equation (1f) and .eta..sub.t .beta..sub.ij (t,x.sub.l.sup.T, .sup..theta..sub.p) are added in step 48 to accumulators according to equation (1d). In step 52, the accumulated sums are sent to step 16 of FIG. 1 which performs the model parameter reestimation. Thus, the prior art requires storage for saving the forward vectors which is proportional to the observation sequence length T.

direct application of this equation, accordingly, usually requires an enormous amount of memory if T is large and .alpha.(x.sub.l.sup.t-1) ›or .beta.(x.sub.t+1.sup.T ! are saved in the computer memory. Alternatively, if both .alpha.(x.sub.l.sup.t-1) and .beta.(x.sub.t+1.sup.T) are calculated on-the-fly, it requires an enormous amount of the processing time.

In contrast, the present invention utilizes recursive algorithms to calculate the sum and, as is clear from the equations below, needs a storage size which is independent of the sequence length. ##EQU8## To calculate this matrix sum recursively, denote ##EQU9## It is easy to see that for any u.ltoreq.v<w

Equation (12) together with equation (6) form the basis for the modified foward-backward algorithms implemented by the present invention and illustrated by the flowchart in FIG. 3. Depending on the order of the matrix evaluation they can be treated as parallel, forward-only, or backward-only algorithms. It follows from equation (1e) that sums of the following form need to be calculated: ##EQU10##

If it is not necessary to calculate S.sub.ijT for some other parameter, to save memory and speed-up the computation recursive algorithms for calculating S.sub.jT directly can be derived. These algorithms can be obtained from the corresponding algorithms for S.sub.ijT by summing both sides of the algorithm equations with respect to i. Thus, for example, equation (12) becomes: ##EQU11## Parallel Algorithms

FIG. 3 is a flowchart of the steps performed by one embodiment of the present invention to perform step 14 of FIG. 1. First, the accumulators (i.e., locations in memory where the sums are accumulated) are initialized in a known manner. Initialization is typically achieved by clearing the memory.

In steps 130-135, the present invention reads training data from storage device 10 shown in FIG. 1. Training data can be read in parallel.

In steps 140-145, P's and .OMEGA.'s are calculated for each training data read in steps 131-133 in parallel. Steps 140-145 are performed as follows:

Equation (12) allows calculations to be performed on parallel processors. To be more specific, denote 1<t.sub.1 <t.sub.2 <. . . <t.sub.k <T a partition of the interval ›1, T!, then it follows that:

These equations allow computations to be performed on parallel processors. Indeed, matrices P(x.sub.t.sbsb.k.sub.+1.sup.t.sbsp.k+1) and .OMEGA..sub.ij (x.sub.t.sbsb.k.sub.+1.sup.t.sbsp.k+1) can be evaluated independently on parallel processors. Then equations (13) can be applied in the following way at steps 150 and 152 of FIG. 3:

Compute .OMEGA..sub.ij (x.sub.l.sup.t.sbsp.l) using any of the previously described algorithms. Then, using (13b) the following is obtained:

and equation (13a) gives

Applying equations (13) again results in

and

and so on.

These equations are valid for any partition of the interval ›1, T!. However, since matrices P(x.sub.t.sbsb.k.sub.+1.sup.t.sbsp.k+1) and .OMEGA..sub.ij (x.sub.t.sbsb.k.sub.+1.sup.t.sbsp.k+1) are the same for the repeated observation patterns, the partition should take advantage of this property.

In the special case in which t.sub.k =t and t.sub.k+1 =t+1,

and equations (13) take the form

Note that evaluation of .OMEGA..sub.ij (x.sub.l.sup.T) is performed in the forward-only fashion. Alternatively, the backward-only algorithm can also be applied starting with .OMEGA..sub.ij (x.sub.t.sbsb.k.sup.T), P(x.sub.t.sbsb.k.sup.T) and recursively computing

However, it is obvious that the actual direction of evaluation is not important. The present invention can compute part of the matrix products by forward-only algorithm and part of the products by backward-only algorithm. In the parallel implementation, the products can be evaluated at the moment when all the matrices in the right hand side of equations (13a) and (13b) are available. In this case the evaluation direction is defined by a tree in which the value .OMEGA..sub.ij (x.sub.u.sup.w) in the parent node is evaluated using the values .OMEGA..sub.ij (x.sub.v.sup.u) and .OMEGA..sub.ij (x.sub.v+1.sup.w) of its children according to equations (13a) and (13b); .OMEGA..sub.ij (x.sub.l.sup.T) is obtained at the root of the tree.

In the special case in which t.sub.k =t and t.sub.k+1 =t+1, equations (16) become

Forward-Only Algorithms

FIG. 4 is a flowchart of the steps performed by another embodiment of the present invention to perform step 14 of FIG. 1. First, the accumulators (i.e., locations in memory where the sums are accumulated) are initialized in a known manner. Initialization is typically achieved by clearing the memory.

In steps 230-235, the present invention reads training data from storage device 10 shown in FIG. 1. Training data can be read sequentially, or in parallel to speed up processing.

In steps 240-245, P's and .OMEGA.'s are calculated for training data read in steps 230-235 sequentially or in parallel. In steps 250-254, .alpha.'s and .omega.'s are calculated using input from steps 240-245 in a forward-only manner.

To speed up calculations and reduce computer memory requirements, matrix equations can be converted into vector equations by multiplying equations (13) from the left by II:

where

.OMEGA..sub.ij (x.sub.t.sbsb.k.sub.+1.sup.t.sbsp.k+1) can still be evaluated on parallel processors as in FIG. 3, but .omega..sub.ij (x.sub.1.sup.t.sbsp.k+1) are evaluated sequentially. Thus, equations (17) represent a forward-only algorithm.

If t.sub.k =t and t.sub.k+1 =t+1, equations (17) become

This specialized version of the forward-only is disclosed in, for example, N. Tan, "Adaptive Channel/Code Matching," Ph.D. dissertation, University of Southern California (Nov. 1993).

The sum in equation (9) is found as

We still need to calculate P(x.sub.1.sup.t) ›using forward-only equation (15a)! for reestimating II according to equation (1a). If we assume that the initial probability vector is fixed, there is no need to use equation (1a) and calculate P(x.sub.1.sup.T).

Since all products of probabilities tend to zero, to increase the calculation accuracy and avoid underflow, it is necessary to scale the equations as disclosed in L. Rabiner et al., "Fundamentals of Speech Recognition", Prentice Hall, Englewood Cliffs, N.J. (1993). Multiplying right-hand sides of equations (18) by the common scale factor c.sub.t we obtain

In principle, c.sub.t can be any sequence since in reestimation equations (1a), (1b), (1c), and (1e) numerators and denominators are multiplied by the same factor II.sub.t c.sub.t. However, it is recommended in L. Rabiner et al., "Fundamentals of Speech Recognition", Prentice Hall, Englewood Cliffs, N.J. (1993) to normalize .alpha.(x.sub.1.sup.t+1). Thus,

and we have the following relations between normalized and non-normalized values

Advantageously, for the selected scale factors, the observation log-likelihood can be evaluated as ##EQU12## and equation (1a) is also simplified:

In some applications, we assume that the process is stationary in which case the initial state probability distribution satisfies the following equations

The stationary distribution can be estimated as a by-product of the Baum-Welch algorithm as disclosed in W. Turin et al., "Modeling Error Sources in Digital Channels", IEEE Jour. Sel. Areas in Commun., 11, No. 3, pp. 340-347 (1993).

Backward-Only Algorithm

FIG. 5 is a flowchart of the steps performed by another embodiment of the present invention to perform step 14 of FIG. 1. First, the accumulators (i.e., locations in memory where the sums are accumulated) are initialized in a known manner. Initialization is typically achieved by clearing the memory.

In steps 330-335, the present invention reads training data from storage device 10 shown in FIG. 1. Training data is read sequentially, or in parallel to speed up processing.

In steps 340-345, P's and .OMEGA.'s are calculated for training data read in steps 330-335 sequentially or in parallel. In steps 350-354, .beta.'s and v's are calculate using input from steps 340-345 in a backward-only manner.

Equations (13) form a basis for the backward-only algorithm. Multiplying both sides of these equations from the right by 1 we replace matrices with column vectors:

where

In particular, if t.sub.k =t and t.sub.k+1 =t+1 we have

The sum in (9) can be written as

Since equations (20) compute .beta.(x.sub.1.sup.t) which is needed for reestimation of II, we do not need to compute P(x.sub.1.sup.T) as in the forward-only alogrithm. Thus, backward-only algorithm (20) is simpler than the forward-only algorithm.

As previously described, it is important to perform scaling to improve the algorithm precision. The scaled version of the algorithm has the form:

This selection of the scale factors allows us to calculate the observation sequence likelihood as by-product of the Baum-Welch algorithm.

Specific Implementation for Discrete Distributions and Mixtures of Densities

In this section a detailed implementation of the present invention is given for the most widely used types of observation probability distributions: discrete distributions and mixtures of densities. In this case of discrete distributions, equation (1c) has the following solution: ##EQU13## where .delta.(x,x.sub.t) is the Kronecker delta function ›.delta.(x,x.sub.t)=1 if x=x.sub.t and .delta.(x,x.sub.t)=0 otherwise!. Thus we can use forward-only or backward-only algorithms with .eta..sub.t =.delta.(x,x.sub.t).

The forward-only algorithm according to equations (18) takes the form

Equation (21b) can be rewritten in the following explicit form ##EQU14##

The initial conditions for the algorithm are ##EQU15## equations (20) can be rewritten as ##EQU16## The scaled version of the algorithm takes the form

The Markov chain stationary distribution can be estimated by ##EQU17##

As pointed out before, the backward-only algorithm is simpler than the forward-only algorithm in the general case. According to equations (20), the backward-only algorithm can be described as follows:

Initialize

For t=T,T-1, . . . , 1 compute

Reestimate parameters: ##EQU18## Repeat iterations over p till convergence.

In the case of mixture densities ##EQU19## where x is the observation vector and c.sub.jk are mixture coefficients. In this case the solution of equation (1c) has the form ##EQU20## Thus, we need to calculate .gamma..sub.t (j,k) based on equation (25b). After substitutions we see that ##EQU21## Thus, we need to calculate the sum in equation (8) with .eta..sub.t =p.sub.jk (x.sub.t ;.PSI..sub.jk)/b.sub.j (x.sub.t ;.PSI.j). Forward-only algorithm (18) takes the form

where

is a matrix whose i,j-th element is a.sub.ij p.sub.jk (x.sub.t ;.PSI..sub.jk).

The Model parameters are reestimated according to equations (1) and (25) which can be written as ##EQU22## The backward-only algorithm has a similar form. Also, the equations should be scaled to avoid numerical underflow as is in the case of discrete distributions considered previously.

For the Gaussian mixtures, we need to calculate ##EQU23## which can be written in the coordinate form: ##EQU24## where x.sub.t.sup.(i) is i-th coordinate of x.sub.t. These sums can also be evaluated using the forward-only or backward-only algorithm with .eta..sub.t =x.sub.t.sup.(i) p.sub.jk (x.sub.t ;.PSI..sub.jk)/b.sub.j (x.sub.t ;.PSI..sub.j) for the first sum and .eta..sub.t =x.sub.t.sup.(i) x.sub.t.sup.(l) p.sub.jk (x.sub.t ;.PSI..sub.jk)/b.sub.j (x.sub.t ;.PSI..sub.j) for the second sum.

The forward-only algorithm is given by the following equations

where

A.sub.j is j-th column of matrix A. The reestimation equations can be written in the coordinate form as ##EQU25## All these equations need to be scaled as previously described. Equations for the backward-only and parallel algorithm can be derived similarly.

Several embodiments of the present invention are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and with in the purview of the appended claims without departing from the spirit and intended scope of the invention. For example, although the present invention is described in conjunction with a speech recognition training system, it also can be used in any other type of training system that uses HMMs, such as a handwriting recognition system. It can also be applied to modeling error sources in communication channels, in signal quantization and compression (e.g., speech or image signals, etc.).