Continuous Speech Recognition Apparatus

The present invention relates to a continuous speech recognition apparatus for recognizing a continuous speech.

It is very important to effectively recognize continuous natural speech with high reliability in a wordprocessor or speech input typewriter which deals with speech input data. Conventionally, a continuous speech recognition apparatus is known wherein a speech segment is used as the minimum unit of input speech to be recognized and a time-sequence of input speech feature parameters is converted to a series of phonemic symbols or segment lattice. However, coarticulation often occurs between two adjacent speech segments (phonemes) in a continuous speech, so that a given speech segment may have different feature parameters from those of an original speech segment. For this reason, it is very difficult to convert a continuous speech pattern to phonemic symbols with high precision.

Another continuous speech recognition apparatus is also known wherein a word unit is used as the minumum unit of input speech to be recognized each word unit is identified based on a sequence of input speech feature parameters, and a series of identified words is recognized as a sentence. According to this speech recognition apparatus, reference speech patterns indicating respective words are used. A feature parameter pattern indicating the input speech is compared with the corresponding reference speech pattern to calculate a similarity therebetween so as to recognize the input speech pattern in each word unit. Therefore, an influence due to the coarticulation described above can thus be substantially reduced. This recognition apparatus employs two word identification methods: one identification method wherein each word interval of an input speech is first detected to identify a word in the word interval; and the other identification method wherein a word is identified without detecting a word interval under the assumption that several words are present during the input speech interval. The word interval is determined by sequentially extracting feature parameters such as acoustic power or power spectrum of the input speech, and detecting a maximal or minimal point of change in the feature parameter. However, when words "I (ai)" and "eat (i:t)" are continuously pronounced to produce a speech input "I eat (ai:t)", the word interval of this speech cannot be correctly detected.

In the latter word identification method described above, reference speech patterns each having feature parameters of a plurality of frames are used to identify a corresponding one of words in the input speech pattern. For each frame, a distance between the feature parameters of the plurality of frames of the input speech and the reference speech pattern is calculated to detect a word giving a shortest distance in each frame. In this case, the distance between the feature parameter pattern of the input speech and the reference speech pattern can be calculated by a dynamic programming method, for example. All possible combinations of a series of words in the speech interval are made, and the input speech is then recognized by detecting one of the series of words giving a minimum total distance.

This word identification method is effective when a speaker is specified and word identification can be performed by using a small number of reference speech patterns. However, when a speaker is not specified, the input speech patterns of a word vary greatly from speaker to speaker. In order to process the speech data from nonspecified speakers, a great number of reference word patterns are required. In practice, it is impossible to prepare reference speech patterns for an indefinite number of nonspecified speakers. Therefore, it is impossible to accurately recognize the input speech patterns of an indefinite number of nonspecified speakers.

Speech data processing is recently proposed wherein a small number of reference patterns are used for the individual words, and speech data of a nonspecified speaker are processed utilizing a clustering technique. However, in this case, the recognition rate of a series of words is greatly decreased. Furthermore, the distance between the reference speech pattern and the feature parameter pattern of the input speech must be calculated in each frame, thus greatly increasing a total number of calculations. Therefore, it is very difficult to effectively recognize the input speech with high reliability.

It is an object of the present invention to provide a continuous speech recognition apparatus capable of effectively and reliably recognizing a continuous speech input of a nonspecified speaker.

In order to achieve the above object of the present invention, there is provided a continuous speech recognition apparatus comprising an acoustic analyzer for extracting feature parameter data of an input speech in each frame; a first memory for storing a plurality of reference pattern data each including reference parameter data of N frames; a partial similarity calculating circuit for calculating a partial similarity between the feature parameter data of each frame which is supplied from said acoustic analyzer and each reference parameter data stored in said first memory; a second memory for sequentially storing partial similarity data from said partial similarity calculating circuit for a predetermined number of frames; an operation circuit for calculating similarities between feature pattern data including N feature parameter data of the input speech and the reference pattern data on the basis of the N partial similarity data read out from said second memory which correspond to each of the reference pattern data and are present in at least one subperiod, and for generating largest similarity data among the calculated similarities; a third memory for storing the largest similarity data from said operation circuit, and reference pattern indication data and subperiod indication data which respectively indicate the reference pattern and the subperiod which are associated with the largest similarity data; and a recognition circuit for detecting a plurality of series of continuous subperiods in the speech interval and for recognizing the input speech on the basis of a series of reference pattern indication data corresponding to a series of continuous subperiods which provide the largest sum of similarity data associated with the continuous subperiods.

In the present invention, the partial similarity between the feature parameter data extracted in each frame and the reference parameter data of the reference pattern data is calculated during a one-frame period. The word similarity can be obtained in accordance with the partial similarity data previously calculated in association with the feature parameter data of a plurality of frames. Therefore, the number of calculations required to obtain the word similarity is greatly decreased.

FIG. 1 is a block diagram of a continuous speech recognition apparatus according to a first embodiment of the present invention;

FIG. 2 is a block diagram of a word similarity calculating circuit shown in FIG. 1;

FIG. 3 shows part of a memory shown in FIG. 2 and storing partial similarlity data;

FIG. 4 is a flow chart for explaining the mode of operation of the continuous speech recognition apparatus shown in FIGS. 1 to 3;

FIG. 5A is a graph for explaining the acoustic power of the input speech;

FIG. 5B shows word-periods obtained at given times in the speech interval; and

FIG. 6 shows an arrangement of word-periods represented by time data obtained by the word similarity calculating circuit shown in FIG. 2.

FIG. 1 shows a continuous speech recognition apparatus according to an embodiment of the present invention. This apparatus includes a microphone 2 for converting an input speech to an electrical speech signal; an acoustic analyzer 4 for extracting feature parameter data of the input speech for every one frame (e.g., 16 msec) in accordance with the electrical speech signal from the microphone 2; and a partial similarity calculating circuit 6 for calculating all similarity data between the feature parameter data of each frame supplied from the acoustic analyzer 4 and all feature parameter data stored in a reference memory 8. The acoustic analyzer 4 divides the frequency bandwidth of the speech signal into M (e.g., an integer within the range of 16 to 30) channels and generates M feature parameters by spectrum analysis using M band-pass filters. Each of reference pattern data stored in the reference memory 8 includes N-frame reference parameter data each having M acoustic parameters. The reference pattern data stored in the reference memory 8 are statistically obtained by processing the same kind of words generated from a number of nonspecified speakers. A number of variance-covariance matrices or correlation coefficient matrices corresponding to feature patterns of M.times.N for each word are calculated. J eigenvectors of these correlation coefficient matrices are sequentially extracted in the order from the largest eigenvalue. These J eigenvectors are used as reference pattern data. Therefore, the reference pattern of each word can be represented by orthogonal feature vectors.

A multiple word similarity Si between a feature pattern data formed of the N-frame feature parameter data and the reference pattern data representing a word i is given by the following equation: ##EQU1## where x is the feature vector constituted by the N-frame feature parameter data, r.sub.ij is the jth eigenvector of the reference pattern data of the word i, and (x,r.sub.ij) is the inner product of the feature vector x and the eigenvector r.sub.ij.

In order to obtain the similarity of the input feature pattern data in accordance with equation (1), for example, subperiods each including a plurality of frame intervals are detected each time one frame interval is elapsed. N feature parameter data are selectively extracted from each subperiod. Similarities between the extracted feature parameter data and the reference parameter data of reference pattern data are sequentially calculated. However, in this case, since it is necessary to selectively extract the N-frame feature parameter data from various subperiods, a large-capacity buffer memory is required to store a plurality of feature parameter data generated during at least a predetermined interval. Even if such a plurality of feature parameter data can be stored, the similarity data during each subperiod cannot be obtained in a real-time manner due to the large number of calculations to be performed.

According to the present invention, every time the one-frame feature parameter data is generated from the acoustic analyzer 4, a partial similarity between the feature parameter data and each of the reference parameter data of the reference pattern data is calculated. The word similarity Si can be obtained in accordance with the partial similarity thus obtained.

Assume that one-frame feature parameter data CP [=(C1, C2, . . . , Cm, . . . , CM)] is generated from the acoustic analyzer 4. Also assume that the eigenvector r.sub.ij of the reference pattern data having the N-frame feature parameter data and representing the word i is given as follows:

The reference parameter data r.sub.ij.sup.n of the nth frame of the eigenvector r.sub.ij in equation (2) is given as follows:

The partial similarity calculating circuit 6 calculates a partial similarity S.sub.ij.sup.n between the feature parameter data CP from the acoustic analyzer 4 and the nth-frame reference parameter data of the jth eigenvector of the reference pattern data representing the word i and stored in the reference memory 8 in accordance with the following equation: ##EQU2##

Assume that a number I of categories of words is given as 10 so as to recognize a numeric value, and that M=10, N=4 and J=5. All the partial similarities of the one-frame feature parameter data CP are obtained by multiplication and addition by 2000 (=M.times.N.times.J.times.I) times. For example, when one frame interval is given as 16 msec, each multiplication/addition is assigned 8 .mu.sec. Therefore, partial similarity calculation can be completed within each frame interval.

The partial similarity calculating circuit 6 calculates the N.times.J.times.I similarity data for the feature parameter data of each frame. These similarity data are sequentially supplied to a word similarity calculating circuit 10 within one frame interval. The word similarity calculating circuit 10 calculates the similarity Si between the feature pattern data of a predetermined number of frames within a subperiod which possibly includes a word in the speech interval and the reference pattern data on the basis of the partial similarity data obtained by the partial similarity calculating circuit 6, in accordance with the following equation: ##EQU3## On the other hand, a word recognition circuit 12 detects word-period series which are detected by the word similarity calculating circuit 10 and which each constitute a speech interval, and calculates a sum of similarities in all word-periods of each word-period series. Thereafter, the word recognition circuit 12 detects the word-period series which gives the largest similarity sum and recognizes as the effective word data the word series associated with the detected word-period series.

The word similarity calculating circuit 10 comprises a memory 10-1 for storing partial similarity data supplied from the partial similarity calculating circuit 6 for 10 frame intervals; a speech interval detecting circuit 10-2 for detecting a speech interval in accordance with the feature parameter data from the acoustic analyzer 4; an address designating circuit 10-3 for the memory 10-1; a control circuit 10-4 for controlling the operation of the memory 10-1 during the speech interval; and an operation circuit 10-5 for sequentially calculating word similarities in accordance with the partial similarity data selectively read out from the memory 10-1, and for sequentially supplying the operation results to a register 10-6.

The memory 10-1 may be constituted by a shift register for shifting and storing the partial similarity data supplied in each frame from the partial similarity calculating circuit 6 into frame areas F1 to F10. In each frame area of the memory 10-1, the I.times.N partial similarity data (I=10 and N=4 in this embodiment) which are supplied from the partial similarity calculating circuit 6 and which are associated with the J eigenvectors, respectively, are sequentially stored in a memory location (i,n) which is determined by the word category i and the number n of a frame in the N reference parameter data constituting the reference pattern.

For example, partial similarity data s.sub.11.sup.1, s.sub.12.sup.1, s.sub.13.sup.1, s.sub.14.sup.1 and s.sub.15.sup.1 (J=5) between the feature parameter data CP of a given frame and the reference parameter data of the first frame of word "0" are stored in a memory location (1,1) of the frame area F1 which is determined by i=1 and n=1. Similarly, partial similarity data s.sub.11.sup.4, s.sub.12.sup.4, s.sub.13.sup.4 s.sub.14.sup.4 and s.sub.15.sup.4 between the feature parameter data CP and the reference parameter data of the fourth frame of the word "0" are stored in a memory location (1,4). The partial similarity data which correspond to the feature parameter data CP and are generated from the partial similarity calculating circuit 6 are sequentially stored in the I.times.N (=40) memory locations of the frame area F1. In this case, the partial similarity data stored in the frame areas F1 to F9 are respectively shifted to the frame areas F2 to F10. The similarity calculation in this embodiment may be carried out in accordance with multiple similarity calculation method disclosed in U.S. Pat. No. 3,688,267, for example.

The operation of the continuous speech recognition apparatus shown in FIGS. 1 to 3 according to this embodiment of the present invention will now be described.

As previously described, the partial similarity calculating circuit 6 calculates the partial similarity data between the feature parameter data of each frame and each reference parameter data in each reference pattern data for every 16 msec, and the calculated results are supplied to the word similarity calculating circuit 10. When the control circuit 10-4 of the word similarity calculating circuit 10 detects in response to the output signal from the speech interval detecting circuit 10-2 that the speech interval has started, the control circuit 10-4 sets the memory 10-1 in the write mode. As shown in the flow chart in FIG. 4, the partial similarity data from the partial similarity calculating circuit 6 are sequentially stored in the memory 10-1. The control circuit 10-4 sets the memory 10-1 in the read mode in response to the output signal generated by the partial similarity calculating circuit 6 each time all the partial similarity data between the feature parameter data of each frame and the I.times.N reference parameter data are calculated and stored in the memory 10-1. The control circuit 10-4 sequentially specifies a plurality of word-periods (6 in this case) each constituted by a corresponding number of frame intervals (e.g., 5 to 10 frame intervals) including the frame interval which is now obtained, so that the partial similarity data associated with the feature parameter data of predetermined frame among the plurality of frames included in the specified word-period are read out from the memory 10-1.

Assume that the speech signal having the acoustic power characteristic shown in FIG. 5A is supplied to the acoustic analyzer 4. In this case, at time t0, when the beginning of the speech interval is detected by the speech interval detecting circuit 10-2, the partial similarity data from the partial similarity calculating circuit 6 are sequentially stored in the memory 10-1. Thereafter, when the interval (i.e., 5-frame interval in this embodiment) corresponding to the shortest word-period has elapsed, the control circuit 10-4 specifies this word-period. For example, at time t1 when the 8-frame interval has elapsed arter time t0, the partial similarity data which are obtained during the 8-frame interval are stored in the frame areas F1 to F8 of the memory 10-1. In this case, the control circuit 10-4 sequentially specifies word-periods WP1-1 to WP1-4 respectively constituted by the 5 to 8 frames and obtained by the time t1 which is given as the reference time. Referring to FIG. 5B, four frames each indicated by a circle represent resampling frames used for word similarity calculation. The number of sampling frames is determined in accordance with the number of frames of the reference pattern data of the reference pattern data.

Assume that the 5-frame word-period is specified. In the 5-frame word-period, the first, third, fourth and fifth frames are specified as the resampling frames. The control circuit 10-4 specifies the frame areas F1, F3, F4 and F5 in the order named. The control circuit 10-4 supplies address data to the address designating circuit 10-3 to specify the frame area F1 and specifies the memory location (1,4) to read out the corresponding partial similarity data. The control circuit 10-4 subsequently specifies the memory location (1,3) of the frame area F3, the memory location (1,2) of the frame area F4 and the memory location (1,1) or the frame area F5 so as to read out the corresponding data therefrom. The operation circuit 10-5 calculates tne word similarity in accordance with equation (5) between the feature pattern data obtained at time t1 and the reference pattern data associated with the word "0" on the basis of the partial similarity data read out from the memory 10-1, and temporarily stores the calculated data. Subsequently, the control circuit 10-4 reads out the partial similarity data from the memory locations (2,4), (2,3), (2,2) and (2,1) of the respective memory areas F1, F3, F4 and F5. The operation circuit 10-5 calculates the word similarity in accordance with equation (5) between the feature pattern data obtained at time t1 and the reference pattern data associated with the word "1 " on the basis of the readout partial similarity data, and temporarily store the calculated data. In this manner, the word similarities between the feature pattern data of the feature parameter data of 4 frames out of 5 frames obtained at time t1 and the reference pattern data respectively representing the words "0" to "9 " are calculated. The largest one of 10 word similarity data which exceeds a predetermined value is stored in the register 10-6. In this case, the word data associated with the largest word similarity and the time data associated with the time t1 and the word-period length are stored together with the largest similarity data in the register 10-6. In this case, the word data and the similarity data can be dealt with as a series of data and the data representing the word-period length can be read out from the control circuit 10-4. For example, the data concerning the time t1 is given by a count signal from a counter (not shown) which is reset in response to the output signal representing the start of the speech interval and generated from the speech interval detecting circuit 10-2 and which counts an output signal which is generated from the partial similarity calculating circuit 6 each trme all the partial similarity data are stored in the memory 10-1.

The control circuit 10-4 then specifies the 6-frame word-period. In this case, in order to calculate the similarity of each word in accordance with equation (5), the control circuit 10-4 reads out the partial similarity data from the memory locations (i,4), (i,3), (i,2) and (i,1) of the respective frame areas F1, F3, F5 and F6. The operation circuit 10-5 calculates the similarities of the words in accordance with equation (5) on the basis of the partial similarity data read out from the memory 10-1 and supplies the largest similarity data to the register 10-6 in the same manner as described above.

The operation as described above is repeated. The control circuit 10-4 specifies the 7-frame word-period and the 8-frame word-period and supplies the largest similarity data, the word data and the time data of each word-period to the register 10-6. Thus, these data are stored in the register 10-6.

When all similarity calculations obtainable for all the word-periods at time t1 are completed, the control circuit 10-4 sets the memory 10-1 in the write mode. Thus, the partial similarity data of the next frame are stored in the frame area F1 of the memory 10-1.

At a predetermined time (e.g., time t2) after the partial similarity data of the 10-frame word-period (i.e., longest word-period) are written, as shown in FIG. 5B, the word-periods WP2-1 to WP2-6 respectively having 5 to 10 frames are sequentially specified, and the similarity calculations are performed in the same manner as described above. As a result. the largest similarity data, the word data and the time data of each word-period are stored in the register 10-6.

The same operation as described above is repeated for every frame until the signal representing the end of the speech interval is generated from the speech interval detecting circuit 10-2.

At the end of the speech interval, the word-period length data, the time data representing the end frame position of each of the word-periods, and similarity data are stored in the register 10-6, as shown in FIG. 6. The word recognition circuit 12 detects all word-period series which constitute the speech interval in accordance with the time data stored in the register 10-6. For example, as shown in FIG. 6, the word recognition circuit 12 detects the word-periods respectively associated with the word data W1, W11, W21 and W25, the word-periods respectively associated with the word data W3, W13, W20 and W25, the word-periods respectively associated with the word data W6, W15, W19 and W25, and the word-periods respectively associated with the word data W12, W21 and W25. Tne word recognition circuit 12 then calculates the sums of similarities (S1+S11+S21+S25), (S3+S13+S20+S25), (S6+S15+S19+S25) and (S12+S21+S25) in the word-periods of the respective word-period series. One of the word-period series having the largest sum of similarities is selected, and the word series corresponding to the largest word-period series is recognized as the effective word data. This word series recognition is performed in accordance with a dynamic programming method or parallel retrieval.

The present invention is exemplifed by the preferred embodiment described above but is not limited to the particular embodiment. In the above embodiment, by way of simplicity, the shortest word-period is constituted by 5 frames and the longest word-period is constituted by 10 frames. However, the shortest and longest word periods can have other number of frames, for example, a 10-frame period and a 30-frame period.

In the above embodiment, after the end of the speech interval, the word recognition, circuit 12 performs word series recognition in accordance with the word similarity data, the word data and the time data which are stored in the register 10-6. However, during the speech interval, the similarity data associated with the continuous word-periods can be sequentially calculated in accordance with the time data and the similarity data which are stored in the register 10-6 for every frame. In this case, a sum of similarity data in the word-period series within the speech interval is obtained for each frame, so that a substantially real-time recognition of the input speech pattern can be performed.

In addition to this modification, the partial similarity data can be represented by a Mahalanobis distance or a statistical discrimination function.

In the above embodiment, the minimum unit of speech recognition is the word, but the minimu unit may be extended to include a syllable or phrase

The parameters M, N, I, J and F used in the above embodiment can be arbitrarily selected in accordance with the types of input speech to be recognized and the required recognition precision.

Furthermore, the speech interval detecting circuit 10-2 can be omitted. In this case, the control circuit 10-4 is operated during a time interval determined independently from the input speech.