Method for correcting error characters in results of speech recognition and speech recognition system using the same

[0002] The invention relates to speech recognition technology and, more particularly, to a method for correcting error characters in results of speech recognition and a speech recognition system using the same.

[0003] Speech recognition technology is a technology for accurately recognizing human speech (for example, character, word, sub-sentence, sentence, etc.) by using computer and digital signal processing technology. Speech recognition is based on collecting the various valuable speech characteristics to be recognized, forming the acoustic model to be recognized, and comparing with the sample model stored in the computer, as well as recognizing what are the characters and words through pattern classification methods. The speech recognition process is a recognition process for syllable or word language composition, etc. No doubt speech recognition is a fast and effective way to input text into a computer. Though a great deal of research for speech recognition has been performed until now, recognition in continuous speech, speaker independent, and long word is in the exploring stage because of the complexity of the language. Thus, error correction for the results of speech recognition is an indispensable step, because the accuracy of speech recognition can never reach 100%.

[0004] Friendliness and efficiency of the alternative input methods in the error correction process is very important since it is a part of the complete speech input process, and it can be a deciding factor regarding user's acceptance of the speech input methods. Generally, different input methods such as handwriting input or various types of stroke-based input have been used to correct the error characters in the results associated with speech recognition because the users of the speech recognition system often do not want to use a keyboard or are not familiar with it, and these users more desirably use the stroke-based handwriting input methods, such as handwriting input, stroke-based input or stroke type-based input, which are approximate to the natural handwriting habits. However, such handwriting recognition technology is not a mature technology, thus the error correction efficiency for the results of speech recognition is reduced. The current various error correction methods so far used for the results of speech recognition do not taken advantage of the useful acoustic information generated from the speech recognition process.

[0005] An object of the invention is to use effectively the useful acoustic information generated from the speech recognition process, so as to improve the error correction efficiency of speech recognition, that is, to improve the reliability and speed of the error correction.

[0006] The invention fully exploits the useful acoustic information obtained in the speech recognition process to maximize the error correction efficiency for the results associated with speech recognition by using the alternative stroke-based input methods. The invention automatically retains and processes the valuable acoustic information from the speech recognition process. This is accomplished via internal data transfer and incorporation of an evaluation procedure involving several statistical models. The invention uses a confusion matrix to generate an acoustic model, and the acoustic model cooperates with character level and word level language models to optimize the error correction processing.

[0007] According to an aspect of the invention, a method for correcting one or more error characters in results of speech recognition comprises the steps of:

[0008] marking the one or more error characters in the speech recognition results;

[0009] inputting one or more correct characters corresponding to the one or more marked error characters by input based on character-shape;

[0010] recognizing the input based on character-shape;

[0011] displaying one or more candidate characters;

[0012] selecting one or more desired characters from the one or more candidate characters in accordance with the user; and

[0013] replacing the one or more error characters with the one or more selected characters;

[0014] the method characterized by further comprising the step of filtering the one or more candidate characters in accordance with acoustic information associated with the one or more error characters.

[0015] According to another aspect of the invention, a speech recognition system capable of correcting one or more error characters in results of speech recognition comprises:

[0016] voice detection means for collecting a speech sample of a user;

[0017] pronunciation probability calculation means, which, for each pronunciation in an acoustic model, gives a probability estimation value of whether the pronunciation is the same as the speech sample;

[0018] word probability calculation means, which, according to a language model, gives a probability estimation value of a word occurring in a current context;

[0019] word matching means for calculating a joint probability through combining a probability value calculated by the pronunciation probability calculation means with a probability value calculated by the word probability calculation means and taking the word with the greatest joint probability value as the result of the speech recognition;

[0020] context generating means for modifying the current context by using the speech recognition result; and,

[0021] word output means;

[0022] the speech recognition system characterized by further comprising error correction means, user marking the one or more error characters in the results of the speech recognition via the error correction means, inputting one or more correct characters corresponding to the one or more error characters by input based on characters-shape, and the error correction means recognizing the input, generating one or more candidate characters and filtering the one or more candidate characters via acoustic information associated with the one or more error characters.

[0023] These and other objects and features of the invention will become more apparent from the detailed descriptions of the preferred embodiments of the invention in conjunction with the accompanying drawings in which:

[0024] FIG. 1 is a particular operation flow for correcting the error characters in the results of speech recognition according to an embodiment of the invention;

[0025] FIG. 2 is a general flow chart of the method of correcting the error characters in the results of speech recognition according to an embodiment of the invention;

[0026] FIG. 3 is a general flow chart for filtering the candidate characters by means of the acoustic information of the error characters obtained from a confusion matrix according to an embodiment of the invention;

[0027] FIG. 4 is a schematic diagram of the composition of the prior art speech recognition system;

[0028] FIG. 5 shows a speech recognition system for correcting the error characters in the results of speech recognition according to an embodiment of the invention;

[0029] FIG. 6 is an error correction screen according to an embodiment of the invention; and

[0030] FIG. 7 is an example of a confusion matrix according to an embodiment of the invention.

[0031] In FIG. 1, there is shown an operation flow for correcting the error characters in the results of speech recognition by handwriting input according to an embodiment of the invention. When error characters are found in the results of the speech recognition, the correction is performed in the following steps:

[0032] Step 101: the user performs dictation, and the dictation may be repeated many times to get correct results;

[0033] Step 102: the speech recognition (SR) results are displayed on the screen;

[0034] Step 103: the user marks the error characters to be corrected;

[0035] Step 104: the system uses the acoustic identity of the error characters to retrieve the acoustic information associated with this error (in the form of statistical models) and incorporate it together with language models for ranking or selecting the candidates;

[0036] Step 105: with the alternative input methods such as handwriting input methods, the user inputs the correct characters corresponding to the marked error characters;

[0037] Step 106: when a recognition step is completed in the alternative input process, the system utilizes the models in Step 104 to rank the current candidate list to achieve better accuracy and speed;

[0038] Step 107: the resulting candidate list is displayed either in part or in total on the screen;

[0039] Step 108: the user selects the correct characters by the cursor etc.

[0040] FIG. 2 shows the error recovery procedure for the results of speech recognition when using the stroke based keyboard or handwriting inputs. As shown in FIG. 2:

[0041] Step 201: the user completes the first pass of dictation;

[0042] Step 202: the speech recognition (SR) results are displayed on the screen;

[0043] Step 203: the user checks the results. If there are no errors in the recognition results, then no correction is necessary and the input procedure is completed. If there exist one or more errors in the recognition results, the user marks the error to be corrected. This may be a word of multiple characters. The user will typically ask for the candidate list. If the correct character is in the list, the user goes to Step 209, otherwise the user goes to Step 204. This step is repeated for each error in the speech recognition results;

[0044] Step 204: the user inputs by speech the correct character (or word) corresponding to the marked error character (or word). The speech recognition machine will decode it with the acoustic model only (i.e., with the language model disabled). If the correct character (or word) appears as a candidate on the screen, the user goes to Step 209;

[0045] Step 205: if the character (or word) that appeared on the screen is not correct, the user may choose to repeat Step 204 again;

[0046] Step 206: when the error persists, the user begins to input the correct character, that is, to input in the sequence of the strokes of the character;

[0047] Step 207: according to the syllable type of this error character from Step 204, the system retrieves the corresponding statistic models associated with this error character from a confusion matrix. These models capture the statistically significant features of the error character, which may consist of the distributions of first Pinyin alphabets and/or the vowels of the error character;

[0048] Step 208: the acoustic models obtained from Step 207 are then combined with character and word level language models, leading to a probabilistic evaluation of the likelihood figures of candidate characters (or words) during the sequential stroke input process. These integrated models are used to rank and order the current set of candidates in the stroke based input system for high error correction efficiency;

[0049] Step 209: the user selects the correct character needed through clicking, and inputs its order number in the candidate list.

[0050] In following, combining with FIG. 3, a process will be described in which, according to a particular embodiment of the invention, the candidate characters are filtered by using the acoustic information obtained from the confusion matrix.

[0051] The acoustic information from the speech recognition process is used to rank and order the candidate characters (or words) effectively. It is described in detail as follows for the given error characters (or words) how to collect the acoustic information from the confusion matrix previously generated. It will be also described how to filter the candidate characters (or words) by combining the language model on the basis of the probability statistical model.

[0052] First, it will be introduced how to generate a confusion model. A confusion matrix is supposed to be generated in advance from speech input error data, and captures the error probabilities of the all syllables in the continuous speech input.

[0053] A syllable set in the Chinese is defined as:

[0054] SSet={S₁, S₂, . . . ,S_N}

[0055] In order to get the candidates for each decoded error E, we need to get the probability of each candidate character given the syllables of the decoded characters and its contexts, i. e.:

[0056] P(C|S_HE,H)

[0057] where C means a certain candidate, S_HEis the sequence of the syllables of the decoded characters, including the syllable of the decoded error itself and its nearest history, i. e., S_HE=S(H)+S(E), where S(H) means the sequence of the syllables of H and S(E) means the syllable of E. H is its history of the language contexts. Then, we rank the candidates according to the value of above probability.

[0058] With Bayes's rule, we can get that: $P\left(C❘S_{\mathrm{HE}},H\right)=\frac{P\left({\mathrm{CS}}_{\mathrm{HE}}H\right)}{P\left(S_{\mathrm{HE}}H\right)}=\frac{P\left(S_{\mathrm{HE}}H❘C\right)P\left(C\right)}{P\left(S_{\mathrm{HE}}H\right)}$

[0059] Because S_HEis the pure acoustic and H is pure language events, so we can regard them as independent variables. And S_HEand H are determined for a given decoded character. So the above equation can by simplified as: $\begin{array}{cc}\begin{array}{c}\mathrm{Rank}P\left(S_{\mathrm{HE}}H❘C\right)\frac{P\left(C\right)}{P\left(S_{\mathrm{HE}}H\right)}=\mathrm{Rank}P\left(S_{\mathrm{HE}}❘C\right)P\left(H❘C\right)P\left(C\right)\\ =\mathrm{Rank}\frac{P\left(C❘S_{\mathrm{HE}}\right)P\left(C❘H\right)}{P\left(C\right)}\end{array}& \left(1\right)\end{array}$

[0060] For practical reason, we simplify P(C|S_HE) as P(C_S|S_E), where C_Srepresents the syllable of C, S_Eis the syllable of the decoded error characters. The simplification indicates we ignore the acoustic context S(H) and group the characters with same syllable into one class.

[0061] While training, we take M test speakers, every speaker reads N test sentences. We decode these sentences for these speakers in syllables disregarding language model.

[0062] For each syllable S_Tin the test sentences, if it is decoded as S_D, where S_Dcould be S_Titself, we increment Count(S_T−S_D) one step in the confusion matrix shown in FIG. 7. Then we can get the probability of S_Tbeing recognized as S_D: $P\left(S_D❘S_T\right)=\frac{\mathrm{Count}\left(S_T-S_D\right)}{\sum \mathrm{Count}\left(S_T-S_M\right)}$ $\mathrm{for}\mathrm{all}{S}_M\in \mathrm{SSet}$

[0063] where S_T, S_DεSSet, Count(S_T−S_D) is the times of S_Tbeing recognized as S_D, ΣCount(S_T−S_M) is the summation of the row S_T, which means the total times S_Tbeing recognized as any syllable S_MεSSet. We can save P(S_D|S_T) in the final confusion matrix.

[0064] At the same time, we get: $\begin{array}{cc}P\left(S_T\right)=\frac{\mathrm{Count}\left(S_T\right)}{\sum \mathrm{Count}\left(S_M\right)}& \left(2\right)\\ \mathrm{for}\mathrm{all}{S}_M\in \mathrm{Training}\mathrm{Data}& \end{array}$

[0065] By using the confusion matrix, we are given the decoded syllable S_D, and we want to get the probability S_Dbeing from a certain S_T, i.e., P(S_T|S_D). With Bayes's rule: $P\left(S_T❘S_D\right)=\frac{P\left(S_D❘S_T\right)P\left(S_T\right)}{P\left(S_D\right)}$

[0066] When we calculate P(C_S|S_E): $P\left(C_S❘S_E\right)=P\left(S_T=C_S❘S_D=S_E\right)=\frac{P\left(S_D=S_E❘S_T=C_S\right)P\left(S_T=C_S\right)}{P\left(S_D=S_E\right)}$

[0067] P(S_D=S_E) is the same for all candidates, so it is no use for ranking. We get P(S_D=S_E|S_T=C_S) and P(S_T=S_E) in equation (2) from the confusion matrix.

[0068] While the above approach is suitable for suitable for losing candidates after a user completes his/her speech input the first time and looks for candidates with respect to a particular error character (Step 203 in the error recovery procedure described above), here we focus on the situation that the user has repeated the voice input for an erroneous character but failed, and is ready to use stroke based input method (Step 206). That means the error persisted when the correct word was uttered in a discrete mode.

[0069] Therefore, the confusion matrix should be trained by character errors recorded in a discrete speech input environment. Such a confusion matrix can be used together with the language models to rank and order the candidates generated in the stroke input process.

[0070] In order to be more robust against inaccuracies in confusion matrix entries (due to insufficient training data, unfamiliar accents, etc.), the following modification of the confusion matrix approach may be used to complement the original confusion matrix method. We may first cluster the syllables into different starting consonant groups (including a group with the starting consonant missing) and generate the consonant confusion matrix in the same way as we described above. We may also cluster the syllables into different vowel groups and generate the vowel confusion matrix. These matrices will give two independent acoustic models, which may be combined with a character (or word) level language model to evaluate the overall likelihood of effective candidates being the correct characters (or words).

[0071] FIG. 3 describes a process for filtering the candidate characters by means of the acoustic information obtained from a confusion matrix according to an embodiment of the invention.

[0072] Step 301: for the user's stroke-based input, recognition is made to generate a candidate list C={C₁, C₂. . . C_n};

[0073] Step 302: for every candidate in the candidate list C={C₁, C₂. . . C_n}, the likelihood of syllable S_Ciof the candidate and syllable S_Eof the recognized erroneous character is obtained from the confusion matrix;

[0074] Step 303: whether the likelihood obtained from the confusion matrix is greater than a threshold S_limor not is judged, if the judgment result is less than S_lim, this candidate is removed from the candidate list, and performing the Step 302 for the next candidate;

[0075] Step 304: if the judgment result is that the likelihood obtained from the confusion matrix is greater than or equal to S_lim, this candidate is remained as a member in the candidate list to be displayed to the user, and performing the Step 302 for the next candidate;

[0076] Step 305: after the above steps are completed for all the candidates in the candidate list C={C₁, C₂. . . C_n}, the candidates remained in the candidate list are displayed to the user.

[0077] When “” is decoded as “”, for example, in a certain speech recognition process, we input the stroke “-” by means of the stroke-based input method in order to correct the error, at this time:

[0079] If the acoustic information is not used, the correct character “” is in the last position, and the candidate list will be displayed as follows to the user via the processing of the above Step 301-305:

[0081] Thereby, it is seen that the error correction efficiency can be improved by means of the acoustic information.

[0082] In addition, we can use the language model (LM) to restrict the candidate list further. To use LM to prune the candidate list, we note that since an error is assumed to occur in a discrete word input environment, we only need to consider the unigram word language model. In other words, we simply incorporate the single word frequency in the evaluation and ordering of active candidates. We also recommend the weighting of acoustic and language models, and the reduced weight on the language model will be more robust with an unfamiliar topic.

[0083] In addition, with a stroke based input, the set of all covered characters are organized as a tree according to the stroke or stroke type sequence. When strokes (stroke types) are input sequentially, the system will trace the tree and keep only the candidates hanging off the particular branch it has traced to be active. The combined acoustic (confusion matrix) and language models will be used to rank the current active candidates and order them according to their overall likelihood value.

[0084] The effectiveness of such acoustically aided stroke inputs may be evaluated as follows. When a stroke/stroke type input system is used independently without the aid of any a priori acoustic information, it takes an average of 5.36 strokes to narrow the set of 6763 commonly used characters down to a candidate list of 10 or under. When we use the character, word level language model to anticipate the likely multi-character words, the average effective number of strokes per character is reduced to 3.12.

[0085] When inputting strokes with the aid of a priori acoustic information, if we assume that 90% (or 60%) of the first Pin-Yin alphabet of the erroneous character is correct, then the average number of strokes for catching the top ten correct candidates in the candidate list is no more than 2.8 (or 3.7). If we assume that 90% (or 60%) of the vowel is correct, then the average stroke number to catch the correct candidate is no more than 2.5 (or 3.4). With both models working together (both consonant and vowel information used), the average stroke number reduces to no more than 1.0027.

[0086] If the a priori acoustic information is provided through the 100-length confusion matrix, while the speech machine cannot provide the correct character in top ten candidates, the stroke based input method is needed. If the 10-100 confusion set can cover 60% of correct candidates of the erroneous characters, our initial confusion matrix data lead to an estimate of the average stroke number to be 3.572. These numbers are obtained using the acoustic models together with a character level language model. With the word level prediction approach, the effective average number of strokes per character will then be further reduced. This figure is estimated to be 2.2-2.56.

[0087] If there are other erroneous characters in the error word, the confusion matrix of other erroneous characters will be used to provide, respectively, character candidates to be evaluated together with the language model. The effective average number of strokes per character may also be estimated.

[0088] Thereby, it is seen that the error correction efficiency of the speech recognition can be greatly improved by means of the acoustic information. A speech recognition system using such speech aided method to perform error correction is described as follows.

[0089] As shown in FIG. 4, a general speech recognition system consists of an acoustic model 7 and a language model 8. The acoustic model 7 includes the pronunciations of common words used in the recognized language. Such pronunciations are statistically summarized from the voices which are pronounced by most people in reading a certain word, and represents the general feature of this word. The language model 8 includes the use method of common words used in the recognized language.

[0090] The operation process of the continuous speech recognition system shown in FIG. 4 includes: a voice detection means 1 collects the user's speech, for example, representing the language as the speech sample, and sends such speech sample to a pronunciation probability calculation means 2. The pronunciation probability calculation means 2, for every pronunciation in the acoustic model 7, gives the probability estimation value of whether it is the same as the speech sample. A word probability calculation means 5, according to the language habits summarized from a great quantity of language materials, gives the probability estimation value of the word occurring in the current context for the language model 8. A word matching means 3 calculates a joint probability (which represents the possibility of recognizing the speech sample as this word) through combining the probability value calculated by the pronunciation probability calculation means 2 with probability value calculated by the word probability calculation means 5, and takes the word with the greatest joint probability value as the result of the speech recognition. A context generating means 4 modifies the current context by using the above recognition result, so as to be used by the recognition of the next speech sample. A word output means 6 outputs the recognized word.

[0091] FIG. 5 shows a speech recognition system for correcting the error characters in the results of speech recognition according to an embodiment of the invention, in which system, the user inputs the correct characters by means of a stroke type based input means 9, and an error correction means 10, according to the acoustic model 7 and the language model 8, filters the candidate list generated by a candidate list generation means 11.

[0092] The stroke type based input means may be either a writing pad or an input stroke or stroke type means. There are several ways as follows to accomplish the stroke input with no additional hardware requirement:

[0093] 1. Design a map of Chinese strokes or stroke types using a sub-area on the common computer keyboard for inputting strokes. Stroke types may be used for simplicity and reliability.

[0094] 2. A virtual key set may be designed on the error correction screen.

[0095] 3. A user may draw with the mouse to identify a desired stoke. A recognition system can be developed to recognize the complete set of strokes or stroke types.

[0096] 4. Speech may be used to input strokes or stroke types.

[0097] In addition, during the error correction process for the speech recognition results, after the user marks an erroneous word, the candidate list will pop up upon request. We describe here a design of the error correction screen. As shown in FIG. 6, the error correction screen consists of a virtual keyboard for five stroke types and the candidate list on its right side. When the user begins to use the virtual keyboard to input stroke types, the candidate list right next to it will automatically be updated to show the new top candidates each time a stroke type is entered. This user interface which integrates candidate list and virtual stroke keyboard on the same screen will greatly improve the speed of the error correction.

[0098] It is to be appreciated that a speech recognition system for implementing the present invention, e.g., as shown in FIG. 5, may comprise a processor, for controlling and performing the methodologies described herein, a memory and a user interface, each coupled to the processor. The term “processor” as used herein is intended to include any processing device, such as, for example, one that includes a CPU (central processing unit) and/or other processing circuitry. For example, the processor may be a digital signal processor, as is known in the art. Also the term “processor” may refer to more than one individual processor. The term “memory” as used herein is intended to include memory associated with a processor or CPU, such as, for example, RAM, ROM, a fixed memory device (e.g., hard drive), a removable memory device (e.g., diskette), flash memory, etc. In addition, the term “user interface” as used herein is intended to include, for example, one or more input devices, e.g., any of the input means described herein, for inputting data to the processing unit, and/or one or more output devices, e.g., CRT display and/or printer, for providing results associated with the processing unit.

[0099] Accordingly, computer software including instructions or code for performing the methodologies of the invention, as described herein, may be stored in one or more of the associated memory devices (e.g., ROM, fixed or removable memory) and, when ready to be utilized, loaded in part or in whole (e.g., into RAM) and executed by a CPU. In any case, it should be understood that the elements illustrated in FIG. 5 may be implemented in various forms of hardware, software, or combinations thereof, e.g., one or more digital signal processors with associated memory, application specific integrated circuit(s), functional circuitry, one or more appropriately programmed general purpose digital computers with associated memory, etc. Given the teachings of the invention provided herein, one of ordinary skill in the related art will be able to contemplate other implementations of the elements of the invention.

[0100] Accordingly, a method for correcting the error characters in the results of speech recognition and a speech recognition system having the error correction function according to the invention is described above in conjunction with the particular embodiment. It is apparent for those skilled in the art that various modifications can be made to the invention without departing from the spirit of the invention, which directs to include all these modifications and changes, and the scope of which is defined by the appending claims.