Method for generating candidate word strings in speech recognition

[0002] 1. Field of the Invention

[0003] The present invention relates to the field of speech recognition and, more particularly, to a node-based method for generating candidate word strings in speech recognition without expanding the strings.

[0004] 2. Description of Related Art

[0005] To achieve a higher recognition accuracy, the output of a speech recognition module is not only a single recognition result, instead, a plurality of possible results are provided so that a subsequent process may select a best one therefrom in current speech recognition system.

[0006] Therefore, a speech recognition module must provide many possible results to the subsequent process. Accordingly, the generation of a plurality of candidate word strings from a speech signal for the subsequent process is a major concern in developing the speech recognition system.

[0007] U.S. Pat. No. 5,241,619 discloses a method for searching candidate word strings in which N candidate word strings are maintained during a matching process of speech signals and words. The N candidate word strings are obtained after the matching process. In such a method, the N candidate word strings maintained previously have to be expanded and modified for each time frame. If there are M words in vocabulary, as shown in FIG. 6, there will be M new candidate word strings generated when a candidate word string is expanded. The best N candidate word strings are selected from all expanded candidate word strings for being used as a basis for the expansion in the next time frame. In this manner, a large memory space is required to store expanded candidate word strings, and a sorting must be performed for each time frame to maintain N possible candidate word strings.

[0008] Another approach for candidate word strings search is implemented in a two-stage design. In the first stage, a modified Viterbi algorithm is employed to generate a word lattice from input speech signal. In the second stage, a stack search is used to generate the candidate word string by tracing back the word lattice generated in the first stage. The detailed description of such method can be found in U.S. Pat. No. 5,805,772, entitled “Systems, Methods and Architecture of Manufacture for Performing High Resolution N-best String Hypothesization” and “A Tree-trellis Based Fast Search for Finding the N Best Sentence Hypotheses in Continuous Speech Recognition” by F. K. Soong and E. F. Huang, ICASSP'91, pp. 705-708, 1991, which are hereby incorporated by reference into this patent application. As known, this method must continuously perform stack operations, such as push and pop, for expanding word strings in order to obtain possible candidate word strings. This method inevitably spends much time on the expanding of candidate word strings.

[0009] Still another method for searching candidate word strings is implemented in a two-stage design similar to above method. In the first stage, 408 Mandarin syllables are used as recognition units for generating syllable lattice. In the second stage, N-best syllables are selected for back-tracing operation with the use of the stack search in order to generate a plurality of candidate word strings. A detailed description of such a method can be found in “An Efficient Algorithm for Syllable Hypothesization in Continuous Mandarin Speech Recognition” by E. F. Huang and H. C. Wang, IEEE transactions on speech and audio processing, pp. 446-449, 1994, which is incorporated herein by reference.

[0010] A further method for searching candidate word strings is also implemented in a two-stage design, in which a word graph algorithm is employed to generate the word graph and a best word string in the first stage. The detailed description can be found in “A Word Graph Algorithm for Large Vocabulary Continuous Speech Recognition” by S. Ortmanns, H. Ney, and X. Aubert, Computer Speech and Language, pp. 43-72, 1997, which is incorporated herein by reference. In the second stage, the searching for candidate word strings is performed on the nodes of the best word string. The output is recorded in a tree structure for saving memory space. A detailed description of this method can be found in U.S. Pat. No. 5,987,409, entitled “Method of and Apparatus for Deriving a Plurality of Sequences of Words From a Speech Signal”, which is incorporated herein by reference.

[0011] Basically, the above methods perform searching operations based on the expansion of word strings. Such operation requires a large memory space to store word strings, and spends a lot of time for to expand word strings. Therefore, it is desired to make an improvement on the search of candidate word strings.

[0012] The object of the present invention is to provide a method for quickly generating candidate word strings without expanding the word strings. This method comprises the steps of: (A) determining an associated maximum string score for each node; (B) sorting all nodes by their associated maximum string scores to group the nodes with the same string score into the same node set; and, (C) selecting the node sets with relative high string scores generated in step (B), so as to connect the nodes by their starting time frame and ending time frame, thereby generating the candidate word strings.

[0013] Other objects, advantages, and novel features of the invention will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0014] FIG. 1 shows a portion of nodes of a word lattice employed in the method for generating candidate word strings;

[0015] FIG. 2 shows the associated maximum string scores for the nodes in FIG. 1;

[0016] FIG. 3 illustrates the node sets for the candidate word strings in FIG. 1;

[0017] FIG. 4 schematically illustrates the use of dummy nodes and forward/backward scoring arrays to store the maximum partial string scores;

[0018] FIG. 5 schematically illustrates the combinations and output of candidate word strings generated by the present method; and

[0019] FIG. 6 schematically illustrates the conventional method for generating candidate word strings.

[0020] To demonstrate the method for generating candidate word strings in speech recognition in accordance with the present invention, an example is given for the search of candidate word strings. In this example, the word is used as the recognition unit. FIG. 1 shows a portion of nodes of a word lattice. Each of the nodes n1 to n9 includes word content, starting time frame, ending time frame, and the node score for the corresponding speech signal. The connections among nodes, n1 to n9, form the possible word strings, for example, n1-n2-n3-n4, n5-n9-n7-n8, or n5-n6-n3-n4. A complete word string is defined as a connection from the starting time frame (where the speech signal always starts from 0) to the ending time frame of the speech signal. In this example, the ending time frame is 300. Furthermore, the string score of a word string is the summation of the scores of all the nodes in the word string.

[0021] For each node shown in FIG. 1, the following characteristics are followed: (1) If the node n_iis one of the components in the best word string, the string score of the best word string is the summation of three scores—the maximum partial string score on the left-hand side of n_i, the node score of n_i, and the maximum partial string score on the right-hand side of n_i; and (2) The associated maximum string scores obtained from each node n_iin the best word string are equivalent through the string score computation. In FIG. 2, the word string n1-n2-n3-n4 is the best word string in this case, and the string score of this word string is −524. Also, the associated maximum string score obtained from any node (e.g., n1, n2, n3, or n4) is −524.

[0022] For the nodes that are not in the best word string, there are still other word strings that include these nodes. And, another point can be found that the associated maximum string scores of the nodes that are not in the best word string will be equivalent if their associated word strings are identical. By using the characteristic (1), the associated maximum string score for each node in FIG. 1 can be obtained. Therefore, as shown in FIG. 2, the method of the present invention first determines the associated maximum string scores for all of nodes. Then, the method sorts all nodes by their associated maximum string scores. As shown in FIG. 3, the nodes are grouped into the same node set if their associated maximum string scores are identical to each other. The candidate word strings can be obtained from the front plurality of node sets with different string scores by the connection of their starting and ending time frames as detailed hereinafter.

[0023] In the calculation of the associated maximum string score for each node, it is implemented by searching the maximum partial string scores on the left-hand side and right-hand side of each node. For example, for the node n2 in FIG. 1, in order to obtain the maximum partial string score on the right-hand side of node n2, it is required to determine the maximum partial string score after frame 64. Similarly, in order to obtain the maximum partial string score after frame 64, it is required to determine the maximum partial string scores after frames 150 and 160, respectively. Such a recursive process is performed until the end of the speech signal, and the maximum partial string score on the right-hand side of node n2 can be obtained. This can be achieved as shown in FIG. 4, wherein two dummy nodes 41 and 42 are created in the front and the rear of the nodes, respectively. A recursive search is performed from the dummy nodes 41 and 42, and the forward scoring array 43 and the backward scoring array 44 are provided to store the maximum partial string scores for the starting and ending time frames of nodes n1 to n9. That is, the forward scoring array 43 is provided to store the maximum partial string scores that are obtained by searching from the starting time frames of nodes n1 to n9 to the end of the speech signal, and to store the indices of nodes that have the maximum partial string scores. Similarly, the backward scoring array 44 is provided to store the maximum partial string scores that are obtained by searching from the ending time frames of nodes n1 to n9 to the beginning of the speech signal, and to store the indices of nodes that have the maximum partial string scores. Hence, the maximum partial string scores for both sides of a specific node can be determined by simply looking up the forward scoring array 43 and the backward scoring array 44. Thus, it is easy to determine the maximum string scores for all nodes in FIG. 1, thereby obtaining the result as shown in FIG. 2.

[0024] For the output of candidate word strings, it can be facilitated by the node indices stored in the forward/backward scoring arrays. Because the node index is used to record the node with the maximum partial string score, it is able to quickly connect the nodes and form the complete word string by referencing the node indices. For example, the node set 2 in FIG. 3, the partial word string can be constructed by node n5 (i.e., silence) and node n6 (i.e., about). But the content in the region from frame 65 to 300 has not been determined. To complete the connection, first, the node index 3 is obtained from frame 65 of the forward scoring array 43 in FIG. 4; that is, the word ‘weather’ and the starting and ending time frames (65, 150) of the node n3. Then, the node index 4 is obtained from frame 151 of forward scoring array 43; that is, the word ‘today’ and the starting and ending time frames (151, 300) of the node n4. Accordingly, the word string consisting of ‘silence’, ‘about’, ‘weather’, and ‘today’ is obtained. Similarly, as shown in FIG. 3, in the node set 4, the partial word string ‘wet day’ can be obtained by the nodes n7 and n8. But the connection from frame 0 to 64 has not been established. To complete the connection, first, the node index 2 is obtained from frame 64 of the backward scoring array 44 in FIG. 4; that is, the word ‘doubt’ and the starting and ending time frames (11, 64) of the node n2. Then, the node index 1 is obtained from frame 10 of the backward scoring array 44; that is, the word ‘silence’ and the starting and ending time frames (0, 10) of the node n1. Accordingly, the word string consisting of ‘silence’, ‘doubt’, ‘wet’, and ‘day’ is obtained.

[0025] In view of the foregoing, it is known that the method of the present invention is provided to obtain the associated maximum string score for each node, and sort the nodes by their associated maximum string scores for obtaining the candidate word strings. The expansion of word strings can be avoided, and thus, the processing time and the memory requirement are significantly reduced. Moreover, the subsequent processes in a spoken dialogue system, such as speech verification and language understanding, may use the number of candidate word strings to control their processing time and the corresponding recognition accuracy by the method of the present invention.

[0026] The following table 1 is provided to illustrate an example for giving the relation among the number of candidate word strings, inclusion rate and word density.

[0027] In this example, 500 spoken utterances for weather inquiry domain, recorded by 10 males and 10 females, are used for testing, where ‘∞’ denotes the case for the full selections of candidate word strings. Although the inclusion rate is 98.85% for the case of full selections, the word density is as high as 130.33. After decreasing the number of candidate word strings to 50, the word density is reduced to 24.62, which is about one-fifth of the case of full selections, and the inclusion rate is 98.33%, which is reduced only by 0.52%. When the number of candidate word strings is decreased, the word density and the inclusion rate are also decreased. The selection for the number of candidate word strings is a trade off computation time, memory space, and correctness, which can be controlled by the post-processing modules.

[0028] Although the present invention has been explained in relation to its preferred embodiment, it is to be understood that many other possible modifications and variations can be made without departing from the spirit and scope of the invention as hereinafter claimed.