Combining N-best lists from multiple speech recognizers

[0002] 1. Field

[0003] This disclosure relates to speech recognition systems, more particularly to methods to combine the N-best lists from multiple recognizers.

[0004] 2. Background

[0005] Speech recognizers are those components used in speech recognition systems that perform the actual conversion from the incoming audio stream to text or commands. The recognizer uses algorithms to match what the user says to elements in a speech model. The recognizer then returns text corresponding to user's speech to the application utilizing the speech recognition. In one example, the algorithms are run on a digital signal processor. However, even with powerful processors and detailed speech models, errors still occur. Word recognition rates are generally better than 90%, but failures occur, especially over sequences of words.

[0006] Because of uncertainties in the recognition process, the speech recognizer may return several possible text results and allow the application that requested the recognition to select the most appropriate result based on knowledge it possesses regarding the user, the task, the context or other factors. Many speech recognizers support this concept of N-best recognition. The recognizer returns a list of elements that the user might have said, typically accompanied by a score of how confident the recognizer is of each potential match. This list will be referred to here as an N-best list. The application software then decides which entry in the N-best list to use.

[0007] Current speech recognition applications use only a single recognizer. However, many speech recognition applications may benefit from the use of several different recognizers. Different recognizers from different manufacturers perform differently even if targeted at the same market. This is due to the use of different algorithms to perform the speech recognition and different training data used to create speech models used by the recognizers. If multiple recognizers are used concurrently, several different N-best lists may be returned to the application. Recognition accuracy could be degraded if the N-best list selected is from a recognizer with poor performance in a particular situation.

[0008] Therefore, it would seem useful to have a process for selecting which recognizers should process an audio stream and one for combining N-best lists from different recognizers into one N-best list prior to the list being returned to the application.

[0009] One aspect of the disclosure is a speech recognition system. The system includes a port for receiving an input audio stream and one or more recognizers operable to convert the input audio stream from speech to text or commands. The system also includes a combiner operable to combine lists of possible results produced by each recognizer into a combined list. Some subset of the combined list is then sent back to the application, allowing the application to select the desired conversion result.

[0010] Another aspect of the disclosure is a method to utilize multiple speech recognizers. An input audio stream is routed to the enabled recognizers. The method of selecting the enabled recognizers is discussed below. A combiner receives a list of possible results from each of the enabled recognizers and combines the lists into a combined list and then returns a subset of that list to the application.

[0011] Another aspect of the disclosure is a method of combining N-best lists from multiple speech recognizers. A combiner receives an N-best list from each enabled speech recognizer and combines the entries in each list into an initial N-best list. The N-best list is then potentially reduced in size and sorted according to at least one sorting criteria. A subset of entries in the resulting sorted N-best list is then returned to the application.

[0012] The invention may be best understood by reading the disclosure with reference to the drawings, wherein:

[0013] FIG. 1 shows one embodiment of a speech recognition system employing multiple recognizers, in accordance with the invention.

[0014] FIG. 2 shows a block diagram of a combiner acting as a proxy for multiple speech recognizers, in accordance with the invention.

[0015] FIG. 3 shows a flowchart of one embodiment of a method for utilizing multiple speech recognizers, in accordance with the invention.

[0016] FIG. 4 shows a flowchart of one embodiment of a method for combining N-best lists from multiple recognizers, in accordance with the invention.

[0017] FIG. 1 shows one embodiment of a speech recognition system employing multiple recognizers, in accordance with the invention. The system 10 includes a port 12 through which an input audio stream is received. For example, if the application provides a voice interface for a user to access e-mail, schedules, etc. over a phone, the port may be a telephone handset. Similarly, for dictation applications, the port may be a microphone on a computing device, such as a personal computer or handheld device. Note that there may be a variety of communication networks and interfaces between the port and the recognizers, including processes to convert the audio from analog to various digital forms, intermediate storage and the like. The invention is in no way limited by details of the port or the paths that the audio, in analog or digital form, may travel.

[0018] The input audio stream is then routed through a connection to the enabled recognizers. Which recognizers are enabled is detailed below. As shown in FIG. 1, there are several recognizers, 14a, 14band 14n. Note that the use of the letter ‘n’ is not to indicate that there are fourteen recognizers (‘n’ is the fourteenth letter of the alphabet), but to indicate that there are ‘n’ recognizers, with the actual value of ‘n’ left up to the system designers.

[0019] The connection 18 may be any kind of connection. For example, the application requesting speech recognition may reside on the same device as the recognition system 10. In this instance, the connection may be through a communication bus on the device. Alternatively, the requesting application may be at one node in a network and the speech recognition system 10 at another node. In this example, the connection 18 would be a network, such as a local area network (LAN) or a wide area network (WAN), and may have wired or wireless network interfaces. No limitation to any particular kind of connection is intended.

[0020] The speech recognizers may be partitioned across a network as in a distributed speech recognition system. In this case, the recognizers 14a-14nmay process raw audio data or a more compact form generated by the front-end speech recognition processes that reside across the network 18. In the following discussions, we refer to the stream of data that the speech recognizers process as the “input stream” or “input audio stream” regardless if it is raw audio data or an intermediate form that may be generated by a distributed speech recognizer or any other form of compression. The invention is in no way limited by the specific architecture of the recognizers.

[0021] The recognizers 14a-14nare operable to receive the routed input stream. The recognizers convert the speech contained within to text or commands used in the requesting application. As mentioned previously, different recognizers offer different performance depending upon several factors. Two recognizers from the same manufacturer may be targeted to different markets, for example. One recognizer from a given manufacturer may be targeted to the command and control functions, and will have high accuracy in one or two word phrases predetermined to be commands in the system. Another recognizer may be targeted to the dictation market and will have a much larger vocabulary. However, that particular recognizer may not have any mechanisms to allow it to handle speech recognition tasks over a noisy connection, such as a cell phone. The variation of recognizer combinations is almost limitless and the actual configuration selected is up to the system designers.

[0022] However, it is this combination of various recognizers that can lead to inaccurate or otherwise less than desirable results. A robust speech recognition system may have many different types of recognizers to handle different types of situations. Combining the results from the different recognizers must be handled carefully to avoid any problems with the resulting converted speech stream returned to the application.

[0023] The combiner 16 handles this combination task. It receives results back from each recognizer to which the input stream was routed. Typically, these results will be in the form of lists of possible results for a given input. Generally, the combiner will receive at least one set of results that is an N-best list. Alternatively, the combiner will produce the combined list in the form of an N-best list. This task of either receiving an N-best list or producing an N-best list will be referred to processing an N-best list. The combiner gathers all of these lists together and coalesces them into a combined list and returns at least a subset of the combined list to the requesting application via output stream 20.

[0024] A flowchart of one embodiment of a method of utilizing multiple speech recognizers is shown in FIG. 3. At 22 the input stream is received. At 24 the stream is routed to one or more selected recognizers. In some situations, the speech recognition system may not route the input stream to all of the recognizers available. For example, the incoming stream may be at a point in the dialogue with the system that is beyond command and control. The speech recognition system may therefore not send the incoming stream to the recognizers that have been identified by the system as being only for command and control. Other types of cues may also be available to the system allowing it to route the stream to a subset of recognizers.

[0025] After the input stream has been routed to the selected recognizers, a result is received from each recognizer at 26. This result will be referred to as individual result sets, even if there are multiple terms returned from each recognizer. This will assist in differentiating the results obtained from each recognizer from the combined result, discussed in more detail later. For example, in one embodiment, the individual result returned from each recognizer is actually a list of multiple entries.

[0026] Note that individual results from recognizers that do not employ a conventional N-best interface may also be employed. Throughout this discussion single entry results will be treated as a form of an N-best interface with N equal to one. For example, a recognizer may be used that returns only a single result. It may return a single result only because the recognizer uses conventional N-best format, but only had one result. Alternatively, the recognizer may not use the conventional N-best format at all, but only return a single result. As used here, the term ‘N-best list’ will include both of these results. The terms ‘dynamic length N-best list’ or ‘N-best list of dynamic length’ will be used to indicate conventional N-best result sets which may contain multiple entries.

[0027] These individual result sets are then coalesced into a combined list at 28. Several options exist for the specific mechanisms used to combine the lists as will be discussed in more detail further. Finally, at least a subset of the combined list will be returned to the requesting application at 30. In some instances, all of the individual results may be returned, such as where there are only five possible results and the requesting application has requested five or more results. In other instances only a subset of the results may be returned. Using the same example above, where the requesting application has requested five results, there may be a total of twelve results from the enabled recognizers, so only five will be sent. Additionally, the combined list returned to the application may be truncated based on other criteria. This is discussed below.

[0028] In one embodiment, the combination of results is based upon an N-best approach. The term ‘N-best’ refers to a ‘top’ number of results, typically those that have the highest confidence values. In many speech recognition systems, results returned from recognizers include a confidence value that indicates how the recognizer views a particular result statistically. For example, if a list of possible results is returned with confidence values ranging from 99% to 78%, and N is equal to three, the results will be sorted by confidence value and the top three will be the N-best list returned to the application.

[0029] Note that while we discuss confidence values as being expressed as percentages, different systems may use different mechanisms, different scales and ranges. We discuss some implications of this below. Other types of utility metrics may be used, and some may be used in those embodiments that do not follow the N-best format, as discussed above.

[0030] Many different alternatives exist for combining the various results into a combined result, as these terms are defined above. A combination may be a simple concatenation, where each result is just added to a list. Combination may be done by either sorting only, or by sorting and reducing, or only by reducing, the various results. An embodiment of a method to combine results from multiple recognizers is shown in flowchart form in FIG. 4. The N-best lists from each recognizer are received at 34. This is analogous to receiving the lists at 26 in FIG. 3. The process after 34 in FIG. 4 is analogous to combining the lists at 28 in FIG. 3. The individual N-best lists from each recognizer are combined into an initial N-best list at 36. This list contains all of the N-best entries provided from each recognizer to which the input stream was routed.

[0031] At 38, the initial N-best list size is reduced. For example, redundant entries could be removed. For example, recognizer A and recognizer B could both have returned the word “blue” for a given input. The combiner will only use one of these, and can eliminate the other.

[0032] In some instances, where confidence values are used, the associated confidence value with the term ‘blue’ may be increased based upon the redundancy. This is similar to a voting scheme, where the recognizers ‘vote’ on the most accurate result. If more recognizers return a given result, the combiner may be more confident of its value. In the case of redundant entries, the confidence value of the version of the entry that is retained may be increased to reflect the number of recognizers that returned that result. An example of this is discussed in Barry, T.; et al., “The simultaneous use of three machine speech recognition systems to increase recognition accuracy,” Aerospace and Electronics Conference, 1994. NAECON 1994, Proceedings of the IEEE 1994 National Page(s): 667-671 vol.2.

[0033] In whatever manner the initial list size is reduced, the N-best list is sorted at 40. The sort is performed according to at least one sorting criteria. Sorting criteria will be discussed in more detail later. At 42, the top N entries will be selected. Again, if N is five, the top five entries in the list are selected and returned to the requesting application. The combiner may not return a list with N entries if confidence values for the entries in the combined list are below a certain threshold.

[0034] The top of the list is determined by the sorting criteria used. As discussed above, the N-best list may be accompanied by confidence values. The confidence values may be used as one of the sorting criteria. Different recognizers may use different formats for their confidence values. One manufacturer may use a percentage as indicated above. Another may use a rating system, where a given rating equals a certain range of accuracy. In this instance, where the recognizers have different confidence values, the confidence values in the results may have to be normalized to a uniform format prior to the sort being performed.

[0035] Another sorting criteria that may be used, either in conjunction with the confidence values, or by itself, is performance values associated with a given recognizer. The performance value can be determined based upon several factors. For example, a user identity may provide information as to the gender of the user, and there may be recognizers that perform differently for women's voices and men's voices. Other characteristics of the user may be available as part of the user identity in the system, including native language, region of the country, history of prior use, etc.

[0036] Another basis for performance values may lie in the dialog state of the interaction, as mentioned above. The dialogue state may not be in a navigational command and control subset, as mentioned above, and therefore the performance values for those recognizers specifically developed for this type of speech might be downgraded. Similarly, different recognizers may be optimized for different types of communication channels, such as cellular phones versus landline phones. Different recognizers may also perform better for specific devices, such as cordless phones, traditional phones, speaker phones, or computer phones, as examples.

[0037] In addition to having a performance value initially assigned to a given recognizer, the performance value may be dynamically adjusted. For example, a given recognizer could have a performance value assigned based upon the assumption that it is not as accurate with regard to audio received from cellular phones. However, over time, that recognizer may return results that are substantially the same as a recognizer with a high performance for cellular phones. The performance value for the first recognizer may be dynamically adjusted to reflect this.

[0038] The performance values may be used to weight information coming from the recognizers, prior to the final sort of the N-best list. For example, recognizers A and B may return first values in each of their respective lists having a confidence level of 99%. However, the performance value of recognizer A may be 90% compared to the performance value for recognizer B of 99%. Therefore, the first value from recognizer A will have an actual confidence value of 0.90*0.99, or 89%, and the first value from recognizer B will have an actual confidence value of 0.99*0.99, or 98%. Note that many algorithms may be applied to normalize confidence levels. This is one example of how confidence levels may be combined, and in no way limits the scope of the invention.

[0039] One aspect of this combined list that becomes especially clear when taking into account performance values is the dynamic nature of the sorted N-best list. Even if the performance values are not used, the combination of N-best lists from the different recognizers results in entries from each of the N-best lists being shifted and intermixed with N-best lists from other recognizers. This can be problematic when a query is received from the application about a particular entry in the sorted N-best list.

[0040] To overcome this problem, the combiner can act as a proxy for the various recognizers. An example of this is shown in FIG. 3. As the initial N-best lists are combined, reduced and sorted into a combined N-best list, the combiner can track the origin of the entries. In this manner, if a query is received about a specific entry or a specific recognizer, the answer can be made available. The information to fulfill the query may be obtained from an individual recognizer or from information cached by the combiner during the combination process. An example of a piece of information that the combiner is likely to cache is the confidence value. Other information that is not used in forming the combined N-Best list, e.g. natural language interpretations or in/out-of-vocabulary confidence values, would likely be obtained from the recognizer.

[0041] The combiner typically will not actually maintain copies of each of the recognizers' N-best lists Instead, the combiner will have an index or map that allows it to point back to the N-best list on a given recognizer and bring that answer ‘forward’ to the requesting application. A block diagram of one embodiment of this situation is shown in FIG. 2.

[0042] The combiner presents an N-best list of entries, where N is 4. The N-best list returned to the application is a list of four words, with no indication from where those words came. However, during the process of combining the lists, the combiner tracks and remembers the order from the recognizers, as shown in the initial list 64. As the initial list 64 was reduced to intermediate list 62 and finally presented, the information as to the origin of the results is maintained. If a query is received desiring that information, the combiner can act as a proxy, go back and fetch that information from the recognizers 14aand 14band return it to the requesting application. Note that instances where the same term has been provided by multiple recognizers, the system may return information from an individual recognizer, such as the one with the highest confidence rating, or may act as an intelligent proxy by consolidating information from multiple recognizers to satisfy the query.

[0043] In this manner, a method and system are provided to intelligently combine results from multiple recognizers. As mentioned above, current speech recognition systems that employ multiple recognizers generally rely on simple voting schemes.

[0044] Note that though this disclosure has discussed the combiner returning a full N-Best list to the application, current systems don't return a full N-best list to the application, rather they provide APIs whereby the application can request specific elements from the list (“give me the 3^rdelement on the list”). Additionally most systems provide an API for the application to determine how many entries are available on the N-best list. The application design determines how far down the N-best list the application fetches. The invention disclosed here is in no way limited to systems which provide the entire N-best to the application at one time, nor is it limited to the current practice of the application querying the recognizer for individual entries.

[0045] Implementation of this invention in existing speech systems may take the form of a software upgrade. Generally, the software would be available on an article such as a floppy diskette, CD-ROM or other optical or magnetic media. The article would then contain code that, when executed, would perform the methods of the invention.

[0046] Although there has been described to this point a particular embodiments for a speech recognition system and method for utilizing multiple speech recognizers, it is not intended that such specific references be considered as limitations upon the scope of this invention except in-so-far as set forth in the following claims.