Method and apparatus for providing unsupervised adaptation of transcriptions

[0002] This invention relates to the field of automatically performing desired actions in response to spoken requests. It is particularly applicable to a method and an apparatus for adapting a speech recognition dictionary, as may be used to automate partially or fully the training of a speech recognition dictionary in a speech recognition system. The method and apparatus may be used to train a speech recognition dictionary for a telephone directory assistance system, voice activated dialing (VAD), credit card number identification and other speech recognition enabled services.

[0003] In addition to providing printed telephone directories, telephone companies provide information services to their subscribers. The services may include stock quotes, directory assistance and many others. In most of these applications, when the information requested can be expressed as a number or number sequence, the user is required to enter his request via a touch tone telephone. This is often aggravating for the user since he is usually obliged to make repetitive entries in order to obtain a single answer. This situation becomes even more difficult when the input information is a word or phrase. In these situations, the involvement of a human operator may be required to complete the desired task.

[0004] Because telephone companies are likely to handle a very large number of calls per year, the associated labour costs are very significant. Consequently, telephone companies and telephone equipment manufacturers have devoted considerable efforts to the development of systems that reduce the labour costs associated with providing information services on the telephone network. These efforts comprise the development of sophisticated speech processing and recognition systems that can be used in the context of telephone networks.

[0005] In typical speech recognition systems, the user enters his request using isolated word, connected word or continuous speech via a microphone or telephone set. The request may be a name, a city or any other type of information for which either a function is to be performed or information is to be supplied. If valid speech is detected, the speech recognition layer of the system is invoked in an attempt to recognize the unknown utterance. Typically entries in a speech recognition dictionary, usually including transcriptions associated to labels, are scored in order to determine the most likely match to the utterance.

[0006] Commonly, the speech recognition dictionary is created by obtaining transcriptions of words, each component of the transcription being associated with an acoustic model. This operation is usually performed when the speech recognition system is built or at the time of installation. However, the initial transcription content of the speech recognition dictionary may inadequately model the pronunciation of certain words. Furthermore, the speech recognition system may be unable to track the time varying aspect of the pronunciation of words in a language leading to performance degradation over time. Typically, many transcriptions are stored for each orthography in the speech recognition vocabulary in order to model the different pronunciations of the orthography. However, to allow real-time performance of the speech recognition system, only a limited number of transcriptions are stored for each word and the correct transcription may not be chosen to be added to the speech recognition dictionary by the training system. Finally, the training of the speech recognition dictionary is limited to information available in a fixed observation period and events not occurring within the observation period are often not well modelled.

[0007] One way to improve the performance of a speech recognition system under these conditions is to make use of past utterance usage. Using this existing knowledge, the speech recognition system is modified to better reflect the utterances received by the system. These techniques are commonly referred to as adaptation techniques. Adaptation attempts to correct the shortcomings of the initial configuration of the speech recogniser. Typical adaptation algorithms can adapt to changes in the speech source, such as changes in the speaker population, as well as changes in the application lexicon, such as modifications made by a system administrator. In principle, almost any parameter in a pattern recognition system can be adapted. Two categories of adaptation can be distinguished namely supervised adaptation and unsupervised adaptation.

[0008] Supervised adaptation involves human intervention, typically that of a phonetician who will correct labels. Supervised adaptation also requires storage of a batch of data, transferring data for validation purposes, and offline computing resources. It requires a large corpus of labelled speech and usually requires a high level of labelling accuracy in order to obtain improved performance of the speech recogniser. In a typical interaction, a phonetician transcribes each sample of speech and labels it with its corresponding orthography and transcription (e.g. Phonetic spelling) and noise codes. The transcriptions generated in this fashion are then entered into the speech recognition dictionary. This operation is time consuming and the labour costs associated with the expert phoneticians are significant. Furthermore, tracking the evolution of word pronunciation requires frequent training sessions and is impractical. In order to reduce the labour costs associated with the generation of transcriptions, systems providing unsupervised or automatic adaptation have been developed.

[0009] Unsupervised adaptation, herein referred to as automatic adaptation, involves little or no human intervention. It requires a large corpus of labelled speech and usually does not require a high level of labelling accuracy in order to obtain improved performance of the speech recogniser. In a specific example the labelled speech is generated by the speech recognizer or by a human operator.

[0010] A common method of finding the most suitable transcription from a set of transcriptions is to find the transcription, T*, which maximises the equation p (U/T) p (T/L) where p (U/T) is the acoustic model score from the speech recognizer (e.g. the probability that the transcription corresponds to a given sample of speech) and p (T/L) is the language model probability (the probability of the transcription given the language model). When the word identity is known with certainty, then p(T/L) becomes p(T/W), the probability of a transcription given a fixed word. Typically, P (T/W) is difficult to assess and a constant value is generally used. When multiple examples of a word are available, then the logarithm of p(U/T) is summed, and the transcription which minimises the summed log p(U/T) is taken. For a more detailed discussion of this method, the reader is invited to consult L. R. Bahl et al. “Automatic phonetic baseform determination”, 1991, IEEE; D. Hunnicutt et al. “Reversible letter-to-sound and sound-to-letter generation based on parsing word morphology”, and R. Haeb-Umbach et al. “Automatic transcription of unknown words in a speech recognition system”, 1995, IEEE. The contents of these documents is hereby incorporated by reference.

[0011] A deficiency of the above described method is that the transcription T* is selected from a fixed set of transcriptions. This does not allow the system to adapt to the time varying nature of speech as well as does not allow the speech recognition system to learn from past speech utterances.

[0012] Thus, there exists a need in the industry to refine the process of automatically adapting a speech recognition system.

[0013] An object of the invention is to provide an improved method and apparatus for generating a transcription capable of being used to create an entry in a speech recognition dictionary for a certain word.

[0014] A more specific object of the invention is to provide a method and apparatus that utilize a signal representative of audio information produced when uttering a certain word that is input to a speech recognition system along with the vocabulary item selected by the speech recognition system as being a match to the uttered word, in order to fine tune the transcription for the certain word that is stored in the dictionary of the speech recognition system.

[0015] A further object of the invention is to provide a computer readable storage medium containing a novel program element that instructs a computer to generate a transcription capable of being used to create an entry in a speech recognition dictionary for a certain word.

[0016] Another object of the present invention is to provide a speech recognition system that offers improved task adaptation capabilities.

[0017] As embodied and broadly described herein the invention provides an apparatus for generating a transcription capable of being used to create an entry in a speech recognition dictionary for a certain word, said apparatus comprising:

[0018] a machine readable storage medium containing a transcriptions bank, said transcriptions bank comprising a plurality of orthographic groups, each orthographic group including a plurality of transcriptions associated with a common word,

[0019] an input for receiving:

[0020] a) a signal representative of audio information produced when a user utters a certain word;

[0021] b) a label element associated with said signal;

[0022] a transcription selector module for:

[0023] a) retrieving on a basis of said label element the orthographic group corresponding to the certain word;

[0024] b) processing said signal to select a certain transcription from said orthographic group corresponding to the certain word;

[0025] an output for delivering the certain transcription capable of being used for creating an entry in the speech recognition dictionary.

[0026] For the purpose of this specification, the expression “word” designates a textual representation of a spoken utterance. In a specific example, a textual representation is a collection of written symbols or characters that are components of an alphabet.

[0027] For the purpose of this specification, the expressions “orthographic representation” and “orthography” are used interchangeably. An “orthography” is a data element in a machine-readable form that is an electronic representation of a word. Typically, an orthography is a collection of symbols mapped to the characters forming the word. The expression “orthography” also includes data structures including solely or in part pointers or links to locations, such as in a memory for example that contain the actual representation of the word.

[0028] For the purpose of this specification, the expression “orthographic group” is used to designate a set of transcriptions (see definition below) associated with a common word. The expression “orthographic group” may also contain the orthography of the word associated with the transcriptions in the group.

[0029] For the purpose of this specification, the expression “utterance” is a sound or combination of sounds that form a meaningful linguistic unit.

[0030] For the purpose of this specification, the expression “transcription” is used to designate a machine readable data element that is a combination of symbols providing information on how a sub-word unit, such as a letter or a combination of letters, may be pronounced. For example, a simple word like “a” is often pronounced as /ey/ in isolation but as /ax/ in context. Another example is a word like “data” which can be pronounced as /d ey t ax/ or /d æ t ax/ depending on the speaker's dialect. Typically a word may have several transcriptions, where each transcription may be a different pronunciation of the word. The symbols that compose a transcription may be phonemes, allophones, triphones, syllables and dyads (demi-syllables). The expression “allophonic transcription” is used to designate a transcription where the sub-word units are allophones and the expression “phonemic transcription” is used to designate a transcription where the sub-word units are phonemes. A phonemic transcription may have multiple corresponding allophonic transcriptions. Although the definition of “transcription” herein refers to a data structure including symbols, it should be understood that what is meant is a data element having constituent parts that are representations of the symbols. The table below illustrates examples of words (or orthographies) and the associated transcriptions.

[0031] In the above table, each alphanumeric character in the transcription is an allophone. The character is a phoneme and the digits following the character indicate a variation of the phoneme in a certain acoustic context (allophone). The “_” character is the inter-word silence and “-” is the syllabic mark.

[0032] For the purpose of this specification, the expressions “model” and “speech recognition model” are used to designate a mathematical representation of a speech signal. Speech modeling is well known in the art of speech recognition. Commonly used models in speech recognition are Hidden Markov Models (HMMs) where each phoneme can be represented by a sequence of states and transition in between these states. For basic information about HMMs the reader is invited to consult “An Introduction to Hidden Markov Models”, L. R. Rabiner and B. H. Juang, IEEE ASSP Magazine, January 1986, pp.4-16 whose content is hereby incorporated by reference.

[0033] For the purpose of this specification the expression “allophone” is used to designate the pronunciation of a phoneme placed in a certain context defined by the surrounding phones. For more information regarding allophonic contexts the reader is invited to consult U.S. Pat. No. 5,195,167 “Apparatus and method of grouping utterances of a phoneme into context-dependent categories based on sound similarity for automatic speech recognition”, L. R. Bahl et al., Apr. 17, 1992. The content of this document is hereby incorporated by reference.

[0034] The methods presented in this specification allow the modification of the speech recognition dictionary of a speech recognition system in order to adapt the system to the word pronunciation of the user population or to variations in the pronunciation of a user of the system.

[0035] In a specific example, the invention is embodied in a transcription adapting apparatus that is associated with a speech recognition system. Preferably, such a system generally comprises a number of functional modules namely a speech source, a speech recognizer and a transcription adapter module. The speech source may be a device such as a microphone or telephone set that converts a spoken utterance into an electric signal. In another preferred embodiment the speech source is a computer readable storage medium containing data elements representative of speech utterances. In a specific example, spoken utterances are digitised and stored on a non-volatile storage medium such as a hard disk. In another specific example, spoken utterances are stored on a magnetic tape. The speech recognition system also includes a computer readable storage medium on which is stored an action table and a speech recognition dictionary. The transcription adapter module interacts with the speech recognition dictionary to modify the latter's contents. Optionally, the speech recognition system features rejection capabilities.

[0036] In a typical interaction, the speech recognition system receives as input a signal representative of the audio information produced when uttering a certain word. The speech recognizer receives the signal, processes it, scores entries in the speech recognition dictionary and outputs the most likely candidate. The recognition candidate is preferably in the form of an orthography.

[0037] Optionally, following the recognition step, a controller unit performs a verification operation in order to eliminate candidates unlikely to match the spoken utterance. In this regard, the controller unit may apply a rejection algorithm such as that described in U.S. Pat. No. 5,097,509 Lennig “Rejection Method for Speech Recognition”, Mar. 17, 1992 whose content is hereby incorporated by reference. If the recognition is successful, the recognized orthography is used to access an entry in an action table. Preferably, the speech token and the recognized orthography are also stored on a computer readable storage medium in the labeled speech storage unit to await further processing.

[0038] Once the speech recognition system is taken offline, the transcription adaptation module is invoked. Alternatively, the transcription adaptation module is invoked after each speech recognition request and its operations are performed on-line. For transcription adaptation modules operating on-line the labeled speech storage unit may be omitted from the system. The transcription adaptation module processes each speech token and orthography. In a first step, for each speech token in the labeled speech storage unit, the selection module locates the corresponding orthographic group and scores all the transcriptions in the group. Once the speech tokens associated with a given orthographic group have been processed, compound scores are computed for each transcription by combining the individual scores obtained from each speech token. Following this, the transcription in a given orthographic groups are sorted on the basis of their compound score and the top N transcriptions are selected to replace the transcriptions in the speech recognition dictionary associated with the same orthography. Alternatively, the N transcriptions selected are stored in the speech recognition dictionary in addition to the original transcriptions.

[0039] In a preferred embodiment of the invention, the speech recognition system comprises a transcription adaptation module. The adaptation module uses the recognized orthography received by the controller unit along with the speech token in order to modify the speech recognition dictionary. In a preferred embodiment of the invention the transcription adaptation module comprises a transcriptions bank and a selection module. Optionally, the transcription adaptation module may comprise an augmentor module and a labeled speech storage unit.

[0040] In a preferred embodiment, the adaptation module comprises a computer readable storage medium containing a repository of transcriptions, herein designated as the transcriptions bank. The transcriptions bank contains a plurality of transcriptions for each of the orthographies in the speech recognition dictionary. The transcriptions bank may be created by using a text to transcription module using information derived from text to phoneme rules. In a most preferred embodiment, text to phoneme rules from many languages are used to include the variations in pronunciations across languages. In another preferred embodiment, the transcriptions in the transcriptions bank are derived from letter to phoneme rules as well as from language dictionaries. The system to generate the transcriptions bank further comprises a language dictionary transcription module and a computer readable storage medium containing a language dictionary. In a most preferred embodiment, language dictionaries from many languages are used to include the variations in pronunciations across languages. Using transcriptions from several languages admits pronunciation variants that are phonetically illegal in a given language but are valid in another language. Transcriptions in the transcriptions bank may be organized any suitable fashion without detracting from the spirit of the invention.

[0041] In a preferred embodiment, the adaptation module comprises a computer readable storage medium containing labeled speech elements. The adaptation of the dictionary is then performed off-line in batch mode. Preferably, a large number of speech tokens are used for adaptation. In a specific example, the number of speech tokens exceeds 20 tokens for each orthography. Alternatively, adaptation may be performed on-line after each recognition result as a speech recognizer post-process. For on-line processing, the labeled speech storage unit may be omitted from the adaptation module. On-line processing typically causes an increase in the computational requirements of the system by allowing the dictionary to be updated for the next time the system is used. The use of batch mode or on-line mode does not detract from the spirit of the invention.

[0042] The transcription adaptation apparatus preferably includes a selection module that selects from the transcriptions bank a set of transcriptions to be included in the speech recognition dictionary. The selection module receives labeled speech from the labeled speech storage unit. Alternatively, if the transcription adaptation occurs on-line the selection module may receive labeled speech items from the speech recognizer or from the controller unit. In a typical interaction, the selection module locates in the transcriptions bank the orthographic group associated with the labeled speech item. Following this, the speech token of the labeled speech item is aligned with the transcriptions within the orthographic group. Each transcription in the orthographic group is then scored. Many methods of assigning scores to transcriptions may be used here without detracting from the spirit of the invention. In a preferred embodiment, for each speech token in the labeled speech storage unit associated with the same orthography a score is obtained. The scores for each speech token are then combined in order to obtain a compound score. Following this, the transcriptions are sorted on the basis of these scores and the top scoring transcriptions are selected. In a preferred embodiment, the selection module outputs the top N scoring transcriptions. In another preferred embodiment, the selection module outputs transcriptions with scores above a given threshold. The N top scoring transcriptions of the word are introduced in the speech recognition dictionary and associated with a label corresponding the new word being added. In a preferred embodiment, the label is the orthographic representation of the word.

[0043] In a preferred embodiment of the invention the transcription adaptation apparatus further comprises an augmentor unit. The augmentor unit receives labeled speech, comprising the speech token and the orthography, from the labeled speech storage unit, from the controller or from the speech recognizer. The output of the augmentor module is a transcription corresponding to the speech token. The augmentor unit may make use of a continuous allophone recognizer to generate the transcription. The automatic transcription of utterances is well known in the art to which this invention pertains. Many algorithms may be used here without detracting from the spirit of the invention. The reader is invited to consult: R. Haeb-Umbach, P. Beyerlein and E. Thelen, “Automatic Transcription of Unknown Words in a Speech Recognition System”, Proc. of the IEEE Inter. Conf. On ASSP, 1995, pp. 840-843; J. Duchateau and D. V. Compernolle, “Automatic Modeling of User specific Words for a Speaker-Independent Recognition System”, Proc. Of the IEEE Inter. Conf. On ASSP, 1996, pp. 869-872. N. Jain, R. Cole, E. Barnard, “Creating Speaker-Specific Phonetic Templates with A Speaker-Independent Phonetic Recognizer; Implications for Voice Dialing” Proc. Of the IEEE Inter. Conf. On ASSP, 1996, pp. 881-884 and T. Holter and T. Svendsen, “Maximum Likelihood Modeling of Pronunciation Variation”, Proc. Of ESCA Workshop on Modeling Pronunciation Variation for ASR, 1998, pp. 63-66. The transcription generated by the augmentor unit is then compared to transcriptions already in the transcriptions bank in the appropriate orthographic group. If the transcription is not within the orthographic group it is added to the transcriptions bank.

[0044] In another embodiment, the speech recognition dictionary created by the above method and apparatus is speaker specific. In this case the utterances are collected from the end user of the system and only 2 or three utterances would be required to supply sufficient precision. Optionally, for a user specific dictionary, the training may take place on-line immediately after the utterances are received from the user since there is no need to gather utterances from a plurality of sources. The dictionary would be updated for the next time it is used.

[0045] As embodied and broadly described herein the invention further provides a method for generating a transcription capable of being used to create an entry in a speech recognition dictionary for a certain word, said method comprising the steps of:

[0046] providing a computer readable storage medium containing a transcriptions bank, said transcriptions bank comprising a plurality of orthographic groups, each orthographic group including a plurality of transcriptions associated with a common word;

[0047] receiving:

[0048] a) a signal representative of audio information produced when uttering a certain word;

[0049] b) a label element associated with said signal;

[0050] retrieving on a basis of said label element the orthographic group corresponding to the certain word;

[0051] processing said signal to select a certain transcription from said orthographic group corresponding to the certain word;

[0052] delivering the certain transcription capable of being used for creating an entry in the speech recognition dictionary.

[0053] As embodied and broadly described herein the invention yet provides a computer readable storage medium containing a program element to direct a computer to generate a transcription capable of being used to create an entry in a speech recognition dictionary for a certain word, said computer including:

[0054] memory means including:

[0055] a) a transcriptions bank, said transcriptions bank comprising a plurality of orthographic groups, each orthographic group including a plurality of transcriptions associated with a common word,

[0056] processor means in operative relationship with said memory means, said program element instructing said processor means to implement functional blocks for:

[0057] a) receiving:

[0058] 1) a signal representative of audio information produced when uttering a certain word;

[0059] 2) a label element associated with said signal;

[0060] b) retrieving on a basis of said label element the orthographic group in the transcriptions bank corresponding to the certain word;

[0061] c) processing said signal to select a certain transcription from said orthographic group corresponding to the certain word;

[0062] d) outputting a signal representative of the certain transcription, the signal representative of the certain transcription capable of being used for creating an entry in the speech recognition dictionary.

[0063] As embodied and broadly described herein the invention also provides an adaptive speech recognition system, said system comprising:

[0064] a first input for receiving audio information representative of an utterance of a certain word;

[0065] a speech recognition dictionary comprising a plurality of vocabulary items potentially recognizable on a basis of a spoken utterance;

[0066] a speech recognition unit for selecting on a basis of the audio information representative of an utterance of the certain word a vocabulary item as being a match to the utterance of the certain word;

[0067] an adaptation module, including:

[0068] a) a transcriptions bank, said transcriptions bank comprising a plurality of orthographic groups, each orthographic group including a plurality of transcriptions associated with a common word,

[0069] b) a second input coupled to said speech recognition unit for receiving:

[0070] 1) the signal representative of audio information produced when uttering the certain word;

[0071] 2) the vocabulary item selected by said speech recognition unit as being a potential match to the utterance of the certain word;

[0072] c) a transcription selector module for:

[0073] 1) retrieving on a basis of said vocabulary item at said second input the orthographic group corresponding to the certain word;

[0074] 2) processing the signal representative of audio information produced when uttering the certain word to select a certain transcription from the orthographic group corresponding to the certain word;

[0075] 3) replacing in said speech recognition dictionary the vocabulary item selected as being a match to the utterance of the certain word by a new vocabulary item generated on a basis of the transcription selected from the orthographic group corresponding to the certain word.

[0076] These and other features of the present invention will become apparent from the following detailed description considered in connection with the accompanying drawings. It is to be understood, however, that the drawings are provided for purposes of illustration only and not as a definition of the boundaries of the invention for which reference should be made to the appending claims.

[0077] FIG. 1 shows a simplified functional block diagram of a speech recognition system in accordance with the invention;

[0078] FIG. 2 shows a block diagram of a speech recognition system with a detailed view of an embodiment of the transcription adaptation module;

[0079] FIG. 3aand FIG. 3bshow embodiments of the transcriptions bank creation unit;

[0080] FIG. 4 shows a flow chart of a method in accordance with the invention for augmenting or replacing a selected set of transcriptions from the transcriptions bank;

[0081] FIG. 5 is a block diagram of an apparatus for performing the training of the speech recognition dictionary in accordance with the invention.

[0082] In a most preferred embodiment of the invention, the transcription adapting apparatus is integrated into a speech recognition system with adaptation capabilities offering transcription adaptation. As shown in FIG. 1, such a system generally comprises a number of functional modules namely a speech source 100, a speech recognizer 104 and a transcription adapter module 112. The speech recognition system with adaptation capabilities also includes a computer readable storage medium comprising an action table 108 and a speech recognition dictionary 110. The transcription adapter module 112 interacts with the speech recognition dictionary 110 to modify the latter's contents. Optionally, the speech recognition system may comprise a controller unit 116 in order to provide the system with rejection capabilities.

[0083] In a preferred embodiment, the speech source unit 100 may be a device such as a microphone or telephone set that converts a spoken utterance into an electric signal. In another preferred embodiment the speech source is a computer readable storage medium containing data elements representative of speech utterances. In a specific example, spoken utterances are digitised and stored on a non-volatile storage medium such as a hard-disk. In another specific example, spoken utterances are stored on a magnetic tape. The speech source 100 processes utterances comprising speech and non-speech events. Speech events, herein referred to as speech tokens, may correspond to entries in the recognition vocabulary or may correspond to entries outside the recognition vocabulary (imposter speech). In a specific example, speech tokens are derived from user utterances in a corporate voice activated dialing system. The recognition vocabulary comprises the names of the employees of the corporation and typical speech tokens comprise the utterances of employee names. In such a situation, speech outside the recognition vocabulary may be due to poor task specification such as unpredicted synonyms or nicknames, or may be due to out of domain utterance. For example, if the name “Mike Fox” is in the recognition vocabulary and the speaker requests “Michael Fox”, then the speech is outside of the recognition vocabulary even though it is a synonym for a valid item. As another example, if a speaker requests “What is the present time” in a Name Dialing system, the speech event is considered outside the recognition vocabulary. Non speech events comprise extraneous noise, silences and others. In a preferred embodiment, the speech source comprises speech tokens originating from a captive population such that substantially the same individuals use the application. In a specific example, a corporate name dialing system may be considered as having a captive user population if only employees of the corporation have access to the system. In another preferred embodiment, the speech source may comprise speech token originating from a slowly varying population. In a specific example, a corporate name dialing system may be considered as having a slowly varying user population due to transfers, promotions, retirements and other events which may alter the composition of the users. In yet another preferred embodiment, the user population may be substantially open ended such that most anyone may have access to the system such as in a directory assistance system. In yet another preferred embodiment, the speech comprises speech tokens from a single user, as may be the case in speaker dependent speech recognition system.

[0084] In a preferred embodiment, the speech recognition system with adaptation capabilities comprises a speech recognition dictionary stored on a machine-readable storage medium. The speech recognition dictionary 110 stores a set of vocabulary items potentially recognizable on the basis of a spoken utterance. Typically, a speech recognition dictionary 110 comprises entries such as orthographies and transcriptions. The transcriptions in the dictionary represent data elements that are a combination of symbols providing information on how a sub-word unit, such as a letter or a combination of letters, may be pronounced. The speech recognition unit 104 tries to match the detected speech signal with entries in the speech recognition dictionary 110 and selects the entry that is the most likely to be what the user is saying. Initially the speech recognition dictionary 110 may be created by a dictionary initialization module on the basis of entries in the recognition vocabulary. The recognition vocabulary typically comprises the words the speech recognition module should recognize. The speech recognition dictionary created on the basis of recognition vocabulary may be speaker dependent or independent without detracting from the spirit of the invention. In speech recognition applications, speech is generally used to invoke an action within the application. Preferably, each transcription in the speech recognition dictionary is associated to a label that can be an orthography. In a preferred embodiment, each label in the speech recognition dictionary is in turn associated to a link allowing the desired action associated to the label to be completed upon selection of a transcription associated to the label by a speech recognition system. In the example of a voice activated dialing system, the link may be a pointer to a data structure containing the telephone number of the entity associated with the label. The system then outputs the telephone number that can be automatically dialed. The link may also be an action link that identifies an action to be taken. For example in a system designed to effect a certain procedure in response to a spoken command, the link designates the action to be performed. This designation may be direct in which case the link contains the information identifying the action, or indirect where the link points to a location containing the information identifying the action. In a specific example, the speech recognition system may be used to operate components on the basis of spoken commands. For instance the user may say “lights on” to indicate that the lighting in a room is to be activated. The action link in this case identifies the specific action to be taken. The link may be a data element that contains the information to the effect that the lights should be activated or it may be a pointer to a table containing the entire list of possible actions that the system can effect, the pointer identifying the entry in the table corresponding to the light activation. Thus, for the purposes of this specification, the expression “link” should be interpreted in a broad manner to designate the structure that constitutes an operative relationship between a label and the desired action that is to be performed when a transcription associated with the label is selected by the speech recognition system as a likely match to the spoken utterance.

[0085] Alternatively, the speech processing system may include an action table 108 establishing a relation between a label and an action to be taken. The speech recognizer 104 releases the selected entry to the action table that locates the appropriate action.

[0086] In a preferred embodiment, the speech recognizer 104 receives from the speech source 100 a speech signal. Typically, the speech signal is processed by a sequence of steps performed by the speech recognizer 104. In a specific example, a pre-processing unit in the speech recognizer 104 converts the signal representative of the utterance into a sequence of feature vectors or other suitable representation. For example mel-based cepstral parameters may be used to compose the feature vectors. Feature vectors are well known in the art to which this invention pertains. A search unit implementing a search algorithm then processes the feature vectors. The search unit scores entries in the speech recognition dictionary 110 and selects potential matches to the spoken utterance, herein referred to as recognition candidates. Any suitable search algorithm may be used here without detracting from the spirit of the invention. In a specific example, the search unit implements the two-pass search algorithm described in U.S. Pat. No. 5,515,475 Gupta et al. “Speech Recognition method using a two-pass search”, May 7, 1996 whose content is hereby incorporated by reference. The operation of the speech recogniser 104 is well known in the art to which this invention pertains. For more information about speech recognisers units, the reader is invited to consult the following patents and articles whose contents are hereby incorporated by reference.

[0087] In a preferred embodiment, the controller module 116 receives the candidate recognition results from the speech recognizer 104 and assigns a confidence measure to them. The controller's 116 purpose is to reject candidates that are less likely to be correct. It may make use of external cues such as call completion information, human correction information and others. Optionally, the controller module 116 may detect the presence of known decoys, also known as imposters, depending on the nature of the application. If the confidence measure indicates that the top candidate is not satisfactory, the recognition result is refused and a rejection handler may be invoked. If the controller was able to detect the presence of decoys and that the top candidate was a decoy, an imposter handler may be invoked. If the confidence metric assigned to the top candidate indicates that the top candidate is satisfactory, the top candidate is used to identify the appropriate action in the action table 108. The top candidate is also forwarded to an adaptation module 112.

[0088] The transcription adaptation module 112 uses the recognized orthography received by the controller unit 116 along with the speech token in order to modify the speech recognition dictionary. In a preferred embodiment of the invention, as shown in FIG. 2, the transcription adaptation module comprises a transcriptions bank 206 and a selection module 204. Optionally, the transcription adaptation module 112 may comprise an augmentor module 202 and a labeled speech storage unit 200.

[0089] In a preferred embodiment, the adaptation module 112 comprises a computer readable storage medium containing a repository of transcriptions 206, herein designated as transcriptions bank. The transcriptions bank 206 contains a plurality of transcriptions for each of the orthographies in a recognition vocabulary, herein designated as a lexicon. Optionally, the transcriptions bank may further comprise weight values associated to each transcription in the transcriptions bank, the weight values being indicative of an a priori probability for the transcription. Initially, all automatically generated transcriptions may be assigned a default probability value. As shown in FIG. 3b, the transcriptions bank 206 is created by using a text to transcription module 300 to process entries in a lexicon 302 in order to obtain a plurality of transcriptions using information derived from text to phoneme rules 304. The transcriptions generated by the text to transcription module 300 are then stored on a computer readable storage medium containing the transcriptions bank 206. The lexicon 302 contains entries for the words the speech recognition system should be able to recognize. As a specific example of a corporate name dialing system, the task lexicon contains an entry for the name of each employee. In a preferred embodiment, the entries in the lexicon are orthographies. The letter to phoneme generator 300 is preferably an automatic letter to transcription generator requiring little or no human intervention. Preferably, the text to transcription generator 300 is configured such that it generates a large number of transcriptions. In a specific example, the text to transcription generator 300 generates 50 to 200 transcriptions per orthography. Variation in the number of transcriptions per orthography does not detract from the spirit of the invention. There are many types of letter to phoneme transcription methods that may be used here such as described in “Modeling Pronunciation Variation for ASR: Overview and Comparison of methods”, Helmer Strik and Catia Cucchiarini, Workshop Modeling Pronunciation Variation, Rolduc, 4-6 May 1998, “Maximum Likelihood Modelling of Pronunciation Variation”, Trym Holter and Torbjorn Svendsen, Workshop Modeling Pronunciation Variation, Rolduc, 4-6 May 1998, pp.63-66 and Automatic Rule-Based Generation of Word Pronunciation Networks, Nick Cremelie and Jean-Pierre Martens, ISSM 1018-4074, pp.2459-2462, 1997 whose contents are hereby incorporated by reference. In a most preferred embodiment, text to phoneme rules 304 from many languages are used to include the variations in pronunciations across languages. In a specific example, for a voice dialing system in North America, three sets of letter to phoneme rules are used for three of the widely spoken languages namely French, English and Spanish. In another preferred embodiment, the transcriptions in the transcriptions bank 206 bank are derived from letter to phoneme rules as well as from language dictionaries. As shown in FIG. 3b, the system to generate the transcriptions bank further comprises a language dictionary transcription module 306 and a computer readable storage medium containing a language dictionary 308. The language dictionary provides transcriptions reflecting the correct pronunciation of words in the lexicon 302. The transcriptions resulting from the language dictionary transcription module 306 are then stored in the transcriptions bank 206. In a most preferred embodiment, language dictionaries 304 from many languages are used to include the variations in pronunciations across languages. In a specific example, for a voice dialing system in North America, three language dictionaries are used for three of the widely spoken languages namely French, English and Spanish. Using transcriptions from several languages admits pronunciation variants that are phonetically illegal in a given language but are valid in another language. In a preferred embodiment, the language dictionary transcription module 306 looks up the entry corresponding to each orthography in the lexicon in the language dictionary in order to generate a transcription to add to the transcriptions bank 206. The table below illustrates the different transcriptions that many be generated from the language dictionaries and the letter to phoneme rules for the orthographies “Brault”, “Jose” and “George” in the lexicon.

[0090] An example of transcriptions bank entries for the orthography “Archimede” is shown in the table below. The set of transcriptions associated with a same orthography is herein referred to as an orthographic group. The expression “orthographic group” may also include the orthography associated with the transcriptions in the group.

[0091] Transcriptions in the transcriptions bank may be organized any suitable fashion without detracting from the spirit of the invention. In a preferred embodiment, the transcriptions bank is organized in the form of a table with transcriptions belonging to the same orthographic group stored in logical proximity. In another preferred embodiment, transcriptions belonging to the same orthographic group are stored in a graph. Graphs are well known in the art to which this invention pertains. For more information about graphs the reader is invited to consult V. N. Gupta et al. “Fast search strategy in a large vocabulary word recognizer”, JASA 84(6), pp.2007-2017, Dec. 1988 whose content is hereby incorporated by reference.

[0092] In a preferred embodiment, the adaptation module 112 comprises a computer readable storage medium containing labeled speech data 200. The utterances are tokenized by the speech source 100 and associated with an orthography in the speech recognition dictionary 110 by the speech recognizer 104. The speech tokens and their associated orthographies are then placed in the labeled speech storage unit 200. The labeled speech storage unit 200 may be stored on a computer readable storage medium such as a hard drive or other non-volatile storage medium. The stored tokens may be arranged in the speech storage unit using any suitable data structure. In a specific example, the labeled speech storage unit is organized as a table. Preferably, speech tokens associated with the same orthography are stored such that they are in logical proximity. This may be achieved by storing the speech tokens associated with the same orthography in adjacent memory locations or by using a data structure such as a linked list in order to provide such a logical proximity. The adaptation of the dictionary 110 is then performed off-line in batch mode. Preferably, a large number of speech tokens are used for adaptation. In a specific example, the number of speech tokens exceeds 20 tokens for each orthography. Alternatively, adaptation may be performed on-line after each recognition result as a speech recognition post-process. For on-line processing, the labeled speech storage unit 200 may be omitted from the adaptation module 112 and the speech tokens and orthography sent directly to the processing modules 204 and 202. On-line processing typically causes an increase in the computational requirements of the system but allows the dictionary to be updated for the next time the system is used. The use of batch mode or on-line mode does not detract from the spirit of the invention.

[0093] In a preferred embodiment of this invention, the transcription adaptation apparatus includes a selection module 204 that selects from the transcriptions bank 206 a set of transcriptions to be included in the speech recognition dictionary 110. The selection module 204 receives labeled speech from the labeled speech storage unit 200. Alternatively, if the transcription adaptation occurs on-line the selection module 204 may receive labeled speech items from the speech recognizer 104 or from the controller unit 116. In a specific example, a labeled speech item comprises a speech token and the orthography most likely to correspond to the speech token.

[0094] In a typical interaction, as shown in FIG. 4, the selection module 204 locates in the transcriptions bank 206 the orthographic group associated with the labeled speech item. Following this, the speech token of the labeled speech item is aligned 400 with the transcriptions within the orthographic group. Each transcription in the orthographic group is then scored 402. Many methods of assigning scores to transcriptions may be used here without detracting from the spirit of the invention. In a specific example, the Viterbi alignment score is used. The Viterbi algorithm is well known in the art to which this invention pertains. In a preferred embodiment, for each speech token in the labeled speech storage unit associated with the same orthography a score is obtained. The scores for each speech token are then combined in order to obtain a compound score.

[0095] In a specific example, the transcriptions in the transcriptions bank are arranged in the form of a graph where each branch is associated to a phoneme or allophone. For each speech token, a graph search algorithm is used to score the graph. For more information about a graph search, the reader is invited to consult V. N. Gupta et al. “Fast search strategy in a large vocabulary word recognizer”, JASA 84(6), pp.2007-2017, Dec. 1988 whose content is herein incorporated by reference. Using the methods presented in the above document, each branch in the graph is assigned a branch log likelihood score for each speech token. Once a branch in the graph has been assigned a score for each speech token, a compound score for each branch is computed by combining the score for each individual speech token. The log likelihood scores associated to a branch are summed to create the compound score for the branch. For example, the branch B^nm_k, where k is the index of the branch, in the transition from state S_nto S_mhas log likelihood scores P^k,nm₁, P^k,nm₂and P^k,nm₃for utterances U₁, U₂and U₃respectively. The compound log likelihood score for the branch B^nm_kis P^k,nm₁+P^kn,m₂+P^k,nm₃.

[0096] Following this, the transcriptions are sorted 404 on the basis of these scores and the top scoring transcriptions are selected 406. In a preferred embodiment, the selection module outputs the top N scoring transcriptions. In another preferred embodiment, the selection module outputs transcriptions with scores above a given threshold.

[0097] Optionally, following this, the top transcriptions are assigned weight values 410. In a preferred embodiment, the rank order method is used to assign weight values to the transcriptions. The rank order method can be used to score the transcriptions. The rank order method assigns a weight value based on the rank of the transcription choice. In a specific example, the top N transcriptions are considered. The top weight is assigned a weight value of 1.0 and the Nth choice is assigned a weight value of 1.0/N. The intermediate weight values are linearly decreasing such that the Kth choice is assigned a weight value of 1.0-K/N. The table below shows the calculated rank order values:

[0098] Other mechanisms may be used to assign weight values to the intermediate results without detracting from the spirit of the invention. In another preferred embodiment, the score ratio method is used to assign weight values to the transcriptions. The score ratio method assigns a weight value to a transcription on the basis of the ratio of top choice Viterbi alignment score to the weight of the Kth choice. Using this method, the weight value of the top transcription is 1.0 and the weight value the Kth transcription is [Viterbi (1)/Viterbi (K)]. Other suitable methods of assigning weight values to the top scoring transcriptions may be used here without detracting from the spirit of the invention.

[0099] Following this, the transcriptions with the top weight values are selected to augment or replace the transcriptions in the speech recognition dictionary. Alternatively, the transcriptions with the top weight values are added to the original transcriptions in the speech recognition dictionary 408. The combined set of top scoring transcriptions and original dictionary transcriptions constitutes the new speech recognition dictionary entry. Optionally, the weight values are used to maintain statistics in the transcriptions bank. As previous mentioned, optionally, to each transcription in the transcriptions bank is associated an a priori weight value. The weight values computed by the above-described method may be added on to the weight value in the transcriptions bank or may replace the weight values.

[0100] In a preferred embodiment of the invention the transcription adaptation apparatus comprises an augmentor unit 202. The augmentor unit 202 receives labeled speech, comprising the speech token and the orthography, from the labeled speech storage unit 200, from the controller 116 or from the speech recognizer 104. The output of the augmentor module 202 is a transcription corresponding to the speech token. The augmentor unit 202 may make use of a continuous allophone recognizer to generate the transcription. Automatic transcription of utterance is well known in the art to which this invention pertains. Many algorithms may be used here without detracting from the spirit of the invention. The reader is invited to consult: R. Haeb-Umbach, P. Beyerlein and E. Thelen, Automatic Modeling of User Specific Words in a Speech Recognition System, Proc. Of the IEEE Inter. Conf. On ASSP, 1995, pp. 840-843. J. Duchateau and D. V. Compernolle, Automatic Modeling of User specific Words for a Speaker-Independent Recognition System, Proc. Of the IEEE Inter. Conf. On ASSP, 1996, pp. 869-872. N. Jain, R. Cole, E. Barnard, Creating Speaker-Specific Phonetic Templates with A Speaker-Independent Phonetic Recognizer; Implications for Voice Dialing. Proc. Of the IEEE Inter. Conf. On ASSP, 1996, pp. 881-884. T. Holter and T. Svendsen, Maximum Likelihood Modeling of Pronunciation Variation, Proc. Of ESCA Workshop on Modeling Pronunciation Variation for ASR, 1998, pp. 63-66. The transcription generated by the augmentor unit is then compared to transcriptions already in the transcriptions bank 206 in the appropriate orthographic group. If the transcription is not within the orthographic group it is added to the transcriptions bank.

[0101] The above-described method for adapting a speech recognition dictionary can be implemented on any suitable computing platform as shown in FIG. 5. Such computing platforms typically include a CPU 502 and a memory 500 connected to the CPU by a data communication bus. The memory 500 stores the data 506508510 and the instructions of the program 504 implementing the functional blocks depicted in the drawings and described in the specification. That program 504 operates on the data 506508 in accordance with the algorithms described above to modify the speech recognition dictionary 510 based on a plurality of spoken utterances.

[0102] Although the present invention has been described in considerable detail with reference to certain preferred embodiments thereof, variations and refinements are possible without departing from the spirit of the invention as have been described throughout the document. The methods described above may also be applied to combination of words without detracting from the spirit of the invention. Therefore, the scope of the invention should be limited only by the appended claims and their equivalents.