Speaker model adaptation via network of similar users

[0002] The present invention is related to U.S. patent application Ser. No. 08/787,031, filed Jan. 28, 1997 entitled “Speaker Recognition Using Thresholded Speaker Class Model Section or Model Adaptation” to Ittycheriah, et al. now issued as U.S. Pat. No. 5,895,447, U.S. patent application Ser. No. 08/788,471, filed Jan. 28, 1997 entitled “Text Independent Speaker Recognition for Transparent Command Ambiguity Resolution and Continuous Access Control” now U.S. Pat. No. 6,073,101 issued Jun. 6, 200, and U.S. patent application Ser. No. 08/787,029, filed Jan. 28, 1997 entitled “Speaker Model Prefetching” both to Stephane Maes now U.S. Pat. No. 6,088,669 issued Jul. 11, 2000, and (Ser. No. 09/422,383) entitled “Language Model Adaptation Via Network of Similar Users” filed Oct. 21, 1999, all assigned to the assignee of the present invention. These patents and patent applications are herein incorporated by reference in their entirety.

[0003] 1. Field of the Invention

[0004] The present invention is related to speech recognition and more particularly to speech recognition on multiple connected computer systems connected together over a network.

[0005] 2. Background Description

[0006] Automatic speech recognition (ASR) systems for voice dictation and the like use any of several well known approaches to for word recognition.

[0007] For example, L. R. Bahl, P. V. de Souza, P. S. Gopalakrishnan, D. Nahamoo, and M. Picheny, “Robust Methods for Using Context-dependent Features and Models in Continuous Speech Recognizer,” Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, vol. I, pp. 533-36, Adelaide, 1994, describe an acoustic ranking method useful for speech recognition. Acoustic decision trees, also useful for speech recognition are described by L. R. Bahl, P. V. de Souza, P. S. Gopalakrishnan, D. Nahamoo, and M. Picheny, in “Decision Trees for Phonological Rules in Continuous Speech,” Proceedings of the1991 International Conference on Acoustic, Speech, and Signal Processing, Toronto, Canada, May 1991. Frederick Jelinek in Statistical Methods for Speech Recognition, The MIT Press, Cambridge, January 1999, describes identifying parameters that control decoding process.

[0008] While generally recognizing spoken words with a relatively high degree of accuracy, especially in a single user system, these prior speech recognition systems still, frequently, make inappropriate recognition errors. Generally, for single user systems, these errors can be reduced with additional user specific training. However, additional training time and increased data volume that must be handled during training are undesirable. So, for expediency, recognition accuracy is traded to minimize training time and data.

[0009] Speaker independent automatic speech recognition systems, such as what are normally referred to as interactive voice response systems, have a different set of problems, because they are intended to recognize speech from a wide variety of individual speakers. Typically, the approach with speaker independent ASR systems is to improve recognition accuracy by assigning individual speakers or recognition system users to user clusters. User clusters are groups of users with similar speech characteristics or patterns. As each speaker or user uses the system, the speaker is identified as belonging to one cluster. For each user cluster, acoustic prototypes are developed and are used for speech decoding.

[0010] For example, speakers may be clustered, according to language or accent. Various techniques for language identification are taught by D. Matrouf, M. Adda-Decker, L. Lamel and J. Gauvain, in “Language Identification Incorporating Lexical Information” in Proceedings of the1998 International Conference on Spoken Language Processing(ICSLP 98), Sydney, Australia, December 1998. A well known method of determining an accent from acoustic features is taught by M. Lincoln, S. Cox and S. Ringland, in “A Comparison of Two Unsupervised Approaches to Accent Identification” Proceedings of the1998 International Conference on Spoken Language Processing(ICSLP 98), Sydney, Australia, December 1998. However, the approach of Lincoln et al., if there is a very large speaker variability, as is normally the case, that variability may not be accounted for in training. Accordingly, speaker clusters that are accumulated in a normal ASR training period, generally, do not provide for all potential ASR users.

[0011] Consequently, to provide some improvement over speaker dependent methods, ASR decoding system approaches are used that are based on various adaptation schemes for acoustic models. These recognition adaptation schemes use additional data that is gathered subsequent to training by the ASR system every time a user dictates to the system. The speaker or user, usually, interactively corrects any errors in the recognition result, and those corrected scripts are used for what is normally referred to as a supervised adaptation.

[0012] See for example, Jerome R. Bellegarda, in “Context-dependent Vector Clustering for Speech Recognition,” in Automatic Speech and Speaker Recognition, edited by Chin-Hui Lee, Frank K. Song, 1996, Kluwer academic Publishers, Boston, pp. 133-153 which teaches an adaptation of acoustic prototypes in response to subsequent speech data collected from other users. Also, M. J. F. Gales and P.C. Woodland, “Mean and variance adaptation within the MLLR framework,” Computer Speech and Language(1996) 10, 249-264 teach incremental adaptation of HMM parameters derived from speech data from additional subsequent users.

[0013] The drawback with the above approaches of Bellegarda or Gales et al. is that during typical dictation sessions the user uses a relatively small number of phrases. So, it may take several user sessions to gather sufficient acoustic data to show any significant recognition accuracy improvement using such a supervised adaptation procedure. As might be expected, in the initial sessions the decoding accuracy may be very low, requiring significant interactive error correction.

[0014] Further, similar or even worse problems arise in unsupervised ASR applications when users do not correct ASR output. For example, unsupervised ASR is used in voice response systems wherein each user calls in to a service that uses ASR to process user voice input. C.H. Lee and J.L. Gauvain, “Bayesian adaptive Learning and MAP Estimation of HMM”, in Automatic Speech and Speaker Recognition, edited by Chin-Hui Lee, Frank K. Song, 1996, Kluwer academic Publishers, Boston, pp. 109-132 describe for supervised and unsupervised acoustic model adaptation methods. While it is still possible to adapt speech recognition for any new users using unsupervised adaptation, sufficient data must be collected prior to unsupervised use to insure adequate decoding accuracy for every new user.

[0015] Thus, there is a need for increasing the amount of usable acoustic data that are available for speech recognition of individual speakers in supervised and unsupervised speech recognition sessions.

[0016] It is a purpose of the invention to improve speech recognition by computers;

[0017] It is yet another purpose of the invention to expand the data available for speech recognition.

[0018] The present invention is a speech recognition system, method and program product for recognizing speech input from computer users connected together over a network of computers, each computer including at least one user based acoustic model trained for a particular user. Computer users on the network are clustered into classes of similar users according their similarities, including characteristics nationality, profession, sex, age, etc. Characteristics of users are collected from databases over the network and from users using the speech recognition system and distributed over the networks during or after user activities. As recognition progresses, similar language models among similar users are identified on the network. The acoustic models include an acoustic model domain, with similar acoustic models being clustered according to an identified domain. Existing acoustic models are modified in response to user production activities. Update information, including information about user activities and user acoustic model data, is transmitted over the network. Acoustic models improve for users that are connected over the network as similar users use their respective voice recognition system.

[0019] The foregoing and other objects, aspects and advantages will be better understood from the following detailed description of a preferred embodiment of the invention with reference to the drawings, in which:

[0020] FIG. 1 is a block diagram of users connected together over network using shared acoustic models during user production activities according to the preferred embodiment of the present invention.

[0021] FIG. 2 is a flow chart of the preferred embodiment speech recognition process operating over the network of FIG. 1;

[0022] FIG. 3 is a flowchart showing how one or more acoustic models are changed;

[0023] FIG. 4 is an illustration of user actions and user production activities;

[0024] FIG. 5 is a flow diagram of the user clustering step;

[0025] FIG. 6 is a flow chart of the acoustic component comparison step;

[0026] FIG. 7 is a flowchart illustrating supervised and unsupervised speech adaptation according to the preferred embodiment of the present invention.

[0027] Referring now to the drawings and, more particularly, FIG. 1 shows a speech recognition network 100, wherein individual user utterances are analyzed to classify the user, the individual user is clustered with other similar users, and data from similar acoustic models for those clustered users are aggregated to provide an expanded or supplemented acoustic model for recognition of that individual user. Computers 102, 104 and 106 connected to the network 100 are each capable of executing an acoustic model (AM) for some type of speech recognition. Also, speech enabled interface devices 108, 110 with embedded processors or smart controllers may be connected to the network 100. In the preferred embodiment networked system, utterances or speech input may be, for example, for command/menu navigation, dictation or transcription.

[0028] Preferably, the speech recognition network 100 is a local area network (LAN) of connected speech recognition computers 102, 104, 106 and speech enabled devices 108, 110. Optionally, the network 100 may be a wide area network (WAN), a connection of computers over what is commonly referred to as the internet or world wide web (www) or over an intranet, an extranet, a radio network or a telephone network, or any combination thereof

[0029] As shown in FIG. 1, by way of example only, connected computers 102, 104, 106 may include what are commonly referred to as personal computers 102, hand held computers 104, and one or more servers 106. Hand held computers 104 may include what is known as a personal digital assistant (PDA). Connected speech enabled interface devices may include, for example, cameras 108, intelligent watches 110 and connected telephones 116. Further, microphones 112, 114 are shown connected to personal computer 102 and PDA 104 and are integrated into speech enabled interface devices 108, 110 for receiving speech input from a user.

[0030] Personal computers 102 also may include an audio capture module that receives audio signals and converts received audio signals into digital signals. Each of the speech recognition computers 102, 104 includes an automatic speech recognition module and a local database containing acoustic models for local users. For speech recognition, each local acoustic model is the direct result of training by a specific local user. Further, a global database is maintained on at least one speech recognition server 106. The global database may include multiple acoustic models for users of connected computers, as well as individual user speech data. Further, as individual user features or characteristics are collected from local user databases in computers 102, 104, the collected features are aggregated in the global databases on servers 106. Interface devices 108, 110 may avail themselves of excess capacity on servers 106, storing local databases on a server 106 and using acoustic models from a server's global database for speech recognition.

[0031] The preferred embodiment of the present invention includes a dialog management unit for conducting a conversation with an individual user. An audio capture module coupled to the dialog management unit captures a speech waveform from utterances spoken by the individual user during the conversation. An acoustic front end coupled to the audio capture module is configured to receive and digitize the speech waveform so as to provide a digital waveform, and to extract, from the digital waveform, at least one acoustic feature. The acoustic front end and audio capture module may be, for example, a microphone connected to an analog-to-digital converter located on a sound board in a personal computer or a telephone connected to an automatic interactive voice response (IVR) system. The dialog management unit can include a telephone IVR system that may be, for example, the same automatic IVR system used to implement the audio capture module. Alternatively, the dialog management unit may simply be an acoustic interface to a human operator.

[0032] The preferred embodiment system includes at least one processing module coupled to the acoustic front end that analyzes the extracted acoustic features to determine user cluster attributes, i.e., to classify the user or speaker. Each processing module includes a speaker clusterer and classifier. Preferably, the processing module is implemented by the processor of the IVR system. Alternatively, dedicated hardware may be used for the processing module such as an application specific integrated circuit (ASIC) or a separate general purpose computer with appropriate software. The classifier can include a speaker clustering and classification module as well as a speaker classification data base. Cluster user attributes from the processing module are stored in a speaker cluster database stores. A speech adaptation module transmits data to other connected user speech systems. Thus, the present invention is an apparatus for collecting data associated with the voice of a user, which is then supplemented by previously collected data and used for speech recognition for the user.

[0033] So, by conducting a conversation with a voice system user, capturing the user's speech waveform and digitizing the speech waveform acoustic features may be extracted from the digitized speech waveform. The extracted features are passed to other connected systems and used to modify the speech recognition systems of similar users clustered in the same user cluster.

[0034] Speaker classification may be supervised or unsupervised. For supervised classification, the classes are decided beforehand based on externally provided information. Typically, such classification employs distinctions between male and female, adult versus child, native speakers versus different potential nonnative speakers, and the like. For unsupervised classification there is no advanced user labeling and classes are developed on the fly with the classification information being extracted from data using very little supervision, if any, and with sounds being clustered as classes develop.

[0035] Preferably, the processing module includes an accent identifier. The accent identifier includes an accent identification module and an accent data base. The accent identifier is employed for native language identification in a manner equivalent, essentially, to accent classification. Meta information about the identified native language of a speaker provides additional to definition each accent/native language model. A dialect can be determined from the user's accent.

[0036] According to the preferred embodiment, a continuous speech recognizor is trained by several speakers with different accents. For accent identification an accent vector is extracted from each individual user's speech and the accent vector is classified. The accent vector is associated with each of the training speakers. Accent vector dimensions represent the most likely component mixture associated with each state of each phoneme or phone. Then, the speakers are clustered based on the distance between corresponding accent vectors, and the clusters are identified by the accent of the member speakers.

[0037] For each cluster various types of speech recognition may be employed for speech recognition, which in combination with the particular speech recognition computer 102, 104, 106 determine the form of the individual acoustic models. Thus, individual model types may include what are known as acoustic prototypes, Hidden Markov Models (HMM) modeling words and phonemes or phones, acoustic ranks, acoustic decision trees. The preferred statistical analysis techniques include analysis of parameters such as weighted mixtures of decoding scores, thresholds that control decoding stacks, duration of phones or words, sizes of previously listed decoding alternatives, and/or the size of decoding trees.

[0038] Further, each acoustic model is directed to a different speech domain. It is important to distinguish between speech domains, because the particular speech domain deeply influences the resulting speech model. Accordingly, in addition to dictation, the speech domains may include telephone speech, speaker independent speech, gender related speech, age related speech, broadcasting speech, speech partially obscured by noise, speech with music, discrete and continuous speech. Further, speech may be the result of different user production activities such as dictation or conversation that is supplemented by error correction and may be partially obscured by noise or music or by some other type of sound generation. As used herein, user production activities refers to, generally, speech or audio related activities that are intended to produce specific computer related responses.

[0039] So, for the networked speech recognition system of the preferred embodiment, as users issue commands, dictate letters, etc., a corresponding local acoustic model on each corresponding user's system, e.g., PDA 104, recognizes the particular user's speech. If the user corrects the results of the recognition, the corrections are stored locally in a local database and used to adjust and refine the local acoustic model. As correction or modifications are made to the local model, the corrections and modifications are passed across the network 100 to the global database on one of the connected recognition servers 106, which in turn distributes the corrections across the network to computers of other clustered similar users.

[0040] The preferred processing module also includes a speaker recognizor, which may be the same as the speech recognizor above. The speaker recognizor is a speaker recognition module that includes a speaker prototype model, a language model and a grammar database. Preferably, the speaker recognizor transcribes queries from the user. In one preferred embodiment, the speaker recognizor is a speaker-independent large-vocabulary continuous-speech recognition system. In a second preferred embodiment, the speaker recognizor is a class-dependent large-vocabulary continuous-speech recognition system. Such speech recognition systems are well known in the art. The output of the normal speech recognizor is complete sentences. However, finer granularity also is selectable, e.g., time alignment of the recognized words.

[0041] As described hereinabove, the acoustic front end extracts acoustic features that are supplied to the speaker clusterer and classifier, the speaker recognizor and the accent identifier. Preferably, the acoustic front end is an eight-dimension-plus-energy front end, such as are well known in the art. For the preferred front end, the speech spectrum is divided into cepstral coefficients using a bank of MEL filters providing what is referred to as MEL cepstra. Thus, for example, MEL cepstra is computed over 25 ms frames with a 10 ms overlap, in combination with deriving first and second finite derivatives, typically referred to as the delta and delta-delta parameters of the speech. Other types of optional acoustic features that may be extracted by the acoustic front end include a running average pitch, a running pitch variance, pitch jitter, running energy variance, speech rate, shimmer, fundamental frequency, and variation in fundamental frequency. Pitch jitter is the number of sign changes of the first derivative of pitch. Shimmer is energy jitter.

[0042] These optional acoustic features may be supplied from the acoustic front end to the classifier. The aforementioned acoustic features, including the MEL cepstra are, essentially, raw, unprocessed features.

[0043] User queries are transcribed by an IVR, for example, and speech features are first processed by a text-independent speaker classification system according to the preferred speaker clusterer and classifier 120 of FIG. 2 which shows a flow chart of the preferred embodiment speech recognition process 120 operating over the network 100 of FIG. 1. This permits classification of the speakers based on acoustic similarities of their voices. Systems and methods of classifying users according to voice similarities is taught in U.S. patent application Ser. No. 08/787,031, filed Jan. 28, 1997 entitled “Speaker Recognition Using Thresholded Speaker Class Model Selection or Model Adaptation” to Ittycheriah, et al. now issued as U.S. Pat. No. 5,895,447, U.S. patent application Ser. No. 08/788,471, filed Jan. 28, 1997 entitled “Text Independent Speaker Recognition for Transparent Command Ambiguity Resolution and Continuous Access Control” and U.S. patent application Ser. No. 08/787,029, filed Jan. 28, 1997 entitled “Speaker Model Prefetching” both to Stephane Maes, all three assigned to the assignee of the present invention and incorporated herein by reference.

[0044] Different user acoustic models are clustered into classes according to acoustic similarities of the users, thereby clustering the speakers based on vocal and verbal similarities. First, in step 122, acoustic profile data for individual users previously accumulated and stored in the local databases are passed over the network 100 to the server 106. The user acoustic data are compared in step 124 in the server 106. In step 126, based on that comparison, users are clustered into classes of similar users according to acoustic voice similarities. Then, in step 128, different acoustic models (i.e., different domains) are compared in sets associated with similar users, to derive cluster update data. Finally, in step 130, acoustic model components for similar users are modified relative to user production activities. As each individual acoustic model is changed, similar acoustic models from different user sets located elsewhere on the network also are modified in the server 106. At appropriate times, such as upon user request, modified acoustic models are transmitted from the server 106 to other sites on the network 100. So, acoustic model components, including data about users and information about user activities, are thereby synchronized in all similar acoustic models across the network.

[0045] FIG. 3 is a flowchart showing how one or more acoustic models are modified in step 130. The input changes to the model may be supervised input 134 or unsupervised input 132, the output of automatic speech recognition 136 or the result of user production activities 138 as described in detail hereinbelow with reference to FIG. 4. Further, the result of the user production activities 138 may be additional speech data (i.e., data collected from other speech related tasks such as speaker identification, speech recording, etc.) 140 or acoustic training data 142.

[0046] Acoustic training data 142 is generated at each initial use by a new user. Acoustic training data 142 includes, for example, acoustic prototypes, or Hidden Markov Models. Alternately, acoustic training data 142 may be employed for growing acoustic decision trees, each decision tree being based on the user's speech training data. Furthermore, acoustic training 142 may include estimating parameters that control the decoding process, estimating parameters that control signal processing and compiling a code book of user speech data. Parameters that control the decoding process may include, but are not limited to, weights of decoding score mixtures, decision thresholds that control decoding stacks, phone or word durations, decoding alternative list sizes, decoding tree sizes.

[0047] After receiving all acoustic input data, user acoustic model components are modified in step 144. Then, in step 146, acoustic prototypes are adapted for any additional or subsequent speech data produced by a user. In step 148, HMM parameters are adapted incrementally to additional user speech data. Finally, in step 150, new words are added to the acoustic vocabulary and new queries are added to acoustic decision trees. Additionally, adding new words in step 150 may entail modifying acoustic ranks, as well as adapting relative weights of language models and acoustic models.

[0048] FIG. 4 is an illustration of user actions and user production activities. User production activities may include activities such as dictation 160, conversation 162, error correction 164, generation of sounds 166 including noise and music. So, dictation 160, conversation 164 and background audio 166 are provided to automatic speech recognition module 168. The automatic speech recognition module 168 generates either text 170 or passes the recognition results to dialog module 172. Error correction 164 operates on the text 170, correcting any recognition errors, providing a supervised adaptation 174 of the input. The dialog module 172 generates system commands 176 and queries 178 in response to recognition results passed to it.

[0049] FIG. 5 is a flow diagram of the user clustering step 126 in FIG. 1. First, in step 180, the user's speaker characteristics, including but not limited to the user's educational level, age, gender, family relationship and nationality are gathered and provided as a user profile in step 182. Network data for all users including user profiles are compared to identify similar users in step 184. Independently, in step 186 an acoustic front end produces acoustic features, e.g., as the result of training. In step 188, corresponding acoustic features are identified in the speaker's voice. As noted above acoustic features may include, for example, accent, vocal tract characteristics, voice source characteristics, fundamental frequency, running average pitch, running pitch variance, pitch jitter, running energy variance, speech rate, shimmer, fundamental frequency, variation in fundamental frequency and MEL cepstra. Then, in step 192, acoustic features collected from various users are compared. Acoustic models from the same domain but from different sets or systems are compared in step 192. Common features are identified in step 194 and passed to step 184 to identify similar users. Similar users identified in step 184 are users that have one or more common characteristics or, one or more common acoustic features. In step 196, user clusters are identified to cluster users with one or several common features, with several similar acoustic components or with similar profile characteristics, thereby classifying such users in the same classes. Additionally, thereafter, user characteristics are recorded, collected and used for further user classification.

[0050] FIG. 6 is a flow chart of the acoustic component comparison step 192 of FIG. 5. In step 200 acoustic vocabularies and features are provided and represented as vectors in step 202. In step 204, the distance, preferably the Euclidean distance between vectors is calculated. Alternately the Kulback distance may be calculated. In step 206, the computed distances are compared against threshold values to identify similar models, similar models being defined as having calculated values that fall below the threshold values. Acoustic user vocabularies, acoustic features, acoustic user components, acoustic prototypes, Hidden Markov Models for words and phones and accent vectors are compared to determine similarities. Also, acoustic vocabularies of similar users may be analyzed to update the user acoustic vocabulary.

[0051] FIG. 7 is a flowchart illustrating supervised and unsupervised speech adaptation according to the preferred embodiment of the present invention, further illustrating aspects of the invention not shown in FIG. 4. Automatic speech recognition module 210 receives speech and, depending on the content of the user's speech, provides commands 212, queries 214 or uncorrected decoded textual data 216. Commands 212 and queries 214 are passed directly to one or more applications 218 and/or to the operating system. Commands 212 may direct an application operation, e.g., “open file . . . , ” “close,” “indent,” or, when passed to the operating system, may direct window navigation. Queries 214 are passed to appropriate applications 218, e.g., queries 214 are passed to a database manager for database searching.

[0052] Further, Commands 212 and queries 214 that are passed to applications 218 elicit a textual output. The textual output is passed to a supervisor 222 for approval. The text may be, for example, used for transmission as e-mail; a decoded document for storage after some period of time that signifies supervisor approval; a decoded document that was corrected; or, a newly decoded document. For supervised applications, uncorrected decoded textual data 216 is corrected in step 224 and passed to the supervisor 222 as corrected text 226. For unsupervised recognition, the decoded textual data 216 is provided directly for unsupervised adaptation 228. Accordingly, the decoded textual data 216 is either uncorrected textual data, unexecuted commands or unissued queries.

[0053] While the invention has been described in terms of preferred embodiments, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.