Predicting Auditory Confusions Using a Weighted Levinstein Distance

The present invention relates generally to speech processing and recognition, and specifically, to methods and systems of predicting the likelihood of a human or machine confusing two different words or phrases.

Words that sound alike tend to be confused with one another. For example, a listener hearing the word "brush" may mistakingly believe she is hearing "rush," while the same listener hearing the word "falcon" is not likely to believe that she is hearing the word "peckham."

Knowing that two words are likely to be confused can be useful in a number of applications. For example, two applications that may benefit from knowing the likelihood of confusion of two words are: speech recognition algorithms, which attempt to associate a spoken word with its intended orthography (i.e., the English written version of the word), and vocabulary rejection algorithms, which determine whether a spoken word is present in a predefined dictionary.

One method to determine the likelihood of confusion between spoken words is to simply have a human expert rate the potential confusability of the words based on intuition and experience. However, this method is laborious and results in subjective and inconsistent confusability measurements.

Thus, there is a need to be able to automatically generate, from a written version of a word, an objective metric of the likelihood of confusing a spoken word with another spoken word.

It is an object of the present invention to automatically generate an objective metric of the likelihood of confusing two words.

A method consistent with the present invention determines the likelihood of confusing a first word for a second word. The method comprises the steps of (1) phonemically transcribing the first and second words into first and second transcriptions; (2) calculating a Levinstein distance between the first and second transcriptions as the number of edit operations required to transform the first transcription into the second transcription; (3) obtaining a phonemic transformation weight for each edit operation of the Levinstein distance; and (4) summing the weights to generate a value indicating the likelihood of confusion between the first and second words.

Additionally, a computer system consistent with the present invention comprises a memory, a processor, and a display. The memory stores a confusability tool program and a transcripted phonemic lexicon. The confusability tool program includes means for calculating a Levinstein distance between first and second phonemic transcriptions from the lexicon as the number of edit operations required to transform the first transcription into the second transcription, obtaining a phonemic transformation weight for each edit operation of the Levinstein distance, and summing the weights to generate a value indicating the likelihood of confusion between the first and second transcriptions.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention. In the drawings,

FIG. 1 is a block diagram of an exemplary computer system used to execute methods consistent with the present invention;

FIG. 2 is high level block diagram illustrating the functional relationships of software components consistent with the present invention;

FIG. 3 is a flowchart illustrating a method of confusability cost calculation for phonemic transcription pairs; and

FIG. 4 is a flow chart illustrating the use of the present invention in a vocabulary diagnostic tool.

Reference will now be made in detail to the embodiments of the invention, examples of which are illustrated in the accompanying drawings.

Generally, methods and systems consistent with the present invention generate, for each of multiple pairs of phonemic transcriptions, a numeric indication of the potential confusability of the spoken version of the phonemic transcriptions. The numeric indication is based on the Levinstein distance, which determines the number of edit operations required to transform a first phonemic transcription into a second. The number of edit operations can be roughly equated to the degree of similarity, or the confusability, of the two phonemic transcriptions. The final numeric indication of confusability is derived from the Levinstein distance by multiplying each required edit operation by a weight corresponding to the significance of the edit operation, and summing the resultant values.

A "phonemic transcription" of a word is a representation of the word comprising a series of phonemes, which are the basic units of sound in a given language that distinguish one word from another. For example, the initial sound in "cat" and "kick" may be represented by the phonemic symbol /k/ while the one in "circus" may be represented by the symbol /s/. Throughout this disclosure, a phonemic alphabet of 40 symbols is used, although other phonemic alphabets could equivalently be used. Further, backslashes will be used to distinguish a symbol as a phonemic one. In contrast to a phonemic transcription of a word, an orthographic transcription of the word refers to the typical spelling of the word.

Phonemic transcriptions are important because English spelling is not phonetic; that is, there is often no direct correspondence between the spelling and the sounds we actually produce. One sound can be represented by a number of different letters or combinations of letters. For example, the first sound in the words "cat," "kick," "quick," and "chemistry" is the same.

FIG. 1 is a block diagram of an exemplary computer system used to execute methods consistent with the present invention. Computer system 102 includes a chassis 110, which holds the computer's main processor and main memory; an input device such as keyboard 112; a storage device such as floppy or hard disk drive 114; and a display such as monitor 116. Preferably, the methods described herein are executed by the processing section of computer system 102. Computer system 102 is shown connected to network 118, and may be operated directly by a user or through the network.

Many variations of computer system 102 are possible. For example: storage device 114 may additionally include storage media such as optical disks, and user input device 112, instead of or in addition to keyboard 112, may include any type of user input device such as: an electronic mouse, a trackball, a lightpen, a touch-sensitive pad, a digitizing tablet, or a joystick.

FIG. 2 is high level block diagram illustrating a functional relationship of the data operated on by computer system 102. As shown, a confusability tool 202 receives, as inputs, a phonemically transcribed lexicon 204 and a phonemic transformation weight matrix 206. Lexicon 204 may be, for example, a pre-selected corpus of orthographies that have been phonemically transcribed either manually or through a known automated transcription method. The phonemic transformation weight matrix 206, W(/x/, /y/), specifies a weight value for each pair of phoneme symbols (/x/, /y/), as well as the phoneme null pairs (/x/,` `) and (` `,/y/).

Based on phonemic lexicon 204 and weight matrix 206, confusability tool 202 generates an ordered set of predicted auditory confusions 208. The generated output set is preferably in the form {orthography 1, orthography 2, cost}, where the cost is a number between zero and one. A small value of cost indicates a high level of potential confusability and a large value of cost indicates a high level of distinctiveness.

In addition to the ordered set of predicted auditory confusions, the tool may output a global confusability parameter. This parameter measures the confusability of the entire input lexicon and is preferably generated by inverting the confusability cost for the most confusable pairs of orthographies and summing the inverted values.

FIG. 3 is a flowchart illustrating the operation of the confusability tool 202 for phonemic transcription pairs from lexicon 204. For each pair of phonemic transcriptions T.sub.1. and T.sub.2, the tool calculates a Levinstein distance and the edits needed to transform T.sub.1 into T.sub.2 (described in more detail below) and uses this information to calculate a weighted Levinstein distance (steps 302, 304, and 306). A weighted Levinstein distance is calculated for each possible Levinstein distance, and the final weighted Levinstein distance, D(T.sub.1, T.sub.2), is defined as the minimum of the possible weighted Levinstein distances (step 308). The final cost value is a normalized version of D(T.sub.1, T.sub.2) (step 310). The normalization is preferably performed by dividing the final weighted Levinstein distance by the length of T.sub.1 plus T.sub.2.

The final weighted Levinstein distance, determined in step 308, can be represented in a single equation as: ##EQU1## Here, T.sub.1 is the source phonemic transcription, T.sub.2 is the target phonemic transcription, N is the number of transformations between T.sub.1 and T.sub.2, i is the index representing the edit operations of the Levinstein distance set (i.e., the particular substitutions, deletions, or insertions), and /x/ and /y/ are the phonemes involved in the edit, and may be the null phoneme ` ` for an insertion or deletion. The function W is the phonemic transformation weight 206 associated with the phonemic pair (/x/, /y/). As described above with reference to step 308, the minimization indicated by "min" is over the possible phonemic transformations between T.sub.1 and T.sub.2.

Some orthographies have more than one phonemic transcription. To calculate the minimum weighted Levinstein distance between a pair of orthographies having different transcriptions, steps 302, 304, 306, and 308 are repeated for each possible pairing of the phonemic transcriptions of the two orthographies. If the weighted Levinstein distance between a pair of orthographies is denoted D(O.sub.1, O.sub.2), where O.sub.1 and O.sub.2 are the two distinct orthographies, then, in terms of equation (1), D(O.sub.1, O.sub.2) can be written as:

Here, the weighted Levinstein distance between the pair of orthographies is defined as the minimum weighted Levinstein distance between the n possible transcriptions of T.sub.1 and m possible transcriptions of T.sub.2.

As discussed above, calculation of the weighted Levinstein distance requires calculation of the Levinstein distance between two phonemic transcriptions (step 304). Suitable methods are known for computing the Levinstein distance between two strings, and include, for example, a method using maximum substring matching and a method using the Viterbi search algorithm across a transcription graph. For purposes of explanation, an exemplary Levinstein distance calculation will now be described.

Calculating the Levinstein distance involves "editing" a first string to transform it into a second string. Three different types of edits are preferably used to perform the transformation: (1) substitution, (2) deletion, and (3) insertion. With these three edit operations, any word can be transformed into another. For example, the substitution transformation {`W`, `f`} may indicate that "wish" should be transformed into "fish"; the deletion transformation {`t`, ` `} may indicate that "tent" is to become "ten"; and the insertion transformation {` `, `h`} may indicate that "tent" is to become "tenth." Each substitution, deletion, or insertion is defined to involve only a single character symbol.

The Levinstein distance is the number of substitutions, deletions, or insertions required to transform the first phonemic transcription, T.sub.1, into the second, T.sub.2. Multiple transformation sets may be possible between T.sub.1 and T.sub.2. For example, "fishing" to "fashion" can be transformed as: (1) {`i`, `a`}, {` `, `o`}, and {`g`, ` `}; or as (2) {`i`, `a`}, {`n`, `o`}, and {`g`, `n`}. In this example, each transformation has a Levinstein distance of three. Note, however, that because W(` `, `o`) may have a value different than W(`n`, `o`), the weighted Levinstein distance of (1) and (2) may vary. Accordingly, between (1) and (2), the chosen weighted Levinstein distance will be the smaller of the two.

Consistent with the present invention, either one of two methods is used to calculate the phonemic transformation weights. Preferably these weights are precalculated and stored in a phonemic transformation weight matrix. In the case of a forty symbol phonemic alphabet (plus a null phoneme), a 41.times.41 phonemic transformation weight matrix would result.

A first method usable with the present invention for calculating the phonemic transformation weight for a phoneme pair (/x/, /y/) is based on descriptive values assigned by an expert to each phoneme. The phonemic transformation weights are given by: ##EQU2## where N is the number of phonological feature types and .alpha..sub.i is the relative significance of the feature type i. The functions f(/x/) and f(/y/) are the feature values of feature type i for phonemes /x/ and /y/, and t(f(/x/), f(/y/)) is the transformation penalty between feature values f.sub.i (/x/) and f.sub.i (/y/). The meaning of feature values, feature types, and transformation cost will be further explained.

As mentioned previously, forty phonemes may be used to describe English language words. Each phoneme sound is described by seven feature types, each of which may take on a number of values. The feature types, along with their permissible values, are shown below in Table I.

The phoneme /a/, for example, as in the word "cat," is assigned the feature values: "low," "front," and "unrounded". These feature values correspond to the feature types "height," "frontness," and "roundness," respectively. The feature types "voicing," "place," "realization," and "manner" are not specifically defined for /a/, and accordingly, /a/ is said to have a null feature value for these feature types. Table II lists the preferred 40 phonemes for English, and their corresponding feature values.

Each feature type relates loosely to the physiological action used to make the phoneme sound. "Place" refers to the position of the primary articulator (e.g., tongue blade, tongue dorsum, lips), "manner" refers to the articulatory method used to generate the sound, "realization" refers to allophonic attributes of the production of the sound, "frontness" and "height" refer to tongue tip position in the production of the vowels, and "roundness" refers to the lip shape during production of the vowels.

The feature types are not all equally significant in the production of a sound. For example, the significance of "manner" is stronger than that of "rounding." The relative significance value, .alpha..sub.i, is used to quantify the significance of the various feature types. The values of .alpha..sub.i were assigned by a speech expert and are shown in Table III.

In addition to the relative significance value, .alpha..sub.i, transformation penalties are used to calculate the phonemic transformation weights. The transformation penalty is a function of two feature values of a feature type, and loosely measures the acoustic consequences of changing from an articulation of one feature type to another. In a manner similar to the significance values, the transformation penalties were determined by an expert. Tables IV-IX illustrate the preferred transformation penalties for the feature values except the feature values for the "realization" feature type.

The "realization" feature type is preferably a sequence of binary features. The particular binary values for the sequence are zero when a "realization" feature value is transformed into the identical feature and one when the "realization" feature value is transformed into a second feature value. In more precise terms, let x be one of the realization attributes X, where X is the set {aspirated, unexploded, syllabic, murmured, long, velarized, labialized, palatalized, rhoticized, pharyngealized, tense}. If x changes from x to not-x (e.g., from aspirated to unexploded), the transformation penalty is one, while if x does not change (e.g., the feature value of both phonemes is aspirated), the transformation penalty is zero.

Using equation (3) and tables II-IX, the phonemic transformation weights can be calculated for any phoneme symbol pair (/x/, /y/). Additionally, insertion and deletion costs, W(/x/, "") and W("", /y/), respectively, are determined by using an expert to estimate the phonemic transformation weights of each phoneme and the null value. As an example transformation weight calculation, consider the phoneme symbol pair (/b/) {voiced bilabial stop}, and /d/ {palato-alveolar}. Applying equation (3), and using the appropriate values from the tables, the feature weight for place is 0.7, and the transformation penalty from "bilabial" to "pal-alv" is 0.3, which leads to a phoneme transformation penalty of D(/b/,/d/)=0.21. Table X, below, lists additional exemplary phonemic transformation weight pairs calculated using phonological features.

Although the values assigned to the phonemes, as shown above in Tables IV-IX, are preferred, other values may also be used. Indeed, the number of phonemes used to describe a language, the feature types, and the feature values were all picked based on the best judgement of language experts, but are, of course, not the only possible representation of a language.

A second method usable with the present invention for calculating the phonemic transformation weights uses the relative entropy between an acoustic model, such as the Hidden Markov Model (HMM). Relative entropy is first found by aligning the states in the HMMs, and then considering the dominant pair of components of the Gaussian mixtures.

Using HMMs to acoustically model phonemes is known. Generally, in a HMM model, phonemes are assigned multiple states. Each state is represented by two probability distributions: the state probability distribution function and the transition probability distribution function. The state probability represents the probability that the state is occupying a certain value, and the transition probability represents the probability that the stochastic process--a derivation of the speech signal--will leave the state.

Consistent with the present invention, each phoneme in the HMM is assigned an equal number of states. The states of the phonemes are aligned in a one-to-one relationship.

The state probability is a mixture of multi-variate Gaussian probability distributions, and is given by ##EQU3## where G(.mu..sub.i, .SIGMA..sub.i) is a multi-variate Gaussian distribution with mean, .mu..sub.i, and covariance matrix, .SIGMA..sub.i, N is the number of distributions in the mixture, and .eta..sub.i is the mixture weight.

The distance between HMM state x.sub.i and state y.sub.i of phoneme /x/ and /y/, respectively, is defined to be the relative entropy between mixtures P.sub.x(i) and P.sub.y(i), denoted H(x.sub.i, y.sub.i). Methods for calculating H(x.sub.i, y.sub.i) are known. One such description is given in the conference report "State Clustering of HMMs with Continuous Mixture Densities," by P. Stubley and J. F. Crespo. IEEE Workshop on Automatic Speech Recognition, Snowbird, Colorado, 1995.

The phonemic transformation weight between phonemes x and y, in terms of H(x.sub.i, y.sub.i), is: ##EQU4## Here, n is the number of HMM states used to model the phoneme (e.g., 3), and v.sub.i is a state weight, set heuristically, with the following guiding principle: because the steady state behavior of a phoneme is usually represented by the central states in the acoustic model, these states are usually assigned a higher weight. Conversely, the initial and final states are usually assigned a smaller weight. Table XI, below, lists exemplary confusability weight pairs calculated using equation (5). In the table, three HMM states were used and the weights, v.sub.i, were set at {0.2, 0.6, 0.2}. Note that the specific value of the state weights V.sub.1 are not restricted to the example weights given in this disclosure.

Computer recognition of human speech is valuable in many applications requiring human/computer interaction. In designing such a speech recognition system, there is some flexibility in the selection of the lexicon the computer is trained to recognize. The confusability tool consistent with the present invention may be used to automatically find word pairings that tend to cause high confusability. The designer can then replace the relevant orthographies with alternate synonyms. In general, simplification of a lexicon by replacing confusable words with non-confusable synonyms can be useful in many different communication areas, such as simplification of a lexicon to facilitate understanding across a communication medium. Alternatively, the creation of a highly confusable lexicon may also be valuable, in, for example, creating a confusable test lexicon to rigorously test a speech recognizer.

FIG. 4 is a flow chart illustrating the use of confusability tool 202 as a vocabulary diagnostic aide. The proposed lexicon of the speech recognizer is phonemically transcribed by a human expert or by an automated transcription tool (step 402) For each pair of orthographies, the weighted Levinstein distance is computed pursuant to equation (1) (step 404). Orthographies may have more than one transcription. For these, the minimum normalized weighted Levinstein distance is computed, pursuant to equation (2), to obtain D(O.sub.1, O.sub.2) (step 406). The vocabulary diagnostic tool then apples a predefined threshold value to the normalized minimum values of each orthography pair, D(O.sub.1, O.sub.2), of each triplet {O.sub.1, O.sub.2, D(O.sub.1, O.sub.2)}(step 408). Finally, the remaining triplets are sorted in ascending order by weight and presented to the designer (step 410).

Accordingly, the designer is automatically presented with a list of the orthographies in the proposed lexicon, sorted so that those orthographies most likely to be confused are presented first. Based on the list, the designer may modify the orthographies in the lexicon, and, if he desires, re-runs the vocabulary diagnostic tool (step 412).

A confusability tool for objectively and automatically generating confusability ratings between pairs of orthographies or phonemic transcriptions has been described. The confusability tool has many potential applications, including use in a vocabulary diagnostic tool and in a speech recognition algorithm. Other uses are possible. For example, the confusability tool can be used to help speakers acquire a second language by indicating word pairings that the speaker should spend extra time learning to pronounce because of their potential confusion, such as the words "tin" and "teen" for speakers whose first language does not differentiate between the sounds /i/ and /l/. As a second example, knowledge of potential confusable words could be used in the speech pathology field to help speakers differentiate or pronounce difficult word pairings.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the scope or spirit of the invention. For example, many-to-many phoneme transformations can be used instead of one-to-one transformations (e.g., T.sub.1 and T.sub.2 in D(T.sub.1, T.sub.2) may include pairs such as /xy/ and /ab/, where /x/, /y/, /a/, and /b/ are all individual phonemes. Other possible modification include: (1) combining the weight and feature transformations using a non-linear combination; (2) the transformation weights may take into account bundles of features rather than a single feature at a time (e.g., mapping from {voiced stop} to {voiceless fricative} pairs rather than cumulating for {voiced} to {voiceless} and {stop} to {fricative}; (3) the table of 41.times.41 phoneme transformation penalties can be assigned heuristically by an expert; (4) and the table of 41.times.41 phoneme penalties can be assigned using results from a speech recognition experiment. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.