US 7,010,486 B2Grant

Speech recognition system, training arrangement and method of calculating iteration values for free parameters of a maximum-entropy speech model

Issue Date:2006-03-07

•15 Claims

•6 Drawing Sheets

Abstract

The invention relates to a speech recognition system and a method of calculating iteration values for free parameters λ_α^ortho(n)of a maximum-entropy speech model MESM with the aid of the generalized-iterative scaling training algorithm in a computer-supported speech recognition system in accordance with the formula λ_α^ortho(n+1)=G(λ_α^ortho(n), m_α^ortho, . . . ), where n is an iteration parameter, G a mathematical function, α an attribute in the MESM and m_α^orthoa desired orthogonalized boundary value in the MESM for the attribute α. It is an object of the invention to further develop the system and method so that they make a fast computation of the free parameters λ possible without a change of the original training object. According to the invention this object is achieved in that the desired orthogonalized boundary value m_α^orthois calculated by a linear combination of the desired boundary value m_α with desired boundary values m_β from attributes β that have a larger range than the attribute α. m_α and m_β are then desired boundary values of the original training object.

Metadata

Assignee

Koninklijke Philips Electronics, N.V.

Inventor

Jochen Peters

Application Information

Application Number:US 10/075,865

Filing Date:2002-02-13

Priority Date:2001-02-13

Art Unit:2655

Classifications

IPC:

G10L15/28G10L15/08G10L15/12G10L15/00

Patent Drawings (6 sheets)

Description

[0001] The invention relates to a method of calculating iteration values for free parameters λ_α^ortho(n)of a maximum-entropy speech model MESM in a speech recognition system with the aid of the generalized iterative scaling training algorithm in accordance with the following formula:
λ_α^ortho(n+1)=G(λ_α^ortho(n), m_α^ortho, . . . ) (1)
where:

[0002] n: is an iteration parameter;
[0003] G: is a mathematical function;
[0004] α: is an attribute in the MESM; and
[0005] m_α^ortho: is a desired orthogonalized boundary value in the MESM for the attribute α.

[0006] The invention further relates to a computer-supported speech recognition system known in the state of the art, as well as a known computer-supported training arrangement in which the method described is implemented.

[0007] A starting point for the formation of a speech model as it is related in a computer-supported speech recognition system for recognizing entered speech is a predefined training object. The training object maps certain statistical patterns in the language of a future user of the speech recognition system into a system of mathematically formulated boundary conditions, which system generally has the following form:

\begin{matrix} \sum_{(h, w)} N (h) \cdot p (w ❘ h) \cdot f_{α} (h, w) = m_{α} & (2) \end{matrix}

where:

[0008] N(h): refers to the frequency of history h in a training corpus;
[0009] p_(w|h): refers to probability p(w|h) with which a predefined word w follows a previous word sequence h (history);
[0010] ƒ_α(h,w): refers to a binary attribute function for an attribute α; and α
[0011] m_α: refers to a desired boundary value in the system of boundary conditions.

[0012] The solution of this system of boundary conditions i.e. the training object is formed by the so-termed maximum-entropy speech model MESM which indicates a suitable solution of the system of boundary conditions in the form of a suitable definition of the probability p(w|h), which reads as follows:

\begin{matrix} p (w ❘ h) = p λ (w ❘ h) = \frac{1}{Z_{λ} (h)} \cdot \exp (\sum_{α} λ_{α} \cdot f_{α} (h, w)) & (3) \end{matrix}

where:

[0013] Zλ(h): refers to a history-dependent standardization factor;
[0014] λα: refers to a free parameter for the attribute α;
[0015] λ: refers to the set of all parameters. For the above parameters hold their above definitions.

[0016] The binary attribute function fα(h,w) makes, for example, a binary decision whether predefined word sequences h,w contain predefined words at certain locations. An attribute α may generally refer to a single word, a word sequence, a word class (color or verbs), a sequence of word classes or more complex patterns.

[0017] FIG. 4 shows predefined attributes in a speech model by way of example. For example, the unigrams shown each represent a single word, the bigrams each represent a word sequence consisting of two words and the trigram shown represents a word sequence consisting of three words. The bigram “ORA” includes the unigram “A” and, in addition, includes a further word; therefore it is referred to as having a larger range compared to the unigram “A”. Analogously, the trigram “A WHITE HOUSE” has a larger range than the unigram “HOUSE” or than the bigram “WHITE HOUSE”.

[0018] The free parameters λ are adapted so that equation 3 represents a solution for the system of boundary conditions according to equation 2. This adaptation is normally made with the aid of known training algorithms. An example for such a training algorithm is the so-termed generalized iterative scaling GIS algorithm as it is described, for example, in J. N. Darroch and D. Ratcliff, “Generalized iterative scaling for log linear models”, Annals Math. Stat., 43(5):1470–1480, 1972.

[0019] This GIS algorithm provides an iterative calculation of the free parameters λ. Traditionally, this calculation is made very slowly, however. For expediting this calculation, there is proposed in the state of the art to substitute orthogonalized attribute functions ƒ_α^ortho(h,w) for the attribute functions ƒ_α(h,w) in the system of boundary conditions in accordance with equation (2); see for this purpose R. Rosenfeld “A maximum-entropy approach to adaptive statistical language modeling”; Computer Speech and Language, 10:187–228, 1996. Because of the substitution of the attribute functions on the left in equation 2, however, also the boundary values m_α on the right are changed. This changes the original system of boundary conditions i.e. the original training object in the customary sets approaches for estimating the boundary values; for this purpose see Rosenfeld at other locations, page 205, first sentence of the last-but-one paragraph.

[0020] In this respect it can be established as a disadvantage of the state of the art that when the calculation of the GIS algorithm is accelerated, the free parameters λ are trained to a changed training object. The parameters λ calculated in this manner are the cause for an inadequate adaptation of the speech model to the original training object when the parameter λ is used in equation 3.

[0021] Starting from this state of the art it is an object of the invention to further develop a known computer-supported speech recognition system, a computer-supported training system and a known method of iteratively calculating free parameters λ_α^ortho(n)of a maximum-entropy speech model in the speech recognition system, so that they make a fast calculation possible of the free parameters λ without a change of the original training object.

[0022] This object is achieved in that with the known above-described method of calculating the free parameters λ according to the GIS algorithm, any desired orthogonalized boundary value m_α^orthois calculated by linearly combining the associated desired boundary value m_α with desired boundary values m_β of attributes β that have a larger range than the attribute α. Here m_α and m_β are desired boundary values of the original training object.

[0023] The use of the boundary values m_α^orthocalculated in this manner makes it possible in an advantageous manner to make an improved approximation of the free parameters λ and thus an improvement of the speech model with a view to the original training object. This qualitative improvement is possible while a high convergence speed continues to realize for the free parameters λ during the iterative calculation with the aid of the GIS algorithm.

[0024] The use of the desired orthogonalized boundary values m_α^orthocalculated according to the invention is recommended for several variants of the GIS training algorithm.

[0025] The object of the invention is furthermore achieved by a speech recognition system based on the maximum-entropy speech model MESM and a training system for training the MESM.

[0026] By implementing the method according to the invention in the training system, compared to the state of the art the MESM in the speech recognition system is adapted more effectively to the individual language peculiarities of a certain user of the speech recognition system; the quote with which the speech recognition system then correctly recognizes the semantic content in the user's speech is improved considerably.

[0027] Otherwise the advantages of this speech recognition system and of the training system correspond to the advantages discussed above for the method.

[0028] The following Figures are added to the description of the invention, in which

[0029] FIGS. 1a and 1b describe a method according to the invention of calculating a desired orthogonalized boundary value m_α^ortho;

[0030] FIGS. 2a and 2b describe a method according to the invention of calculating an orthogonalized attribute function ƒ_α^ortho;

[0031] FIG. 3 describes a block diagram of a speech recognition system according to the invention;

[0032] FIG. 4 describes an attribute tree.

[0033] In the following first a detailed description is given of an example of embodiment of the invention while reference is made to FIGS. 1a and 1b.

[0034] FIGS. 1 and 1b illustrate a method according to the invention of calculating an improved desired orthogonalized boundary value m_α^orthofor an attribute α=β₀in a speech model. In a first step of the method all the attributes β_iwith i=1 . . . g that have a so-termed larger range than the predefined attribute α=β₀i.e. which include this at a predefined position are determined in accordance with this method. Subsequently, in a method step S2 a desired boundary value mβ_iof the original training object is calculated for all the attributes β_iwith i=0 . . . g, thus also for the attribute α=β₀.

[0035] For the calculation of such a desired boundary value mβ_i, several methods are known in the state of the art.

[0036] According to a first method the calculation is made in that first a frequency N(β_i) is determined with which the associated binary attribute function fβ_iyields the value 1 when a training corpus of the speech model is used and that, subsequently, the thus determined frequency value N(β_i) is smoothed.

[0037] According to a second, alternative method, the calculation is performed by reducing the quantities of attributes in the speech model until the boundary conditions no longer demonstrate conflicts. This sort of reduction in the quantity of attributes must be very extensive in practical situations, since otherwise the generated speech model will no longer represent a solution to the original training object. According to a third method, the calculation is made by using a so-called induced speech model as it is described in J. Peters and D. Klakow, “Compact Maximum Entropy Language Models”, Proc. ASRU, Keystone, Colo., 1999.

[0038] In a method step S3 all the attributes β_iiare subsequently sorted according to their range where an attribute β_ithat has the largest range is assigned the index i=g. It may then certainly happen that individual classes of ranges thus, for example, the class of bigrams or the class of trigrams are assigned a plurality of attributes β_i. In these cases a plurality of attributes β_ihaving different, but successive indices i are assigned to one and the same class of ranges i.e. these attributes then always have the same RW and belong to the same class of ranges.

[0039] For the method to be carried out, in which in the successive steps the individual attributes β_iare evaluated one after the other, it is important for the attributes to be processed according to decreasing (or constant) range. In the first run of the method a start is therefore made with an attribute βi which is assigned to the highest class of ranges; preferably i is set equal to g (see method step S4 and S5 in FIG. 1a).

[0040] In a subsequent method step S6 a check is then made whether larger-range attributes β_koccur with i<k≦g for the currently selected attribute β_i, which include the attribute β_i. With the first run the attribute β_iwith i=g automatically belongs to the class that has the largest range, as observed above, and therefore the query in the method step S6 is to be answered in the negative for this attribute β_i. In this case the method jumps to method step S8 where a parameter X is set to zero. Then a calculation is made of an improved desired orthogonalized boundary value m_βi^orthofor the attribute β_i(with a first run with i=g) in accordance with method step S9. As can be seen there, this boundary value for the attribute β_iis set equal to the desired boundary value mβ_icalculated in step S2, if the parameter X=0 (this is the case, for example, during the first run).

[0041] The method steps S5 to S11 are then successively repeated for all the attributes β_i−1 with i−1=g−1 . . . 0. In the method step S10 the index i is re-initialized, which is necessary, and in method step S11 a query is made whether all the attributes β_iwith i=0 . . . g have been processed.

[0042] For all attributes β_ifor which there are attributes β_kwith i<k≦g that have a larger range, the query in method step S6 must be answered with “Yes”. The parameter X is then not set to zero but is instead calculated according to method step S7 by totaling the corresponding improved desired orthogonalized boundary values m_βk^orthoeach calculated in previous run-throughs in method step S9 for the respective attributes β_kthat have a larger range.

[0043] Once it has been determined in method step S11 that the desired orthogonalized boundary value m_β0^orthohas been calculated in method step S9, this is then output in method step S12 as m_α^ortho. The method according to the invention extensively described just now for the calculation of the improved desired orthogonalized boundary value m_α^orthomay be shortened to the following formula:

\begin{matrix} m_{α}^{ortho} = m α - \sum_{(*)} m_{β}^{ortho} . & (4) \end{matrix}

[0044] The sum (*) includes all attributes β that have a larger range and contain the predefined attribute α. For calculating the boundary value m_β^orthosaid formula can be used in an almost recursive manner for each attribute β again and again until the sum term disappears for certain attributes, that is, for those with the largest range, because there are no more attributes that have a larger range for them. The desired orthogonalized boundary values for the attributes βk that have the largest range then correspond to the respective originally desired boundary values mβk.

[0045] The implementation of the method according to the invention and as shown in FIGS. 1a and 1b will be further explained hereinafter while use is made of the following training corpus of a speech model used by way of example. The training corpus reads:

[0046]


“THAT	WAS	A	RED
	OR	A	GREEN HOUSE
	OR	A	BLUE HOUSE
THIS	IS	A	WHITE HOUSE AND
THAT	IS	THE	WHITE HOUSE”

[0047] The training corpus consists of N=23 individual words. It is assumed that in the speech model the desired unigram, bigram and trigram attributes are predefined according to FIG. 4.

[0048] Then, by using the normal attribute function fα for the training corpus it may be established that the unigrams, bigrams and trigrams according to FIG. 4 occur in the training corpus with the following frequencies:

[0049]


Unigrams:
A	4
HOUSE	4
IS	2
OR	2
THAT	2
WHITE	2
Bigrams:
A	WHITE	1
OR	A	2
WHITE	HOUSE	2
Trigrams:
A	WHITE	HOUSE	1

[0050] In the example shown here the improved desired orthogonalized boundary value m_α^orthois to be calculated for the attribute α=“HOUSE”. For this purpose first according to method step S1 in FIG. 1a all attributes that have a larger range are to be determined for the attribute α. They are according to FIG. 4 the bigram “WHITE HOUSE” and the trigram “A WHITE HOUSE”. According to method step S2 the normal desired boundary values are to be calculated for these attributes that have a larger range but also for the attribute α, for example, in that the respective frequencies established above are smoothed. This smoothing is effected here, for example, by subtracting the value 0.1. Thus the following normal desired boundary values are the result:

[0051] m_α: “HOUSE”=4−0.1=3.9
[0052] m_β1: “WHITE HOUSE”=2−0.1=1.9
[0053] m_β2: “A WHITE HOUSE”=1−0.1=0.9.

[0054] The attributes α, β₁, β₂are now sorted according to their range and—starting with the widest ranging attribute—the respective improved desired orthogonalized boundary values are calculated according to formula (6) or according to method step S7–S9 in FIGS. 1a and 1b:
m_β2^ortho=m_β2=0.9 (5)
m_β1^ortho=m_β1−m_β2^ortho=1.9−0.9=1 (6)

[0055] Finally, the improved desired orthogonalized boundary value m_α^orthois calculated for the attribute α to:
m_α^ortho=m_α−m_β1^ortho−m_β2^ortho=3.9−1−0.9=2 (7)

[0056] The orthogonalized boundary value m_α^orthocalculated according to the invention makes a sufficiently accurate calculation possible of the free parameters λ and thus of the probability according to formula (1) with a view to an original training object while the calculation velocity remains the same when used in the GIS training algorithm.

[0057] Hereinafter the use of the boundary value m_α^orthocalculated according to the invention will be represented for three different variants of the GIS training algorithm.

[0058] With a first variant of the GIS training algorithm the mathematical function G has the following form according to equation 1 when the orthogonalized boundary value m_α^orthocalculated according to the invention is used:

\begin{matrix} \begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho},, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \\ \log (\frac{[t_{α}^{ortho} \cdot m_{α}^{ortho} + b_{α}]}{[t_{α}^{ortho} \cdot m_{α}^{ortho (n)} + b_{α}]} \cdot \frac{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho (n)} + b_{γ}]}{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho} + b_{γ}]}) \end{matrix} & (8) \end{matrix}

where:

[0059] n: refers to an iteration parameter;
[0060] α: refers to a just considered attribute;
[0061] γ: refers to all the attributes in the speech model;
[0062] t_α^ortho, t_γ^ortho: refer to the size of the convergence step;
[0063] m_α^ortho, m_γ^ortho: desired orthogonalized boundary values in the MESM for the attributes α and γ;
[0064] m_α^ortho(n), m_γ^ortho(n): refers to iterative approximate values for the desired boundary values m_α^ortho, m_γ^ortho; and
[0065] bα and bγ: refer to constants.

[0066] The calculation of the convergence step sizes t and of the iterative approximate values for the desired boundary values m is effected—as will be shown hereinafter—by the use of an orthogonalized attribute function ƒ_α^orthodefined according to the invention, which reads as follows:

\begin{matrix} f_{α}^{ortho} = f α - \sum_{(*)} f_{β}^{ortho} & (9) \end{matrix}

[0067] It should be observed at this point that the orthogonalized attribute function ƒ_α^orthocalculated according to the invention in accordance with equation 9 corresponds as regards value to the attribute function proposed by Rosenfeld at other locations. However, their calculation according to the invention is effected totally different as can be seen in FIGS. 2a and 2b. The calculation method is effected analogously to the method described in FIGS. 1a and 1b for the calculation of the desired orthogonalized boundary values m_α^orthowhere only the symbol for the boundary value m is to be replaced by the symbol for the attribute function f and the parameter X by the function F. To avoid repetitions, reference is made here to the description of FIGS. 1a and 1b for explanations of the method according to FIGS. 2a and 2b.

[0068] With the orthogonalized attribute function ƒ_α^orthoor ƒ_β^orthothus calculated according to the invention, the size of the convergence steps t_α^orthoand t_γ^orthois calculated in equation 8 as follows:

\begin{matrix} t_{α}^{ortho} = t_{γ}^{ortho} = 1 / M^{ortho} with M^{ortho} = \max_{(h, w)} (\sum_{β} f_{β}^{ortho} (h, w)) & (10) \end{matrix}

where Mortho for binary attribute functions ƒ_β^orthorepresents the maximum number of functions which yield the value 1 for the same argument (h,w).

[0069] Furthermore, with the attribute function ƒ_α^orthodefined according to the invention, the iterative approximate value m_α^ortho(n)can be calculated for the desired orthogonalized boundary value m_α^orthowhen the following equation (2) is used:

\begin{matrix} m_{α}^{ortho (n)} = \sum_{(h, w)} N (h) \cdot p^{(n)} (w ❘ h) \cdot f_{α}^{ortho} (h, w); & (11) \end{matrix}

where:

[0070] N(h): refers to the frequency of the history h in the training corpus; and
[0071] p⁽ⁿ⁾(w|h): refers to an iteration value for the probability p(w|h) with which a predefined word w follows a previous word sequence h (history);
[0072] Here p⁽ⁿ⁾(w|h) uses the parameter values λ_α^ortho(n).

[0073] The use of the improved desired orthogonalized boundary value m_α^orthocalculated according to the invention is furthermore recommended for a second variant of the GIS training algorithm. Here the attributes of the MESM are subdivided into m groups Ai and for each iteration only the parameters λ_α^orthoof the attributes α from one of the groups are changed according to the following formula:

\begin{matrix} \begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho}, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \log (\frac{m_{α}^{ortho}}{m_{α}^{ortho (n)}} \cdot \frac{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho (n)})}{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho})}) \end{matrix} & (12) \end{matrix}

where:

[0074] n: represents the iteration parameter
[0075] A_i(n): represents an attribute group A_i(n)with 1≦i≦m selected in the n^thiteration step;
[0076] α: represents a just considered attribute from the just selected attribute group Ai(n);
[0077] β: represents all attributes from the attribute group Ai(n);
[0078] t_α^ortho, t_β^ortho: represent the size of the convergence step with t_α^ortho=t_β^ortho=1/M_i(n)^orthowith $M_{i (n)}^{ortho} = \max_{(h, w)} (\sum_{β \in Ai (n)} f_{β}^{ortho} (h, w))$
where
[0079] M_i(n)^orthofor binary functions ƒ_β^orthorepresents the maximum number of functions from the attribute group Ai(n) which yield the value 1 for the same argument (h,w);
[0080] m_α^ortho, m_β^ortho: represent the desired orthogonalized boundary values in the MESM for the attributes α and β respectively;
[0081] m_α^ortho(n), m_β^ortho(n): represents iterative approximate values for the desired boundary values m_α^ortho, m_β^ortho.

[0082] The group Ai(n) of attributes α whose parameters λ_α^orthoare adapted in the current iteration step, then cyclically runs through all the m groups in accordance with i(n)=n(mod m).

[0083] The use of the desired orthogonalized boundary value m_α^orthocalculated according to the invention is further recommended for a third variant of the GIS training algorithm which distinguishes itself from the second variant only in that the attribute group Ai(n) to be used for each iteration step is not selected cyclically but according to a predefined criterion D_i⁽ⁿ⁾.

[0084] FIG. 3 finally shows a speech recognition system 10 of the type according to this invention which is based on the so-termed maximum-entropy speech model. It includes a recognition device 12 which attempts to recognize the semantic content of supplied speech signals. The speech signals are generally supplied to the speech recognition system in the form of output signals from a microphone 20. The recognition device 12 recognizes the semantic content of the speech signals by mapping patterns in the received acoustic signal on two predefined recognition symbols such as specific words, actions or events, using the implemented maximum-entropy speech model MESM. Finally, the recognition device 12 outputs a signal which represents the semantic content recognized in the speech signal and can be used to control all kinds of equipment—for example a word-processing program or telephone.

[0085] To make the control of the equipment as error-free as possible in terms of the semantic content of speech information used as a control medium, the speech recognition system 10 must recognize the semantic content of the speech to be evaluated as correctly as possible. To do this, the speech model must be adapted as effectively as possible to the linguistic peculiarities of the speaker, i.e. the user of the speech recognition system. This adaptation is performed by a training system 14 which can be operated either externally or integrated into the speech recognition system 10. To be more accurate, the training system 14 is used to adapt the MESM in the speech recognition system 10 to recurrent statistical patterns in the speech of a particular user.

[0086] Both the recognition device 12 and the training system 14 are normally, although not necessarily, in the form of software modules and run on a suitable computer (not shown).

Claims

The invention claimed is:

1. A method of calculating iteration values for free parameters λ_α^ortho(n)of a maximum-entropy speech model MESM in a speech recognition system with the aid of the generalized iterative scaling training algorithm, the method comprising the step of

iteratively determining:

λ_α^ortho(n+1)=G(λ_α^ortho(n), m_α^ortho, . . . )

where:

n: is an iteration parameter;

G: is a mathematical function;

α: is an attribute in the MESM; and

m_α^ortho: is a desired orthogonalized boundary value in the MESM for the attribute α,

characterized in that the desired orthogonalized boundary value m_α^orthois calculated by linearly combining the desired boundary value m_α with desired boundary values m_β of attributes β that have a larger range than the attribute α.

2. A method as claimed in claim 1, characterized in that the calculation of the desired orthogonalized boundary value m_α^orthofor the attribute α=β₀comprises the following steps:

a) Selecting all the attributes β_iwith i=1 . . . g in the speech model that have a larger range RW than the attribute α=β₀and include the latter;

b) Calculating desired boundary values mβi for the attributes β_iwith i=0 . . . g;

c) Sorting the attributes β_iwith i=0 . . . g according to their RW;

d) Selecting one of the attributes β_ihaving the largest RW;

e) Checking whether there are other attributes β_kwhich include the attribute β_iand have a larger RW than the selected attribute β_i;

f1) If so, defining a parameter X as a linear combination of the orthogonalized boundary values m_βk^orthocalculated in step g) during the last run of the steps e) to g) for all the attributes β_ithat have a larger range and are determined in the most recently run step e);

f2) If not, defining the parameters X to X=0;

g) Calculating the desired orthogonalized boundary value m_βk^orthofor the attribute β_iby arithmetically combining the desired boundary value mβ_iwith a parameter X; and

h) Repeating the steps e) to g) for the attribute β_i−1 whose RW is smaller than or equal to the RW of the attribute β_iuntil the desired orthogonalized boundary value m_β0^ortho=m_α^orthowith i=0 has been calculated in step g).

3. A method as claimed in claim 2, characterized in that the calculation of the parameter X in step f1) is made according to the following formula:

X = \sum_{k} m_{β k}^{ortho} .

4. A method as claimed in claim 3, characterized in that the calculation of the desired orthogonalized boundary value m_βi^orthois made in step g) according to the following formula:

m_βi^ortho=m_βi−X.

5. A method as claimed in claim 2, characterized in that the calculation of the desired boundary values m_βifor the attributes β_iwith i=0, . . . , g is made in step b) by respectively calculating the frequency N(β_i), with which the attribute β_ioccurs in a training corpus and by subsequently smoothing the calculated frequency value N(β_i).

6. A method as claimed in claim 5, characterized in that the calculation of the frequency N(β_i) is made by applying a binary attribute function fβ_ito the training corpus where fβ_iis defined as:

f_{β i} (h, w) f_{β i} (h, w) = {\begin{matrix} 1 if β_{i} fits in the word sequence (h, w) \\ otherwise 0 \end{matrix}

and where ƒ_βi(h,w) indicates whether the attribute β_icorrectly describes a pattern predefined by the word sequence (h,w).

7. A method as claimed in claim 1, characterized in that the mathematical function G has as a further variable the magnitude of a convergence step t_α^orthowith:

t_α^ortho=1/M^ortho

where

M^ortho: represents for binary functions ƒ_α^orthothe maximum number of functions which yield the value 1 for the same argument (h,w).

8. A method as claimed in claim 7, characterized in that the attribute function ƒ_α^orthois calculated by linearly combining an attribute function ƒ_α with orthogonalized attribute functions ƒ_β^orthois calculated from attributes β that have a larger range than the attribute α.

9. A method as claimed in claim 8, characterized in that the calculation of the orthogonalized attribute function ƒ_α^orthofor the attribute α=β₀comprises the following steps:

a) Selecting all the attributes β_iwith i=1 . . . g in the speech model that have a larger range RW than the attribute α=β₀and include the latter;

b) Calculating boundary values fβ_ifor the attributes β_iwith i=0 . . . g;

c) Sorting the attributes β_iwith i=0 . . . g according to their RW;

d) Selecting one of the attributes β_ihaving the largest RW;

e) Checking whether there are other attributes βk which include the attribute β_iand have a larger RW than the selected attribute β_i;

f1) If so, defining a function F as a linear combination of the orthogonalized attribute function ƒ_βk^orthocalculated in step g) during the last run of the steps e) to g) for all the attributes β_kthat have a larger range determined in the most recently run step e);

f2) If not, defining the function F to F=0;

g) Calculating the orthogonalized attribute function ƒ_βk^orthofor the attribute βi by arithmetically combining the attribute function fβ_iwith the function F; and

h) Repeating the steps e) to g) for the attribute β_i−1 whose range is smaller than or equal to the range of the attribute β_iuntil the orthogonalized attribute function ƒ_β0^ortho=ƒ_α^orthowith i=0 has been calculated in step g).

10. A method as claimed in claim 9, characterized in that the calculation of the function F in step f1) is made according to the following formula:

F = \sum_{k} f_{β k}^{ortho} .

11. A method as claimed in claim 9, characterized in that the calculation of the orthogonalized attribute function ƒ_βi^orthoin step g) is made according to the following formula:

ƒ_βi^ortho=ƒ_βi−F.

12. A method as claimed in claim 1, characterized in that the mathematical function G has the following form:

\begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho}, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \log (\frac{[t_{α}^{ortho} \cdot m_{α}^{ortho} + b_{α}]}{[t_{α}^{ortho} \cdot m_{α}^{ortho (n)} + b_{α}]} \cdot \frac{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho (n)} + b_{γ}]}{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho} + b_{γ}]}) \end{matrix}

where:

α: refers to a just considered attribute;

γ: refers to all the attributes in the speech model;

t_α^ortho, t_γ^ortho: refer to the size of the convergence step with t_α^ortho=t_γ^ortho=1/M^orthowith

M^{ortho} = \max_{(h, w)} (\sum_{β} f_{β}^{ortho} (h, w));

where M^orthofor binary functions ƒ_β^orthorepresents the maximum number of functions which yield the value 1 for the same argument (h,w);

m_α^ortho, m_γ^ortho: refers to desired orthogonalized boundary values in the MESM for the attributes α and γ;

m_α^ortho(n), m_γ^ortho(n): refers to iterative approximate values for the desired boundary values m_α^ortho, m_γ^ortho(n); and

bα and bγ: refer to constants.

13. A method as claimed in claim 1, characterized in that the mathematical function has the following form:

\begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho},, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \log (\frac{m_{α}^{ortho}}{m_{α}^{ortho (n)}} \cdot \frac{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho (n)})}{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho})}) \end{matrix}

where:

n: represents the iteration parameter;

A_i(n): represents an attribute group A_i(n)with 1≦i≦m selected in the n^thiteration step;

α: represents a just considered attribute from the just selected attribute group A_i(n);

β: represents all the attributes of the attribute group Ai(n);

t_α^ortho, t_β^ortho: represents the size of a convergence step with t_α^ortho=t_β^ortho=1/M_i(n)^orthowith

M_{i (n)}^{ortho} = \max_{(h, w)} (\sum_{β \in Ai (n)} f_{β}^{ortho} (h, w))

where M_i(n)^orthorepresents for binary functions ƒ_β^orthothe maximum number of functions from the attribute group A_i(n), which yield the value 1 for the same argument (h,w);

m_α^ortho, m_β^ortho: represent desired orthogonalized boundary values in the MESM for the attributes α and β respectively;

m_α^ortho(n), m_β^ortho(n): represent iterative approximate values for the desired boundary values m_α^ortho, m_β^ortho;

where the selection of the group=A_i(n)of attributes α, whose associated parameters λ_α^orthoare adapted to a current iteration step is made either cyclically or according to a predefined criterion.

14. A speech recognition system comprising: a recognition device for recognizing the semantic content of an acoustic signal captured and rendered available by a microphone, more particularly a speech signal, by mapping parts of this signal onto predefined recognition symbols as they are offered by the implemented maximum-entropy speech model MESM, and for generating output signals which represent the recognized semantic content; and a training system for adapting the MESM to recurrent statistical patterns in the speech of a certain user of the speech recognition system; characterized in that the training system calculates free parameters λ in the MESM in accordance with the method as claimed in claim 1.

15. A training system for adapting the maximum-entropy speech model MESM in a speech recognition system to recurrent statistical patterns in the speech of a certain user of this speech recognition system, characterized in that the training system calculates free parameters λ in the MESM in accordance with the method as claimed in claim 1.

Patent Citations (1)

Patent	Date	Inventor	Cited By
US6304841(B1)	2001-10-01	Berger et al.	Examiner

Non-Patent Literature (17)

Adam Berger, The Improved Iterative Scalling Algorithm: A Gentle Introduction, Dec., 1997, http://citeseer.nj.nec.com/31826.html.Examiner
Chen, S., Rosenfeld, Ronald. “Efficient Sampling and Feature Selection in Whole Sentence Maximum Entropy Language Models”, Acoustics, Speech and Signal Processing, ICASSP-99, IEEE Conference on, vol. 1, Mar. 15-19, 1999, pp. 549-552.Examiner
Beyerlein, Peter. “Discrimitive Model Combination”, Automatic Speech Recognition and Understanding, IEEE Workshop on, Dec. 14-17, 1997, pp. 238-245.Examiner
Simons, M., Ney, H., Martin, S.C. “Distant Bigram Language Modeling Using Maximum Entropy”, Acoustics, Speech and Signal Processing, ICASSP-97, IEEE Conference on, vol. 2, Apr. 21-24, pp. 787-790.Examiner
Peters, Jochen. Dietrich, Klakow. “Compact Maximum Entropy Language Models”, Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, Dec. 1999.□□.Examiner
Kneser, Reinhard. Peters, Jochen. “Semantic Clustering for Adaptive Language Modeling”, Acoustics, Speech and Signal Processing, vol. 2, pp. 779-782, 1997.□□.Examiner
Berger, Adam L. Della Pietra, Vincent J. Della Pietra, Stephan A. “A Maximum Entropy Approach to Natural Language Processing” Computational Linguistics vol. 22, Mar. 1996.□□.Examiner
Mikheev, Andrei. “Feature Lattices for Maximum Entropy Modelling” International Conference on Computational Linguistics vol. 2, pp. 848-854 1998.□□.Examiner
Greiff, Warren R. Ponte, Jay M. “The Maximum Entropy Approach and Probabilistic IR Models” ACM Transactions on Information Systems, vol. 18, pp. 246-287, Jul. 2000.Examiner
By J.N. Darroch & D. Ratcliff, Entitled: “Generalized Iterative Scaling for Log Linear Models” Annals Math. Stat., 43(5): 1972, 1470-1480.Applicant
By R. Rosenfeld Entitled: “A Maximum-Entropy Approach to Adaptive Statistical Language Modeling”; Computer Speech and Language, 10: pp. 187-228, 1996.Applicant
R.Rosenfeld, “A Maximum Entropy Approach to Adaptive Statistical Language Modelling;” Computer Speech and Language, 1996, Academic Press Limited; 4 pgs.Applicant
J.N.Darroch et al., “Generalized Iterative Scaling For Log-Linear Models;” The Annals of Mathematical Statistics, vol. 43, No. 5; 1972; 10 pgs.Applicant
By J.N. Darroch & D. Ratcliff, Entitled: “Generalized Iterative Scaling for Log Linear Models” Annals Math. Stat., 43(5): 1972, 1470-1480, no month or day.Applicant
By R. Rosenfeld Entitled: “A Maximum-Entropy Approach to Adaptive Statistical Language Modeling”; Computer Speech and Language, 10: pp. 187-228, 1996, no month or day.Applicant
R.Rosenfeld, “A Maximum Entropy Approach to Adaptive Statistical Language Modelling;” Computer Speech and Language, 1996, Academic Press Limited; 4 pgs, no month or day.Applicant
J.N.Darroch et al., “Generalized Iterative Scaling For Log-Linear Models;” The Annals of Mathematical Statistics, vol. 43, No. 5; 1972; 10 pgs, no month or day.Applicant

Abstract

Description

[0002] n: is an iteration parameter;
[0003] G: is a mathematical function;
[0004] α: is an attribute in the MESM; and
[0005] m_α^ortho: is a desired orthogonalized boundary value in the MESM for the attribute α.

\begin{matrix} \sum_{(h, w)} N (h) \cdot p (w ❘ h) \cdot f_{α} (h, w) = m_{α} & (2) \end{matrix}

where:

[0008] N(h): refers to the frequency of history h in a training corpus;
[0009] p_(w|h): refers to probability p(w|h) with which a predefined word w follows a previous word sequence h (history);
[0010] ƒ_α(h,w): refers to a binary attribute function for an attribute α; and α
[0011] m_α: refers to a desired boundary value in the system of boundary conditions.

\begin{matrix} p (w ❘ h) = p λ (w ❘ h) = \frac{1}{Z_{λ} (h)} \cdot \exp (\sum_{α} λ_{α} \cdot f_{α} (h, w)) & (3) \end{matrix}

where:

[0013] Zλ(h): refers to a history-dependent standardization factor;
[0014] λα: refers to a free parameter for the attribute α;
[0015] λ: refers to the set of all parameters. For the above parameters hold their above definitions.

[0024] The use of the desired orthogonalized boundary values m_α^orthocalculated according to the invention is recommended for several variants of the GIS training algorithm.

[0025] The object of the invention is furthermore achieved by a speech recognition system based on the maximum-entropy speech model MESM and a training system for training the MESM.

[0027] Otherwise the advantages of this speech recognition system and of the training system correspond to the advantages discussed above for the method.

[0028] The following Figures are added to the description of the invention, in which

[0029] FIGS. 1a and 1b describe a method according to the invention of calculating a desired orthogonalized boundary value m_α^ortho;

[0030] FIGS. 2a and 2b describe a method according to the invention of calculating an orthogonalized attribute function ƒ_α^ortho;

[0031] FIG. 3 describes a block diagram of a speech recognition system according to the invention;

[0032] FIG. 4 describes an attribute tree.

[0033] In the following first a detailed description is given of an example of embodiment of the invention while reference is made to FIGS. 1a and 1b.

[0035] For the calculation of such a desired boundary value mβ_i, several methods are known in the state of the art.

\begin{matrix} m_{α}^{ortho} = m α - \sum_{(*)} m_{β}^{ortho} . & (4) \end{matrix}

[0046]


“THAT	WAS	A	RED
	OR	A	GREEN HOUSE
	OR	A	BLUE HOUSE
THIS	IS	A	WHITE HOUSE AND
THAT	IS	THE	WHITE HOUSE”

[0047] The training corpus consists of N=23 individual words. It is assumed that in the speech model the desired unigram, bigram and trigram attributes are predefined according to FIG. 4.

[0049]


Unigrams:
A	4
HOUSE	4
IS	2
OR	2
THAT	2
WHITE	2
Bigrams:
A	WHITE	1
OR	A	2
WHITE	HOUSE	2
Trigrams:
A	WHITE	HOUSE	1

[0051] m_α: “HOUSE”=4−0.1=3.9
[0052] m_β1: “WHITE HOUSE”=2−0.1=1.9
[0053] m_β2: “A WHITE HOUSE”=1−0.1=0.9.

[0055] Finally, the improved desired orthogonalized boundary value m_α^orthois calculated for the attribute α to:
m_α^ortho=m_α−m_β1^ortho−m_β2^ortho=3.9−1−0.9=2 (7)

[0057] Hereinafter the use of the boundary value m_α^orthocalculated according to the invention will be represented for three different variants of the GIS training algorithm.

\begin{matrix} \begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho},, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \\ \log (\frac{[t_{α}^{ortho} \cdot m_{α}^{ortho} + b_{α}]}{[t_{α}^{ortho} \cdot m_{α}^{ortho (n)} + b_{α}]} \cdot \frac{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho (n)} + b_{γ}]}{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho} + b_{γ}]}) \end{matrix} & (8) \end{matrix}

where:

[0059] n: refers to an iteration parameter;
[0060] α: refers to a just considered attribute;
[0061] γ: refers to all the attributes in the speech model;
[0062] t_α^ortho, t_γ^ortho: refer to the size of the convergence step;
[0063] m_α^ortho, m_γ^ortho: desired orthogonalized boundary values in the MESM for the attributes α and γ;
[0064] m_α^ortho(n), m_γ^ortho(n): refers to iterative approximate values for the desired boundary values m_α^ortho, m_γ^ortho; and
[0065] bα and bγ: refer to constants.

\begin{matrix} f_{α}^{ortho} = f α - \sum_{(*)} f_{β}^{ortho} & (9) \end{matrix}

\begin{matrix} t_{α}^{ortho} = t_{γ}^{ortho} = 1 / M^{ortho} with M^{ortho} = \max_{(h, w)} (\sum_{β} f_{β}^{ortho} (h, w)) & (10) \end{matrix}

where Mortho for binary attribute functions ƒ_β^orthorepresents the maximum number of functions which yield the value 1 for the same argument (h,w).

\begin{matrix} m_{α}^{ortho (n)} = \sum_{(h, w)} N (h) \cdot p^{(n)} (w ❘ h) \cdot f_{α}^{ortho} (h, w); & (11) \end{matrix}

where:

[0070] N(h): refers to the frequency of the history h in the training corpus; and
[0071] p⁽ⁿ⁾(w|h): refers to an iteration value for the probability p(w|h) with which a predefined word w follows a previous word sequence h (history);
[0072] Here p⁽ⁿ⁾(w|h) uses the parameter values λ_α^ortho(n).

\begin{matrix} \begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho}, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \log (\frac{m_{α}^{ortho}}{m_{α}^{ortho (n)}} \cdot \frac{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho (n)})}{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho})}) \end{matrix} & (12) \end{matrix}

where:

[0074] n: represents the iteration parameter
[0075] A_i(n): represents an attribute group A_i(n)with 1≦i≦m selected in the n^thiteration step;
[0076] α: represents a just considered attribute from the just selected attribute group Ai(n);
[0077] β: represents all attributes from the attribute group Ai(n);
[0078] t_α^ortho, t_β^ortho: represent the size of the convergence step with t_α^ortho=t_β^ortho=1/M_i(n)^orthowith $M_{i (n)}^{ortho} = \max_{(h, w)} (\sum_{β \in Ai (n)} f_{β}^{ortho} (h, w))$
where
[0079] M_i(n)^orthofor binary functions ƒ_β^orthorepresents the maximum number of functions from the attribute group Ai(n) which yield the value 1 for the same argument (h,w);
[0080] m_α^ortho, m_β^ortho: represent the desired orthogonalized boundary values in the MESM for the attributes α and β respectively;
[0081] m_α^ortho(n), m_β^ortho(n): represents iterative approximate values for the desired boundary values m_α^ortho, m_β^ortho.

[0082] The group Ai(n) of attributes α whose parameters λ_α^orthoare adapted in the current iteration step, then cyclically runs through all the m groups in accordance with i(n)=n(mod m).

[0086] Both the recognition device 12 and the training system 14 are normally, although not necessarily, in the form of software modules and run on a suitable computer (not shown).

Claims

The invention claimed is:

iteratively determining:

λ_α^ortho(n+1)=G(λ_α^ortho(n), m_α^ortho, . . . )

where:

n: is an iteration parameter;

G: is a mathematical function;

α: is an attribute in the MESM; and

m_α^ortho: is a desired orthogonalized boundary value in the MESM for the attribute α,

2. A method as claimed in claim 1, characterized in that the calculation of the desired orthogonalized boundary value m_α^orthofor the attribute α=β₀comprises the following steps:

a) Selecting all the attributes β_iwith i=1 . . . g in the speech model that have a larger range RW than the attribute α=β₀and include the latter;

b) Calculating desired boundary values mβi for the attributes β_iwith i=0 . . . g;

c) Sorting the attributes β_iwith i=0 . . . g according to their RW;

d) Selecting one of the attributes β_ihaving the largest RW;

e) Checking whether there are other attributes β_kwhich include the attribute β_iand have a larger RW than the selected attribute β_i;

f2) If not, defining the parameters X to X=0;

g) Calculating the desired orthogonalized boundary value m_βk^orthofor the attribute β_iby arithmetically combining the desired boundary value mβ_iwith a parameter X; and

3. A method as claimed in claim 2, characterized in that the calculation of the parameter X in step f1) is made according to the following formula:

X = \sum_{k} m_{β k}^{ortho} .

4. A method as claimed in claim 3, characterized in that the calculation of the desired orthogonalized boundary value m_βi^orthois made in step g) according to the following formula:

m_βi^ortho=m_βi−X.

6. A method as claimed in claim 5, characterized in that the calculation of the frequency N(β_i) is made by applying a binary attribute function fβ_ito the training corpus where fβ_iis defined as:

f_{β i} (h, w) f_{β i} (h, w) = {\begin{matrix} 1 if β_{i} fits in the word sequence (h, w) \\ otherwise 0 \end{matrix}

and where ƒ_βi(h,w) indicates whether the attribute β_icorrectly describes a pattern predefined by the word sequence (h,w).

7. A method as claimed in claim 1, characterized in that the mathematical function G has as a further variable the magnitude of a convergence step t_α^orthowith:

t_α^ortho=1/M^ortho

where

M^ortho: represents for binary functions ƒ_α^orthothe maximum number of functions which yield the value 1 for the same argument (h,w).

9. A method as claimed in claim 8, characterized in that the calculation of the orthogonalized attribute function ƒ_α^orthofor the attribute α=β₀comprises the following steps:

a) Selecting all the attributes β_iwith i=1 . . . g in the speech model that have a larger range RW than the attribute α=β₀and include the latter;

b) Calculating boundary values fβ_ifor the attributes β_iwith i=0 . . . g;

c) Sorting the attributes β_iwith i=0 . . . g according to their RW;

d) Selecting one of the attributes β_ihaving the largest RW;

e) Checking whether there are other attributes βk which include the attribute β_iand have a larger RW than the selected attribute β_i;

f2) If not, defining the function F to F=0;

g) Calculating the orthogonalized attribute function ƒ_βk^orthofor the attribute βi by arithmetically combining the attribute function fβ_iwith the function F; and

10. A method as claimed in claim 9, characterized in that the calculation of the function F in step f1) is made according to the following formula:

F = \sum_{k} f_{β k}^{ortho} .

11. A method as claimed in claim 9, characterized in that the calculation of the orthogonalized attribute function ƒ_βi^orthoin step g) is made according to the following formula:

ƒ_βi^ortho=ƒ_βi−F.

12. A method as claimed in claim 1, characterized in that the mathematical function G has the following form:

\begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho}, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \log (\frac{[t_{α}^{ortho} \cdot m_{α}^{ortho} + b_{α}]}{[t_{α}^{ortho} \cdot m_{α}^{ortho (n)} + b_{α}]} \cdot \frac{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho (n)} + b_{γ}]}{1 - \sum_{γ} [t_{γ}^{ortho} \cdot m_{γ}^{ortho} + b_{γ}]}) \end{matrix}

where:

α: refers to a just considered attribute;

γ: refers to all the attributes in the speech model;

t_α^ortho, t_γ^ortho: refer to the size of the convergence step with t_α^ortho=t_γ^ortho=1/M^orthowith

M^{ortho} = \max_{(h, w)} (\sum_{β} f_{β}^{ortho} (h, w));

where M^orthofor binary functions ƒ_β^orthorepresents the maximum number of functions which yield the value 1 for the same argument (h,w);

m_α^ortho, m_γ^ortho: refers to desired orthogonalized boundary values in the MESM for the attributes α and γ;

m_α^ortho(n), m_γ^ortho(n): refers to iterative approximate values for the desired boundary values m_α^ortho, m_γ^ortho(n); and

bα and bγ: refer to constants.

13. A method as claimed in claim 1, characterized in that the mathematical function has the following form:

\begin{matrix} λ_{α}^{ortho (n + 1)} = G (λ_{α}^{ortho (n)}, m_{α}^{ortho},, \dots) \\ = λ_{α}^{ortho (n)} + t_{α}^{ortho} \cdot \log (\frac{m_{α}^{ortho}}{m_{α}^{ortho (n)}} \cdot \frac{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho (n)})}{1 - \sum_{β \in Ai (n)} (t_{β} \cdot m_{β}^{ortho})}) \end{matrix}

where:

n: represents the iteration parameter;

A_i(n): represents an attribute group A_i(n)with 1≦i≦m selected in the n^thiteration step;

α: represents a just considered attribute from the just selected attribute group A_i(n);

β: represents all the attributes of the attribute group Ai(n);

t_α^ortho, t_β^ortho: represents the size of a convergence step with t_α^ortho=t_β^ortho=1/M_i(n)^orthowith

M_{i (n)}^{ortho} = \max_{(h, w)} (\sum_{β \in Ai (n)} f_{β}^{ortho} (h, w))

where M_i(n)^orthorepresents for binary functions ƒ_β^orthothe maximum number of functions from the attribute group A_i(n), which yield the value 1 for the same argument (h,w);

m_α^ortho, m_β^ortho: represent desired orthogonalized boundary values in the MESM for the attributes α and β respectively;

m_α^ortho(n), m_β^ortho(n): represent iterative approximate values for the desired boundary values m_α^ortho, m_β^ortho;

Patent

Date

Inventor

Cited By

US6304841(B1)

2001-10-01

Berger et al.

Examiner

Non-Patent Literature (17)

Adam Berger, The Improved Iterative Scalling Algorithm: A Gentle Introduction, Dec., 1997, http://citeseer.nj.nec.com/31826.html.Examiner

Chen, S., Rosenfeld, Ronald. “Efficient Sampling and Feature Selection in Whole Sentence Maximum Entropy Language Models”, Acoustics, Speech and Signal Processing, ICASSP-99, IEEE Conference on, vol. 1, Mar. 15-19, 1999, pp. 549-552.Examiner

Beyerlein, Peter. “Discrimitive Model Combination”, Automatic Speech Recognition and Understanding, IEEE Workshop on, Dec. 14-17, 1997, pp. 238-245.Examiner

Simons, M., Ney, H., Martin, S.C. “Distant Bigram Language Modeling Using Maximum Entropy”, Acoustics, Speech and Signal Processing, ICASSP-97, IEEE Conference on, vol. 2, Apr. 21-24, pp. 787-790.Examiner

Peters, Jochen. Dietrich, Klakow. “Compact Maximum Entropy Language Models”, Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, Dec. 1999.□□.Examiner

Kneser, Reinhard. Peters, Jochen. “Semantic Clustering for Adaptive Language Modeling”, Acoustics, Speech and Signal Processing, vol. 2, pp. 779-782, 1997.□□.Examiner

Berger, Adam L. Della Pietra, Vincent J. Della Pietra, Stephan A. “A Maximum Entropy Approach to Natural Language Processing” Computational Linguistics vol. 22, Mar. 1996.□□.Examiner

Mikheev, Andrei. “Feature Lattices for Maximum Entropy Modelling” International Conference on Computational Linguistics vol. 2, pp. 848-854 1998.□□.Examiner

Greiff, Warren R. Ponte, Jay M. “The Maximum Entropy Approach and Probabilistic IR Models” ACM Transactions on Information Systems, vol. 18, pp. 246-287, Jul. 2000.Examiner

By J.N. Darroch & D. Ratcliff, Entitled: “Generalized Iterative Scaling for Log Linear Models” Annals Math. Stat., 43(5): 1972, 1470-1480.Applicant

By R. Rosenfeld Entitled: “A Maximum-Entropy Approach to Adaptive Statistical Language Modeling”; Computer Speech and Language, 10: pp. 187-228, 1996.Applicant

R.Rosenfeld, “A Maximum Entropy Approach to Adaptive Statistical Language Modelling;” Computer Speech and Language, 1996, Academic Press Limited; 4 pgs.Applicant

J.N.Darroch et al., “Generalized Iterative Scaling For Log-Linear Models;” The Annals of Mathematical Statistics, vol. 43, No. 5; 1972; 10 pgs.Applicant

By J.N. Darroch & D. Ratcliff, Entitled: “Generalized Iterative Scaling for Log Linear Models” Annals Math. Stat., 43(5): 1972, 1470-1480, no month or day.Applicant

By R. Rosenfeld Entitled: “A Maximum-Entropy Approach to Adaptive Statistical Language Modeling”; Computer Speech and Language, 10: pp. 187-228, 1996, no month or day.Applicant

R.Rosenfeld, “A Maximum Entropy Approach to Adaptive Statistical Language Modelling;” Computer Speech and Language, 1996, Academic Press Limited; 4 pgs, no month or day.Applicant

J.N.Darroch et al., “Generalized Iterative Scaling For Log-Linear Models;” The Annals of Mathematical Statistics, vol. 43, No. 5; 1972; 10 pgs, no month or day.Applicant