User interface for speech recognition system grammars

[0002] The present application claims priority from U.S. provisional application No. 60/102,500, filed Sep. 30, 1998, which is hereby incorporated herein by reference.

[0003] The present invention relates to speech recognition systems, and more particularly, to a graphical user interface for interacting with multiple application-associated grammars.

[0004] A computer operating system graphical user interface allows a user to control multiple applications by manually operating a pointing device to position a cursor over application-associated icons. The pointing device is also employed within a specific application to control operation of the application, typically by drop-down menus and dialog boxes.

[0005] Speech recognition systems may also perform such application control by spoken input rather than manually operated pointing device. Multiple computer applications may be registered with a speech recognition engine, e.g.,word processing, e-mail, etc. Each such application may have an associated grammar of valid vocabulary and usage. These grammars vary in their complexity. A grammar may be as simple as a list of command words such as “open,” “close,” “save,” “print,” etc. Such a simple grammar may also be visually displayed to a user in the familiar structure of drop-down menus. More sophisticated grammars emulate natural language usage and may parse complex phrases such as: “Italicize the first paragraph.”

[0006] A grammar is properly viewed as the specification of word sequence structures permitted in a given language or application. The most common kinds of grammars are known as context-free grammars (CFGs) which contain a set of terminal symbols that appear in final sequences (e.g.,words and punctuation), a set of non-terminal symbols that are expanded into other symbols (e.g., “NP” for a noun phrase, “VP” for a verb phrase), a specific non-terminal designated as the starting symbol (e.g., “s-maj”, for major sentence), and a set of rewrite rules each of which has a single non-terminal symbol on the left-hand side (LHS) and one or more symbols on the right-hand side (RHS). A context-free grammar thus assigns one or more structures to every valid word sequence in a language.

[0007] Context-free grammars are often presented in Backus-Naur Form (BNF) notation. In BNF, non-terminal symbols are enclosed in <brackets> and the LHS and RHS of a rewrite rule are separated by an indicator symbol ‘::=’ which may be read as “includes”. In an example of a simple grammar:

[0008] <SENTENCE>::=<SUBJECT><VERBPHRASE>

[0009] <SUBJECT>::=John|Mary

[0010] <VERBPHRASE>::=<VERB><OBJECT>

[0011] <VERB>::=eats|drinks

[0012] <OBJECTS>::=wine|cheese

[0013] A speech recognition system user may have some difficulty operating multiple applications by spoken command when each application has its own different associated grammar of valid vocabulary and usage. As a result, at any given instant, such a user may not know what commands may be spoken and recognized by a given application.

[0014] A preferred embodiment of the present invention provides a graphical user interface of a speech recognition system for user interaction with at least one grammar of valid vocabulary and usage for an application which is accessible to the speech recognition system. The interface includes an interactive dialog box in a portion of a display screen, a grammar identifier displayed within the dialog box which describes an active grammar, a search identifier displayed within the dialog box which describes a grammar search mode for determining valid inputs according to the active grammar, a current input displayed within the dialog box which describes a spoken input, and a valid grammar list displayed within the dialog box which lists examples of valid vocabulary and usage according to the grammar search mode for the active grammar. The dialog box may further include an active grammar display within the dialog box which displays the active grammar, and which may be used to interactively select the current input.

[0015] In a further embodiment, the active grammar or the grammar search mode, or both are interactively selectable using the dialog box. Similarly, one of the examples displayed in the valid grammar list may be interactively selectable as an input to the application of the active grammar.

[0016] In addition, or alternatively, the grammar search mode may determine all word phrases of at least one word which are valid in the active grammar, or word phrases of at least one word which contain the current input and which are valid in the active grammar. Such word phrases may be further constrained to start with the current input.

[0017] A preferred embodiment may also bias towards shortness the examples displayed in the valid grammar list by use of a matrix-based algorithm. For example, the matrix-based algorithm may determine an average generated example length to bias the examples towards shortness.

[0018] A preferred embodiment also includes a method of user interaction via a graphical user interface with at least one grammar of valid vocabulary and usage for an application accessible to a speech recognition system. The method includes creating an interactive dialog box in a portion of a display screen; displaying within the dialog box a grammar identifier which describes an active grammar, a search identifier which describes a grammar searching mode for determining valid inputs according to the active grammar, and a current input which describes a spoken input; and generating and displaying within the dialog box a valid grammar list which lists examples of valid vocabulary and usage according to the grammar search mode for the active grammar. The dialog box may further include an active grammar display within the dialog box which displays the active grammar, and which may be used to interactively select the current input.

[0019] In a further embodiment, the active grammar or the grammar search mode, or both, are interactively selected using the dialog box. Similarly, one of the examples displayed in the valid grammar list may be selected as an input to the application of the active grammar.

[0020] In addition, or alternatively, the grammar search mode may determine all word phrases of at least one word which are valid in the active grammar, or word phrases of at least one word which contain the current input and which are valid in the active grammar. Such word phrases may be further constrained to start with the current input.

[0021] A preferred embodiment may also bias towards shortness the examples displayed in the valid grammar list by use of a matrix-based algorithm. For example, the matrix-based algorithm may determine an average generated example length to bias the examples towards shortness.

[0022] The present invention will be more readily understood by reference to the following detailed description taken with the accompanying drawings, in which:

[0023] FIG. 1 illustrates a graphical user interface of a speech recognition system according to a preferred embodiment of the present invention.

[0024] For each application registered with a speech recognition system, the total allowable vocabulary at any given moment is referred to as “the active grammar,” or more simply, “the grammar.” As used herein, “grammar” is shorthand for a more specific form of a factored BNF grammar wherein rewrite rules always have a single non-terminal symbol on the LHS and a sequence of symbols (terminal, non-terminal, or both) on the RHS, and wherein there are no other “special” symbols such as Booleans or punctuation.

[0025] Each grammar has the ability to divide itself into “topics.” A topic is a non-terminal symbol in a grammar whose parent is also a topic, such as is well-known in the art. The start symbol of a grammar is by definition a topic, and thus, all grammars must have at least one topic.

[0026] The idea of topics is to define rules in the grammar that map to understandable areas of functionality that the language covers. For a simple calculator example:

[0027] Start→Addition|

[0028] Subtraction|

[0029] Multiplication|

[0030] Division|

[0031] Compute

[0032] Addition→“Add” Num AddAnd Num

[0033] Subtraction→“Subtract” Num SubtractFrom Num

[0034] Multiplication→“Multiply” Num MultiplyWith Num

[0035] Division→“Divide” Num DivideBy Num

[0036] “Divide” Num DivideWith Num

[0037] Compute→“Compute” ComputeExpr

[0038] . . . and so on. The topics in this example extend only one level into the grammar, but complex grammars may use this approach to divide the language they define into various subsets. It should be noted that some, but not necessarily all non-terminal symbols are topics. The start symbol is required to be a topic. Any other non-terminal symbol B can be a topic if, in the grammar, there exists a rewrite rule:

[0039] A::=B, and A is also a topic.

[0040] A preferred embodiment of the present invention, a “What Can I Say” (WCIS) window, gives a user the ability to view the topics of the various grammars associated with the applications which are active in a speech recognition system. Each application grammar which is registered with the system contains information regarding how to display itself in the WCIS window. The WCIS provides to the user visual indication of allowable vocabulary and usage, including words, phrases and sentences.

[0041] As a preliminary matter, it is noted that the complete WCIS window is not intended to be displayed at all times. The amount of information available to and processed by the WCIS demands considerable system computational resources. Moreover, the amount of information displayed requires significant display screen space. As a result, the continuous display of the entire WCIS is impractical, and the WCIS window acts like a dialog box in that it is only temporarily displayed when invoked by the user.

[0042] A preferred embodiment, as shown in FIG. 1, includes a WCIS window 10 which occupies a portion of the display screen. At the top of the WCIS window 10, FIG. 1 shows a grammar identifier 12 which contains a grammar topic control that allows the user to interactively select an active grammar. The grammar identifier 12 uses a drop-down menu list to alphabetically display all the grammars. In addition, one level of depth into each grammar may be displayed in the grammar identifier 12. For instance, the grammar identifier 12 may contain grammars for all active commands (a superset of all active command sets), global commands that are active anywhere within the speech application, for various word-processors, for general menu navigation, e-mail applications, etc.

[0043] Since the grammars may include large, complex natural language grammars, the WCIS 10 provides multiple operating modes, including word-by-word search, example search, tree search, and all-words list. Display pages associated with the various operating modes are contained in a WCIS Search Tab Control which displays a search identifier 14. The search identifier 14 describes various available grammar search modes for determining currently valid inputs according to the active grammar.

[0044] A current input 16 is also displayed within the WCIS 10 which describes a current spoken input. Below the current input 16 is the valid grammar list 18 which lists examples of valid vocabulary and usage according to the grammar search mode for the active grammar. Typically, the examples in the valid grammar list 18 are also constrained by the current input 16.

[0045] Operation of the WCIS 10 varies depending on the search mode selected and displayed in the search identifier 14. Tree search presents the user with a visual display of the active grammar starting from the current input 16. The active grammar is displayed in the valid grammar list 18. The user can expand the topics in the valid grammar list 18 to see deeper into the grammar. Working deeper into the grammar through the valid grammar list 18 also changes the current input 16 display. By manipulating the search tree, the user can see how the current input 16 changes. The display of the topics in the valid grammar list 18 may include use of ellipses after topics which are continuable and a “X” mark before the topic to indicate the topic is a partial form of a valid topic and a “”0 mark before the topic to indicate that it is a complete. form of a valid topic.

[0046] Selecting example search with the search identifier 14 generates examples of valid vocabulary and usage contained in the currently selected topic as shown in the current input 16. The user can limit the example search by inputting keywords that the generated sentences must contain. For example, clicking on the “Addition” topic will generate sentences like “Add three and five”, “Add four to ten”, and “Add seventy three with two hundred and eight.” However, typing “and” into the keyword field will limit the generated examples to “Add three and five” and “Add seventy three with two hundred and eight.” “Add four to ten” would not be generated: The user can control the number of examples generated and this enter of search is limited to the topic selected in the grammar topic control.

[0047] Word-by word search allows the user to build a valid sentence by selecting from a list of start words, and then from words that follow what is already present. Alternatively, the user can enter a sentence to check if it is valid. As the user enters a sentence, all the valid next words are displayed below the search window in a list box. The search is limited to the topic selected in the grammar topic control.

[0048] Other search and display forms are also possible. All-words list displays all the words in the grammar. In addition, a WCIS window can display a dictation grammar window. This dictation grammar window has a single dictation topic in the grammar topic control and it does not have any sub-topics. When the user selects the dictation topic with the topic selector, the list box displays the dictation words, and the user can search the dictation grammar for valid vocabulary.

[0049] The examples displayed in the WCIS are produced by stochastic generation. From the start symbol, grammar rules are chosen at random to generate a random valid parse tree in which the terminal symbols form the generated sentence. Such stochastic generation algorithms are well-known in the art.

[0050] To generate example sentences with a single keyword, a rule is chosen with the keyword somewhere on the right hand side. Then, the grammar is recursively processed back to get to a rule that has the start symbol on the left hand side, saving all the parse tree information. This produces a partial parse tree with non-terminal symbols as leaves of the parse tree. The non-terminals are filled in using the plain stochastic generator described above, using the non-terminal in question as the start symbol. For generation of sentences with more than one keyword, many sentences are generated using the single keyword technique. Then all the examples that do not have all the keywords are discarded.

[0051] To bias the generated example sentences toward shortness, the grammar is converted into a matrix. A square matrix M is initialized which is composed of integers with zero in every cell. The square matrix has as many rows and columns as there are non-terminal symbols in the grammar. For each time that a non-terminal symbol <j> appears on the RHS of a rewrite rule, take that rule's LHS non-terminal symbol <i> and add one to the matrix cell M(i,j). Then, for each time a non-terminal symbol <i> appears on the LHS of any rewrite rule, subtract one from the diagonal M(i,j).

[0052] The matrix is then solved as a set of linear equations using one of the well-known matrix diagonalization methods. When the system is solved, a diagonal matrix is obtained wherein all the values are positive on the diagonal cells and zero elsewhere. Then, for each rule, an average generated example length is determined by summing over the symbols on the RHS where each terminal symbol has a value of 1 and each non-terminal symbol has the value indicated in the square matrix M(i,j). This average generated example length is used to bias the stochastic generator to improve the usefulness of the example sentences generated by the WCIS. Otherwise, the examples may tend to be overly long and hard to understand.

[0053] One of the biggest problems with a natural language user interface which does not contain all of natural English is that the user may try to say something that is not defined by the grammar. The WCIS window gives the user access to application grammar searching capabilities alongside the normal correction mechanism, so the user can determine if what they are trying to say is valid, and if it is not, what they can say to accomplish the desired task. As the user enters a sentence, a list box below the edit field displays possible completions to the keystrokes already entered. The user can double-click or say “take” to replace what they have entered with a specific completion. The completions take into account both the dictation grammar and the Application grammar. As a user enters a word, the system will gather all the individual words in both dictation and the Application grammar that begin with the letters entered. If the letters entered complete a word in the Application grammar, the system will display all the words that can come after the entered word and still be part of a valid sentence.

[0054] The following examples show what would be displayed as the user enters:

[0055] Current sentence: b

[0056] 1. baby

[0057] 2. bath

[0058] 3. bet

[0059] 4. bind

[0060] 5. bold

[0061] 6. butter

[0062] 7. bold . . .

[0063] 8. boldface

[0064] Current sentence: bo

[0065] 1. boat

[0066] 2. bold

[0067] 3. bone

[0068] 4. bold . . .

[0069] 5. boldface . . .

[0070] 6. boldtype . . .

[0071] Current sentence: bold

[0072] 1.bold the . . .

[0073] 2. bold this

[0074] 3. bold on

[0075] 4. bold off

[0076] Ellipses after a word in the list indicates that the word is an incomplete command in the Application grammar. When that word is selected, the words which follow it are listed below it. The use of bold, italics, color, etc. may be incorporated into this list box as needed to clarify things such as actions, commands and dictation words.

[0077] In addition, any valid sentence displayed in the WCIS window is trainable. That is, the user can provide examples or tokens of the user speaking the sentence which the speech recognition system integrates into the recognition engine. Right-clicking on the sentence brings up a context menu with a selection for training the sentence. The user can also train an item on the grammar topic control of the WCIS window. Clicking on “train” on the context menu will allow the user to perform training on randomly generated sentences within the selected topic.

[0078] The WCIS window also allows for editing of client grammars. However, the grammars are not owned by the speech recognition engine with the WCIS window, but rather are part of the speech applications programming interface (SAPI) client associated with the grammar. Therefore, the grammars cannot be directly edited by the WCIS, but must be edited indirectly by the WCIS. When a SAPI client registers a grammar with the speech recognition engine, the client provides information as to how that grammar is edited by specifying which of the WCIS's standard edit methods (Add, Edit, Delete, Explore, Properties, etc.) are valid. When the user right-clicks on a grammar in WCIS, the system checks to see what edit modes that grammar supports. Any supported methods are reflected in the context menu that is displayed as a result of the mouse click. When the user clicks on one of these methods, WCIS notifies the SAPI client of the requested modification through an appropriate notify sink. The client can then pop up an appropriate dialog box or perform appropriate actions.