Conversion between data representation formats

[0002] The present invention pertains to a method and an arrangement for conversion between data representation formats, said data comprising sound or text information. The invention specifically provides for a method and arrangement means for word and sound processing.

[0003] Natural language understanding has been the topic of research since the first days of Artificial Intelligence. The present invention is primarily intended for understanding spontaneous utterances, in written or spoken form, within a limited domain.

[0004] One current approach to this problem is to model a dialog flow for each operation that can be performed in a specific system, dividing each dialog into modes. For each mode, valid inputs and their consequences are listed. For example, the Philips SpeechMania® 99 product has been demonstrated with a pizza ordering application, where a user goes through dialog modes involving for instance selecting pizza toppings. A disadvantage of this type of technology is that the system will only understand the utterances expected in a given mode. If a user changes his drink order while he is expected to select pizza toppings, the system may fail to understand this. The degree to which the system ‘understands’ the utterances in this kind of iteration is limited; each mode and the utterances valid therein must be anticipated by the developers, and directly related to the action the system takes as a response to user input.

[0005] Other speech recognition systems, such as those using the Java™ Speech Grammar Format (JSGF), provide tags attached to an (often handwritten) grammar. The tags normally have some semantic meaning, regardless of how that semantic meaning was expressed. Words or phrases without semantic interest to the application (politeness phrases, articles, etc.) are ignored. Such an application then has a very simple parsing of the tags in order to act on the speech input. This requires manual adaptation of the application and grammar so that they work together, and cannot be said to ‘understand’ the spoken utterances.

[0006] More advanced approaches to natural language understanding make use of formalisms developed within the field of linguistics and computational linguistics. One currently popular formalism is Head-Driven Phrase Structure Grammars, which associates groups of lexical features with words, resulting in a grammar structure which can be used for parsing general natural language. Many of these linguistic formalisms could be used to perform some of the steps described in the present invention, but require much more work to be integrated into a complete language understanding interface to an application, and also substantial adaptation to new domains.

[0007] Some speech recognition systems use word spotting. This entails listening for certain key words and ignoring the rest of the spoken utterance. This may simplify the parsing component of a system, but does not allow the system to understand the details of the user's utterance.

[0008] No commercial applications use the same grammar for both natural language generation and natural language understanding. Most current applications either understand a very simple subset of natural language, or require substantial manpower to adapt the natural language understanding system to a given application.

[0009] The present invention sets forth a method and an arrangement for word and sound processing. It solves the problem of simple natural language understanding, allowing users to interact with (for instance by giving commands and asking questions) machines using natural language, for instance in spoken or written form. Additionally, the method provides language independence by transforming linguistic utterances by the user to a semantic form which is independent of the original language used. This form may later be converted into another human language, thereby resulting in a simplistic translation. Furthermore, the present invention can automate the process of adapting domain specific natural language understanding to an application of the same.

[0010] The present invention does not attempt to solve the highly complex problem of general natural language understanding, but rather the understanding of a limited subset of natural language—utterances which can be straight forwardly mapped to the domain of one or several data models or computer applications.

[0011] To achieve these aims and objectives, the present invention provides a method for conversion between data representation formats, where the data includes sound or text information. The data representation formats are text, sound, words, phrases, and logic. Those are combined as conversions between text or sound to words or vice versa, words to phrases or vice versa, and phrases to logic or vice versa. The formats include a string of characters in the text format, a digital representation of an acoustic wave form in the sound format, a reference to a data structure containing information about a word in the word format, a tree-like representation of grammatical structure of a phrase in the phrase format, where the leaf nodes of the tree-like representation are referenced to meanings of constituent words and conjunction information, and references to function, objects and attributes in an underlying data model in the logic format. The method includes the steps of converting text to words by using characters with separate words, converting words to text by concatenating the spelling of the constituent words, converting sounds to words by providing a continuous speech recognition system, converting words to sounds by providing a speech synthesis system, converting words to phrases by parsing, converting phrases to words by transversing the tree-like representation preferably from left to right and converting each leaf node to a word, converting phrases to logic by resolving or binding verb phrases to functions and noun phrases to objects in the underlying data model and converting logic to phrases by using knowledge of the grammar of the language used to create a phrase expressing the same semantics as in the original logic format.

[0012] The method further comprises:

[0013] converting text to words by using characters which separate words;

[0014] converting words to text by concatenating the spelling of the constituent words;

[0015] converting sound to words by providing a continuous speech recognition system;

[0016] converting words to sound by providing a speech synthesis system;

[0017] converting words to phrases by parsing

[0018] converting phrases to words by traversing said tree-like representation preferably from left to right and convert each leaf node to a word;

[0019] converting phrases to logic by resolving or binding verb phrases to functions and noun phrases to objects in said underlying data model;

[0020] converting logic to phrases by using knowledge of the grammar of the used language to create a phrase expressing the same semantics as the original logic form; and thus providing a computer word processing and sound processing means.

[0021] The invention here also includes the apparatus for accomplishing the conversion. There are means for converting text to words, words to text, sounds to words, words to sounds, words to phrases, phrases to words, phrases to logic and logic to phrases, typically in form of software or a combination of hardware and software.

[0022] The arrangement further comprises:

[0023] converting means from text to words, which uses characters which separate words;

[0024] converting means from words to text, which concatenates the spelling of the constituent words;

[0025] converting means for sounds to words, providing a continuous speech recognition system;

[0026] converting means for words to sound, providing a speech synthesis system;

[0027] converting means for words to phrases, which uses parsing;

[0028] converting means for phrases to words, which traverses said tree-like representation preferably from left to right and converts each leaf node to a word;

[0029] converting means for phrases to logic, which resolves or binds verb phrases to functions and noun phrases to objects in said underlying data model; and

[0030] converting means for logic to phrases, which uses knowledge of the grammar of the used language to create a phrase expressing the same semantics as the original logic form.

[0031] Further embodiments of the present invention are set out through the attached dependent claims. Also, said arrangement is able to provide the embodiments relating to said method.

[0032] Henceforth reference is had to the attached figures for a better understanding of the present invention and its examples and embodiments, wherein:

[0033] FIG. 1 is a diagrammatic illustration of conversions between reorientations according to the present invention; and

[0034] FIG. 2 is a diagrammatic illustration of a tree-like structure used in the present invention.

[0035] The present invention relates to how information can be converted between data representation formats. The conversions can be done using known techniques, where the uniqueness lies in the way the present invention is used and how to combine representations, and the conversions between them.

[0036] FIG. 1 illustrates conversions 10 between representations 12, 14, 16, 18, 20 according to the present invention, whereby the arrows represent the conversions.

[0037] Provided below is an explanation of the representations corverted between. As an example, the phrase “print the file” will be used.

[0038] The format text 12 depicts a string of characters, such as “print the file”, and sound 14 is a digital representation of an acoustic waveform. A digital recording of a person or a speech synthesizer saying “print the file” would be such a representation.

[0039] The format words 16 is a reference to a data structure containing information about a word, such as its spelling, pronunciation and meanings. In the “words” representation, the example phrase is a list of references to data structures representing the words “print”, “the” and “file”. No indication exists of the meaning intended for each word. For instance, the word “file” has at least two meanings, both as the verb “to file”, and the noun “a file”.

[0040] The format phrases 18 is a tree-like representation 30, FIG. 2, of the grammatical structure of a phrase. The leaves or leaf nodes of such a tree-like representation are references to meanings of the constituent words, and conjugation information such as verb tense, noun case, etc. Using this representation, it is not exactly clear which word is used to express each meaning, because any one of several synonyms may be chosen to verbalize a common noun, for instance.

[0041] FIG. 2 illustrates a phrase representation 18 of the example phrase, “print the file”. Supposing that ‘document’ is a synonym of ‘file’ in a given domain, then either word can be used in realizing the phrase.

[0042] A logic format representation 20 is similar to predicate logic, a well known basis for the proof of mathematical theorems. In the logic representation 20, all references are had to functions, objects and attributes in an underlying data model. This means that all abstract references, such as “the file”, must be resolved to determine which individual file is referred to. The logic representation 20 is language independent, as opposed to the other representations, which are specific to a certain human or spoken language.

[0043] The example phrase could in some context be represented as:

[0044] print(‘/etc/passwd’)

[0045] in a computer language. This is a function call to a function called “print” with the filename “/etc/passwd” as its argument. This assumes that it is resolved that “the file” refers to a file with the name “/etc/passwd”, and that there is a function called “print” which corresponds to the verb “print” as it is used in natural language.

[0046] With an utterance in a logic representation, it is possible to execute it (by calling the functions with their arguments), evaluate the truth of a statement, or enumerate free variables to answer questions. Performing these operations (execution, evaluation and enumeration) can be done by standard techniques implemented by persons skilled in the art.

[0047] For instance, answering the utterance “Is Joe happy?” might be performed by generating the logic statement “isHappy(joe)”, and evaluating it by calling the “isHappy” function with the argument “joe”, and letting the value of the statement be the Boolean result of this function call. Another possible logic representation might be “mood(joe)=happy”.

[0048] To answer the question “Who is happy?”, the logic representation “isHappy(X)” might be used (where X is an unbound variable). Answering the question would be equivalent to finding for which objects the function “is Happy” has the value true. Another possible logic representation 20 would be “is Happy(X) AND is Person (X)” (interpreting the word “who” to limit the query to persons).

[0049] Converting text 12 to words 16 can be accomplished by first using characters which separate words, such as spaces and punctuation, to extract the spelling of each word. In the example, “print the file” would be converted to “print”, “the”, and “file.” These strings can then be used to look up the representations of the words in a dictionary, lexicon, or the like through well known searching and sorting algorithms in the art. Such a conversion is deterministic, meaning that the same text will always yield the same list of words.

[0050] Converting words 16 to text 12 consists of concatenating the spelling of the constituent words. In some implementations, words may have several spellings. Examples of this are American and British spelling of “color” vs. “colour”. In this case, converting words to text may use different methods to choose which spelling to use, varying in complexity from choosing one at random, to more advanced methods.

[0051] Converting sounds 14 to words is what is provided through continuous speech recognition systems. Such systems are available in both commercial and free implementations, including IBM's Viavoice®, Dragon Dictate® from Dragon Systems, and the open source ISIP Artificial Speech Recognition system from the University of Mississippi, and are often based on well known techniques such as Hidden Markov Models (HMMs).

[0052] Speech synthesis systems perform conversions from word 16 to sound 14. Such systems can vary in complexity from simply concatenating stored recordings of each word spoken by a person, to advanced synthesis taking prosody and emotions into account. Speech synthesis systems have been available for many years, and include the freely available Festival Speech Synthesis System from the University of Edinburgh.

[0053] Converting words 16 to phrases 18 is known as parsing and is accomplished by a device known as a parser. Parsing is a classical task in computer science and linguistics. Grammar is used to specify possible word sequences, and a parser matches words to possible sequences using the grammar. Grammars can take many forms. Deterministic grammars, using the common Backus Naur Form (BNF) notation, probabilistic n-gram grammars and Head-Driven Phrase Structure Grammars are some examples.

[0054] Converting a phrase 18 structure to words 16 consists of traversing the tree-like representation 30 from left to right, in this embodiment, and converting each leaf node to a word. To convert a node to a word, look-up tables together with information about conjugation of verbs, noun cases, etc., can be used to select words which can represent a given meaning.

[0055] A primary issued in going from a phrase 18 to logic 20 involves to resolve or bind verb phrases to functions and noun phrases to objects in the underlying data model. Binding verb phrases to functions can be accomplished by using a look-up table relating each verb to one or possibly several functions implementing the functionality of that verb. Different interpretations of the same verb may reduce ambiguity by the number and type of phrase arguments that it takes. The exact mechanisms for such resolution may be realized in different ways for person skilled in the art.

[0056] Resolving noun phrases to objects may require tracking recently mentioned objects, i.e., to now which file “the file” refers to in the example phrase. The exact mechanism for doing this is known to persons skilled in the art.

[0057] Going from logic 20 to phrase 18 uses and requires knowledge of the grammar of the language, word order, and the like, to create a phrase expressing the same semantics as the original logic form. Similarly to the resolution of noun phrases, tracking recently mentioned objects can help in good logic-to-phrase conversion. This can for instance recognize when the pronoun “it” can be used instead of generating a long description of the object.

[0058] In order to make use of the present invention an apparatus including converting means to go from text to words which uses characters which separate words, converting means to go from words to text which concatenates the spelling of the constituent words, converting means to go from sounds to words using a continuous speech recognition system, converting means to go from words to sounds using a speech synthesis system, converting means to go from words to phrases which uses parsing, converting means to go from phrases to words which traverses the tree-like representation and converts each leaf node to a word, converting means to go from phrases to logic which resolves or binds verb phrases to functions and noun phrases to objects in the underlying data model and converting means to go from logic to phrases which uses knowledge of the grammar of the language used to create a phrase expressing the same semantics as the original logic form.

[0059] The converting means mentioned in the present description can be software or a combination of hardware and software. The hardware may be a computer and peripheral devices.

[0060] The present invention has been described with non-limiting examples and embodiments. It is the attached set of claims that describe all possible embodiments for a person skilled in the art.