US 7,295,965 B2Grant
Method and apparatus for determining a measure of similarity between natural language sentences
Issue Date:2007-11-13
•38 Claims
•12 Drawing Sheets
Abstract
Systems and methods for classifying natural language (NL) sentences using a combination of NL algorithms or techniques is disclosed. Each NL algorithm or technique may identify a different similarity trait between two or more sentences, and each may help compare the meaning of the sentences. By combining the various similarity factors, preferably by various weighting factors, a distance metric can be computed. The distance metric provides a measure of the overall similarity between sentences, and can be used to assign a sentences to an appropriate sentence category.
Metadata
Assignee
- Honeywell International Inc.
Inventors
- Karen Z. Haigh
- Kevin M. Kramer
Application Information
Application Number:US 09/896,846
Filing Date:2001-06-29
Priority Date:2001-06-29
Art Unit:2654
Classifications
IPC:
G06F17/27G06F17/30
Patent Drawings (12 sheets)
Description
Field of the Invention
[0001] The present invention generally relates to methods and systems for classifying natural language text or sentences, and more particularly to methods and apparatus for determining a measure of similarity between natural language sentences or text.
Background of the Invention
[0002] Computers and computer technology have contributed greatly to our personal and professional lives. Computers now help perform many tasks that were previously performed by humans only a few years ago. While computer systems and computer technology have made significant penetration into our lives, some tasks still require significant human intervention.
[0003] A recurring problem in the computer field is the difficulty of developing computer systems that can perform tasks that have complex, unpredictable or undefined input data. To date, these tasks typically require human intervention, and often, intervention by scarce human workers that have particular knowledge or expertise. An essential problem is the translation of knowledge and skill of a human expert to a computer system in such a manner that the computer system, when provided with the same fact pattern, reaches the same conclusion or decision as the expert.
[0004] The first implementations of such systems used conventional, sequential computers that perform a sequence of operations on a very limited number of data elements, such as an add or compare operation of two data elements. A sequential system that works with large numbers of data elements often requires prohibitively long computation times, even for very fast computer systems.
[0005] To reduce the number of data items to be dealt with, “expert” systems have been developed. One form of “expert” system attempts to implement human “expertise” in a number of rules. In a rule based expert system, knowledge engineers attempt to elicit from experts a set of rules that implement the reasoning of the experts when given a set of facts. The rules typically attempt to codify, for example, the knowledge, methodology and reasoning process used by experts to solve a particular problem. The rules are programmed as a sequence of decision steps and, given a fact pattern, the system executes the programmed sequence of rule decisions in an attempt to reach the same conclusion as the expert.
[0006] A limitations of such rule based “expert” systems is that much of the expertise expressed in the rules is based on knowledge from a large number of individual fact patterns, which are of necessity and purpose more general than individual cases. This results in a loss of a significant amount of information. In addition, it is often difficult to determine whether the correct set of rules has been implemented, particularly since many experts do not consciously know and understand their own methodology and reasoning processes, and may unconsciously create “rules” that do not in fact reflect their methodology.
[0007] Another limitation of many rule based “expert” systems is that a substantial investment of knowledge engineer and expert time is required to determine and implement the appropriate set of rules. Furthermore, if the rules do not produce a desired result, or the input data changes in a material way, rewriting or updating the rules is often an extremely difficult and time consuming process. The rules often interact with one another, and a change in one rule may require corresponding changes in other related rules.
[0008] One application where such difficulties have arisen is in generating knowledge repositories (or data bases) from legacy documents. Many companies are currently processing legacy documents for use in automated reasoning systems. In one example, a knowledge repository may be used as a diagnostic fault model for an airplane. In another example, a knowledge repository may describe company business practices.
[0009] In order to create a knowledge repository, the legacy documents are typically parsed and the relevant information is identified, often by hand. Some of the information can be easily identified from the document context or by pattern-matching techniques. However, much of the information can be more efficiently and accurately identified by having some level of understanding of the textual meaning of the information in the document. To illustrate this, it is known that some legacy maintenance manuals may include certain fault descriptions. To identify and extract the fault descriptions from the the legacy document, some level of understanding of the meaning of the text within the document would be extremely helpful. For example, the sentence “REPLACE THE GO-AROUND SWITCH, S2 ON THE RIGHT THRUST LEVER” may describe a fault, while “DO THE IRU BITE PROCEDURE (MM 34-21-00, FIG. 107)” may describe a procedure.
[0010] It is known that English sentences are extremely complex, and are subject to stylistic variation. Using tediously hand-generated rules that rely on regular expressions to identify the fault statements would be extremely difficult. In general, the more complex the pattern, the more difficult it is to write a regular expression to recognize the pattern. Thus, not only would it be difficult to generate rules that rely on regular expressions to identify fault statements with a legacy document, but such a system would likely be dependent on the writing style of the author, making it difficult to transfer the hand-generated rules and/or regular expressions to different legacy documents or even to different chapters within the same document. Thus, such systems may be highly brittle and error-prone. Because of the foregoing, there is often a large cost associated with creating knowledge repositories from legacy documents.
[0011] The field of search engines is another application where it is often desirable to identify and categorize certain information within documents. Search engines typically accept a user specified search expression, and compare the search expression to text in selected documents, databases or web pages. Using rules or regular expressions to identify and categorize text within documents can be difficult, time consuming, and tedious.
Summary of the Invention
[0012] The present invention provides improved systems and methods for classifying natural language text or sentences by first determining a measure of similarity between natural language sentences or text. The present invention preferably uses Machine Learning (ML) algorithms to categorize and/or identify the similarity between sentences. The ML algorithms are preferably trainable, so that they can be easily customized to any particular domain. By using trainable ML algorithms, the system may be incremental and robust, which allows the accuracy of the system to be increased with additional training. One embodiment of the present invention exploits the semantic information captured by Natural Language Processing (NLP) techniques along with statistical modelling to calculate the similarity between texts.
[0013] It is believed that the present invention may provide as much as a 10-fold reduction or more in the time required to accurately categorize sentences in certain legacy documents, resulting in a cost reduction of 30% to 50%. It is also believed that the present invention may be used to substantially increase the accuracy of many search engines.
[0014] In one illustrative embodiment, the present invention is applied to categorizing sentences in a legacy document to ultimately generate a knowledge repository. In this embodiment, a number of training sentences are selected, and one or more ML algorithms are used to assign each of the training sentences to one or more predefined categories. The algorithms are preferably dependent on a number of operating parameters (e.g., weights) over its constituent components. Once assigned, a correspondence is displayed between each of the training sentences and each of the one or more predefined categories. A user may then review the categorization of each of the training sentences, and change the categorization where appropriate. Based on the changes made by the user, the algorithm updates the operating parameters to reflect the change in the assigned categories, resulting in one iteration of training. Thereafter, the algorithm may be further trained by repeating the assigning, displaying, changing and updating steps until a desired accuracy level is achieved. Once adequately trained, all of the sentences in a legacy document may be processed.
[0015] The assigning step described above preferably uses a distance metric to determine which predefined category to assign a particular sentence. For example, the distance metrics can provide a measure of similarity between a particular sentence and one or more sentences that have already been assigned to a particular category. The particular sentence is preferably assigned to the category that includes the training sentences that are most similar to it. One possible assignment is to assign the sentence to the category of the single most similar sentence (smallest distance). Another possible assignment is to assign the particular sentence to the category that contains the majority of the k most similar sentences.
[0016] There are a number of illustrative algorithms for determining a measure of similarity between a first sentence and a second sentence. One illustrative method calculates the similarity between a pair of sentences based on inputs from several similarity factors. The final similarity value is preferably calculated by a weighted combination of each of the constituent similarity factors. The constituent similarity factors may be functions or methods based on sentence grammar, sentence semantics, statistics, or other specific techniques developed for the language or domain. Some illustrative methods or functions may include, for example, a string match, a keyword match, a phrase match, or a match based on the statistical significance of words, or a domain model match.
[0017] In order to utilize sentence grammar or semantics as a basis for a similarity factor, the sentences are preferably first parsed by a Natural Language Parsing engine. The NLP engine preferably assigns words to their pre-defined grammatical categories (e.g. nouns, verbs, adjectives, . . . ). The engine may also build complete sentence diagramming trees (e.g. noun phrase, prepositional phrase, . . . ). The engine may also assign words to pre-defined semantic categories (e.g. subject, object, modifier, . . . ). The NLP may also identify other natural language elements, including for example, proper names, currency, or sentence voice (passive/active).
[0018] The information extracted by the NLP engine can then be used by any number of similarity factors to assist in the calculation of the similarity between sentences. It is believed that by calculating the natural language properties of the text, and then comparing words or groups of words from similar classes, a more accurate estimation of the similarity of the sentences can be achieved. Some illustrative methods or functions include, for example, a string overlap match, a keyword match, and a phrase match.
[0019] The string overlap match function may contribute to the measure of the overall similarity of the two sentences by identifying the number of overlapping words in the sentences, regardless of word type or class. For example, the string overlap match function may return the number of words in the first sentence that match a word in the second sentence, as a percentage of the number of words in the longest of the first sentence and second sentence.
[0020] The keyword match function may provide a measure of the overall similarity of the two sentences by identifying the number of overlapping words in the sentences, where only words with a common syntactic or semantic type are compared. For example, the keyword match function may return the number of words in the first sentence that match a word in the second sentence, where both words share a common type, as a percentage of the maximum number of keywords in the first sentence and the second sentence. The phrase match function may be similar to the keyword match function, but may be applied to phrases or words. This method allows, as one possible example, noun phrases to be compared, including (or not) information from any constituent prepositional phrases.
[0021] Other techniques may be incorporated to improve the robustness of the natural language metrics. For example, to compensate for spelling errors or the like, it is contemplated that a word in the first sentence may be deemed to match a word in the second sentence when, for example, the word in the first sentence and the word in the second sentence each exceed a minimum number of characters, and the word in the first sentence shares a predetermined minimum number of characters with the word in the second sentence. Similarly, words or phrases may be considered similar if they are related according to some pre-defined knowledge (either domain dependent or independent). For example, the related words may include an original word with the prefix or suffix removed, various tenses of the original word, and/or different words with similar meanings as identified by, for example, a domain thesaurus. In addition, it is contemplated that one or more named entities may be extracted from the first sentence and the second sentence; the named entities may include a complex groups of nouns. The named entities may then be treated as a single object when performing word or phrase comparisons.
[0022] A second general category or factor that can be used for the overall similarity metric is statistical information. Certain words are more common or more indicative of a sentence's meaning than other words. For example, the general noun “thing” indicates much less meaning about a sentence than a more specific noun. Statistical information can be gathered from the language the text was written in (i.e. a general database of word usage), or built from the text under consideration (i.e. domain-specific). There are different methods of building and using this statistical information; one possible technique is to calculate the correlation factor of words in the training text to the category of the sentence, and this correlation factor can then be used to predict the category of new sentences. In the current embodiment, the statistical information is captured in a domain model that returns a similarity value, wherein the similarity value is increased by the correlation factor of a model word when both sentences include the model word, and is decreased by the correlation factor of a model word when only one of the first and second sentences include the model word. By decreasing the similarity value when only one of the two sentences includes the model word, a negative probability or correlation is applied.
[0023] Statistical information can augment the natural language metric, and vice versa. For example, extremely common words (and hence information-free words) can be identified as “stop” words by the statistical methods, and then the natural language methods can exclude them from the similarity analysis. Inversely, the natural language information can be used to augment the statistical calculation, such as when a word can be used as both a noun and a verb; each type of usage may have a different statistical correlation.
[0024] The two above-mentioned categories of similarity factors (natural language and statistical) and any combinations are independent of the domain of the text. In some domains, there may be domain-specific information that will augment the accuracy of the similarity metric. In one embodiment, for example, certain airline maintenance words are “more interesting” to the human users of the system than others. Any domain-specific factors can be added to the similarity metric in the same manner as the domain-independent factors, and moreover, factors that mix the domain-independent methods with domain-dependent methods can be created.
[0025] Each of these factors can then be combined to create a measure of the similarity of two sentences. The metric is flexible enough to incorporate any additional techniques as they are designed or developed. Each factor contributes to the final measure through a weighted function that can be tuned to the domain under consideration.
[0026] While the above illustrative embodiments may be used to categorize sentences in legacy documents, other applications are also contemplated. For example, the illustrative algorithms of the present invention may be applied to search engines. Often, search engines compare the similarity of a search expression with sentences or to text in search documents. The present invention can advantageously perform such a comparison, using some notion of the meaning of the sentences to be compared, which in many applications is highly desirable.
Brief Description of the Drawings
[0027] Other objects of the present invention and many of the attendant advantages of the present invention will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, in which like reference numerals designate like parts throughout the figures thereof and wherein:
[0028] FIG. 1 is a screen shot of an illustrative graphical user interface in accordance with the present invention;
[0029] FIG. 2 is a schematic diagram showing the architecture of an illustrative embodiment of the present invention;
[0030] FIG. 3 is a high level diagram showing an illustrative algorithm for assigning sentences to one of several categories;
[0031] FIG. 4 is a schematic diagram showing a sentence diagram of a sentence using a natural language parser;
[0032] FIG. 5 is a diagram showing the calculation of a similarity factor between two sentences using an illustrative string overlap function;
[0033] FIG. 6 is a diagram showing the calculation of a similarity factor between two sentences using an illustrative keyword match function;
[0034] FIG. 7 is a diagram showing the calculation of a similarity factor between two sentences using an illustrative phrase match function;
[0035] FIG. 8 is table showing illustrative correlation factors for several model words;
[0036] FIG. 9 is a diagram showing the calculation of a similarity factor between two sentences using an illustrative model match function;
[0037] FIG. 10 is a diagram showing the calculation of an illustrative distance metric in accordance with a preferred embodiment of the present invention;
[0038] FIG. 11 is a table showing the calculation of the distance metric of FIG. 10 using the similarity factors of FIGS. 5 , 6, 7 and 9 and various weighting factors; and
[0039] FIG. 12 is a graph showing the accuracy of sentence categorization versus the number of sentences used for training for an Airbus A320 legacy maintenance manual.
Detailed Description of the Preferred Embodiments
[0040] FIG. 1 is a screen shot of an illustrative graphical user interface in accordance with the present invention. The illustrative graphical user interface provides an interactive and iterative environment for machine learning in the context of placing sentences into desired categories. At each iteration of the learning process, the user is presented with a set of sentences, which are highlighted with best guesses for sentence classification. In FIG. 1 , a number of training sentences are displayed at 10a–10e, with the “fault” type sentences, such as sentence 10c, highlighted as shown. The user may then make any necessary corrections, preferably by clicking on the appropriate sentences to toggle the highlights. Control is then returned to the ML learner by depressing the learn button 12. The ML learner then tunes one or more weights and/or other operating parameters to the new correlation data. The algorithm then uses all previously labeled sentences including those just categorized by the user, and categorizes a next set of sentences. This may be repeated until the desired accuracy is achieved. When the user quits via the quit button 14, the algorithm may categorize all remaining sentences in the document.
[0041] FIG. 2 is a schematic diagram of the architecture of an illustrative embodiment of the present invention. Before the ML algorithm is applied, every sentence in the document is initially parsed, as shown at 20. Each sentence is parsed to identify, for example, words, parts-of-speech, phrases, etc. Any parser or tagger may be used. One illustrative parser is known as the Text Tokeniser Toolkit (TTT), which is available from the Language Technology Group of the University of Edenburgh, located in Edenburgh, Scotland. Another illustrative parser is known as the Link Grammar Parser (LGP), available from Carnegie Mellon University, Pittsburgh, Pa. Yet another illustrative parser is Barker's ML-based semantic recognizer, described in Ken Barker's PhD thesis, Ottawa-Carleton Institute for Computer Science, University of Ottawa, Ottawa, Ontario, June 1998. Others are also available.
[0042] For some documents, such as maintenance manuals, there are often many grammatical oddities. At a minimum, many maintenance manuals are not written with standard English, but rather often contain abbreviations and/or strange structure. For example, some maintenance manuals may contain parenthetical cross-references, and a large vocabulary unfamiliar to standard parsers. Moreover, sentences may be truncated by the character recognition software used to read the manuals into computer readable form. It has been found that TTT parses these sentences reasonably well, giving fairly accurate part-of-speech tags and syntactic chunks. The LGP provides a reasonable identification of prepositional phrases.
[0043] After parsing each sentence, block 22 calculates pair-wise distance metrics between sentences. Then, and as shown at block 24, a small set of sentences are presented to the user for labeling, such as described above with respect to FIG. 1 . The labels are used to build a statistical domain model (e.g., word-to-category correlations), as shown at block 26. The domain models may also help identify, for example, a set of appropriate weights and/or operating parameters to tune the metric to the relevant data set.
[0044] Using the tuned metric, the user may enter into a full interactive training mode. The ML algorithm presents the user with a set of training sentences, along with an estimated category for each sentence. The user can make any necessary corrections. After the corrections are made, the ML algorithm may update the model, and then categorize another set of sentences. This iterative process may continue until the user is satisfied with the accuracy of the model. The ML algorithm may then be allowed to categorizes all remaining sentences in the document.
[0045] The fully categorized sentences can be used to build fault models for automated reasoning systems. Even after a fault model has been built, the user can still make corrections to the sentence categorizations via block 24 to produce an updated model via block 26. This may be used to increase the accuracy of sentence categorization at any time.
[0046] FIG. 3 is a high level diagram showing an illustrative algorithm for assigning sentences to one of several categories. The ML algorithm of the present invention builds clusters of training examples according to their labeled categories. The n-dimensional clusters 30a–30c depend on features of each categorized sentence in the training examples. For each new target sentence, such as sentence 32, the ML algorithm may select a category 30a–30c based on the similarity of the target sentence 32 to each of the categorized sentences in the training examples. The categorized sentences in the training examples are represented by dots, squares and triangles in FIG. 3 . The similarity between the target sentence 32 and the categorized sentences is preferably calculated using the measure of similarity, or distance metric, as described herein.
[0047] The distance metric is preferably a mathematical combination of a set of relevant high-level features that are transformed from a low-level feature space, e.g., the statistical domain model and the natural language parse information. The transformation preferably enables a knowledge-based distance-metric to be built, i.e., one that uses knowledge about the structure and meaning of the sentence. Such a distant metric may enable the ML algorithm to powerfully differentiate between sentence categories.
[0048] One approach for building the distance metric is to leverage recent advances in Natural Language processing technology. Other domain information may also be leveraged wherever possible. By combining one or more NL processing techniques, along with statistical models and any other available information, a single distance metric may be constructed that sufficiently differentiates between sentence categories.
[0049] In one illustrative embodiment, several NL techniques and statistical information are used to provide similarity factors for the distance metric. Some of the NL techniques may include parts-of-speech tagging and syntactic chunks. Parts-of-speech tagging identifies nouns, adverbs, adjectives, etc. in the sentences. Syntactic chunks identify such things as noun phrases, prepositional phrases and verb phrases.
[0050] FIG. 4 is a schematic diagram showing the sentence diagram of a sentence using a natural language parser. A part-of-speech tagger may identify and tag the various parts-of-speech in each sentence. The parts-of-speech may include, for example, nouns, adverbs, adjectives, articles, etc. Referring to FIG. 4 , in the sentence “Check the wiring of the ground signal”, the word “the” is a determiner, “of” is a preposition, the words “wiring”, “ground” and “signal” are nouns, and the word “check” is a verb. A parser may provide additional natural language syntactic information such as noun, verb and prepositional phrases, and also semantic information such as subject and object. A parser may provide an output of “(check_VB) [ [ the_DT wiring_NN ] of_PP [the_DT ground_NN signal_NN ] ], where each of the words has a suffix that identifies the part-of-speech of each word, each noun phrase is enclosed in square brackets and each verb phrase is enclosed in parentheses.
[0051] A word model may also be developed and used. A word model is a statistical analysis of word-to-category correlations. Each of these techniques help identify relevant parts of the sentences. By using one of more of these NL techniques, a single distance metric may be constructed that sufficiently differentiates between sentence categories.
[0052] A number of illustrative methods for calculating similarity factors for use in some distance metric calculations are now discussed. FIG. 5 is a diagram showing the calculation of a similarity factor between two sentences using a string overlap function. The illustrative string overlap function calculates how many words match between a first and second sentence, as a percentage of the number of words in the longest of the two sentences. Referring to FIG. 5 , and for illustration purposes, a first sentence may be “Do a check and repair the unserviceable wiring”, and a second sentence may be “Repair the above wiring” The words that match between the two sentences are “Repair”, “the” and “wiring”. The maximum number of words in the longest sentence is eight (8). Thus, the similarity factor produced by the string overlap function is ⅜, or 0.375.
[0053] To reduce the effects of spelling errors, stylistic changes, abbreviations, and other variations, a longest common subsequence (LCS) technique may be used when comparing the word in each sentence. The LCS technique may, for example, accept a pair of word as equal if each of the words has at least four (4) letters and the words share more than 80% of their characters.
[0054] A keyword match function may contribute to the similarity metric by comparing only words of the same part-of-speech “type”; hence the similarity of the meaning of the two sentences can be more accurately determined, as further shown and described below. FIG. 6 is a diagram showing the calculation of a similarity factor between two sentences using an illustrative keyword match function. A keyword is a particular part-of-speech “type” as described above. The illustrative keyword match function calculates how many keywords match between the two sentences, as a percentage of the maximum number of keywords in the longest sentence. A keyword match occurs when the keyword types are equal and the words match according to the longest common subsequence (LCS) technique described above. To increase the importance of certain part-of-speech “types”, weights can be applied to each of the keyword types (e.g., noun, verb, adverb, etc).
[0055] Referring to FIG. 6 , and for illustration purposes, the same two sentences described above with reference to FIG. 5 are used. Namely, and after being processed by the part-of-speech tagger, the first sentence recites “Do_VB a_DT check_NN and_CONJ repair_VB the_DT unserviceable_ADJ wiring_NN”, and the second sentence recites “Repair_VB the_DT above_ADJ wiring_NN”. In this example, verbs are assigned a weight of 2.0, nouns are assigned a weight of 1.0, and all other word types are assigned a weight of zero. Since the _DT, _CONJ and _ADJ word types have a weight of zero, they do not contribute to the overall similarity factor produced by the keyword match function. The words that match between the two sentences and have the same word type are thus “Repair_VB” and “wiring_NN”. In addition, the maximum number of keywords in either sentence (with a non-zero weight) is four (4). Thus, the similarity factor produced by the illustrative keyword match function is (2.0+1.0)/4, or 0.75.
[0056] A phrase match function may contribute to the similarity metric by comparing only phrases of the same type, hence the similarity of the meaning of the two sentences can be more accurately determined, as further shown and described below. FIG. 7 is a diagram showing the calculation of a similarity factor between two sentences using an illustrative phrase match function. The illustrative phrase match function calculates how many phrases match between the two sentences, as a percentage of the maximum number of phrases in the longest sentence. Like the keyword match function discussed above, a phrase match may occur when the phrase types are equal and the phrases match according to a longest common subsequence (LCS) technique, as described above. To increase the importance of certain phrase “types”, weights can be applied to each of the phrase types (e.g., noun-phrase, verb-phrase, prepositional-phrase, etc). Within each phrase, the overlap or keyword match functions may be applied.
[0057] Referring to FIG. 7 , and for illustration purposes, the same two sentences described above with reference to FIG. 5 and FIG. 6 are used. Namely, and after being processed by the part-of-speech and grammar tagger, the first sentence recites “(Do_VB) [a_DT check_NN] and_CONJ (repair_VB) [the_DT unserviceable_ADJ wiring_NN]”, and the second sentence recites “(Repair_VB) [the_DT above_ADJ wiring_NN]”. In this example, verb-phrases are assigned a weight of 2.0, noun-phrases are assigned a weight of 1.0, and all other phrase types are assigned a weight of zero. Also in this example, a longest common subsequence (LCS) technique is used only requiring 80% of the characters to be the same. The two verb phrases from the first sentence are compared with the one verb phrase from the second sentence; the longest common subsequence (LCS) algorithm is applied over the keywords in each phrase, yielding a match between the “repair” words (total weight 2.0). The two noun phrases from the first sentence are compared with the one noun phrase from the second sentence; the longest common subsequence (LCS) algorithm is applied over the keywords in each phrase, yielding a match between the “wiring” words (total weight 1.0). The maximum number of phrases in either sentence (with a non-zero weight) is four (4). Thus, the similarity factor produced by the illustrative phrase match function is (2.0 (the weight of verb phrases)*2.0 (one verb match found in the phrases, with weight of 2.0)+1.0 (the weight of noun phrases)*1.0 (one noun match found in the phrases))/4, or 1.25.
[0058] Another illustrative method for calculating a similarity factor between two sentences uses a model match function. In this particular embodiment, the model match function uses a statistical model that is based on word-to-category statistics based on the observation that certain words or phrases tend to correlate strongly to a particular sentence category. For example, the words “repair”, “replace” and “install” may correlate strongly to “fault” type statements in a maintenance manual, while the words “test”, “procedure” and “check” may correlate strongly to no-fault type statements.
[0059] FIG. 8 is table showing some illustrative correlation factors for several model words including the words repair, check, replace and install. The word “repair” has a correlation factor of 1.00 to “fault” type statements. The word “check” has a correlation factor of 0.95 to “no-fault” type statement. The word “replace” has a correlation factor of 0.98 to “fault” type statements, and the word “install” has a correlation factor of 1.00 to “fault” type statements. The statistical model may capture NL information to distinguish between different uses of the same word (e.g. “service” may appear as a noun or a verb).
[0060] Using these correlation factors, an illustrative model match function may return a similarity value between the two sentences. In doing so, a similarity value is initialized to zero, and is increased by the correlation factor of a model word when both the first sentence and the second sentence include the model word, and is decreased by the correlation factor of a model word when only one of the first or second sentences include the model word. The similarity value may be divided by the number of model words detected in either of the sentences, if desired. By decreasing the similarity value when only one of the first and second sentences include the model word, a negative probability or correlation is applied. This method has proven to be fairly powerful in categorizing sentences.
[0061] FIG. 9 is a diagram showing the calculation of a similarity factor between two sentences using the correlation factors shown in FIG. 8 . For illustration purposes, the same two sentences described above with reference to FIG. 5 , FIG. 6 and 7 are used. Both sentences include the word “repair”, but only the first sentence includes the word “check”. Neither of the sentences include the words “replace” or “install”. Thus, the illustrative model match function may return a value of (repair
(+1)+check (−0.95))/2=0.05/2=0.025.
(+1)+check (−0.95))/2=0.05/2=0.025.
[0062] The various sentence similarity values discussed above may be combined to calculate a distance metric. FIG. 10 is a diagram showing the calculation of an illustrative distance metric. The illustrative distance Dpq 46 between two sentences p and q is determined by a weighted sum of the votes of each underlying technique. In FIG. 10 , “d1” may correspond to the similarity factor calculated using the string overlap function, “d2” may correspond to the similarity factor calculated using the keyword match function, “d3” may correspond to the similarity factor calculated using the phrase match function, and “d4” may correspond to the similarity factor calculated using the model match function. Each of the similarity factors is weighted by a corresponding weighting factor wi.
[0063] As shown at 48, some of the similarity factors di may rely on some of the information provided by one or more of the other similarity factors dj. When so provided, separate weights wij may be applied to each of the techniques, if desired. In one example, the similarity factor “d3” for the phrase match function may rely on the similarity factor “d2” of the keyword match function, if desired. Experiments were conducted with several of the above ML techniques, and obtained the best results from a nearest-neighbor approach, using k=10 neighbors and a weighted distance voting strategy.
[0064] FIG. 11 is a table showing the calculation of the distance metric Dpq 46 of FIG. 10 using the similarity factors calculated with reference to FIGS. 5 , 6, 7, and 9. The two sentences used to calculate the illustrative similarity factors in FIGS. 5 , 6, 7, and 9 are reproduced along the top of FIG. 11 . The first column of FIG. 11 recites the four illustrative techniques described above, including the string overlap function 50, the keyword match function 52, the phrase match function 54 and the model match function 56. The second column, labeled “score”, recites the similarity factors calculated for each of the NL techniques. The third column 60 includes a first set of illustrative weights wi for each of the NL techniques, wherein all the weights are the same and equal to one. Using these weights, the distance metric Dpq 46 of FIG. 10 produces a distance metric value of 2.40, as shown at 62.
[0065] The fourth column 64 includes a second set of illustrative weights wi for each of the NL techniques, wherein the string overlap function has a weight of 0.0, the keyword match function and model match function of a weight of 1.0, and the phrase match function has a weight of 2.0. Using these weights, the distance metric Dpq 46 of FIG. 10 produces a distance metric value of 3.275, as shown at 66. Finally, the fifth column 68 includes a third set of illustrative weights wi for each of the NL techniques, wherein the string overlap function has a weight of 1.0, the keyword match function and model match function of a weight of 2.0, and the phrase match function has a weight of 0.0. Using these weights, the distance metric Dpq 46 of FIG. 10 produces a distance metric value of 1.9, as shown at 70. During the learning process described above, the weights wi for each of the techniques are preferably updated to reflect the current categorization of sentences.
[0066] The above-described NL techniques are only meant to be illustrative and that numerous other possible techniques can be used in the present invention. For example, to improve the robustness of the system, it is contemplated that related words may be identified and added to a sentence. The related words may include, for example an original word with the prefix or suffix removed, various tenses of the original word, and/or different words with similar meanings as identified by, for example, a domain thesaurus. In addition, it is contemplated that one or more named entities may be extracted from the first sentence and the second sentence; the named entities may include a complex groups of nouns. The named entities may then be treated as a single object when performing word or phrase comparisons. In some embodiments, one or more stop words may also be excluded from the first and second sentences. The stop words may be identified in a predefined stop list, and may include common words that are not likely to help identify the similarity of the first and second sentences.
[0067] FIG. 12 is a graph showing the accuracy of sentence categorization versus the number of sentences used for training for an Airbus A320 legacy maintenance manual. In this experiment, a complete set of 67,000 sentences were obtained from an Airbus A320 maintenance manual. For this data set, the present invention achieved over 99% accuracy after training the sentence with 820 sentences (with 20 sentences per interaction with the user), or approximately one hour of training time. Using current manual methods, only an 80% accuracy level was achieved with 400 hours of conversion time.
[0068] Having thus described the preferred embodiments of the present invention, those of skill in the art will readily appreciate that the teachings found herein may be applied to yet other embodiments within the scope of the claims hereto attached.
Claims
What is claimed is:
1. A method for categorizing a sentence into one of two or more sentence categories, wherein each sentence category has at least one associated categorized sentence, and wherein the sentence and each categorized sentence has a number of words and/or phrases, the method comprising:
calculating one or more similarity factor between the sentence and at least one categorized sentence in each of the two or more sentence categories based on selections of words in the sentence and the least one categorized sentence in each of the two or more sentence categories; generating a distance metric for each of the sentence categories using one or more of the similarity factors, the distance metric representing a measure of the similarity between the sentence and the least one categorized sentence in each of the two or more sentence categories;
categorizing the sentence into one of the sentence categories based on the distance metrics; and
providing a result that at least in part reflects the categorization of the sentence to a user and/or computer.
2. A method according to claim 1, further comprising the step of processing the sentence and the at least one categorized sentence with a natural language algorithm to identify and assign the words in each sentence to one or more natural language categories.
3. A method according to claim 1, wherein one or more similarity factors provide a measure of similarity according to natural language information.
4. A method according to claim 1, wherein one or more similarity factors provide a measure of similarity according to statistical methods.
5. A method according to claim 1, wherein one or more similarity factors provide a measure of similarity according to domain-specific information.
6. A method according to claim 1, wherein one or more similarity factors provide a measure of similarity according to other relevant information.
7. A method according to claim 1, wherein one or more similarity factors provide a measure of similarity according to a combination of any of natural language, statistical, or other technique.
8. A method according to claim 1, where the one or more similarity factors are combined to generate an overall estimate of the similarity of the sentences.
9. A method according to claim 1, wherein the one or more similarity factors result from one or more of a string overlap function, a keyword match function, a phrase match function, or a model match function.
10. A method according to claim 9, wherein the string overlap function returns how many words in the sentence match a word in the at least one categorized sentence, as a percentage of the number of words in the longest of the sentence and the at least one categorized sentence.
11. A method according to claim 9, wherein the keyword match function returns how many words in the sentence match a word in the at least one categorized sentence with both words having a common word type, as a percentage of the maximum number of keywords in the sentence and the at least one categorized sentence.
12. A method according to claim 11, wherein the common word type is a natural language category, including a noun, a verb, an adverb, a preposition, an object, a direct object, and a subject.
13. A method according to claim 9, wherein the phrase match function returns a sentence similarity value, wherein the sentence similarity value is related to the similarity of one or more selected phrases that exist in both the sentence and the at least one categorized sentences, with both phrases having a common phrase type.
14. A method according to claim 13, wherein the phrase type is one of a noun phrase, a verb phrase, and a prepositional phrase.
15. A method according to claim 9, wherein the model match function includes at least one set of model words, each model word having a corresponding correlation factor that correlates the model word to a category or domain.
16. A method according to claim 15, wherein the model match function returns a sentence similarity value, wherein the sentence similarity value is increased by the correlation factor of a model word when both the sentence and the at least one categorized sentence include the model word, and is decreased by the correlation factor of a model word when only one of the sentence and the at least one categorized sentences include the model word.
17. A method according to claim 1, further comprising the step of identifying words related to those in the sentences, and using the related words to complete the similarity factor.
18. A method according to claim 11, wherein the related words are identified using a domain thesaurus.
19. A method according to claim 17, wherein the related words are identified by removing a suffix or a prefix from selected words in the sentence and the at least one categorized sentences.
20. A method according to claim 1, further comprising the step of extracting one or more named entities from the sentence and the at least one categorized sentences to allow complex groups of nouns to be treated as a single object.
21. A method according to claim 1, further comprising the step of excluding from the comparing step selected words that are identified in a predefined stop list.
22. A method for determining a distance metric that indicates the similarity of a first sentence and a second sentence, wherein the first sentence and the second sentence are extracted from one or more documents, the method comprising the steps of:
comparing the first sentence and the second sentence using a first comparing function, the first comparing function including a string overlap function and providing a first similarity indicator, the string overlap function returning how many words in the first sentence match a word in the second sentence, as a percentage of the number of words of the longest of the first sentence and second sentence;
comparing the first sentence and the second sentence using a second comparing function, the second comparing function providing a second similarity indicator;
calculating the distance metric by combining the first similarity indicator and the second similarity indicator; and
providing a result that at least in part reflects the distance metric to a user and/or computer.
23. A method according to claim 22, wherein a word in the first sentence is determined to match a word in the second sentence if the word in the first sentence and the word in the second sentence each exceed a minimum number of characters, and the word in the first sentence shares a predetermined minimum number of characters with the word in the second sentence.
24. A method according to claim 23, wherein a word in the first sentence is determined to match a word in the second sentence if the word in the first sentence and the word in the second sentence each have a minimum of four characters, and the word in the first sentence shares at least 80 percent of the characters with the word in the second sentence.
25. A method for determining a distance metric that indicates the similarity of a first sentence and a second sentence, wherein the first sentence and the second sentence are extracted from one or more documents, the method comprising the steps of:
comparing the first sentence and the second sentence using a first comparing function, the first comparing function including a keyword match function and providing a first similarity indicator, wherein the keyword match function returns how many words in the first sentence match a word in the second sentence with both words having a common word type, as a percentage of the maximum number of keywords in the first sentence and second sentence;
comparing the first sentence and the second sentence using a second comparing function, the second comparing function providing a second similarity indicator;
calculating the distance metric by combining the first similarity indicator and the second similarity indicator; and
providing a result that at least in part reflects the distance metric to a user and/or computer.
26. A method according to claim 25, wherein a word in the first sentence matches a word in the second sentence if the word in the first sentence and the word in the second sentence each exceed a minimum number of characters, and the word in the first sentence shares a predetermined minimum number of characters with the word in the second sentence.
27. A method according to claim 26, wherein the word type of each word is assigned by a part-of-speech tagger function.
28. A method according to claim 25, wherein the keyword match function only attempts to match those words that have a selected word type.
29. A method for determining a distance metric that indicates the similarity of a first sentence and a second sentence, wherein the first sentence and the second sentence are extracted from one or more documents, the method comprising the steps of:
comparing the first sentence and the second sentence using a first comparing function, the first comparing function including a phrase match function and providing a first similarity indicator, wherein the phrase match function determines a phrase similarity value for each phrase pair by identifying how many words in a first phrase match a word in a second phrase, as a percentage of the maximum number of words in the first phrase and second phrase;
comparing the first sentence and the second sentence using a second comparing function, the second comparing function providing a second similarity indicator;
calculating the distance metric by combining the first similarity indicator and the second similarity indicator; and
providing a result that at least in part reflects the distance metric to a user and/or computer.
30. A method according to claim 29, wherein a weighting factor is applied to each phrase similarity value, depending on the phrase type, to compute the phrase match similarity factor.
31. A method according to claim 29, wherein a word in the first phrase matches a word in the second phrase when the word in the first phrase and the word in the second phrase each exceed a minimum number of characters and share a predetermined minimum number of characters.
32. A method for categorizing sentences in a document, comprising:
processing a number of selected sentences, said processing step assigning each of the number of selected sentences to one or more predefined categories without using a user entered query, the particulars of the assignment of the number of selected sentences being dependent on a number of operating parameters;
displaying a correspondence between the one or more predefined categories and the selected sentences;
allowing a user to change the assigned category for selected sentences;
updating one or more of the operating parameters to reflect the change in the assigned categories; and repeating the processing, displaying, allowing and updating steps until a desired accuracy level is achieved.
33. A method according to claim 32, further comprising the step of processing all sentences in the document once the repeating step is completed.
34. A method for providing a measure of similarity between a first sentence and a second sentence, each sentence having a number of words, the method comprising:
comparing one or more words of the first sentence with one or more words of the second sentence, the comparing step returning how many words in the first sentence match a word in the second sentence, as a percentage of the number of words in the longest of the first sentence and second sentence;
providing a result that is related the comparing step to a user and/or computer.
35. A method according to claim 34, wherein a word in the first sentence is determined to match a word in the second sentence if the word in the first sentence and the word in the second sentence each exceed a minimum number of characters, and the word in the first sentence shares a predetermined minimum number of characters with the word in the second sentence.
36. A method for providing a measure of similarity between a first phrase and a second phrase, each phrase having a number of words, the method comprising:
comparing one or more words of the first phrase with one or more words of the second phrase, the comparing step returning how many words in the first phrase match a word in the second phrase, as a percentage of the number of words in the longest of the first phrase and second phrase; and
providing a result that is related the comparing step to a user and/or computer.
37. A method according to claim 36, wherein a word in the first phrase matches a word in the second phrase when the word in the first phrase and the word in the second phrase each exceed a minimum number of characters and share a predetermined minimum number of characters.
38. A method of comparing a first sentence and a second sentence, the method comprising the steps of:
comparing a word in the first sentence with a word in the second sentence; and
determining that the word in the first sentence matches the word in the second sentence if the word in the first sentence and the word in the second sentence each have a minimum of four characters, and the word in the first sentence shares at least 80 percent of the characters with the word in the second sentence; and
providing a result to a user and/or computer.
Patent Citations (15)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4916614(A) | 1990-04-01 | Kaji et al. | Applicant |
| US5056021(A) | 1991-10-01 | Ausborn | Applicant |
| US5251131(A) | 1993-10-01 | Masand et al. | Applicant |
| US5418948(A) | 1995-05-01 | Turtle | Applicant |
| US5555408(A) | 1996-09-01 | Fujisawa et al. | Applicant |
| US5559693(A) | 1996-09-01 | Anick et al. | Applicant |
| US5682539(A) | 1997-10-01 | Conrad et al. | Applicant |
| US5893092(A) | 1999-04-01 | Driscoll | Applicant |
| US5963940(A) | 1999-10-01 | Liddy et al. | Applicant |
| US5987404(A) | 1999-11-01 | Della Pietra et al. | Applicant |
| US6076058(A) | 2000-06-01 | Chengalvarayan | Applicant |
| US6076088(A) | 2000-06-01 | Paik et al. | Applicant |
| US6078914(A) | 2000-06-01 | Redfern | Applicant |
| US6088692(A) | 2000-07-01 | Driscoll | Applicant |
| WO1/33414(A1) | 2000-10-01 | Examiner |
Non-Patent Literature (16)
- “Creating Multiple Choice Type Questions”, URL address: http://itsinfo.tamu.edu/workshops/handouts/pdf—handouts/multiple—choice—questions.pdf, May 9, 2001.Examiner
- Hearst, Marti A. et al, “Subtopic Structuring for Full-Length Document Access.”□□SIGIR 1993, pp. 59-68.Examiner
- G. Salton, et al., “Automatic Analysis, Theme Generation and Summarization of □□Machine-Readable Texts,” Science, vol. 264, Jun. 3, 1994□□.Examiner
- G. Salton, C. Buckley, “Automatic Text Structuring and Retrieval—Experiments in Automatic □□Encyclopedia Searching” Proc. of the 14th International ACM/SIGIR Conf. on Research and Development in Information Retrieval, pp. 21-30 ACM New York, 1991.Examiner
- Salton, et al., “Global Text Matching for Information Retrieval,” Aug. 30, 1991, Science, vol. 253, pp. 1012-1015.Examiner
- U.S. Appl. No. 09/522,483, filed Mar. 10, 2000, entitled “Trainable, Extensible, Automated Data-To-Knowledge Translator”.Applicant
- Claire Grover et al., http://www.1tg.ed.ac.uk/software/ttt/tttdoc.html, “TTT: Text Tokenisation Tool”, dated prior to Jun. 29, 2001, 67 pages.Applicant
- http://www.inxight.com/products—eu/index.html, “Inxight Software—Products for End Users” dated prior to Jun. 29, 2001, 1 page.Applicant
- http://www.inxight.com/products—sp/index.html, “Inxight Software—Products for Software Producers”, dated prior to Jun. 29, 2001, 2 pages.Applicant
- http://www.inxight.com/products—wb/index.html, “Inxight Software—Products for Web Builders” dated prior to Jun. 29, 2001, 2 pages.Applicant
- http://bobo.link.cs.cmu.edu/dougb/playground.html, “The Natural Language Playground”, dated prior to Jun. 29, 2001, 2 pages.Applicant
- http://dir.yahoo.com/Computers—and—Internet/Software/Natural—Language—Processing/, “Yahoo! Computers and Internet > Software > Natural Language Processing”, dated prior to Jun. 29, 2001, 1 page.Applicant
- http://bobo.link.cs.cmu.edu/grammar/html/dict/summarize-links.html, “Summary of Link Types”, dated prior to Jun. 29, 2001, 7 pages.Applicant
- http://bobo.link.cs.cmu.edu/grammar/html/, “Link Grammar”, dated prior to Jun. 29, 2001, 2 pages.Applicant
- http://www.attensity.com/data/index—support/menu.html, “Untitled Document” dated prior to Jun. 29, 2001, 1 page.Applicant
- http://www.cs.utexas.edu/users/kbarker/thesis/, Ken Barker, Ph.D. Thesis Page, “Semi-Automatic Recognition of Semantic Relationships in English Technical Texts”, dated Jul. 18, 1998, 151 pages.Applicant