System for Detecting and Correcting Contextual Errors in a Text Processing System

This invention relates to text processing systems and more particularly to methods in text processing systems for automatically detecting and correcting contextual errors related to confusable words in text.

With the advent of office systems and the improvements in computing power and storage, it becomes appropriate to consider various applications which offer the operator the benefit of the system scanning documents to find anomalies and possible errors in the text. It has been posed in the prior art that computer systems review documents and compute a "foggyness factor" for the respective sentences, paragraphs and chapters. This advises the operator on a high level how consistent and in a bulk manner infers how lucid a composition is. Such "foggyness factors" derive their inference based on a correlation against sentence length, number of punctuations, length of words and other superficial factors. The technology does not necessarily say something is wrong, but rather how far from a postulated ideal the author's composition style is.

Another known method for examining text for errors is to scan the document and check each word against a spelling dictionary to detect misspelled words. A technology suitable for miniprocessor implementation and consistent with office system technology is disclosed in the "Digital Reference Matrix For Word Verification", U.S. Pat. No. 3,995,254, issued Nov. 30, 1976 to W. S. Rosenbaum. This technology determines if a word is misspelled but not whether it is misused. No intelligence is exercised about the word, rather only about its character content.

A simple method for examining a document with contextual significance is to examine each word in the document against a dictionary composed of homophones or confusable words. Each time a word in the dictionary is encountered in the document, it is flagged and brought to the operator's attention for manual review, and if necessary, corrective intervention.

However, a major detraction of such content analysis is that for such common homophones or confusable forms as, there/their, fore/four/for, to/too/two, etc. the noise level of successive flagged words, regardless of the obviousness of their correct usage, is more of a nuisance factor than a remedial composition aid. The ability to automatically detect homophones and confusion prone words that appear to be syntactically mismatched and only prompt the operator with those suspicious cases has not been addressed in the prior art.

It has been discovered that improved text proofing for purposes of word use validation in a text processing system can be achieved by coupling a specialized dictionary of sets of homophones and confusable words where each word has related to it a set of word di-gram or N-gram conditions whereby proper or improper usage of the words can be statistically determined. This system is dictionary-driven and the operator's composition is reviewed word-by-word against the dictionary of homophones and confusable words. When a match is found against the dictionary, the related list of syntactic word N-grams is examined relative to the immediate region of text where the subject homophone or confusable word is found. If the syntax in the immediate context of the homophone or confusable word conflicts with the prestored N-gram word conditions, the homophone or confusable word is flagged for the operator on the system display. In an appropriate region of the display screen, the definition of the flagged homophone or confusable word is shown along with the spelling of the possibly intended alternate forms with their respective definitions. The operator can then examine the word used and the possible alternatives indicated with their respective definitions and make the final determination as to whether an error has been made and if a correction of the text is required. If correction is required, the operator can cause the error word to be replaced by the desired word by positioning the display cursor under the desired word and depressing the appropriate key on the system keyboard.

FIG. 1 is a block diagram of an interactive text processing system embodying the present invention;

FIG. 2 is a functional diagram of the microprocessor shown in FIG. 1.;

FIG. 3 is a functional diagram illustrating the data flow path between portions of the memory and the microprocesor and the display refresh buffer;

FIG. 4 is a diagrammatic view of the display shown in FIG. 1;

FIG. 5 illustrates the presentation of a text document on the display screen of FIG. 4; and

FIG. 6 illustrates the display of the document as it would appear on the display device during operation of the proofreading function of the present invention.

The invention will now be described as embodied in an interactive text processing system of the type shown in FIG. 1. As shown in FIG. 1, the text processing system illustrated therein comprises a keyboard 10, a microprocessor 11, a display refresh buffer 12, a display device 14, a printer 15, and an auxiliary direct access storage device 16 such as a disk or diskette drive. A clock 17, for keeping the various components of the system in synchronism, is also shown in FIG. 1 and is effectively coupled to each of the units.

Keyboard 10 comprises a normal set of graphic symbol keys such as letters, numbers, punctuation marks, and special character keys, plus text format or control keys including carriage return, tab, index, etc. In addition, the keyboard 10 includes a second set of control keys for issuing special control commands to the system. The control keys include cursor movement keys, keys for setting the keyboard 10 into a number of different modes, etc.

The keyboard 10 is connected to the microprocessor 11 by means of a bus 20. The microprocessor 11, as shown in FIG. 2, comprises an input port 21, an output port 22, a random access memory 23, and a process execution unit 24.

Functionally, memory unit 23 stores both instructions and data in specified sections which will be described in more detail later in the specification. Data is entered into memory 23 from the keyboard 10 as bytes of binary information through input port 21. As shown in FIG. 3, the section of RAM 23 which receives the keystroke data from the keyboard 10 is designated keystroke queue 26. Data to be displayed is transferred by a series of instructions from queue 26 to the text buffer section 27 and then to the display refresh buffer 12 through output port 22 of the microprocessor 11. This is achieved in a conventional way by the microprocessor 11 executing a series of move instructions.

The microprocessor 11 may be an IBM Series 1, INTEL model 8086, or any of the recognized functionally equivalent, currently available microprocessors.

The display refresh buffer 12 is shown as a separate buffer connected between the output port 22 and the display device 14. Buffer 12, in practice, is normally a part of the display device 14 and functions to control the generation of characters on the screen of the display device 14 by exercising on-off control of the beam as it traces a series of horizontal lines across the screen.

The output port 22 also supplies data stored in memory 23 to the printer 15 and disk storage unit 16, each of which may have their own internal buffers which are not shown. Commands to transfer data from the random access memory 23 to the printer 15 or storage unit 16 are sent to the microprocessor 11 by the operator from the keyboard 10.

Printer 15 may be any suitable printer known in the art. In most text processing systems, the printer 15 is basically a standard input/output terminal printer having a type ball element or a daisy-wheel print element.

Disk storage 16 may also be any suitable disk storage device which is capable of storing serial by byte data supplied to it at determined sector address locations, each of which are randomly addressable by the microprocessor 11 to retrieve the data. Spatially related data supplied from disk drive 16 is stored in the display data area 28 of the memory 23 in encoded form. Another section of memory 23 shown in FIG. 3 is the display format buffer area 29 which is involved in the handling of spatially related data in decoded form in accordance with the method of the present invention.

FIG. 4 is a schematic representation of the screen 40 of display device 14. As shown in FIG. 4, the screen 40 has, for example, the capability of displaying 25 lines of characters where each line consists of 80 character column positions. In practice, one character position consists of a matrix of dot positions or picture elements sometimes referred to as pels. A typical character matrix for a display of the type represented by device 14 would be a matrix of six wide by ten high pels, which has been designated by reference character 42 in FIG. 4. The interaction of the refresh buffer 12 and the display 14 is to convert the characters stored at a location in the buffer 12 to the corresponding character as formed in a 6.times.10 dot matrix at the equivalent location on the display screen 40. Display 14 generally is provided with its own set of electronics to achieve that conversion. The microprocessor 11 need only supply the address and load the buffer 12 with the appropriate characters.

The disk storage device 16 also is generally provided with its own set of electronics for converting a byte of data supplied from the display data area 28 of memory 23 through the output port 22 to a serial by bit stream of data to be recorded at a predetermined sector of the one addressed concentric recording track on the disk. Data from the device 16 is supplied to the microprocessor 11 serial by byte from the addressed sector and storage tracks when requested.

It will be understood that all of the above described functions and interactions involving the microprocessor 11 are achieved through suitable programs and data which are also stored in memory 23 shown as memory blocks 30, 31, 32, 33 and 34 of FIG. 2 and which are called into operation in response to data from the keyboard 10 or interrupt signals generated by the various components of the system shown in FIG. 1.

The preferred embodiment of the present invention comprises a set of instructions or programs for controlling a text processing system for reviewing the input screen of words or data segments against the entries in a confusable word list in memory 31 until a match is found. The match condition against a word from the list causes the processor 11 under control of the instructions in memory 30 to flag the word and apply rules associated with the word. This processing may involve interaction with the contents of the parts of speech information in memory 33 and/or with the special word lists in memory 32 and the RAM memory 23. The processing continues under the control of the instructions of this invention until a conclusion can be drawn as to whether conditions are met in the immediate region of the previously flagged word to accept its use. If the conditions are met then no further processing of the flagged word is performed. However, if the conditions are not met, the processor 30 will obtain from memory 34 the alternate words and definitions related to the flagged word and display them to the operator. Flagging the word may be done by displaying the word in reverse video, underscoring the word, boxing the word, etc.

The confusable words are stored in sets in the memory section 31. Table 1 shows a sample listing of confusable words which are stored in the memory 31. Each set of confusable words has a related set of definitions shown in Table 2 and stored in memory 34. The sets of confusable words and definitions represent popular usage and definitions for the words.

The rules for testing the confusable words are shown in Table 3 and stored in the memory section 32 as special words. The confusable word identified in text along with predefined sections of text preceding the confusable word and following the confusable word are examined to determine whether the confusable word is misused. It has been determined that a string of 11 words generally provides sufficient information to determine whether a word has been used properly. The confusable word is positioned as the "pivot" word in the string with 5 preceding and 5 following words. The rules in memory section 32 provide means for inspecting the text surrounding the pivot word for specific words, words ending in specific characters, words terminated by a blank, comma, or period, words which are a particular part of speech, capitalized words, numbers, or proper nouns.

Each set of confusable words has stored in the memory 32 rules specific to each member of the set. These follow the general rules in the memory 32 and serve as pointers to the applicable general rules for each word. The specific rules for a sample set of confusable words is shown in Table 4. The "-" designations in the rules indicates that the word following the equal sign is tested for preceding the confusable word. For example, -1="please" translates to "please accept" for the first word in Table 4. The "+" designation indicates that the term following the equal sign will be tested for following the confusable word. For example, "+1=PRONOUN means the word immediately following the confusable word (pivot word) will be examined to determine if it is a pronoun. This determination is made by examining the word against a dictionary of correctly spelled words stored in the RAM 23 in accordance with the teachings of U.S. Pat. No. 3,995,254 issued to Nov. 30, 1976 to W. S. Rosenbaum wherein the dictionary includes an additional code appended to each word to indicate its part of speech. The rule +1=#1 means that the word immediately following the pivot word will be examined to determine if it matches one of the words in set number 1 of Table 3.

As previously stated, a string of words surrounding the pivot word may be examined. This means that the rule may contain compound conditions. Compound conditions preceding the pivot word are designated as -1 followed by -2, -3, etc. For example, the last two terms in the rules for "course" are -1=`the` and -2=`in`. This translates to "in the course". Likewise, compound conditions following the pivot word are designated as +1 followed by +2, +3, etc. It follows that the rule condition may also be negative as well as positive. For example, satisfaction of a rule condition may require rejection of the pivot word's use rather than acceptance.

Referring now to FIG. 5, there is shown an example of a letter displayed on the screen 40 of the display 14. The display has highlighted the words "affect", "confidant" and "coarse" as possible misused words. In operation, the contextual error detection and correction function may be invoked in a text processing system through a conventional "menu" task selection function such as disclosed in U.S. Pat. No. 4,308,582, issued Dec. 29, 1981 to David A. Berger and assigned to International Business Machines Corp.

In a menu task selection function, the text processing system displays to the operator a menu of functions that the system can perform. The operator may then select one or a combination of such functions through operation of cursor control and select or enter keybuttons on the system keyboard 10. For example, functions to paginate, check spelling, hyphenate for formatting, and check contextual errors in a document could be invoked singularly or in any combination. As an alternative to menu task selection, the text processing system could be provided with a separate keybutton to invoke each of the functions that the system is capable of performing.

At any rate, once invoked, the contextual error routine sequentially scans each word in the text document and compares the words to the confusable words stored in the confusable word memory 31. A program listing for the main context analyzer routine is shown in Table 5 in pseudo code. The application and use of pseudo code is described in "An Introduction to Programming", (A Structural Approach Using PL/1 and PL/C-7), Second Edition, by Richard Conway and David Gries, Winthrop Publishers, Cambridge, Mass., Copyright 1975.

As was previously stated, this routine is entered following a menu setup procedure which is overseen by the system control program which operates in a manner similar to that described in U.S. Pat. No. 4,308,582, previously referenced. The main context analyzer routine of Table 5 is structured to control the processor 11 to access from the disk storage file 16 the confusable words file, special words list file, parts of speech file, and definitions file and store these files in the sections 31, 32, 33 and 34 of the random access memory 23. In conventional information processing systems, program instructions and data are typically stored in direct access storage devices such as diskette and disk files until needed in the system random access memory. However, where the system random access memory size permits, the program instructions and data may be permanently stored in system electronic read-only memory or stored in system random access memory during initial program load.

Once the confusable words, special words, parts of speech and definitions have been stored in the memory sections 31, 32, 33 and 34 respectively, the context routine scans the displayed document word-by-word in the display refresh buffer 12 and compares each word to the words in the confusable words list. When a word in a document is found to match a word in the confusable words list then the context routine invokes the rules check routine shown in Table 6 to determine whether the word is properly used by comparing the context of the word with the rules in the special words memory 32. For example, referring to FIG. 5, the word "affect" is potentially misused in the second paragraph of the letter on display screen 40. The rules for determining the proper usage of affect are shown in Table 4. A comparison of the rules shown in Table 4 and stored in memory section 33 to the context surrounding affect in FIG. 5 shows no matches. This causes the rules check subroutine to set a condition to reject the use of the word "affect" and return control to the context routine where the word is flagged on the display and in the parameter list as having unacceptable use. The flagging on the display screen takes the form of boldface so the word can be easily located on the display screen by the operator. Alternatively, the word may be overstruck, or underscored, or enclosed in a box, or any other suitable means of highlighting the word on the display screen.

The context routine continues to scan the document word-by-word until each word in the document has been checked against the confusable word list. As can be seen in FIG. 5, the words "confidant" in the third paragraph and "coarse" in the last paragraph are also highlighted (flagged) as being potentially misused.

When the end of the document is reached, the main context routine scans to determine if any confusable words were identified. If confusable words were identified as appearing in the document the context routine invokes the prompt preparation routine shown in Table 7 to display the confusable words and definitions for the potentially misused word. Referring to FIG. 6, the confusable words "affect" and "effect" along with their short definitions are shown in bold-face and surrounded by a dotted line rectangle following the second paragraph of the document. The operator can select the proper word for substitution into the document in place of the highlighted word by placing the display cursor underneath the word to be selected and pressing an enter key on the keyboard 10. The cursored word will then be substituted for the highlighted word in the text document and the program routine will proceed to display the confusable words and definitions for the next highlighted word in text. The routine then continues processing the highlighted words until the end of the document is reached.

While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the invention.