System and method for providing user-directed constraints for handwriting recognition

[0002] 1. Technical Field

[0003] The present application relates generally to machine recognition and, more particularly, to system and methods for providing user-directed recognition constraints, wherein a user interface allows a writer to select one or more recognition constraints which temporarily modify default recognition parameters so as to decode uncharacteristic and/or special data with increased accuracy.

[0004] 2. Description of the Related Art

[0005] In general, conventional machine recognition systems recognize input data by decoding the data using a plurality of trained models. For instance, speech recognition systems decode input speech utterances using trained speech models to recognize spoken words. Likewise, handwriting recognition systems recognize input handwriting data using trained character models to convert the handwriting data to machine printed text.

[0006] One example of a handwriting recognition system is the CROSSPAD™ personal digital notepad (PDN) device (marketed by the A.T. Cross Company) which is sold with International Business Machines Corporation's INK MANAGER™ handwriting text recognition software package. The CROSSPAD device allows a user to produce handwritten records on a standard paper notepad (which is placed over a digitizing tablet) using an electronic inking stylus (i.e., pen). A data stream representing the handwriting strokes is generated by the digitizing tablet (simultaneously with the inking) by detecting RF emissions from the stylus, thereby capturing and storing an electronic carbon copy of the handwritten record. The handwriting data that is electronically recorded by the CROSSPAD device can be subsequently uploaded to a computer and processed by the INK MANAGER system which is stored in the computer.

[0007] Typically, machine recognition of input data (e.g., speech or handwriting) is performed using predetermined default recognition parameters which are automatically applied by the recognition engine for decoding the input data. These default recognition parameters may be those default parameters which are originally programmed in the system (i.e., “out-of-the-box” default recognition parameters). In addition, the default recognition parameters may also comprise user-selected default recognition parameters (i.e., user preferences) which modify and/or replace the original out-of-the-box default parameters. For instance, handwriting recognition systems as the INK MANAGER system allow the user to initially indicate, for example, the user's normal style of writing (e.g., a user preference) prior to using the system. In particular, before using the INK MANAGER system, the user will be prompted to select (if the user desires) one of the following writing style preferences: pure discrete, pure upper case, and cursive (which includes all other handwriting such as printing and script mixed). During recognition, the INK MANAGER system will automatically apply any default parameters associated with a previously selected user preference. The INK MANAGER system includes other default settings to determine which, if any, word lists will be used for decoding handwriting.

[0008] In most situations, utilizing the default recognition parameters will result in optimal data recognition. In certain circumstances, however, utilizing the default recognition parameters may result in sub-optimal recognition accuracy. For example, with regard to handwriting recognition, such special circumstances occur when a user writes in an uncharacteristic manner (e.g., when a user writes an address using all capital letters rather than the user's typical style of writing addresses with printed mixed case). Decoding such uncharacteristic handwriting data using default recognition parameters can result in poor recognition accuracy.

[0009] There is a need, therefore, for a system and methods for use with machine recognition that allow a user to temporarily modify default recognition parameters by selecting one or more recognition constraints to decode uncharacteristic and/or special data so as to obtain increased recognition accuracy.

[0010] The present application is directed to system and methods which can be implemented in a machine recognition device to provide a user (via a user interface) the option of selecting one of a plurality of recognition modes including a “default recognition” mode and a “constrained recognition” mode. In the default recognition mode, the recognition engine utilizes predetermined default recognition parameters (e.g., out-of-the-box and/or user-selected default parameters) to decode selected data. In the constrained recognition mode, the user can select one or more of a plurality of recognition constraints which are temporarily applied by a recognition engine for decoding uncharacteristic and/or special data. When the user selects a particular recognition constraint, default recognition parameters are temporarily modified to reflect the selected constraint. The recognition parameters associated with the selected constraint enable the recognition engine to temporarily utilize specific information for decoding the data, thereby providing increased recognition accuracy.

[0011] These and other features and advantages of the present system and method will become apparent from the following detailed description of preferred embodiments, which is to be read in connection with the accompanying drawings.

[0012] FIG. 1 is a block diagram of a handwriting recognition system which provides user-selected recognition constraints according to an embodiment of the present invention;

[0013] in FIG. 2 is a flow diagram of a method for selecting and applying recognition constraints in accordance with one aspect of the present invention;

[0014] FIG. 3 is a flow diagram of a method for selecting and applying recognition constraints in accordance with another aspect of the present invention;

[0015] FIGS. 4aand 4bare diagrams of exemplary cascading menus for providing handwriting recognition constraints in accordance with one aspect the present invention; and

[0016] FIG. 5 is a diagram of an exemplary dialog for selecting recognition constraints in accordance with one aspect of the present invention.

[0017] It is to be understood that, notwithstanding that the illustrative embodiments herein are directed to handwriting recognition, the present invention may be implemented in other machine recognition applications such as speech recognition.

[0018] Referring now to FIG. 1, a block diagram illustrates a system for providing user-selected recognition constraints for handwriting recognition in accordance with an embodiment of the present invention. The system 10 includes an input device 12 (or “data collection device”) for collecting data, e.g., handwritten text. The input device 12 can be any conventional device such as a digitizing tablet (i.e., pen-based computer) for real-time digitization and recognition of text which is directly written on the tablet, or an OCR (optical character recognition) scanner for inputting handwritten text. A memory 14 is included for storing handwriting data (hereinafter, alternatively referred to as “ink”) which is input via the input device 12 for “non-real-time” processing. The ink may be stored in any suitable conventional format such as a bitmap image (which is typically generated with “off-line” OCR scanning, where “off-line” refers to recognition of handwriting that has already been written on paper using, e.g., a pencil or pen) or as a sequence of X-Y coordinates which represent the location of a pen writing tip on the input device 12 at successive points in time (which is generated with an “on-line” pen-based computer, where “on-line” refers to recognition of handwriting written with an electronic stylus onto a tablet or pen computer). It is to be understood that memory 14 may also be memory (such as flash memory) included within the input device 12 for storing the ink prior to being uploaded to, and decoded by, a handwriting recognition engine 22. It is to be appreciated that “real-time” recognition may also be performed by decoding handwriting data received by input device 12 as it is being collected by the input device 12.

[0019] The system 10 further includes a user interface 16 which may be, for example, a computer display having a suitable GUI (graphic user interface) which allows a user to perform handwriting recognition in accordance with the present invention. The user interface 16 provides the user with the option of selecting a “default recognition” mode, whereby predetermined default recognition parameters 18 are applied to the recognition engine 22. The user interface 16 also provides the user the option of selecting a “constrained recognition” mode, whereby the user can select one or more available recognition constraints 20 that are temporarily applied to the recognition engine 22. Specifically, the recognition constraints 20 present the user with task-oriented choices, the selection of which determines the recognition parameters that are temporarily utilized by the recognition engine 22 for decoding uncharacteristic and/or special text.

[0020] A word list/grammar store 24 contains a plurality of word lists and associated grammars which may be applied to the recognition engine 22 based on the selected recognition mode. Each word list may contain, for example, words or numbers that are associated with a particular application (e.g., for each of the available recognition modes), or even for particular fields in a data-entry tool. By way of example, a word list for telephone numbers may contain all the numeric characters, as well as hyphen and parentheses characters. In addition, a word list for in recognizing dates may contain years, months and days both numerically and in word form. A grammar is a constraint which may be applied in connection with a word list. By way of example, a grammar associated with a “date” word list may be employed to limit the order in which each data field of a date (e.g., month-day-year) can be recognized. In particular, when recognizing a handwritten date, the first field “month” of the grammar would constrain the universe of available words in the word list to those pertaining only to months, the second field “day” of the grammar would constrain the universe of words in the word list to days, and, likewise, the third field “year” of the grammar would constrain the universe of allowable words in the word list solely to years.

[0021] A writer prototypes store 26 contains one or more writer prototypes (models). Each writer prototype contains one or more sets of character models (e.g., upper case character model, lower case character model). It is to be understood that the system of FIG. 1 presumes a trained handwriting recognition system. The trained recognition system may be a writer-independent system (i.e., a system which is trained to recognize the handwriting of many different writers), in which case the writer prototypes store may contain a writer-independent prototype (model). Alternatively, the recognition system may be a writer-dependent recognition system (i.e., a system trained to recognize the handwriting of a particular writer), in which case the writer prototypes store 26 may contain one or more writer-dependent prototypes, as well as a writer-independent prototype.

[0022] The recognition engine 22 will decode handwriting data using recognition parameters associated with the selected recognition mode. For example, if a user selects the default recognition mode to decode the user's handwriting, predetermined default recognition parameters (which can be out-of-the-box or user-specific default constraints (user preferences) as discussed above) are temporarily applied to the recognition engine. For instance, assume the user's default constraints (e.g., user preference) indicate that the user writes in cursive and that the writer-dependent prototype of the user is trained on cursive data. In the default recognition mode, the recognition engine 22 will decode the user's handwriting data using the writer-dependent prototype 26 associated with the user, as well as the relevant default word lists/grammars 24, regardless of whether the handwriting to be decoded includes upper case letters or numbers. This may result in decreased recognition accuracy.

[0023] On the other hand, if the user selects the “constrained recognition” mode, the recognition constraint(s) selected by the user will temporarily modify the predetermined default recognition parameters utilized by the recognition engine 22 and/or cause one or more relevant word lists/grammars to be retrieved from the word list/grammar store 24. For example, one type of constraint may cause the recognition engine 22 to temporarily utilize a particular writer-prototype 26 or use a particular character set (e.g., upper case character model) of a given writer prototype (“model constraint”). For instance, assume in the above example that the user (whose user preference indicates cursive writing) has written in upper case. The user can override his/her user preference by selecting an “upper case” model constraint. In this situation, assuming the user's writer-dependent prototype does not include an upper case character model, the selected “upper case” constraint may cause the recognition engine 22 to “swap out” the writer-dependent prototype for the writer-independent prototype and apply an upper case model included within the writer-independent prototype to decode the special text.

[0024] Another type of constraint may cause the recognition engine 22 to temporarily utilize a specific word list/grammar 24 during decoding (“word list constraint”). For instance, assume further (in the above example) that the handwriting data of the user contains one or more written dates. A word list constraint may be selected for decoding the written dates, whereby the recognition engine 22 applies the “dates” word list so as to constrain the universe of allowed words to the words contained in such word list. In this example, the selected word list constraint causes the writer's user preference to be adjusted (constrained) and not overridden.

[0025] Another type of constraint may be temporarily applied to limit the types of characters which may be recognized (“character set constraint”). For instance, if certain characters for a particular field or word type can only be, for example, letters “a” “b” or “c” or numbers “1”, “2” or “3,” a character set constraint may be selected to limit the characters which may be recognized for the field or word type.

[0026] Another type of recognition constraint may cause the recognition engine 22 to temporarily modify or adjust the default decoding algorithm and/or a specific parameter of the decoding process (“decoding algorithm constraint”). For instance, if the handwriting data contains a plurality closely written words, a constraint may be selected to modify the process by which the recognition engine 22 detects spaces, and thereby differentiates, between different words. Advantageously, the recognition constraints allow the recognition engine 22 to achieve higher decoding accuracy when recognizing handwriting data comprising special and/or uncharacteristic text. The decoding results of the system are output via the output device 28 (which may be a computer monitor or printing device).

[0027] Referring now to FIGS. 4a, 4band 5, diagrams illustrate several examples of user interfaces which may be utilized for handwriting recognition in accordance with the present invention. In particular, FIG. 4aillustrates one embodiment of a GUI comprising a cascading menu for selecting one or more available recognition constraints in the handwriting recognition system of the present invention. A recognition menu option (e.g., “Ink”) may be selected from a menu bar (or toolbar) to provide recognition functions. Such functions include, for instance, a menu function for allowing a user to mark the desired text to be recognized (e.g., “Select”), a menu function (e.g., “Recognize”) for selecting the default recognition mode, and a menu function (e.g., “Recognize as”) for selecting the constrained recognition mode, which cascades to a set of recognition constraints (e.g., “printed mixed case,” “printed all caps,” etc. as shown in FIGS. 4a).

[0028] Another embodiment for the GUI interface is illustrated in FIG. 4b., which is similar to the GUI of FIG. 4aexcept that rather than displaying the recognition constraint menu items as a second pull down (i.e., cascade), the recognition options (e.g., “Recognize Dates”) are displayed at the same level in the menu hierarchy as the default recognition mode (e.g., “Recognize”). menu item. A further embodiment of the GUI in accordance with the present invention is illustrated in FIG. 5, wherein a dialog box is produced (in response to selecting the constrained recognition mode) which allows the user to make easily understood choices. Advantageously, the GUI utilized in accordance with the present invention allows even a computer-naive user to easily find and select one or more of the available recognition constraints.

[0029] It is to be appreciated that the selection of desired recognition constraints may also be implemented via the input device 12. For example, with the CROSSPAD/INK MANAGER system described above, rather than decoding all the uploaded handwriting data at once, the user can mark (or select, highlight) specific handwritten text using the CROSSPAD device (as well as the INK MANAGER system), and then subsequently direct the INK MANAGER system to recognize the marked text. Specifically, as shown in FIG. 1, the input device 12 (e.g., CROSSPAD) contains a user interface 12bwhich allows a user to mark (or select) certain ink and “tag” (or label) selected recognition constraints 12ato the marked ink. The user interface 12bof the input device 12 may comprise, for example, an LCD display which displays 2 to 3 lines of text to indicate the state of the device or provide menu options for performing various functions such as selecting ink and labelling the selected ink with desired recognition constraints. Advantageously, the user can “tag” (label) the selected ink with desired recognition constraint, store the “tagged” ink in memory 14, and upload the “tagged” ink to the recognition engine 22 for processing at a later time.

[0030] Another example for illustrating user-selected recognition constraints will now be discussed with reference to FIG. 5. Assume the user has written down a set of part numbers for an inventory and wants them to be recognized. Assume further that, following the user's typical convention, the user has written the alphabetic portions of the part numbers using uppercase print. This special case can be addressed by adding, for example, a “Part numbers” menu item as one of the “text style” options shown in the dialog user interface of FIG. 5. In this case, the selected handwriting would be decoded using the writer-independent models for interpreting text printed as all caps, the system may retrieve the word list 24 which corresponds the target text being identified as part numbers. Furthermore, additional “text style” options (such as “Address” and “Normal text”) would allow a user to apply the available “handwriting styles” menu items (shown for example in FIG. 5) so as to realize a more specific task-oriented recognition of particular handwriting.

[0031] It is to be understood that the various recognition constraints illustrated in the GUI interfaces of FIG. 4a, 4band 5 are not meant to exhaustively list all possible handwriting styles and text types, nor all the possible classes of constraints that may be available for selection by a user. For example, text styles could be extended to include other types of text, such as numbers-only or symbols. The recognition constraints may include other types of constraints such as “left-handed” and “right-handed” constraints, as well as “paper ruling” constraints to provide information regarding the vertical spacing between lines on a piece of paper (which is used in conjunction with the CROSSPAD discussed above). Other type of constraint categories, such as Field Description, could include items such as dates, URLS, math, e-mail, addresses, social security numbers, phone numbers, zip codes, state abbreviations, and numerous other possibilities that one of skill in the art may envision. Since the universe of potential recognition constraints that may be utilized is virtually limitless (except for practical considerations), the term “recognition constraints” as used herein refers, collectively, to the universe of all possible constraints. In particular, the term “recognition constraint” as used herein refers to any generic type of handwriting that has some feature which distinguishes it from other types of writing.

[0032] It is to be further understood that the system and methods described herein may be implemented in various forms of hardware, software, firmware, or a combination thereof. In particular, the present invention is preferably implemented in software as an application program which is executed on a general purpose computer having any suitable and preferred microprocessor architecture. It is to be further understood that because some of the elements of the present system are preferably implemented as software modules (such as the recognition engine 22) the actual connections shown in FIG. 1 may differ depending upon the manner in which the system is programmed. Of course, special purpose microprocessors may be employed to implement the present system. Given the teachings herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations of the present invention.

[0033] Preferably, the system and methods described herein are implemented on a computer platform including hardware such as one or more central processing units (CPU), a random access memory (RAM), and input/output (I/O) interface(s). The computer platform also includes an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or application programs which are executed via the operating system. In addition, various other peripheral devices may be connected to the computer platform such as an additional data storage device and a printing device.

[0034] Various methods of operation of the system shown in FIG. 1 will now be discussed in detail with reference to the flow diagrams of FIGS. 2 and 3. In particular, the flow diagram of FIG. 2, illustrates a “one shot” method for selecting recognition modes, wherein selected recognition constraints are applied to marked ink only once and then reset, thereby allowing the user to select different recognition constraints for subsequently marked text. On the other hand, the method illustrated in FIG. 3 implements a “sticky constraint”, wherein a selected recognition constraint is set and applied to all marked ink until the user selects another recognition constraint.

[0035] Referring now to FIG. 2, initially, the handwriting data will be collected by, and uploaded from, the input device 12 (step 200). Next, the user will select the “recognition” user interface when the user desires to perform machine recognition of the uploaded handwriting data (step 201). For example, as illustrated in FIGS. 4aand 4b, the user would select the “Ink” menu (or toolbar button) to provide the initial recognition options. Next, the user will select (or mark) the ink to be recognized (step 202) via the “Select” option shown in FIGS. 4aand 4b, for example, or any other suitable menu option which allows a user to select the text to be recognized. Next, the user will select either the “default recognition” mode (step 203) or the “constrained recognition” mode (step 205) from the displayed recognition menus. If the “default recognition” mode is selected (affirmative result in step 203) (via the “Recognize” menu item shown in FIGS. 4aand 4b, for example), then the recognition engine 22 (FIG. 1) will decode the selected ink using the default recognition parameters 26 (step 204). After default recognition is finished, the recognition results are output (step 211).

[0036] On the other hand, the user can select the “constrained recognition” mode (affirmative result in step 205) (via the “Recognize as” menu item shown in FIGS. 4aand 4b, for example). The user will then be presented with a list of available “recognition constraints” and the user will select one or more desired constraints (step 206). After the user selects the desired constraint(s), a determination is made as to whether recognition of the selected ink is deferred (step 207). In particular, the system provides the user the option (e.g., via a menu or tool bar item or default setting) of selecting between having the selected ink recognized immediately (step 208) or having recognition deferred to a later time (step 209).

[0037] If recognition is not deferred (negative determination is step 207), the selected ink will be recognized in accordance with the selected constraints (step 208) and the recognition results will be output (step 211). If the user wants additional ink recognized (affirmative result in step 212), the user will select the desired ink (return to step 202) and the recognition process may be repeated for the newly selected ink (steps 203, 204 and 211 or steps 205, 206, 208, and 211).

[0038] On the other hand, if recognition is deferred (affirmative determination in step 207), the selected ink will be tagged with one or more labels associated with the selected constraint(s) that are to be applied during subsequent recognition (step 209). If the user wants to label additional ink for subsequent recognition (affirmative result in step 210), the user will select the desired ink (return to step 202) and the labelling process will be repeated for the newly selected ink (steps 205, 206, and 209). Accordingly, in the “one shot” method of FIG. 2, after the selected ink is either labelled (step 209) or recognized (step 208), since the selected recognition constraints are applied to marked ink only once and then reset, the user must actively select a desired recognition mode for subsequently marked ink.

[0039] Referring now to FIG. 3, a method for selecting recognition modes in accordance with another aspect of the present invention is shown. Initially, the handwriting data will be collected by, and uploaded from, the input device 12 (step 300). Next, the user will select the “recognition” user interface when the user desires to perform machine recognition of the uploaded ink (step 301). Next, the user will select either the “default recognition” mode (step 302) or the “constrained recognition” mode (step 305) from the displayed recognition menus. If the “default recognition” mode is selected (affirmative result in step 302), the user will select (or mark) the ink to be recognized (step 303), and the recognition engine 22 will decode the selected ink using the default recognition parameters 26 (step 304). After default recognition is finished, the recognition results are output (step 313).

[0040] On the other hand, if the “constrained recognition” mode is selected (affirmative result in step 305), the user will then be presented with a list of available recognition constraints, and the user will select one of more of the constraints (step 306). After the user selects the desired constraint(s), the user will select the ink to which the constraints are then applied for recognition (step 307). A determination is then made as to whether recognition of the selected ink is deferred (step 308).

[0041] If recognition is not deferred (negative determination is step 308), the selected ink will be recognized in accordance with the selected constraints (step 309) and the recognition results will be output (step 313). If the user wants additional ink recognized (affirmative in result in step 314) using the previously selected constraints (affirmative result in step 312), the user will select the desired ink (return to step 307) and the recognition process will be repeated for the selected ink using the previously selected constraints (steps 309 and 313). In this manner, a “sticky constraint” is applied to each ink selection.

[0042] On the other hand, if the user wants to recognize additional ink (affirmative result in step 314) but does not want to apply the previously selected constraint(s) (negative result in step 312), the user may select either the default recognition mode (step 302) or the constrained recognition mode and desired constraints (steps 305 and 306) before selecting additional ink (steps 303 and 307, respectively).

[0043] If recognition is deferred (affirmative determination in step 308), the selected ink will be tagged with the label associated with the selected constraint(s) (step 310). If the user wants to label additional ink (affirmative result in step 311), using the previously selected constraint(s) (affirmative result in step 312), the user will select the desired ink (step 307) and repeat the labelling process (step 310). In this manner, the user can apply the same recognition constraint(s) to a plurality of ink selections. On the other hand, if the user wants to label additional ink with different constraints (affirmative result in step 311 and negative result in step 312), the user may select another constraint (steps 305 and 306) selecting additional ink (step 307), and then label the selected ink with the new constraint (step 310).

[0044] In summary, the illustrative embodiments described above may be implemented in any handwriting recognition system to afford a user the option of selecting specific recognition constraints to provide increased recognition accuracy for decoding uncharacteristic handwriting styles or text. The exemplary user interfaces described above facilitate the selection of such recognition constraints by allowing a user to readily find and select the available recognition options.

[0045] It is to be appreciated that the present system and methods may be employed in other machine recognition applications to allow a user to select certain recognition constraints for decoding data, notwithstanding that the illustrative embodiments herein are directed to handwriting recognition. For instance, the present invention may be implemented in a speech recognition system in which the user interface provides a user the option of performing constrained speech recognition. Such constraints may include, for example, the option of selecting between adult/child and male/female speech models or selecting special vocabularies (i.e., word lists) for scientific, legal, and medical applications. One of ordinary skill in the art could envision additional constraints which may be applied in speech recognition applications.

[0046] With regard to speech recognition, using methods well-known to those skilled in the art, the input utterances (spoken words) are initially processed by extracting spectral features to produce a plurality of feature vectors which, typically, represent the envelope of the speech spectrum. Each feature vector is computed for a given frame (or time interval) of the digitized speech, with each frame representing, typically, 10 ms to 30 ms of the digitized speech. In addition, each feature vector includes “n” dimensions (parameters) to represent the sound within the corresponding time frame.

[0047] In accordance with an illustrative embodiment discussed above, after the feature data is extracted from the input utterances, the feature data may decoded using speech models (and default recognition parameters) and the decoding results (i.e., the machine text representing the decoded feature data) is displayed. Thereafter, the displayed text can be selected by the user and decoded once again using one or more selected recognition constraint to achieve increased recognition accuracy. In particular, the frames of feature data which are aligned to, and associated with, the selected text can either be re-decoded using the selected recognition constraints or labelled with the selected constraints for re-decoding at a subsequent time.

[0048] Alternatively, in accordance with another illustrative embodiment discussed above, the user can select one or more recognition constraints which are applied in the first instance for performing recognition. Moreover, it is to be understood the input utterances can be recorded and processed using the input device 12 (FIG. 1), and/or subsequently uploaded to a remote speech recognition system for further processing.

[0049] Although illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present invention is not limited to those precise embodiments, and that various other changes and modifications may be affected therein by one skilled in the art without departing from the scope or spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as defined by the appended claims.