Customized tokenization of domain specific text via rules corresponding to a speech recognition vocabulary

[0002] (Not Applicable)

[0003] (Not Applicable)

[0004] 1. Technical Field

[0005] This invention relates to the field of text tokenization and more particularly to a method and system for supporting customized tokenization of domain-specific text.

[0006] 2. Description of the Related Art

[0007] Tokenization is the process of separating text into words, punctuation and optionally, phrases. Tokenization can include case folding of words at the beginning of a sentence and special formatting modifications to an input string, such as is sometimes done with numbers. Tokenization plays a critical role in the building of speech recognition vocabularies. Tokenization can also be used in coordination with other components of a speech recognition system, for instance with a speech correction tool or a speech analysis tool for updating a system language model. To ensure consistency, it is essential to have one common tokenizer for all applications needing a particular type of token processing so that the concept of what makes up a word remains in agreement.

[0008] The inherent difficulty associated with processing a variety of electronic text can cause the expansion of the complexity of the source code forming the tokenization program. Typically, several hundred lines of source code are needed to form a tokenization program able to convert written forms to spoken forms, dividing character streams at logical word boundaries. Tokenization code can become particularly complex and troublesome in view of the multiple uses for common symbols, such as the apostrophe, comma, period and numbers. Since general purpose tokenizers cannot correctly process text in all domains, it is essential that vocabulary builders have the flexibility to customize this process. Specifically, when building a new vocabulary, it is common to make minor modifications to the general purpose tokenizer in order to correctly tokenize domain-specific strings.

[0009] Present systems address the need for context-specific tokenization, also referred to as domain-specific tokenization, in two ways. First, a vocabulary requiring special tokenization can be distributed without a vocabulary-specific tokenizer. The general-purpose tokenizer can then be used by the speech recognition system when the vocabulary becomes active, for instance during correction. In consequence, the tokenization used in building the vocabulary can differ from the tokenization used for updating the system language model. Second, as an alternative, a vocabulary requiring special tokenization can be distributed with a vocabulary-specific tokenizer which includes general purpose rules in addition to domain-specific rules.

[0010] Where a vocabulary requiring special tokenization is distributed without a vocabulary-specific tokenizer, inconsistencies can arise between the vocabulary and the personal language model. In the alternative case, where a vocabulary is distributed with a vocabulary-specific tokenizer, improvements or bug fixes directed toward future versions of the general purpose tokenizer will require rebuilding and redistributing the vocabulary-specific tokenizer of the domain-specific vocabulary. Furthermore, external software developers building vocabularies will not be able to develop vocabulary-specific tokenizers because the external developers lack the proprietary knowledge of the speech recognition system necessary for the development of the general-purpose tokenizer. Thus, no present system provides for a flexible customized tokenization system, capable of processing vocabulary-specific tokenization schemes while treating all vocabularies uniformly. Accordingly, there is a long-felt need for a flexible tokenization system, capable of processing vocabulary-specific tokenization schemes while treating all vocabularies uniformly.

[0011] A flexible tokenization system, in accordance with the inventive arrangement, satisfies the long-felt need of the prior art by processing vocabulary-specific tokenization schemes while treating all vocabularies uniformly. Thus, the inventive arrangements provide a method and system for supporting customized tokenization of domain-specific text. The inventive arrangements have advantages over all known tokenization methods, and provide a novel and nonobvious system, including apparatus and method, for supporting customized tokenization of domain-specific text. A method for supporting customized tokenization of domain-specific text in accordance with the inventive arrangements comprises the steps of: loading domain-specific tokenization rules corresponding to the customized tokenization of the domain-specific text; tokenizing the domain-specific text using the loaded domain-specific tokenization rules; and, further tokenizing the domain-specific text using general purpose tokenization rules.

[0012] The loading step can comprise: loading a speech recognition vocabulary; and, loading domain-specific tokenization rules corresponding to the speech recognition vocabulary. In addition, the tokenizing step can comprise identifying each substring in the domain-specific text matching a regular expression having a corresponding replacement pattern in the loaded domain-specific tokenization rules, and replacing each substring identified in the identifying step with the replacement pattern corresponding to the matched regular expression. However, to accomodate the circumstance where the exclusion of particular domain-specific text from domain-specific tokenization is desired, the tokenizing step can comprise identifying substrings in the domain-specific text matching a regular expression having a corresponding replacement pattern in the second loaded domain-specific tokenization rules; excluding from further processing the identified substrings having a do-not-replace marker associated with the identified substring; and, replacing each non-excluded identified substring with the replacement pattern corresponding to the matched regular expression.

[0013] Alternatively, the loading step can comprise first loading an active vocabulary, identifying domain-specific tokenization rules corresponding to the active vocabulary, and second loading the domain-specific tokenization rules identified in the identifying step. In the alternative case, the tokenizing step can comprise checking for the second loaded domain-specific tokenization rules, and processing the domain-specific text using the second loaded domain-specific tokenization rules only if the second loaded domain-specific tokenization rules are identified in the checking step. Moreover, the processing step can comprise identifying each substring in the domain-specific text matching a regular expression having a corresponding replacement pattern in the second loaded domain-specific tokenization rules, and replacing each substring identified in the identifying step with the replacement pattern corresponding to the matched regular expression. As before, to accomodate the circumstance where the exclusion of particular domain-specific text from domain-specific tokenization is desired, the tokenizing step can comprise identifying substrings in the domain-specific text matching a regular expression having a corresponding replacement pattern in the second loaded domain-specific tokenization rules; excluding from further processing the identified substrings having a do-not-replace marker associated with the identified substring; and, replacing each non-excluded identified substring with the replacement pattern corresponding to the matched regular expression.

[0014] There are presently shown in the drawings embodiments which are presently preferred, it being understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown.

[0015] FIG. 1 is a pictorial representation of a computer system for speech recognition on which the system of the invention can be used.

[0016] FIG. 2 is a block diagram showing a typical high level architecture for the domain-specific tokenization support system in accordance with the inventive arrangements and as implemented on the computer system of FIG. 1.

[0017] FIG. 3 is a block diagram illustrating a process for supporting the customized tokenization of domain-specific text.

[0018] FIGS. 4A and 4B, taken together, are a flow chart illustrating a process for supporting customized tokenization of domain-specific text.

[0019] FIG. 1 shows a typical computer system 1 for use in conjunction with the present invention. The system preferably comprises a computer 3 including a central processing unit (CPU), one or more memory devices and associated circuitry. The system also includes a microphone 7 operatively connected to the computer system through suitable interface circuitry or “sound board” (not shown), a keyboard 5, and at least one user interface display unit 2 such as a video data terminal (VDT) operatively connected thereto. The CPU can include any suitable microprocessor or other electronic processing unit, as is well known to those skilled in the art. An example of such a CPU would include the Pentium or Pentium II brand microprocessor available from Intel Corporation, or any similar microprocessor. Speakers 4, as well as an interface device, such as mouse 6, can also be provided with the system, but are not necessary for operation of the invention as described herein. The various hardware requirements for the computer system as described herein can generally be satisfied by any one of many commercially available high speed multimedia personal computers offered by manufacturers such as International Business Machines (IBM), Compaq, Hewlett Packard and Apple Computers.

[0020] FIG. 2 illustrates a presently preferred architecture for a domain-specific tokenization support system in accordance with the inventive arrangements and as implemented in the speech recognition system in computer 1. As shown in FIG. 2, computer system 1 includes one or more computer memory devices 8, preferably an electronic random access memory 8A and a bulk data storage medium, such as a magnetic disk drive 8B. The system can include an operating system 9, a speech recognition system 11, a domain-specific tokenization support engine 10 in accordance with the inventive arrangements, and a loaded active vocabulary 12. In a preferred embodiment described herein, operating system 9 is one of the Windows family of operating systems, such as Windows NT, Windows 95 or Windows 98 which are available from Microsoft Corporation of Redmond, Wash. However, the system is not limited in this regard, and the invention can also be used with any other type of computer operating system.

[0021] In FIG. 2, the domain-specific tokenization support engine 10, the speech recognition system 11, and the loaded active vocabulary 12 are shown as separate modules. It should be noted, however, that the invention is not limited in this regard, and these various modules can, of course, be implemented as a single, more complex applications program. Domain-specific tokenization support engine 10 preferably comprises a general purpose tokenizer 13 having general purpose tokenization rules 16. Moreover, domain-specific tokenization support engine 10 maintains an association with the loaded active vocabulary 12. The loaded active vocabulary 12 can include two individual components: customized vocabulary 15 and domain-specific tokenization rules 14, to be used in conjunction with customized vocabulary 15. The system as disclosed herein can be implemented by a programmer, using commercially available development tools for the operating systems described above.

[0022] FIG. 3 is a block diagram illustrating the inventive process for supporting the customized tokenization of domain-specific text. FIG. 3 shows the contents of domain-specific tokenization rules 14. Domain-specific tokenization rules preferably can include an aggregation of replacement patterns corresponding to regular expressions. When applying tokenization rules, in general, a tokenizer preferably can search through text for substrings matching any one of the regular expressions contained in the aggregation of tokenization rules. Upon identifying a matching substring, the tokenizer preferably can replace the matching substring with a replacement pattern corresponding to the matching regular expression.

[0023] In FIG. 3, unprocessed domain-specific text 17 can be preprocessed using domain-specific tokenization rules 14. Specifically, each substring contained in domain-specific text 17 preferably can be replaced with a corresponding replacement pattern contained in domain-specific tokenization rules 14 resulting in preprocessed domain-specific text 18. For instance, as shown in domain-specific text 17, the RegularExpressionMatch1 substring matching RegularExpression1 in domain-specific tokenization rules 14 will be replaced with ReplacementPattern1 in domain-specific tokenization rules 14. Similarly, the RegularExpressionMatch4 substring matching RegularExpression4 in domain-specific tokenization rules 14 will be replaced with ReplacementPattern4 in domain-specific tokenization rules 14. However, where excluding particular substrings in domain-specific text 17 from tokenization is desired, the particular substrings can be enclosed with the special do-not-replace markers, “<<AS-IS” and “AS-IS>>”. In that instance, the replacement pattern inserted into the preprocessed domain-specific text 18 will not be further tokenized by the general purpose tokenizer 13.

[0024] Subsequent to the preprocessing step, preprocessed domain-specific text preferably can be passed to the general purpose tokenizer 13 where each substring contained in preprocessed domain-specific text 18 preferably can be replaced with a corresponding replacement pattern contained in general purpose tokenization rules 16 resulting in fully tokenized domain-specific text 19. For example, RegularExpressionMatch8 substring matching RegularExpression8 in general purpose tokenization rules 16 will be replaced with ReplacementPattern8 in general purpose tokenization rules 16. One skilled in the art will recognized that replacement pattern inherently can be regarded as a regular expression. For instance, the substring ReplacementPattern4 in preprocessed domain specific text 18 will be replaced with ReplacementPattern7 in general purpose tokenization rules 16. Notwithstanding, the substring ReplacementPaftern5 surrounded by the do-not-replace markers in preprocessed domain specific text 18 will not be replaced by the ReplacementPattern5 rule in general purpose tokenization rules 16.

[0025] FIGS. 4A and 4B, taken together, are a flow chart illustrating a process for supporting customized tokenization of domain-specific text. In FIG. 4A, the method in accordance with the inventive arrangements begins in block 20 in which the active vocabulary 12 is loaded. Following path 21 to decision block 22, if vocabulary 12 includes domain-specific tokenization rules 14 for use with customized vocabulary database 15, the inventive method preferably can load the domain-specific tokenization rules 14 in block 24. Following path 27 to decision block 27, if no domain-specific text remains to be tokenized, the inventive method terminates along path 29 in exit block 36. But, if in decision block 27 domain-specific text remains to be tokenized, following path 31 to block 30, the inventive method positions a text pointer at the beginning of the domain-specific text to be tokenized. Following path 33 to decision block 32, if domain-specific tokenization rules 14 have not been identified in block 22 and loaded in block 24, the inventive method simply passes the text to be tokenized to general purpose tokenizer 13 for general purpose tokenization, before returning along path 39 to decision block 26. In contrast, if in decision block 32, domain-specific tokenization rules 14 have been identified in block 22 and loaded in block 24, the inventive method continues along path 35 to jump circle A leading to decision block 38 in FIG. 4B, where the inventive method preferably can preprocess the domain-specific text according to domain-specific tokenization rules 14.

[0026] With reference to FIG. 4B, if, in decision block 38, any substrings pointed to by the text pointer match regular expressions contained in the domain-specific tokenization rules 14, following path 43 to block 40, the inventive method can replace the first and longest matching substring with a replacement pattern corresponding to the regular expression matched in decision block 38, the replacement being referred to generically as a maximal munch. Subsequently, following path 45 to block 42, the inventive method can advance the text pointer to the next character following the matched regular expression. Returning along path 47 to decision block 38, the inventive process preferably repeats until no more substrings pointed to by the text pointer match regular expressions contained in the domain-specific tokenization rules 14. In that event, following path 41 to jump circle B leading to block 34 in FIG. 4A, the inventive method passes the preprocessed text to general purpose tokenizer 13 for general purpose tokenization, before returning along path 39 to decision block 26.

[0027] Thus, the inventive arrangements negate the need to distribute a vocabulary requiring special tokenization without a vocabulary-specific tokenizer. Moreover, the inventive arrangements further free the vocabulary developer from the burden of distributing a domain-specific vocabulary with a vocabulary-specific tokenizer. Rather, the present invention permits vocabulary developers to distribute with the vocabulary, only domain-specific tokenization rules to be applied during a preprocessing step. Consequently, vocabulary developers need not obtain proprietary knowledge of the internal speech recognition system Moreover, the present invention, inherently, is language independent. Thus, the present system provides for a flexible customized tokenization system, capable of processing vocabulary-specific tokenization schemes while treating all vocabularies uniformly.