Stored phrase reutilization when testing speech recognition

[0001] The present invention relates to the field of speech processing technologies and, more particularly, to stored phrase reutilization when testing speech recognition grammars.

[0002] Voice user interfaces (VUI) and multimodal interfaces accept spoken phrases as input. These spoken phrases are speech recognized using an associated speech recognition grammar. In many implementations, different application states of the VUI are associated with different permissible spoken phrases. These phrases are recognized by a context dependent speech recognition grammar, where the context is based upon the VUI state. As changes are made to a VUI interface, the set of permitted spoken phrases and corresponding context dependent speech recognition grammars can change. Thus, from version to version of a VUI, related speech recognition grammars can change.

[0003] Testing VUIs can be a challenge, especially with regards to testing speech recognition accuracy and precision, which can involve an accuracy and precision of underlying speech recognition grammars. Typically, each speech recognition grammar is tested using a large number of pre-recorded phrases. The pre-recorded phrases are typically stored in a database as an audio file, which is associated with a text representation of each stored phrase. When a speech recognition engine using the speech recognition grammar is able to generate a text result from the audio file, which matches the stored text representation, a successful test has occurred. Ideally, the set of test phrases used to test a speech recognition grammar should cover a statistically significant portion, if not all, of the allowed phrases.

[0004] Current VUI testing techniques focus upon maximizing phrase coverage and minimizing complexities of testing. Many VUI testing techniques select a set of phrases for a given version of a VUI and store a version specific test set of phrases consisting of audio files and textual representations for each of the selected phrases. When a common phrase is used across more than one VUI version, multiple copies of audio files for that common phrase are stored, one copy per each version specific test set. Additionally, each test set for a VUI version can be produced through a VUI version specific recording session. These practices result in significant storage and recording costs.

[0005] Often a minimization of recording costs is attempted by relying upon one or more external sources of audio recordings. A large manual effort is involved in selecting which phrases from the external sources are to be used to test each specific speech recognition grammar. The cost, time, and confusion resulting from manual efforts of selecting phrases for grammar testing is one reason many opt for the previous solution of version specific recordings, which result in version specific test sets.

[0006] One aspect of the present invention can include a method, apparatus, computer program product, and system for maximizing a use of stored phrases for speech recognition purposes. In the aspect, a set of audio phrases and corresponding phrase characteristics can be maintained, such as in a database. The phrase characteristics can include a translation of speech in the associated audio phrase. A finite state grammar that includes a set of textual phrases can be received. A software algorithm can execute to compare the set of textual phrases against the translations associated with the maintained audio phrases. A result of the software algorithm execution can be produced, where the result indicates phrase coverage for the finite state grammar based upon the audio phrases.

[0007] Another aspect of the present invention can include a system for maximizing a set of stored phrases. The system can include a data store and a coverage software tool. The data store can include digitally encoded information specifying a set of audio phrases and associated phrase characteristics. The phrase characteristics can include a translation of speech in the associated audio phrase. The coverage tool can receive as input an identifier for a finite state grammar. The finite state grammar can include a set of textual phrases associated with a voice user interface (VUI). The coverage software tool can compare the textual phrases against the translations associated with the audio phrases and can produce a result. The result can indicate phrase coverage for the finite state grammar based upon the audio phrases.

[0010] The present inventions stores speech phrases in a data store, each of which are associated with a text translation of the related phrase. A software tool can analyze the stored speech phrases in context of a particular finite state grammar. This finite state grammar represents a vocabulary of a speech recognition grammar associated with a version and/or application state of a voice user interface. That is, the finite state grammar defines a set of valid phrases for a speech enabled application. A test phrase coverage tool can analyze the set of speech phrases in the data store and determine which are valid for the finite state grammar. The determined valid phrases from the data store can be used as a test set to test the finite state grammar.

[0011] In one embodiment, when coverage of pre-existing phrases in the data store is insufficient to test a finite state grammar, new phrases can be added to the data store as needed. Each time a finite state grammar changes and/or the phrases in the data store changes, the coverage tool can be re-run to determine an updated coverage report. Configurable settings can exist for the coverage tool, which permits software developers to be automatically notified whenever coverage falls below established limits.

[0012] The present invention may be embodied as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, the present invention may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0013] Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to the Internet, wireline, optical fiber cable, RF, etc.

[0014] Any suitable computer usable or computer readable medium may be utilized. The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or Flash memory, a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W) and DVD. Other computer-readable medium can include a transmission media, such as those supporting the Internet, an intranet, a personal area network (PAN), or a magnetic storage device. Transmission media can include an electrical connection having one or more wires, an optical fiber, an optical storage device, and a defined segment of the electromagnet spectrum through which digitally encoded content is wirelessly conveyed using a carrier wave.

[0015] Note that the computer-usable or computer-readable medium can even include paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0016] Computer program code for carrying out operations of the present invention may be written in an object oriented programming language such as Java, Smalltalk, C++ or the like. However, the computer program code for carrying out operations of the present invention may also be written in conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0017] A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

[0018] Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.

[0019] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.

[0020] The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0021] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

[0022] The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0023] FIG. 1 is a schematic diagram of a system 100 for maximizing phrase coverage by comparing a set of previously stored phrases against one or more finite state grammars in accordance with an embodiment of the inventive arrangements disclosed herein. In system 100, a coverage tool 130 can receive a grammar identifier 137 as well as one or more optional constraints 138 as input. The coverage tool 130 can then compare a set of in-grammar phrases from grammar sources 120 against a set of previously stored phrases of the phrase sources 110 to generate a report 150 and/or a test run package 160. The report 150 and/or the package 160 each contain information showing which of the phrases of the phrase sources 110 are valid/invalid phrases for the selected grammar associated with grammar identifier 137. In one embodiment, the package 160 can be used as input for a recognition grammar tester 165, which produces test results 170. Each time a set of phrases of the phrase sources 110 changes 119 and/or the grammar changes 126, the coverage tool 130 can re-execute to generate a new report 150, to generate a package 160, and/or to take another programmatic action related to comparison results.

[0024] Although one contemplated use of system 100 is for testing and/or evaluating speech recognition grammar performance, other situations exist in which system 100 can be used. For example, the coverage tool 130 can be used as a filter to filter all phrases possible in a grammar. The phrases that fail the filter can be considered “out of vocabulary phrases” and those that pass the filter can be considered “in vocabulary” phrases. Constraints 138 can be imposed on the filter as desired. A filter embodiment can be useful in identifying potential expansion phrases for a grammar or even potential phrases to remove from a grammar.

[0025] The phrase sources 110 of system 100 can include one or more linked data stores, each containing a set of audio phrases. These data stores can be geographically distributed and interconnected via a network. In one embodiment, a database 113 can be used to manage the phrases and to expedite search operations. For example, the database 113 can be a relational database management system (RDBM) that includes indexed records for each audio segment 117 and associated phrase characteristics 118. The database 113 is an optional component of the phrase sources 110, which is not necessary in some implementation, especially those that manage a relatively modest quantity of audio phrases.

[0026] Associations between an audio file 117 and its corresponding characteristics 118 can be maintained in numerous fashions. One fashion is to utilize records of a relational database. Another is to include the characteristics as metadata linked to the audio files 117. For example, a header of the audio file can specify the set of characteristics 118. At a minimum, these characteristics 118 can include a textual representation of the audio 117, which is a translation of the audio 117. Other characteristics 118 can be useful in constraining a coverage comparison and/or in providing more detailed report 150 information.

[0027] These characteristics 118 can include, but are not limited to, a gender, an age, a native language, a region, environment characteristics, audio capture voice identifier (e.g., identifies the voice talent providing a recording), a recording session identifier, audio capture characteristics, a signal to noise ratio (SNR) value, a voice identifier, a recording session identifier, and the like. The gender can refer to a gender of a speaker of the audio 117. The age is the age of a speaker. The native language is a native language of a speaker. The region refers to a region where a speaker grew up, which can also be defined as a dialect characteristic and/or an accent characteristic. The environmental characteristic can refer to ambient noise characteristics of the recording, such as noises representing an office background, a moving automobile, etc. The audio capture characteristics refer to a microphone type for capturing audio, a microphone brand, a distance from a speaker to a microphone, noise suppression characteristics, a fidelity of captured audio and the like. The SNR is a measure of ambient noise. The voice identifier can identify a specific voice talent who made a recording, which can be useful when supplemental recording sessions are necessary. The recording session identifier can identify and provide details of a recording session from which the audio 117 was produced. Any desired characteristics 118 can be added to system 100.

[0028] In one embodiment, a search engine 112 can exist, which is designed to search all the phrase sources. This engine 112 can execute SQL queries in a configuration where a database 113 is used. In an implementation lacking a database 113, the engine 112 can search for key phrases and terms, such as those present within a header, which define characteristics 118. One of the searches of engine 112 can generate a “master listing” of text phrases included in the phrase sources 110. Moreover, engine 112 can permit the master listing (through use of filters and/or additional query parameters) to be limited by selected characteristic 118 values, which can be specified within constraints 138 for a given comparison instance.

[0029] A phrase handling engine 114 can perform all programmatic actions necessary to maintain a currency of the phrases from the phrase sources 110. For example, when multiple different data stores of phrases exist, the phrase handling engine 114 may have to normalize a format of the phrases and characteristics 118 contained in the different data stores. The phrase handling engine 114 can also periodically synchronize content from different data stores and can handle conflicting or redundant entries. Engine 114 can also extract metadata content 118 from audio files and use this content to maintain database 113 records. Further, the engine 114 can perform appropriate index, database 113, and other updates when phrases change 119 to ensure a currency of phrase information.

[0030] The grammar sources 120 can include one or more linked data stores of grammars. A database 123 can optionally be used to centrally manage information of these data stores. The grammars of the sources 120 can be finite state grammars (FSG), which define a vocabulary. This vocabulary can be associated with a specific context or speech enabled application. A versioning component 124 can maintain records indicating which FSG grammars are associated with which speech-enabled application version. Grammar editor 122 can permit an authorized user to add, delete, and edit FSGs of the data stores. Grammar changes 126 can also be received from external sources. In one embodiment, the grammar editor 122 and/or the versioning component 124 can be integrated components of a software development platform.

[0031] The coverage tool 130 can be communicatively linked to the phrase sources 110 and the grammar sources 120. Tool 130 can include an invocation engine 132, a comparator 134, and a report generator. Additionally, the tool 130 can be able to access a data store 140, which can include user settings 142, application settings 144, and the like, which can affect operations of the tool 130. The settings 142, 144 of data store 140 can be user or administrator configurable in one contemplated arrangement of system 100.

[0032] Invocation engine 132 can invoke an execution of the coverage tool 130, which can occur responsive to a manually issued user command and/or based upon an occurrence of a detectable event. A manual command can, for example, be issued from a testing application, a software development application, a grammar creation application, and/or any other application to which the coverage tool 130 is programmatically linked. Events monitored by the invocation engine 132 can include a change 119 in the phrases, a grammar change 126 to the vocabularies, and the like. The events and conditions for triggering invocation actions can be modified and configured by a user or authorized administrator. For example, a setting can be applied to the invocation engine 132, where an execution occurs only if the phrases changed 119 exceed a percentage threshold (e.g., five percent or more of the total phrases have changed), a time threshold (e.g., one percent or greater change and no coverage update in at least three days), or any other definable set of conditions.

[0033] Comparator 134 can compare the phrases of a given set of grammars (i.e., defined by Grammar_ID 137) against a set of phrases stored in the phrase sources 110. In one embodiment, constraints 138 can be applied, which limit the scope of comparison performed by the comparator 134. For example, the constraints 138 can limit matched phrases of the phrase sources 110 to those that are male (i.e., gender=male using characteristics 118). Constraints 138 can set any definable limitation on the comparison, which is not limited to values of the characteristics 118. For example, a constraint 138 can limit the phrase sources 110 being compared to those resulting from recordings specifically made for previous incarcerations (versions) of an application (assuming the grammar being compared is for a new version of an application). In various configurations, settings of the comparator 134 can cause the comparison to determine in-grammar phrases (e.g., contained in the previously stored phrase sources 110 and the grammar being compared), out-of-grammar phrases (contained in one or more of the data stores of the phrase sources 110 but missing from the grammar being compared), non-covered phrases (contained in the grammar being compared but absent from the phrase sources 110), and the like.

[0034] The report generator 136 can generate a set of reports 150 including customizable reports 150 relating to results of comparisons performed by the comparator 134. In one embodiment, SQL based reports can be configured. Additionally, in one implementation a report creation GUI can be included with the tool 130 to permit a user to establish custom reports. The report generator 136 can provide various statistics within the reports 150, which help identify holes in phrase coverage, an adequacy of phrase coverage, and to provide a level of confidence for phrases which are available. For example, the reports 150 can include how many utterances of a particular phrase were selected, how much unique phrase coverage there is for a particular grammar, and the like.

[0035] The various reports 150 can be intended for different audiences and purposes. For example, a report 150 to an application developer can specify an expected accuracy of a test based upon the phrase coverage. A report 150 for a sound technician can specify the audio capture characteristics, and environmental characteristics of the available phrases, can provide summary statistics like average environmental characteristics, range of available environmental characteristics for complete environmental coverage, etc. In one embodiment, the reports 150 can produce intermediary files, which are designed to be consumed by executing programs. For example, one report 150 can define unique phrases 110, their locations, etc, which can be consumed by a recognition grammar tester 165 during a test (i.e., the report can define a test run package 160, which the recognition grammar tester 165 can utilize). In another example, a report 150 can be an automatically generated script to be read by a voice talent to generate additional phrases for the phrase sources 110 (i.e., the script can include needed phrases for a grammar not originally contained in the phrase sources 110). One or more of the reports 150 can combine information from the comparator 134 with other information, such as a report of test results 170, which combines results from the tester 165 and the comparator 134.

[0036] FIG. 2 is a flow chart of a method 200 for maximizing a use of stored phrases to test speech recognition grammars in accordance with an embodiment of the inventive arrangements disclosed herein. The method 200 can be performed in a context of system 100, or any other system designed to compare a set of recorded phrases against one or more speech enabled grammars.

[0037] Method 200 can begin in step 205, where multiple audio files containing speech phrases and associated textual translations are stored in at least one searchable data store, such as a database. Multiple data stores can be linked, each containing a unique set of speech phrases and translations. In one embodiment, additional phrase defining attributes other than just the textual translation of the phrase can be recorded. Characteristics can include, but are not limited to, a gender of the speaker, an age of the speaker, a native language of the speaker, a region of the speaker, environmental characteristics of the audio file, audio capture characteristics of the audio file, signal to noise ratio of the audio file, and the like. These additional characteristics will often apply to a set of audio files, such as audio files produced in a single recording session, which can be stored in entries of a database table linked to multiple ones of the audio files.

[0038] In step 210, a master listing of textual phrases associated with audio files can be obtained. In an RDBMS implementation, this master listing can be generated using a SQL query. In another implementation, the textual phrases can be stored in metadata fields (e.g., within a header, for example) of the audio files. Creation of the master listing of textual phrases can parse the textual phrases from each file. These two approaches can be combined, where the textual phrases are first extracted and placed in a database table in a manner indexed against the audio file from which the extraction occurred. In one embodiment, the master listing can be restricted to one or more of the audio characteristics, such as restricted to only include phrases spoken by males or phrases having a SNR under a fixed threshold. In step 215, a check can be made to see if any changes have been made to the stored audio files since a last time steps 205 and 210 executed. If so, the method can re-execute steps 205 and 210 to ensure the master listing of textual phrases is current.

[0039] In step 220, a finite state grammar can be identified, which contains a list of grammar phrases. In step 225, a comparison can be made between the grammar phrases and the phrases of the master listing. In step 230, a result list of valid phrases from the master listing satisfying corresponding grammar phrases can be produced. This produced list of valid phrases can be further utilized to drive a variety of other programmatic actions, as shown by step 235. For example, a test set of grammar phrases can be generated, which can be used to test the finite state grammar. In another example, a list of needed phrases (not covered by an existing phrase in the master listing) can be created, which can form a basis for a recording script. The recording script can be a script that a voice talent is to read during a recording session to generate supplemental audio phrases to be utilized to test the finite state grammar. These supplemental recordings can be added to the data store, which would trigger a corresponding change (step 215) to the master listing. In step 240, a check for changes to the finite state grammar can be made. When the grammar changes, the method can repeat steps 220-235 to obtain a current listing of valid phrases.

[0040] The method can repeat steps 220-240 for any number of different finite state grammars. In one embodiment, the steps of the method can be implemented by a computing device in accordance with a set of programmatic instructions. Further, these programmatic instructions can be integrated into a software development environment. For example, the toolset can include a finite state grammar creation and editing component, which is used to make changes to the finite state grammar of step 220. Each time a change to a finite state grammar is made, steps 220-235 can automatically execute. In another example, the programmatic instructions can be integrated with a testing tool, which automatically tests a finite state grammar based upon the audio files associated with phrases in the produced listing of valid phrases. Coverage limitations can be established, which causes notifications for additional phrases to be automatically generated and conveyed to responsible parties as part of the programmatic actions of step 235. Thus, the method 200 can be part of a dynamic software development process, which automatically determines a set of test phrases for recognition grammars and/or deficiencies in testing coverage as changes to underlying software development projects are made.

[0041] The diagrams in FIGS. 1-2 illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0042] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0043] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.