Automatic search of audio channels by matching viewer-spoken words against closed-caption/audio content for interactive television

[0002] The present invention relates generally to interactive television and more particularly, to a system that allows the user to select channels by spoken request.

[0003] Interactive television promises to allow two-way communication between the viewer and his or her television set. Although the technology is still in its infancy, digital television is expected to greatly enrich the prospects for interactive TV, because the digital technology makes possible a far more efficient use of available channel bandwidth. Through digital technology, broadcasters can pack a significantly larger number of programs into the available bandwidth of the delivery infrastructure (e.g. cable or satellite).

[0004] While the new interactive, digital television technology offers a significant number of benefits to both viewers and broadcasters, it is not without problems. The prospect of having 200 or more channels simultaneously available for viewing boggles the mind. Conventional on-screen electronic program guides are likely to prove inadequate in assisting viewers to find programs they are interested in. Interactive digital television demands a more sophisticated system of interaction if the viewers are ever going to be able to fully utilize this rich new resource.

[0005] The present invention provides a speech-enabled interactive system through which a user can specify a desired program content through natural language speech. The system extracts both keyword and semantic content from the user's speech, prompting the user to furnish additional information if the meaning is unclear.

[0006] The system then monitors closed caption information on multiple channels simultaneously and switches the active channel tuner or auxiliary tuner to the channel carrying information matching the user's request. If closed caption information is not available, the system will alternatively employ speech recognition upon the audio signal of the channels being monitored. Once the channel has been switched, the program may be displayed in full screen mode, or in split-screen or picture-in-picture mode, or recorded for later viewing.

[0007] The speech recognition system works with a semantic analyzer that is able to discriminate between speech intended to describe program content and speech intended to supply meta-commands to the system. By extracting meaning as well as keywords and phrases from the spoken input, the system will find matching content even when the spoken words do not match the closed caption text verbatim.

[0008] For a more complete understanding of the invention, its objects and advantages, refer to the following specification and to the accompanying drawings.

[0009] FIG. 1 is a block diagram of a presently preferred embodiment of the invention.

[0010] FIG. 2 is a data flow diagram illustrating the word selector and semantic analyzer component of the preferred embodiment.

[0011] Referring to FIG. 1, the interactive content searching system of the invention may be integrated into the television set 10, or into a set top box 12. In either embodiment, the system is designed to monitor one or more channels not currently being viewed, to detect closed caption text or audio channel speech that matches the user's previously spoken request. In FIG. 1, a plurality of tuners has been illustrated, including an active channel tuner 14 and a plurality of auxiliary tuners 16. In the illustrated embodiment it is assumed that there are n auxiliary tuners (where n is an integer number greater than 0). In its simplest form, the invention may be implemented using a single auxiliary tuner.

[0012] The active channel tuner 14 is tuned to a channel set by the user and this tuner thus selects the channel the user is currently watching on television set 10. If desired, one or more of the auxiliary tuners may also supply program content for viewing on television set 10, such as in a split-screen mode or picture-in-picture mode. In FIG. 1, the auxiliary tuner, labeled tuner n, is connected to supply program content to television set 10.

[0013] Using current tuner technology, the active channel tuner 14 and auxiliary tuners 16 select the desired channel by selecting the corresponding frequency band through bandpass filtering of the RF signal. While tuners of this type may be employed to implement the invention, other forms of digital “channel” selection are also envisioned, whereby the desired program content is extracted from the video data stream in the digital domain. For purposes of implementing the invention, the manner of channel selection depends upon the manner in which the television signals are encoded and broadcast.

[0014] Regardless of the form of the signals used to broadcast program material, the auxiliary tuners 16 are each set to monitor a different program channel, so that the closed caption text information and audio signal may be monitored by the system. The user selects which channels to monitor, using either on-screen menu selection or voiced meta-commands

[0015] The system employs a speech recognizer 18 with which the user communicates through a suitable microphone 20. Microphone 20 may be incorporated into the television set or set top box, however the presently preferred embodiment incorporates the microphone into a hand-held remote control unit 22, which communicates with the television set or set top box by suitable link, such as an infrared or hard wired link.

[0016] Speech recognizer 18 works in conjunction with a set of speech models 24 representing all words recognizable by the system. The speech recognizer may be based on Hidden Markov Model (HMM) technology, or other suitable model-based recognition technology. The dictionary or lexicon of words recognizable by the system may include not only words, but letters of the alphabet, thereby allowing the system to recognize letters spoken by the user in spelling other new words. As will be more fully explained below, inclusion of speech models for letters of the alphabet allows the user to train the speech recognizer to learn new words even if a keyboard is not available for typing.

[0017] Speech recognizer 18, in effect, converts spoken utterances into text corresponding to the most probable word or phrase candidates (or letter candidates) recognized by the system. In the presently preferred embodiment, speech recognizer 18 outputs the N-best sentence candidates for each sentence utterance spoken by the user. The recognizer generates a probability score for each sentence, indicative of the likelihood that the sentence corresponds to the spoken utterance. The top N candidates are selected and fed to the word selector and semantic analyzer block 26 for further processing.

[0018] Word selector and semantic analyzer block 26 performs several functions. First, it resolves which of the N-best recognition candidates were actually intended by the user. Second, it analyzes the semantic content of the user's entire utterance, to determine additional information about the user's request that may not be gleaned from the individual words, themselves. Third, the semantic analyzer also analyzes the user's input to resolve recognition errors and to determine whether the user's input speech represents description of program content or represents meta- commands intended as instructions to effect system operation.

[0019] The word selector and semantic analyzer uses a combined local parser and global parser to select the correct candidate from the N-best candidates and also to perform semantic analysis. The details of these parser components are described more fully below. The word selector and semantic analyzer works with a dialog manager 28 that helps resolve ambiguities by prompting the user to supply additional information to specify either the program content or the meta-command.

[0020] Dialog manager 28 can supply either text prompt or voiced prompts. Text prompts are generated as alphanumeric text that is suitably injected into the video signal for on-screen display. Voiced prompts are supplied by a speech synthesizer within the dialog manager and may be injected into the audio stream for replay through the television speaker system.

[0021] If desired, a word history data store 30 may be provided to store a record of previously resolved word ambiguities, allowing the system to “learn” the user's viewing habits, thereby assisting the word selector in resolving subsequent word recognition ambiguities.

[0022] The word selector and semantic analyzer is designed to extract the meaning behind the user's request for a channel selection and it will automatically select applicable synonyms to improve the text matching process. Thus, if the word selector and semantic analyzer determines that the user is interested in watching a football game, synonyms and related words, such as “touch down,” “kick-off,” “NFL,” “Superbowl,” and the like are extracted from the word selector's synonym database 32.

[0023] The extracted words along with the user's originally spoken word are then sent to a word list buffer 34 that serves as a dynamic dictionary for the text matching processor 36. Text matching processor 36 receives individual streams of closed caption text data and/or audio data from the auxiliary tuner's 16 as that information is broadcast live and selected by the respective tuners. If audio data is supplied by an auxiliary tuner, text matching processor 36 employs the services of speech recognizer 18 to convert the audio stream into text data.

[0024] Text matching processor 36 compares each of the incoming text streams from the auxiliary tuners 16 with the words contained in word list buffer 34. If a match is detected, processor 36 signals the channel switcher 38, which, in turn triggers a number of different actions, depending upon the mode set by the user.

[0025] In a first mode, channel switcher 38 sends a command to the active channel tuner 14, causing the active channel tuner to immediately switch to the channel on which the detected word match occurred. The user is thus immediately switched to the channel containing the content he or she previously requested.

[0026] In a second mode, channel switcher 38 switches one of the auxiliary tuners (such as tuner n) to the channel that triggered the word match. In this mode, the viewer continues to watch the active channel, but is also presented with a picture-in-picture or a split screen view of the other channel detected.

[0027] In a third mode, the channel switcher activates a recorder 40, such as a DVD recorder, that will record the program on the tuner that triggered the word match. This mode allows the viewer to continue watching the active channel, while the system records the other selected channel for later viewing.

[0028] The speech recognizer that forms the heart of the word recognition system of the invention is preferably provided with a set of speech models 24 representing speaker independent word and letter templates for the most popular words used to describe program content. However, to give the system added flexibility, a model training processor 42 may be provided to allow an individual user to add words to the speech model dictionary. The model training processor 42 takes as its input two pieces of information: (a) speech information corresponding to new words the user wishes to add to the dictionary and (b) text information representing the spelling of those new words. Speech information is provided via microphone 20, in the same fashion as speech information is provided to recognizer 18. Text information may be provided via a keyboard 44 or other suitable text entry device, including on-screen text entry system employing the keypad buttons of the remote control 22.

[0029] As an alternate means of inputting text information, the speech recognizer 18 may be used. In this alternate mode, the speaker both speaks the new word and then spells it, by speaking into microphone 20. Speech recognizer 18 uses its speech models of spelled letters to interpret the spelled word input and correlate that with the spoken utterance representing the word itself. The model training processor 42 then constructs speech models using the same model parameters upon which the initially supplied speech models are based.

[0030] The word selector and semantic analyzer 26 performs the important function of making sense of the user's natural language spoken input. The task of the word selector and semantic analyzer is thus more complex than merely spotting keywords within a stream of speech recognized text. The analyzer extracts not only the important keywords but also the context of those words, so that the semantic content or meaning of the spoken input can be determined. The word selector and semantic analyzer employs a dual parser system for this purpose. That system is shown diagrammatically in FIG. 2.

[0031] Referring to FIG. 2, the analyzer maintains a frame data store 50 in which a plurality of task-based frames or templates are stored. The data structure of these templates is illustrated diagrammatically at 52. Each frame comprises a plurality of slots 54 into which extracted keywords are placed as the word selector and semantic analyzer operates.

[0032] A local parser 56, based on an LR grammar 58, parses the text stream 60 supplied by the speech recognizer 18 (FIG. 1). The LR grammar allows the local parser to detect and label sentence fragments within the text stream that contain important keywords used to select words for filling the word list buffer 34 (FIG. 1). For example, local parser 56 contains an LR grammar to extract the keyword “football” from the following sentence:

[0033] “I think I would like to watch a football game tonight.”

[0034] Using its LR grammar, the local parser decodes the above sentence by examining the structure of the sentence and determines that the object of the sentence is “football game” and that the user has also specified a time frame parameter, namely “tonight”.

[0035] Local parser 56 then accesses a data store of keyword tags 62 to extract meaning from the keywords and phrases. The keyword tags data store may be structured to give a frame tag and slot tag identifier for each phrase or keyword. The keyword “football” might have a frame tag of “sports” and a slot tag of “sports type.” These keyword tags allow the local parser to determine which frame within data store 52 to use and which slot 54 the identified phrase or keyword should be assigned.

[0036] Each of the frames within frame data store 50 is goal-oriented. That is, each frame corresponds to a different media content selection task or system operation task. The range of tasks can be as varied as the user wishes. In a typical embodiment suitable for consumer applications, the system may be provided with a predefined set of frames corresponding to each of the available system operation commands and to a variety of typical program content requests. The user could thus speak into the system to perform a system command, such as instructing the system to record an identified program instead of displaying it through the active channel tuner. A user command such as:

[0037] “I want to record the Seinfeld re-run tomorrow night.” would cause the system to enter a record mode. The above command would also be parsed by the local parser to identify the users requested program content, namely the Seinfeld re-run.

[0038] Similarly, the user could utter:

[0039] “I want to watch Seinfeld now.”

[0040] This would cause the system to immediately switch channels to the one carrying the Seinfeld broadcast.

[0041] In some instances, the LR grammar of the local parser may not be sufficient to resolve the user's input without ambiguity. This will occur where the local parser identifies sentence fragments that, taken out of context, may have several meanings. For example, the following input:

[0042] “I want to watch Seinfeld and record it.”presents the following ambiguity. The local parser may determine with equal validity that the program requested by the user is either (a) “Seinfeld” or (b) “Seinfeld And Record It.”

[0043] To resolve such ambiguities, the system includes a second parser, the global parser 70. The global parser 70 also monitors the text stream as well as receiving input from the local parser 56. The global parser has a set of decision trees 72 that it uses to resolve ambiguities such as the one illustrated above. More specifically, global parser 70 has a set of decision trees 72, one decision tree for each meaning. Each decision tree is also in charge of solving ambiguities in the meaning represented. Each decision tree is a binary tree structure in which the root node and intermediate nodes each contain a question that may be answered either YES or NO. Answering a given question branches left or right to a successively lower node, depending on whether the answer was YES or NO. The final nodes or leaf nodes contain the determination of the meaning that has been expressed. The system uses this decision information to resolve ambiguities in selecting the proper frame from frame data store 50 and in assigning keywords to the proper slots.

[0044] After the frame data store has been populated by the local and global parsers, the word selector module 74 accesses the data store 50 to obtain the applicable list of keywords for sending to the word list buffer 34. The selector module may employ the services of an electronic thesaurus 76 to generate synonyms or additional words to enrich the key word list supplied to the word list buffer. The word list selector module might, for example, extract the word “football” from frame data store 50 and obtain additional words such as “touchdown,” “Green Bay Packers,” or “NFL” from the thesaurus 76. In this regard, note that the additional words selected need not necessarily constitute synonyms in the dictionary sense. Rather, they may constitute additional words or related words that are often found in natural language speech involving the subject of the user-specified key word.

[0045] From the foregoing, it will be appreciated that the automatic search mechanism of the invention greatly eases the task of identifying program material in a television system having access to many channels of information. While the invention has been described in its presently preferred embodiment, it will be understood that the invention is capable of modification and change without departing from the spirit of the invention as set forth in the appended claims.