PHRASE SPOTTING SYSTEMS AND METHODS

[0001] Methods and systems for phrase spotting are provided.

[0002] In various applications, it is desirable to detect when a specific phrase has been spoken. However, current phrase spotting products can suffer from inaccurate phrase spotting. In addition, such products often do not provide the speaker with means to guide the system and improve its performance. In particular, the individual accents of speakers can adversely affect the accuracy of spotting specific phrases.

[0003] In order to improve the accuracy of phrase spotting systems, a training mode can be utilized. During training, a user is asked to provide speech samples in response to prompts. While such training can be effective at increasing the accuracy of speech to text systems for individual users, such training is time consuming. Additionally, when attempting to spot specific phrases spoken by an unknown or random speaker, traditional training as described above may prove to be impractical. In order to provide improved accuracy for a larger group of users such as callers into a contact center, individual users can be assigned to profile categories. For example, a user with an Australian accent can be associated with a profile that is intended to accurately spot phrases spoken with that accent.

[0004] In certain contexts, such as in contact centers, it can be desirable to monitor audio signals comprising speech for certain key words or phrases. For example, an enterprise might be interested in monitoring conversations between contact center agents and customers for certain words. As a particular example, a contact center server can monitor calls in real time for the word “supervisor.” If that word is detected, it can be used as a trigger to a supervisor to intervene in the call, or to monitor the ongoing call. As another example, a financial institution may routinely record customer calls, so that an accurate record of customer instructions can be maintained. If a question later arises as to the content of a customer's earlier instructions, it can be desirable to search through the recordings made of earlier conversations between the customer and contact center agents, to locate and play back the earlier instructions. However, in such situations, there is little or no opportunity to train the system to accurately recognize the speech being monitored. Accordingly, previous techniques for training systems have been ineffective in these other contexts.

[0005] Systems and methods that provide for improved phrase spotting accuracy in connection with word or phrase spotting are provided. More particularly, speech is monitored in real time for one or more words or phrases. When the system determines that a certain word or phrase has been uttered, a message can be displayed that asks for confirmation that the phrase identified by the system was in fact spoken. In response to confirmation that the specific phrase has been spoken, a copy of the phrase can be stored in a library of phrases and used to increase the accuracy of speech recognition with respect to the individual speaker, another user, or a group of users with some commonality.

[0006] In accordance with at least some embodiments of the present disclosure, an audio stream, for example, but not limited to a telephone conversation, is monitored by a monitor application. The monitor application listens to the audio stream for one or more specified phrases. As used herein, a phrase can include a phrase, word, phoneme, or other portion of speech. In response to a determination by the monitor application that a specified phrase has been identified, a hidden Markov model can be created. Additionally, verification from a user that the phrase was included in the audio stream is requested. This request for verification can be made to the user providing the speech. Moreover, the request for verification can be presented as a displayed message asking whether the specified phrase was in fact spoken by the user. If the user answers in the affirmative, the portion of the audio stream that includes the specified phrase can be stored. The stored phrase can then be used by the monitor application to identify future utterances of that phrase. The user who is asked to verify that a specific phrase has been spoken does not have to be the same person whose voice stream is monitored.

[0007] Systems implementing embodiments of the present disclosure can provide a communication server that executes a monitor application. The monitor application can be provisioned by an administrator or supervisor to monitor one or more audio streams for a specified phrase. The monitor application then operates to “listen” to voice communications for the one or more specified phrases. In response to determining that a specified phrase has been uttered, the monitor application requests confirmation from a human listener or participant who may be an agent or supervisor user that a specified phrase has in fact been uttered. The request may also include presenting text to a contact center customer user stating the phrase has been identified by the monitor application, and further requesting acknowledgement from the customer user. In additional embodiments, the user may be one or more users and may have different relationships that the ones used above. In response to input confirming the presence of the specified phrase in the audio stream, the monitor application can store the portion of the audio stream that includes the specified phrase. The stored phrase can then be used to identify subsequent utterances of that phrase. By thus providing an actual example of the customer user's utterance of the specified phrase, that phrase can be identified with greater accuracy in future voice communications from that customer user or from other customer users with a similar accent or speech pattern. Alternatively, if the agent or customer user indicates that the specified phrase was not uttered, the monitor application can continue to use the standard text file applied to users generally. Additionally, the voice segment which was mistakenly identified as containing the specific phrase may be stored for future reference and used to reduce future false detection of the specific phrase. In accordance with further embodiments of the present disclosure, a library of specified phrases can be developed for individual users or classes of users with similar accent or speech patterns. Moreover, the library can be parsed into a library of phonemes that can be used to identify specified phrases included in or comprising words or phrases other than those for which samples have been obtained. Similarly, the system provides users with the ability to alert the system whenever the user determines that a specific phrase has been uttered and the system has not spotted it. When this occurs, the system may store the relevant voice stream segment for future reference in a manner similar to the one described above.

[0008] Additional features and advantages of embodiments of the present invention will become more readily apparent from the following description, particularly when taken together with the accompanying drawings.

[0013] FIG. 1 is a block diagram depicting components of a communication system 100 in accordance with embodiments of the present invention. In this example, the communication system 100 includes a contact center 104. In general, the contact center 104 can be in communication with one or more communication endpoints or devices 108 via one or more communication networks 112. Examples of communication endpoints 108 include but are not limited to smartphones, cellular telephones, desktop telephones, desktop computers, laptop computers, thin client devices, tablet computers, or any other device capable of supporting communications, including but not limited to voice communications, between a user 110 (e.g., a customer) and a human user or automated agent user 122 associated with the contact center 104. In accordance with further embodiments, a communication endpoint 108 can exchange non-voice data between a customer user 110 and the contact center 104 or other components or endpoints. Accordingly, a communication endpoint 108 can include various user output and user input devices or facilities, including but not limited to speakers, microphones, text displays, display screens, video cameras, keypads, keyboards, porting devices, and touch screen displays. The communication network 112 can include the Internet, a local area network (LAN), wide area network (WAN), public switched telephone network (PSTN), wireless networks, cellular telephony networks, satellite telephony networks, virtual or cloud networks, remote or off-premise networks, or a plurality of networks in any combination.

[0014] The contact center 104 generally includes a communication server 116. The communication server 116 can include (or be attached to) an automatic contact (or call) distribution system (ACD) server. The communication server 116 is illustratively the Communication Manager™ enterprise communication-based ACD system available from Avaya, Inc. The communication server 116 is interconnected to a plurality of agent workstations or endpoints 120. For example, the agent workstations 120 may be connected to the ACD server 116 by an enterprise network or networks 128. As examples, the agent workstations 120 can comprise general purpose computers, telephones, thin client devices, or other devices, that are operable to place an associated agent 120 in voice communication with a customer user 110 of a communication endpoint 108. As further examples, an agent workstation 120 can comprise an automated resource, such as an interactive voice response (IVR) system or server. In accordance with another example embodiment, an agent workstation 120 can comprise an automated resource, such as a virtual assistant (also known as a virtual agent or automated chat agent).

[0015] The communication server 116 generally functions to connect agent workstations 120 to communication devices or endpoints 108 through the communication network 112, to allow the agent users 122 to service customer 110 contacts. In accordance with embodiments of the present disclosure, the contacts comprise voice communications. For example, contacts can be in the form of voice telephony communications. As can be appreciated by one of skill in the art after consideration of the present disclosure, the communication server 116 can maintain one or more queues for organizing and maintaining or holding contacts waiting for handling by a contact center agent user 122. For example, a single queue or plurality of queues can be provided to sort contacts according to various parameters. Agent users 122, automated attendants, or other resources associated with or provided by the agent workstations 120 are assigned to provide services to contacts that have been placed within one or more of the queues based on availability and/or weighting factors. In addition to supporting voice communications, the workstations 120 can include a user output in the form of a display and a user input in the form of a keypad, keyboard, touch screen input, pointing devices, video cameras, or the like.

[0016] In addition, embodiments of a system 100 as described herein can include one or more supervisor or administrator devices 124. The supervisor device 124 is generally in communication with the communication server 116 via the communication network 112 and/or the enterprise network 128. For example, if the supervisor device 124 is on the premises of the contact center 104, communications with the communication server 116 may be over a portion of the enterprise network 128 comprising a wireline or wireless network. As another example, the supervisor device 124 may be in communication with the communication server 116 over the communication network 112, for example via a cellular telephony data network, a wired or wireless connection outside of the enterprise network 128, or the like. In general, the supervisor device 124 comprises functionality that allows a supervisor user 126 to monitor communications between a communication device 108 and the contact center 104, to join in-process communications, to initiate communications, and to control aspects of the operation of the contact center 104. Accordingly, the supervisor device 124 can comprise any device, including a mobile device, capable of presenting information to a supervisor user 126. Examples of the supervisor device 124 include, but are not limited to, a telephone, a tablet computer, a smartphone, a laptop computer, a desktop computer, a netbook, or the like. A supervisor may use one or more supervisor devices 124 to monitor communications.

[0017] FIG. 2 is a block diagram depicting components of a communication server 116 in accordance with embodiments of the present disclosure. The communication server 116 includes a processor 204 capable of executing program instructions. The processor 204 can include any general purpose programmable processor or controller for executing application programming. Alternatively, the processor 204 may comprise a specially configured application specific integrated circuit (ASIC). The processor 204 generally functions to run programming code implementing various functions performed by the communication server 116. For example, the processor 204 can implement functions including but not limited to monitoring in-process or recorded audio streams for specified phrases through execution of a monitor application 132 as described herein.

[0018] The communication server 116 additionally includes memory 208. The memory 208 can be used in connection with the execution of programming by the processor 204 of the communication server 116, and for the temporary or long term storage of data and/or program instructions. In accordance with embodiments of the present disclosure, the monitor application 132 may be stored on and executed in connection with the memory 208 of the communication server 116. As a further example, the memory 208 can store user profile data 220, and recorded voice communications 224. As still other examples, the communication server 116 can include the automatic call distribution (ACD) application 228, call queue data 232, and one or more communication applications 236 such as chat, co-browsing, etc. The memory 208 of the communication server 116 can include solid state memory that is resident, removable and/or remote in nature, such as DRAM and SDRAM. Moreover, the memory 208 can include a plurality of discrete components of different types and/or a plurality of logical partitions. The memory 208 can also include or be associated with a database or storage system that is integrated with or separate from but interconnected to the communication server 116. In accordance with still other embodiments, the memory 208 comprises a non-transitory computer readable storage medium. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, NVRAM, or magnetic or optical disks. Volatile media includes dynamic memory, such as main memory. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, magneto-optical medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, a solid state medium like a memory card, any other memory chip or cartridge, or any other medium from which a computer can read.

[0019] In addition, user input devices 212 and user output devices 216 may be provided. With respect to the communication server 116, such devices 212 and 216 can be used in connection with the monitoring and control of the contact center 104 and associated communications by a supervisor user 126 and/or an agent user 122. However, a supervisor user 126 or agent user 122 typically interfaces with the communication server 116 through a supervisor device 124 or agent workstation 120, where the supervisor device 124 or agent workstation 120 each are associated with one or more user inputs and one or more user outputs. Examples of user input devices 212 include a keyboard, a numeric keypad, a touch screen, a microphone, mouse, video camera, scanner, and pointing device combined with a screen or other position encoder. Examples of user output devices 216 include a display, a touch screen display, a speaker, and a printer. The communication server 116 also generally includes a communication interface 240 to interconnect the communication server 116 to the networks 112 and/or 128.

[0020] FIG. 3 depicts a user interface 304 in accordance with embodiments of the present disclosure. The user interface 304 can be provided by or in connection with a user output device (e.g., a display) of a communication endpoint 108, an agent workstation 120, and/or supervisor device 124. The user interface 304 can be generated through or in connection with the operation of the monitor application 132 running on the communication server 116, and/or in connection with a companion application, such as a specially provided application and/or a browser application, provided as part of a communication endpoint 108, an agent workstation 120, or supervisor device 124. Accordingly, the user interface 304 is generally presented to a customer user 110, an agent user 122, and/or a supervisor user 126. Moreover, the user interface 304 can be interactive in that it can provide fields, buttons, menus, or other features to enable the user interface 304 to receive input from a customer user 110, an agent user 122, or a supervisor user 126, as well as to present information to the customer user 110, agent user 122, or supervisor user 126 graphically, by means of an audio signal, or through any other suitable means.

[0021] The user interface 304 can operate to present a query to a customer user 110, an agent user 122, or a supervisor user 126 regarding whether a specified phrase has been voiced in a speech confirmation area 308. In the example query shown in the figure, the question 312 “Did you say: ‘How may I help you’?” is presented in the query area 308. This is an example of a query 312 that might be presented to an agent user 122 in response to the monitor application 132 monitoring for the specified phrase “how may I help you”. The query area 308 can additional include input buttons, including a yes button 316 and a no button 320. Accordingly, the agent user 122 may confirm that the specified phrase was in fact voiced by clicking the yes button 316. Alternatively, the agent user 122 can indicate that the specified phrase has not been voiced by clicking the no button 320. In accordance with still other embodiments, similar queries 312 can be presented to a customer user 110 and/or a supervisor user 126, as described in greater detail elsewhere herein. Continuing the example of a user interface 304 that is presented to an agent user 122, active contact information 324 can also be presented. Such active contact information 324 can include the name of a customer user 110, and other information about the customer user 110 that may be available to the contact center 104. In addition, a user interface 304 can present other controls, for example controls typically provided as part of a soft phone application, and controls that allow access to other communication applications 236, such as but not limited to email, text messaging, instant messaging, or the like. Accordingly, the user interface 304 allows the monitor application 132 to collect feedback from a participant in or an observer of a communication, regarding whether a specified word or phrase has been voiced. In general, any participant, whether a user, agent, supervisor, etc., may be asked to acknowledge that a specific phrase was uttered by other participants.

[0022] With reference now to FIG. 4, aspects of a method for monitoring an audio stream for a specified phrase are depicted. Generally, the method begins with a step of provisioning a monitor application 132, in step 404. This includes specifying one or more spoken phrases that the monitor application is to listen for. A given phrase can be specified by a system administrator or supervisor user 126 as a text string. As used herein, a phrase can include a single word or multiple words. In actual use, a system 100 is configured to monitor for a specified phrase that acts as a trigger for an event, or that is otherwise significant. In addition, monitoring of audio streams is often desirable in connection with the operation of a contact center 104, although embodiments can be applied in different contexts. As examples, the detection of specific phrases within an audio stream can be used to identify when intervention in a communication session by a supervisor user 126 is desirable, to ensure that customer service guidelines are being met by agent users 122, or to identify instructions received from a customer user 110.

[0023] At step 408, an audio stream is monitored. As used herein, an audio stream can be received by a communication server 116 running the monitor module or application 132 as part of a voice telephony session between a customer user 110 and resource associated with the communication server 116, such as a human agent user 122, or an automated agent. Accordingly, an audio stream can include a real time voice communication session. In accordance with still other embodiments, the audio stream can include a recorded audio stream.

[0024] A determination can then be made as to whether a specified phrase has been identified within the monitored audio stream, in step 412. In accordance with a specific example implementation, the occurrence of a specified phrase is, at least in an initial instance, determined by comparing a file representing the sound profile of the specified phrase that is generated from a text to speech facility of the monitor application 132 to the sound profile of the monitored audio stream. Moreover, identification of a phrase is determined probabilistically. For example, the monitor application 132 can determine that a phrase is included in a monitored audio stream if it concludes with some minimum degree of certainty that the phrase was in fact uttered. Until a specified phrase has been identified, the monitor application 132 can continue to monitor the audio stream.

[0025] The monitor application may be based on sub-word hidden Markov models (HMMs), which are a sophisticated way of averaging multiple phrases to obtain a robust pattern to match. When a specified phrase has been identified, a new model may be constructed in step 416. Additional HMMs may be built for each word or phrase later in the method using an associated training set (in step 432). In response to a determination that the specified phrase has been identified, a request for confirmation that the specified phrase was in fact included in the audio stream is generated, in step 416. For example, the monitor application can operate to cause a user interface 304 to display a request for verification that is presented to a user, such as a customer user 110 comprising a customer or other third party, an agent user 122, or a supervisor user 126, as text. Typically, the request for verification is sent to the user who provided the portion of the audio stream in which the monitor application 132 identified the phrase. However, in other embodiments, a human listener other than the user who provided the portion of the audio stream in which the phrase was identified can be queried for verification.

[0026] At step 420, a determination is made as to whether confirmation that the audio stream included the phrase has been received from a user. For example, where a query regarding the occurrence of a specified phrase is presented to a user through a user interface 304, the user can confirm the utterance of the phrase by clicking a “yes” button 316. If user confirmation has not been received, the process can return to step 408, and the audio stream can continue to be monitored. If user confirmation has been received, an audio portion that included the phrase may be stored, in step 424.

[0027] HMMs may correspond to any level of representation of the speech: phoneme, word, phrase, etc., and may be concatenated to construct a higher level model. A dynamic algorithm may be structured to determine and record an optimum alignment of frames from the phrase to the corresponding HMM. The HMM can be trained using a variety of methods. For example, a canonical approach uses “maximum likelihood” to train the HMM, and “discriminative training” takes close mismatches into account during training, in step 428. In response to receiving confirmation that the specified phrase was uttered in the monitored audio stream, a profile for the user that provided the phrase can be created or updated, in step 432. Creating or updating the user profile can include storing a portion of the audio stream containing the phrase as user profile data 220. As can be appreciated by one of skill in the art after consideration of the present disclosure, the portion of the audio stream can be stored as a representation of the audible signal. For example, the portion of the audio stream can be stored as a .wav file, mp3 file, or other format. Moreover, storing the portion of the audio stream can include parsing in order to separate different phrases, words, and/or phonemes from one another. In addition, the audio file can be filtered or otherwise processed as part of creating or updating the user profile. Additional HMMs may be created and stored in the library.

[0028] In a preferred embodiment, the segment of the media stream containing the phrase or the speech parameters extracted from the segment of the media stream are stored and utilized to spot the phrase in the media stream of the same speaker or alternatively in media streams originating from other speakers. In this aspect, the segment of the media stream or the associated speech parameters are used in conjunction with the text that specifies the phrase the system attempts to spot. As the library of media stream segments or its corresponding parameters increase, the system may discontinue the use of the text which identifies the desired phrase and switch to using the specific media stream segment or its associated parameters. The library associated with each phrase may be constructed from phrases originating from multiple users over time, which may provide improved phrase spotting for a variety of users who may speak with different accents.

[0029] At step 436, a determination can be made as to whether operation of the monitor application 132 should be continued. If operation is not to be continued, the process can end. If operation is to continue, the created or updated user profile data 220 and/or the updated phrase library/profile can be applied, in step 440. The process can then return to step 408, and the audio stream can continue to be monitored with improved phrase recognition accuracy, since the continued monitoring uses the created or updated user profile data 220 and/or the updated phrase library/profile. As can be appreciated by one of skill in the art after consideration of the present disclosure, the user profile data 220 for a particular user and/or the updated phrase library/profile can be applied during the continuation of the communication session that resulted in the creation of or update to the user profile data 220 for that user and/or the updated phrase library/profile. Alternatively or in addition, user profile data 220 generated in connection with an earlier communication session and/or the updated phrase library/profile can be applied in connection with a later communication session with the same user or with other users. In accordance with still other embodiments, user profile data 220 generated in connection with a later communication session can be applied to an earlier, recorded communication session. For example, where a later communication session references an earlier communication session for which a recording 224 is available, a sample of a phrase obtained from a customer user 110 during the later communication session can be used to more accurately identify the recorded communication session, and/or the portion of a recorded communication session in which the phrase was uttered.

[0030] Embodiments of the present disclosure provided systems and methods to improve the accuracy of word or phrase spotting systems. Although various examples have been discussed that relate to use in connection with contact centers, other applications are possible. For example, systems and methods as disclosed herein can be used to train any system for identifying phrases in any real time or recorded speech.

[0031] For simplicity of explanation, the notion of storing (and using) media string segments representing specific phrases may be used interchangeably with the notion of storing speech parameters associated with the media stream segments.

[0032] The foregoing discussion of the invention has been presented for purposes of illustration and description. Further, the description is not intended to limit the invention to the form disclosed herein. Consequently, variations and modifications commensurate with the above teachings, within the skill or knowledge of the relevant art, are within the scope of the present invention. The embodiments described hereinabove are further intended to explain the best mode presently known of practicing the invention and to enable others skilled in the art to utilize the invention in such or in other embodiments and with various modifications required by the particular application or use of the invention. It is intended that the appended claims be construed to include alternative embodiments to the extent permitted by the prior art.