Analyzing audio input for efficient speech and music recognition

[0001] This application claims priority from U.S. Provisional Ser. No. 61/993,709, filed on May 15, 2014, entitled ANALYZING AUDIO INPUT FOR EFFICIENT SPEECH AND MUSIC RECOGNITION, which is hereby incorporated by reference in its entirety for all purposes.

[0002] This relates generally to audio analysis and, more specifically, to analyzing audio input for efficient speech and music recognition.

[0003] Audio recognition, such as speech and music recognition, has become increasingly significant in electronic devices, and in particular, portable electronic devices (e.g., portable media players, cellular telephones, and tablet computers). For example, virtual assistant applications running on electronic devices can apply speech recognition to decipher spoken user requests and deliver relevant services based on the deciphered spoken user request. In another example, music identification software applications running on electronic devices can apply music recognition to analyze unknown excerpts of music and provide music identification services. Currently, applications providing speech recognition related services typically operate independently from those providing music recognition related services. Users must therefore select different applications depending on the type of service desired and the type of audio input (e.g., speech or music) provided, which negatively impacts user experience.

[0004] Systems and processes for analyzing audio input for efficient speech and music recognition are provided. In one example process, an audio input can be received. A determination can be made as to whether the audio input includes music. In addition, a determination can be made as to whether the audio input includes speech. In response to determining that the audio input includes music, an acoustic fingerprint representing a portion of the audio input that includes music is generated. In response to determining that the audio input includes speech rather than music, an end-point of a speech utterance of the audio input is identified.

[0009] In the following description of examples, reference is made to the accompanying drawings in which it is shown by way of illustration specific examples that can be practiced. It is to be understood that other examples can be used and structural changes can be made without departing from the scope of the various examples.

[0010] The present disclosure relates to systems and processes for audio analysis for efficient speech and music recognition. As described above, applications providing speech recognition related services typically operate independently from those providing music recognition related services. In accordance with various examples described herein, speech recognition related services can be integrated with music recognition related services in a single application to provide enhanced user experience. Specifically, a virtual assistant can be configured to provide both speech recognition and music recognition related services. In order for both types of services to be provided seamlessly, it can be desirable for the virtual assistant to determine whether a received audio input includes speech or music and thereby provide the appropriate services to the user automatically based on the determination. Further, by determining whether the audio input includes speech or music, system resources can be used more efficiently. This can be because processing audio input for music recognition incurs significantly greater computational costs than processing audio input for speech recognition. Computational resources can thus be conserved by processing audio input for music recognition only when the audio input is determined to include music. In this way, audio input can be analyzed more quickly and efficiently for speech and music recognition.

[0011] In one example process for analyzing audio input for efficient speech and music recognition, an audio input can be received. A determination can be made as to whether the audio input includes music. In addition, a determination can be made as to whether the audio input includes speech. In response to determining that the audio input includes music, an acoustic fingerprint representing a portion of the audio input that includes music is generated. In response to determining that the audio input includes speech rather than music, an end-point of a speech utterance of the audio input is identified.

1. Process for Analyzing Audio Input

[0012] FIG. 1 illustrates exemplary process 100 for analyzing an audio input for efficient speech and music recognition according to various examples. In some examples, process 100 can be performed using a system implementing a client-server model. Specifically, process 100 can be performed using a system capable of implementing a virtual assistant (e.g., system 200, described below). In other examples, process 100 can be performed using a stand-alone electronic device.

[0013] At block 101 of process 100, an audio input can be received. The audio input can include various combinations of speech, music, background noise, and silence. In particular, the audio input can include speech only, music only, speech with background music, speech with background noise, music with background noise, or neither speech nor music. In some examples, the audio input can be received by a user device (e.g., user device 202, described below) or by a remote system (e.g., server system 210 or virtual assistant server 214, described below). The audio input can be received in the form of sound waves, an audio file, or a representative audio signal (analog or digital).

[0014] At block 103 of process 100, a determination can be made as to whether the audio input includes music. In some examples, block 103 can be performed by a user device (e.g., user device 202), while, in other examples, block 103 can be performed by a remote server (e.g., server system 210 or virtual assistant server 214). Distinguishing music from non-music in the audio input can be advantageous in determining whether computationally intensive audio processing (e.g., acoustic fingerprinting of block 107) associated with music recognition is necessary. This enables greater efficiencies in audio processing for speech and music recognition. Additionally, services such as music identification can be automatically provided to the users based on the determination at block 103, thereby enhancing user experience.

[0015] The determination at block 103 can be made using any known audio classifier that can distinguish music from non-music based on features extracted from the audio input. In some examples, the audio input can be analyzed on a segment-by-segment basis. In these examples, the audio input can be divided into a sequence of audio segments where each audio segment has a duration of 0.5 to 3 seconds. The audio segments can be shifted such that adjacent audio segments overlap. Each audio segment can be analyzed to extract one or more features that characterize the audio segment. The one or more extracted features of each audio segment can be analyzed using the audio classifier to determine whether or not each audio segment includes music. In some cases, the audio segments are analyzed sequentially.

[0016] The one or more extracted features of each audio segment can include any quantifiable feature that characterizes the audio segment. In some examples, the one or more extracted features can include standard low-level features such as, root mean square (RMS) amplitude, zero crossing rate (ZCR), spectral centroid, spectral roll-off, spectral flux, and spectral flatness. In other examples, the one or more extracted features can include frequency cepstrum coefficients, linear predictive cepstral coefficients, bark scale frequency cepstral coefficients, mel-frequency discrete wavelet coefficients, mel-frequency cepstral coefficients, or the like.

[0017] The one or more extracted features of each audio segment can include statistical features (e.g., mean, variance, or standard deviation) of feature values extracted from multiple audio frames within each audio segment. For example, an audio segment can include a predetermined number of audio frames and the one or more extracted features of the audio segment can include the variance of the RMS amplitudes extracted from each of the audio frames within the audio segment.

[0018] The audio classifier can operate to analyze the one or more extracted features of an audio segment and provide an output manifesting the presence or absence of music in the audio segment. In some examples, the audio classifier can include a neural network classifier, such as a multi-layer perceptron, having one or more hidden layers. Neural network classifiers can be advantageous in providing a more robust and accurate determination of whether or not the audio input includes music. However, neural network classifiers can incur higher computational costs compared to other classifiers, which increases latency and system power consumption.

[0019] In other examples, the audio classifier can include a rule-based classifier that employs logical operations to determine the presence or absence of music. Specifically, the ruled-based classifier can apply any suitable logical condition to the one or more extracted features of the audio segment to determine whether or not an audio segment includes music. In some examples, the rule-based classifier can determine whether or not an audio segment includes music by comparing the one or more extracted features of the audio segment to one or more predetermined thresholds. In one such example, the rule-based classifier can determine that the audio segment includes music based on a first extracted feature (e.g., variance in amplitude) exceeding a first predetermined threshold and a second extracted feature (e.g., zero crossing rate) exceeding a second predetermined threshold. It should be recognized that in other examples, the first extracted feature and the second extracted feature can be any extracted feature described above. Rule-based classifiers can have lower computational costs than neural network classifiers. However, rule-based classifiers can be less accurate than neural network classifiers.

[0020] At block 105 of process 100, a determination can be made as to whether the audio input includes speech. In some examples, block 105 can be performed by the user device (e.g., user device 202), while, in other examples, block 105 can be performed by a remote server (e.g., server system 210 or virtual assistant server 214). Determining whether the audio input includes speech at block 105 can be used to identify an end-point of a speech utterance (e.g., block 113, described below) and thereby determine when to stop receiving audio input at block 101. In addition, distinguishing speech from non-speech in the audio input can be advantageous in determining the relevant portions of the audio input to perform speech recognition upon. This enables greater accuracy in deciphering a speech utterance and better efficiencies in audio processing.

[0021] In some examples, determining whether the audio input includes speech can include determining whether the one or more extracted features of the audio input satisfy one or more predetermined logical conditions. In one such example, determining whether the audio input includes speech can include determining whether the one or more extracted features exceed one or more predetermined thresholds for longer than a predetermined threshold duration.

[0022] In some examples, determining whether the audio input includes speech can be performed on a frame-by-frame basis. The audio input can be divided into a sequence of audio frames with each audio frame having a duration of 5 to 50 ms. In some examples, the duration of an audio frame can be different from the duration of an audio segment. The audio frame can be shifted such that adjacent audio frames overlap. Each audio frame can be analyzed to extract one or more features that characterize the audio frame. The one or more extracted features of each audio frame can be analyzed to determine whether or not the audio frame includes speech. In some cases, the audio frames can be analyzed sequentially.

[0023] An audio frame can be determined to include speech by comparing the one or more features of the audio frame to a predetermined threshold. For example, the audio frame can be determined to include speech when the short-term energy level of the audio frame is greater than a predetermined threshold. Conversely, the audio frame can be determined to not include speech if the short-term energy level of the audio frame is less than the predetermined threshold. In some cases, the audio input can be determined to include speech if the number of consecutive audio frames determined to include speech exceeds a predetermined number.

[0024] It should be recognized that in other examples, the one or more extracted features for determining whether an audio frame includes speech can include any quantifiable feature that characterizes the audio frame. For example, the one or more extracted features of each audio frame can include time domain features such as, zero crossing rates, spectral centroid, spectral roll-off, spectral flux, spectral flatness, and autocorrelation. In addition, the one or more extracted features for each audio frame can include frequency domain features. Further, various suitable predetermined logical conditions can be applied to the one or more extracted features to determine whether or not the audio frame includes speech.

[0025] In some examples, blocks 103 and 105 can be performed independently with the audio input being analyzed separately for music and speech. In these examples, separate software modules or algorithms can be used to perform blocks 103 and 105. In other examples, blocks 103 and 105 can share one or more common functions, software modules, or algorithms. In a specific example, dividing the audio input into audio frames or segments and extracting one or more features from the audio frames or segments of blocks 103 and 105 can be performed by one or more common software modules or algorithms.

[0026] In some examples, the determining of whether the audio input includes music at block 103 can continue even after it is determined that the audio input includes speech. In particular, the determining at block 103 can continue even after one or more audio frames are determined to include speech and can cease at a predetermined time after an end-point of a speech utterance in the audio input is identified (e.g., block 113, described below). This can enable any music that is subsequent to the speech to be analyzed.

[0027] At block 107 of process 100, an acoustic fingerprint representing a portion of the audio input that includes music can be generated in response to determining that the audio input includes music. Specifically, in response to determining that an audio segment of the audio input includes music, the audio segment and subsequent audio segments of the audio input can be processed to generate an acoustic fingerprint. The acoustic fingerprint can be used to provide a music identification service to the user. This can be desirable for enhancing user experience by providing information that is relevant to the user's environment.

[0028] An acoustic fingerprint can be a compact digital signature that summarizes the perceptually relevant characteristics of the audio input. The acoustic fingerprint can be generated from an uncompressed representation of the portion of the audio input having music in order to reduce errors in the acoustic fingerprint. Generating the acoustic fingerprint from a portion of the audio input can include dividing the portion into audio frames and extracting a set of features (e.g., mel-frequency cepstral coefficients, spectral flatness measure, band representative vectors, linear predictive coding coefficients, modulation frequency, etc.) from each audio frame. The extracted features can be assembled into a feature vector, which can be transformed using derivatives, means, or variances. The feature vector can be mapped into a more compact representation using algorithms (e.g., Hidden Markov Model or Principal Component Analysis) and quantization to produce the acoustic fingerprint. The acoustic fingerprint can be a single fingerprint that summarizes the audio input or can be a series of sub-fingerprints generated for each audio frame or segment of the audio input. The acoustic fingerprint can subsequently be used to obtain an identity of the music in the audio input.

[0029] Examples for generating acoustic fingerprints and performing music identification using acoustic fingerprints are described in Haitsma et al., “A highly robust audio fingerprinting system”, In Proceedings of the International Symposium on Music Information Retrieval (ISMIR), 2002, and Avery Li-Chun Wang, “An industrial strength audio search algorithm”, In Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2003, both of which are incorporated herein by reference in their entirety.

[0030] In some examples, in response to determining that the audio input includes music, receiving the audio input at block 101 can continue and generating the acoustic fingerprint at block 107 can continue until after the identity of the music is obtained or until after a predetermined duration in which the identity of the music cannot be successfully obtained.

[0031] Generating the acoustic fingerprint at block 107 can be performed by the user device (e.g., user device 202) or by the remote server (e.g., server system 210 or virtual assistant server 214). However, generating the acoustic fingerprint using the user device can be advantageous in reducing the error rate in the acoustic fingerprint. This can be because the user device receives an uncompressed representation of the audio input and generating the acoustic fingerprint using the uncompressed representation of the audio input minimizes errors and inaccuracies in the acoustic fingerprint. In contrast, the server can receive a compressed representation of the audio input, which can be less desirable for generating an accurate acoustic fingerprint.

[0032] Generating an acoustic fingerprint incurs significant computation costs. It can thus be desirable to generate an acoustic fingerprint only when music recognition is to be performed on the audio input. Accordingly, in this example, an acoustic fingerprint can be generated from the audio input only in response to determining that the audio input includes music. Thus, in this example, an acoustic fingerprint may not be generated when it is determined that the audio input includes only speech or non-music audio. In this way, computational resources can be conserved, which results in efficient audio analysis for speech and music recognition.

[0033] At block 109 of process 100, the determining of whether the audio input includes speech at block 105 can be discontinued in response to determining that the audio input includes music. As described above, blocks 103 and 105 can be performed in parallel. In some examples, the determining of whether the audio input includes speech at block 105 can still be in progress when it is determined that the audio input includes music. In these examples, the determining at block 105 can be discontinued in response to determining that the audio input includes music. This can be advantageous in conserving computational resources. Further, any speech that may be commingled with music in the audio input can subsequently be identified and processed at block 117, described below, using robust speech recognition algorithms.

[0034] In some examples, the determining at block 105 can be used to identify an end-point of a speech utterance and thereby determine when to cease receiving the audio input at block 101. In these examples, by ceasing or discontinuing the determining at block 105, audio input can continue to be received at block 101 until after the music in the audio input is identified or until after a predetermined duration in which the music cannot successfully be identified.

[0035] At block 113 of process 100, an end-point of a speech utterance of the audio input can be identified in response to determining that the audio input includes speech rather than music. In some examples, the end-point can be identified based on predetermined logic or rules applied to the one or more features extracted from the audio input at block 105. In one example, the end-point can be based on the time at which an extracted feature transitions from a value greater than a predetermined threshold to a value less than the predetermined threshold and remains below the predetermined threshold for longer than a predetermined duration. In another example, the end-point can be based on the time at which an extracted feature value decreases by a predetermined amount within a predetermined duration. By identifying the end-point, the portion of the audio input containing the speech utterance can be identified, which can be advantageous in enabling more efficient and accurate speech recognition and speech processing (e.g., speech processing at block 117, described below). In addition, the receiving of the audio input at block 101 can cease or discontinue after the end-point is identified.

[0036] In some examples, the determining of whether the audio input includes speech at block 105 can be stopped or discontinued in response to identifying the end-point at block 113. In some instances, the determining of whether the audio input includes music at block 103 can also be stopped or discontinued in response to identifying the end-point at block 113. In other examples, the determining at blocks 103 and/or 105 can continue after identifying the end-point at block 113.

[0037] At block 115 of process 100, the receiving of the audio input can be ceased or discontinued in response to determining that the audio input includes neither speech nor music for a predetermined duration. For example, a signal to begin receiving an audio input can be received via input to the user device and in response to receiving the signal, the user device can begin to receive the audio input at block 101. At blocks 103 and 105, it can be determined that the audio input includes neither speech nor music for a duration greater than a predetermined duration. The duration can be measured from the beginning of the audio input. In response to such a determination, the receiving of the audio input at block 101 can be ceased or discontinued. In some examples, the determining of blocks 103 and 105 can also be ceased or discontinued in response to determining that the audio input includes neither speech nor music for a predetermined duration. Further, in some examples, any communication from the remote server (e.g., server system 210 or virtual assistant server 214) can be disregarded and any requested processes running on the remote server can be terminated in response to determining that the audio input includes neither speech nor music for a predetermined duration. The absence of music and speech in the audio input can suggest that the user does not intend to receive any services related to speech or music recognition. Therefore, process 100 for analyzing audio input can be ceased or discontinued to enable efficient use of computational resources.

[0038] At block 117 of process 100, the audio input can be processed for speech in response to determining that the audio input includes music or speech. Processing the audio input for speech can include one or more of identifying a speech utterance of the audio input, performing speech recognition (e.g., speech to text conversion) to obtain a textual representation of the speech utterance, interpreting the textual representation of the speech utterance to obtain an inferred user intent, and identifying a task flow with steps and parameters designed to accomplish the inferred user intent.

[0039] In some examples, the entire audio input can be processed for speech in response to determining that the audio input includes music or speech. In other examples, only the portion of the audio input after which either music or speech is initially detected, can be processed for speech. In one such example, it can be determined that an audio segment of the audio input is the first audio segment that includes music and that the audio input does not include speech prior to the audio segment. In this example, the portion of the audio input including and subsequent to the audio segment can be processed for speech.

[0040] In another example, it can be determined that an audio frame of the audio input is the first audio frame among several audio frames that includes speech (e.g., a start-point) and that the audio input does not include music prior to the audio frame. In such an example, the portion of the audio input including and subsequent to the audio frame can be processed for speech. In yet another example, an end-point of a speech utterance in the audio input can be identified at block 113 and the audio input can be determined to not include any music prior to the end-point. In such an example, the portion of the audio input prior to the end-point can be processed for speech at block 117. In some cases, a start-point and an end-point of a speech utterance in the audio input can be identified based on the one or more extracted features of block 105 and it can be determined that the audio input includes no music prior to the end-point. In these cases, the portion of the audio input between the start-point and the end-point can be processed for speech at block 117.

[0041] Block 117 can be performed at the user device (e.g., user device 202), the remote system (e.g., server system 210 or virtual assistant server 214), or both (e.g., system 200). In examples where block 117 can be performed at least in part at the remote system, processing the audio input for speech can include transmitting the audio input to the remote system for speech processing and obtaining from the remote system an output such as, the textual representation of the speech utterance, the inferred user intent, or the task flow to accomplish the inferred user intent. In some examples, the output obtained from the remote system can be used to generate and output a relevant dialog response to the speech utterance or to provide relevant services to the user.

[0042] Processing the audio input for speech can incur significant computational resources. Therefore, it can be desirable to perform the speech processing of block 117 only when there is sufficient likelihood that the audio input includes speech. Accordingly, in some examples, the speech processing at block 117 can be performed on the audio input in response to determining that the audio input includes speech or music and not in response to determining that the audio input includes neither speech nor music. In other examples, the speech processing at block 117 can be performed on the audio input regardless of whether it is determined that the audio input includes speech or music.

[0043] In some examples, an inferred user intent can be obtained at block 117 based on a speech utterance of the audio input. At block 119 of process 100, a determination can be made as to whether the inferred user intent (e.g., obtained at block 117) includes identifying music. In some examples, the inferred user intent can be determined to include identifying music in the audio input. For example, the speech utterance can include the user request, “What is this song?” and thus the inferred user intent based on this speech utterance can be determined to include identifying the music in the audio input. In response to determining that the inferred user intent includes identifying music in the audio input, the acoustic fingerprint generated at block 107 can be used to obtain the identity of the music. Further, audio input can continue to be received at block 101 and the acoustic fingerprint can continue to be generated at block 107 until the identity of the music is obtained or until a predetermined duration is reached.

[0044] In other examples, the inferred user intent (e.g., obtained at block 117) can be determined to not include identifying music in the audio input. For example, the audio input can include the speech utterance, “What's the weather like tomorrow?” and thus the inferred user intent based on this speech utterance can be determined to include searching the weather rather than identifying the music in the audio input. At block 121 of process 100, in response to determining that the inferred user intent does not include identifying music in the audio input, the generating of the acoustic fingerprint at block 107 can be ceased or discontinued. Discontinuing the generation of the acoustic fingerprint at block 107 when music identification is not desired by the user can conserve computational resources, thereby enabling efficient audio analysis.

[0045] Although blocks 101 through 121 of process 100 are shown in a particular order in FIG. 1, it should be appreciated that these blocks can be performed in any order and that some blocks can be performed simultaneously. For instance, in some examples, blocks 103 and 105 can be performed, at least in part, simultaneously. In other examples, blocks 103 and 105 can be performed serially.

[0046] Further, it should be appreciated that in some cases, one or more blocks of process 100 can be optional and additional blocks can also be performed. For instance, in some examples, process 100 can further include receiving a signal to begin receiving the audio input at block 101. The signal can be received via input to the user device. Receiving the audio input at block 101 can thus be performed in response to receiving the signal to begin receiving the audio input.

[0047] In some examples, process 100 can further include performing a service or presenting a relevant output based on a speech utterance of the audio input. For example, at block 117, a user intent can be inferred based on a speech utterance of the audio input and a task flow for accomplishing the inferred user intent can be obtained. Process 100 can thus include executing the task flow obtained at block 117 by invoking programs, methods, services, APIs, or the like. Additionally, process 100 can include presenting relevant output responses in an audible (e.g., speech) and/or visual form. For example, a relevant dialog response to the speech utterance of the audio input can be presented.

[0048] As described above, the acoustic fingerprint generated at block 107 can be used to identify the music in the audio input. Thus, in some examples, process 100 can include obtaining an identity of the music in the audio input using the acoustic fingerprint generated at block 107. In some examples, the identity of the music can be obtained from a remote system (e.g., server system 210, virtual assistant server 214, or external services 224). In these examples, the acoustic fingerprint can be transmitted to the remote system where the acoustic fingerprint can be used to search a music library database containing a collection of acoustic fingerprints of identified music to obtain the identity of the music in the audio input. The obtained identity of the music can be received from the remote server and subsequently displayed to the user on the user device. In other examples, the identity of the music can be obtained by performing music recognition on the user device itself (e.g., user device 202). In these examples, the user device can use the acoustic fingerprint to search a music library database containing a collection of acoustic fingerprints of identified music to obtain the identity of the music in the audio input. The music library database can be stored locally on the user device or at a remote system. The obtained identity of the music can then be displayed to the user on the user device.

[0049] Further, in some examples, process 100 can include displaying the obtained identity of the music. Information and resources related to the identity of the music can also be displayed. For example, the cost associated with acquiring a copy of the music or a link to a webpage for purchasing a copy of the music can be displayed.

2. System and Environment

[0050] FIG. 2 illustrates exemplary system 200 for carrying out various aspects of analyzing audio input for efficient speech and music recognition according to various examples. System 200 can be capable of analyzing audio input for speech and/or music recognition in order to provide various services to users. For example, system 200 can implement a virtual assistant. The terms “virtual assistant,” “digital assistant,” “intelligent automated assistant,” or “automatic digital assistant” can refer to any information processing system (e.g., system 200) that can interpret natural language input in spoken and/or textual form to infer user intent, and perform actions based on the inferred user intent.

[0051] The virtual assistant can be capable of performing speech recognition on a spoken input in order to obtain a textual representation of the spoken input. The textual representation can be analyzed to infer user intent. The virtual assistant can then act on the inferred user intent by performing one or more of the following: identifying a task flow with steps and parameters designed to accomplish the inferred user intent; inputting specific requirements from the inferred user intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, or the like; and generating output responses to the user in an audible (e.g., speech) and/or visual form.

[0052] Further, the virtual assistant can be capable of interpreting various other forms of input and inferring user intent based on the received input. For example, the virtual assistant can receive audio input that includes music and infer that the user would like to identify the music in the audio input. The virtual assistant can then perform music recognition on the music in the audio input and display the identity of the music to the user.

[0053] An example of a virtual assistant is described in Applicants' U.S. Utility application Ser. No. 12/987,982 for “Intelligent Automated Assistant,” filed Jan. 10, 2011, the entire disclosure of which is incorporated herein by reference.

[0054] As shown in FIG. 2, in some examples, a virtual assistant can be implemented according to a client-server model. The virtual assistant can include a client-side portion executed on a user device 202 and a server-side portion executed on a server system 210. User device 202 can include any electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set-top box, wearable electronic device, or the like, and can communicate with server system 210 through one or more networks 208, which can include the Internet, an intranet, or any other wired or wireless public or private network. A detailed description of user device 202 is provided below with reference to FIG. 3. The client-side portion executed on user device 202 can provide client-side functionalities, such as user-facing input and output processing and communications with server system 210. Server system 210 can provide server-side functionalities for any number of clients residing on a respective user device 202.

[0055] Server system 210 can include one or more virtual assistant servers 214. As shown in FIG. 2, virtual assistant server 214 includes memory 228, one or more processors 226, a client-facing I/O interface 222, and an I/O interface to external services 216. The various components of virtual assistant server 214 can be coupled together by one or more communication buses or signal lines. Memory 228, or the computer readable storage media of memory 228, can include one or more processing modules 218 and data and model storage 220. The one or more processing modules 218 can include various programs and instructions. The one or more processors 226 can execute the programs and instructions of the one or more processing modules 218 and read/write to/from data and model storage 220. In the context of this document, a “non-transitory computer readable storage medium” can be any medium that can contain or store the program for use by or in connection with the instruction execution system, apparatus, or device.

[0056] In some examples, the one or more processing modules 218 can include various programs and instructions for performing various aspects of process 100 described above. In one example, the one or more processing modules 218 can include separate instructions for blocks 103 and 105 such that blocks 103 and 105 can be performed independent of each other. In some examples, the one or more processing modules 218 can include a speech-to-text processing module, a natural language processing module, a task flow processing module, and a service processing module for performing various aspects of block 117 described above. The speech-to-text processing module can include instructions for transcribing a speech utterance in an audio input, and the natural language processing module can include instructions for inferring user intent from the transcribed speech utterance. The task flow processing module and the service processing module can include instructions for identifying a task flow to accomplish the inferred user intent, inputting specific requirements from the inferred user intent into the task flow, executing the task flow, and outputting relevant responses to the speech utterance. Data and models 220 can include various user data and models that can be accessed or referenced when performing various aspects of process 100. For example, data and models 220 can include a music library database containing a collection of acoustic fingerprints of identified music. Further, data and models 220 can include speech models, task flow models, and service models.

[0057] In some examples, virtual assistant server 214 can communicate with external services 224, such as telephony services, calendar services, information services, messaging services, navigation services, and the like, through network(s) 208 for task completion or information acquisition. The I/O interface to external services 216 can facilitate such communications. In some examples, external services 224 can include music identification services from which virtual assistant server 214 or user device 202 can obtain an identity of the music in the audio input. In particular, the acoustic fingerprint generated at block 107 can be transmitted to the music identification service and the identity of the music associated with the acoustic fingerprint can be obtained by server system 210 or user device 202 from the music identification service. In other examples, server system 210 can include a music identification server (not shown) for providing such music identification services. In these examples, user device 202 or virtual assistant server 214 can communicate with the music identification server to obtain the music identity associated with the acoustic fingerprint generated at block 107.

[0058] Server system 210 can be implemented on one or more stand-alone data processing devices or a distributed network of computers. In some examples, server system 210 can employ various virtual devices and/or services of third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing resources and/or infrastructure resources of server system 210.

[0059] Although the functionality of the virtual assistant is shown in FIG. 2 as including both a client-side portion and a server-side portion, in some examples, the functions of the assistant can be implemented as a stand-alone application installed on a user device. In addition, the division of functionalities between the client and server portions of the virtual assistant can vary in different examples. For instance, in some examples, one or more processing modules 218 and data and models 220 can be stored in the memory of user device 202 to enable the user device to perform a greater proportion or all of the functionalities associated with the virtual assistant. In other examples, the client executed on user device 202 can be a thin-client that provides only user-facing input and output processing functions, and delegates all other functionalities of the virtual assistant to a back-end server.

3. User Device

[0060] FIG. 3 is a block diagram of a user device 202 according to various examples. As shown, user device 202 can include a memory interface 302, one or more processors 304, and a peripherals interface 306. The various components in user device 202 can be together coupled by one or more communication buses or signal lines. User device 202 can further include various sensors, subsystems, and peripheral devices that are coupled to the peripherals interface 306. The sensors, subsystems, and peripheral devices gather information and/or facilitate various functionalities of user device 202.

[0061] For example, user device 202 can include a motion sensor 310, a light sensor 312, and a proximity sensor 314 coupled to peripherals interface 306 to facilitate orientation, light, and proximity sensing functions. One or more other sensors 316, such as a positioning system (e.g., a GPS receiver), a temperature sensor, a biometric sensor, a gyroscope, a compass, an accelerometer, and the like, are also connected to peripherals interface 306 to facilitate related functionalities

[0062] In some examples, a camera subsystem 320 and an optical sensor 322 can be utilized to facilitate camera functions, such as taking photographs and recording video clips. Communication functions can be facilitated through one or more wired and/or wireless communication subsystems 324, which can include various communication ports, radio frequency receivers and transmitters, and/or optical (e.g., infrared) receivers and transmitters. An audio subsystem 326 can be coupled to speakers 328 and a microphone 330 to facilitate audio-enabled functions, such as voice recognition, music recognition, voice replication, digital recording, and telephony functions. For example, user device 202 can receive audio input (e.g., at block 101) via microphone 330. Microphone 330 can convert the audio input into a representative audio signal (digital or analog). The representative audio signal can be temporarily stored as an audio file in a memory buffer before undergoing audio analysis or processing. The memory buffer can include memory 350 or can include separate memory of the audio subsystem 326.

[0063] In some examples, user device 202 can further include an I/O subsystem 340 coupled to peripherals interface 306. I/O subsystem 340 can include a touch screen controller 342 and/or other input controller(s) 344. Touch-screen controller 342 can be coupled to a touch screen 346. Touch screen 346 and the touch screen controller 342 can, for example, detect contact and movement or break thereof using any of a plurality of touch sensitivity technologies, such as capacitive, resistive, infrared, surface acoustic wave technologies, proximity sensor arrays, and the like. Other input controller(s) 344 can be coupled to other input/control devices 348, such as one or more buttons, rocker switches, a thumb-wheel, an infrared port, a USB port, and/or a pointer device such as a stylus. In some examples, a signal to begin receiving an audio input can be received by user device 202 via input to touch screen 346 (e.g., a virtual button) or other input/control devices 348.

[0064] In some examples, user device 202 can further include a memory interface 302 coupled to memory 350. Memory 350 can include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM) (magnetic), a portable optical disc such as CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as compact flash cards, secured digital cards, USB memory devices, memory sticks, and the like. In some examples, a non-transitory computer-readable storage medium of memory 350 can be used to store instructions (e.g., for performing process 100, described above) for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In other examples, the instructions (e.g., for performing process 100, described above) can be stored on a non-transitory computer-readable storage medium of server system 210, or can be divided between the non-transitory computer-readable storage medium of memory 350 and the non-transitory computer-readable storage medium of server system 210.

[0065] In some examples, memory 350 can store an operating system 352, a communication module 354, a graphical user interface module 356, a sensor processing module 358, a phone module 360, and applications 362. Operating system 352 can include instructions for handling basic system services and for performing hardware dependent tasks. Communication module 354 can facilitate communicating with one or more additional devices, one or more computers, and/or one or more servers. Graphical user interface module 356 can facilitate graphic user interface processing. Sensor processing module 358 can facilitate sensor related processing and functions. Phone module 360 can facilitate phone-related processes and functions. Application module 362 can facilitate various functionalities of user applications, such as electronic-messaging, web browsing, media processing, navigation, imaging, and/or other processes and functions.

[0066] As described herein, memory 350 can also store client-side virtual assistant instructions (e.g., in a virtual assistant client module 364) and various user data and models 366 to provide the client-side functionalities of the virtual assistant. The virtual assistant client module 364 can include modules, instructions, and programs for performing various aspects of process 100 described above. For example, virtual assistant client module 364 can include programs or instructions associated with the audio classifier described above with respect to block 103. In some cases, the instructions for performing various aspects of process 100 can be stored in a separate module in memory 350. User data and models 366 can include user-specific vocabulary data, preference data, and/or other data such as the user's electronic address book, to-do lists, shopping lists, and the like. In addition, user data and models 366 can include speech models, task flow models, and service models. In some examples, user data and models 366 can include a music library database containing a collection of acoustic fingerprints of identified music.

[0067] In various examples, virtual assistant client module 364 can include instructions for accepting audio input (e.g., speech and/or music input), text input, touch input, and/or gestural input through various user interfaces (e.g., I/O subsystem 340, audio subsystem 326, or the like) of user device 202. Virtual assistant client module 364 can also include instructions for providing output in audio (e.g., speech and/or music output), visual, and/or tactile forms. For example, output can be provided as voice, music, sound, alerts, text messages, menus, graphics, videos, animations, vibrations, and/or combinations of two or more of the above. During operation, user device 202 can communicate with the virtual assistant server using communication subsystems 324 to perform the functionalities associated with the virtual assistant.

[0068] In various examples, memory 350 can include additional instructions or fewer instructions. Furthermore, various functions of user device 202 can be implemented in hardware and/or in firmware, including in one or more signal processing and/or application specific integrated circuits.

4. Electronic Device

[0069] FIG. 4 shows a functional block diagram of an electronic device 400 configured in accordance with the principles of the various described examples. The functional blocks of the device can be, optionally, implemented by hardware, software, or a combination of hardware and software to carry out the principles of the various described examples. It is understood by persons of skill in the art that the functional blocks described in FIG. 4 can be, optionally, combined or separated into sub-blocks to implement the principles of the various described examples. Therefore, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

[0070] As shown in FIG. 4, electronic device 400 can include touch screen display unit 402 configured to display a user interface and to receive touch input, and audio receiving unit 404 configured to receive audio input. In some examples, electronic device 400 can include speaker unit 406 configured to generate sound and input unit 408 configured to receive user input. Input unit 408 can be configured to generate a signal in response to receiving user input. In one example, the signal can be a signal to begin receiving an audio input using audio receiving unit 404. In another example, the signal can be a signal to stop receiving the audio input using audio receiving unit 404. Electronic device 400 can further include processing unit 410 coupled to touch screen display unit 402 and audio receiving unit 404 (and, optionally, coupled to speaker unit 406 and input unit 408). In some examples, processing unit 410 can include receiving unit 412, music determining unit 414, speech determining unit 416, acoustic fingerprint generating unit 418, and end-point identifying unit. In some examples, processing unit 410 can further include speech processing unit 422, speech determination ceasing unit 424, receipt of audio input ceasing unit 426, acoustic fingerprint generation ceasing unit 428, outputting unit 430, music identity obtaining unit 432, and intent determining unit 434.

[0071] Processing unit 410 can be configured to receive an audio input (e.g., from audio receiving unit 404 and using receiving unit 412). Processing unit 410 can be configured to determine (e.g., using music determining unit 414) whether the audio input includes music. Processing unit 410 can be configured to determine (e.g., using speech determining unit 416) whether the audio input includes speech. In response to determining that the audio input includes music, processing unit 410 can be configured to generate an acoustic fingerprint (e.g., using acoustic fingerprint generating unit 418) representing a portion of the audio input that includes music. In response to determining that the audio input includes speech rather than music, processing unit 410 can be configured to identify an end-point of a speech utterance of the audio input (e.g., using end-point identifying unit 420).

[0072] In some examples, the audio input can comprise a sequence of audio segments. In these examples, processing unit 410 can be configured to extract from an audio segment of the sequence of audio segments (e.g., using music determining unit 414) one or more features that characterize the audio segment. Processing unit 410 can be further configured to determine (e.g., using music determining unit 414) whether the audio segment includes music using an audio classifier and based on the one or more features that characterize the audio segment.

[0073] In some examples, the one or more features that characterize the audio segment can include at least one of root mean square amplitude, zero crossing rate, spectral centroid, spectral roll-off, spectral flux, and spectral flatness. In other examples, the one or more features that characterize the audio segment can include at least one of frequency cepstrum coefficients, linear predictive cepstral coefficients, bark scale frequency cepstral coefficients, mel-frequency discrete wavelet coefficients, mel-frequency cepstral coefficients, or the like.

[0074] In some examples, the audio classifier can be a neural network classifier. In other examples, the audio classifier can be a rule-based classifier. In some examples, the rule-based classifier can determine whether the audio segment includes music by comparing the one or more features to one or more predetermined thresholds.

[0075] In some examples, the audio input can comprise a sequence of audio frames. In these examples, processing unit 410 can be configured to extract from an audio frame of the sequence of audio frames (e.g., using speech determining unit 416) one or more features that characterize the audio frame. Processing unit 410 can be further configured to determine (e.g., using speech determining unit 416) whether the audio frame includes speech based on the one or more features that characterize the audio frame and one or more predetermined thresholds. In some examples, the duration of the audio frame can be different from the duration of the audio segment.

[0076] In some examples, the one or more features that characterize the audio frame can include at least one of short-term energy level, zero crossing rate, spectral centroid, spectral roll-off, spectral flux, spectral flatness, and autocorrelation.

[0077] In some examples, processing unit 410 can be configured to determine whether the audio input includes music (e.g., using music determining unit 414) independent of determining whether the audio input includes speech (e.g., using speech determining unit 416).

[0078] In some examples, processing unit 410 can be configured to determine whether the audio input includes music (e.g., using music determining unit 414) and whether the audio input includes speech (e.g., using speech determining unit 416) at least in part simultaneously.

[0079] In some examples, the acoustic fingerprint can be generated (e.g., using acoustic fingerprint generating unit 418) from an uncompressed representation of a portion of the audio input.

[0080] In some examples, in response to determining that the audio input includes speech, processing unit 410 can be configured to present (e.g., using touch screen display unit 402 or speaker unit 406 and using outputting unit 430) a relevant dialog response to a speech utterance of the audio input.

[0081] In some examples, in response to determining that the audio input includes music, processing unit 410 can be configured to obtain an identity of the music in the audio input (e.g., using music identity obtaining unit 432) based on the acoustic fingerprint. Processing unit 410 can be further configured to display (e.g., on touch screen display unit 402 and using the outputting unit 430) the identity of the music.

[0082] In some examples, in response to determining that the audio input includes music, processing unit 410 can be configured to identify a speech utterance of the audio input (e.g., using speech processing unit 422) and to determine an inferred user intent based on the speech utterance (e.g., using speech processing unit 422). Processing unit 410 can be further configured to determine (e.g., using intent determining unit 434) whether the inferred user intent includes identifying music in the audio input. In response to determining that the inferred user intent does not include identifying music in the audio input, processing unit 410 can be further configured to cease generating the acoustic fingerprint (e.g., using acoustic fingerprint generation ceasing unit 428).

[0083] In some examples, processing unit 410 can be configured to begin receiving the audio input (e.g., using receiving unit 412) in response to receiving a signal (e.g., from input unit 408) to begin receiving the audio input. In response to determining that the audio input includes neither speech nor music for a predetermined duration, processing unit 410 can be further configured to cease receiving the audio input (e.g., using receipt of audio input ceasing unit 426).

[0084] In some examples, in response to determining that the audio input includes music, processing unit 410 can be configured to cease determining whether the audio input includes speech (e.g., using speech determination ceasing unit 424).

[0085] In some examples, processing unit 410 can be configured to begin receiving the audio input (e.g., using receiving unit 412) in response to receiving a signal (e.g., from input unit 408) to begin receiving the audio input. In response to determining that the audio input includes speech rather than music, processing unit 410 can be further configured to cease receiving the audio input (e.g., using receipt of audio input ceasing unit 426) a predetermined duration after the end-point is identified.

[0086] Although examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the various examples as defined by the appended claims.

[0087] In some cases, the systems, processes, and devices described above can include the gathering and use of data available from various sources to improve the delivery to users of invitational content or any other content that may be of interest to them. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, home addresses, or any other identifying information.

[0088] The present disclosure recognizes that the use of such personal information data in connection with the systems, processes, and devices described above, can be used to the benefit of users. For example, the personal information data can be used to deliver targeted content that is of greater interest to the user. Accordingly, use of such personal information data enables calculated control of the delivered content. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure.

[0089] The present disclosure further contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and/or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining personal information data private and secure. For example, personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection should occur only after receiving the informed consent of the users. Additionally, such entities would take any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices.

[0090] Despite the foregoing, the present disclosure also contemplates examples in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of advertisement delivery services, the systems and devices described above can be configured to allow users to select to “opt in” or “opt out” of participation in the collection of personal information data during registration for services. In another example, users can select not to provide location information for targeted content delivery services. In yet another example, users can select to not provide precise location information, but permit the transfer of location zone information.

[0091] Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed examples, the present disclosure also contemplates that the various examples can also be implemented without the need for accessing such personal information data. That is, the various examples disclosed herein are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, content can be selected and delivered to users by inferring preferences based on non-personal information data or a bare minimum amount of personal information, such as the content being requested by the device associated with a user, other non-personal information available to the content delivery services, or publicly available information.