Method and apparatus for speech synthesis using paralinguistic variation

[0001] The present invention relates generally to speech synthesis systems. More particularly, this invention relates to generating variations in synthesized speech to produce speech that sounds more natural.

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings hereto: Copyright© 2002, Apple Computer, Inc., All Rights Reserved.

[0003] Speech is used to communicate information from a speaker to a listener. In a computer-user interface, the computer generates synthesized speech to convey an audible message to the user rather than just displaying the message as text with an accompanying “beep.” There are several advantages to conveying audible messages to the computer user in the form of synthesized speech. In addition to liberating the user from having to look at the computer's display screen, the spoken message conveys more information than the simple “beep” and, for certain types of information, speech is a more natural communication medium.

[0004] Due to the nature of computer systems, the same message may occur many times. For example, the message “Attention! The printer is out of paper” may be programmed to repeat several times over a short period of time until the user replenishes the printer's paper tray. Or the message “Are you sure you want to quit without saving?” may be repeated several times over the course of using a particular program. In human speech, when a person says the same words over and over again, he or she does not produce exactly the same acoustic signal each time the words are spoken. In synthesized speech, however, the opposite is true; a computer generates exactly the same acoustic signal each time the message is spoken. Users inevitably become annoyed at hearing the same predictable message spoken each time in exactly the same way. The more often a particular message is spoken in exactly the same way, the more unnaturally mechanical it sounds. In fact, studies have shown that listeners tune out repetitive sounds and, eventually, a repetitive spoken message will not be noticed.

[0005] One way to overcome the problems of sound repetition is to alter the way the computer produces the acoustic signal each time the message is spoken. Altering a computer-generated sound each time it is produced is known in the art. For example, alteration of the sound can be achieved by changing the sample playback rate, which shifts the overall spectrum and duration of the acoustic signal. While this approach works well for non-speech sounds, it does not work well when applied to speech sounds. In human speech, the overall spectrum of sound stays the same because a human speaker's vocal tract length does not vary. Thus, in order to sound like human speech, the overall spectrum of the sound of synthesized speech needs to stay the same as well. Another prior art example of altering a computer-generated sound each time it is produced is found in computer-generated music. In computer music a small random variation in the timing of each note is sometimes made to achieve a less mechanical sound. However, as with changing the sample playback rate, changing the timing of the components of speech does not work well for speech sounds because, unlike music, speech does not consist of easily identifiable note-onset and note-duration events. Rather, speech consists of tonal patterns of pitch, syllable stresses, overlapped gestures of the articulators (tongue, lips, jaw, etc.), and timing to form the rhythmic speech patterns that comprise the spoken message. Thus, it is not so clear exactly what parameters in speech synthesis should be varied to achieve a more natural sound. A more detailed analysis of the components of speech is required.

[0006] Speech is the acoustic output of a complex system whose underlying state consists of a known set of discrete phonemes that every human speaker produces. A phoneme is the basic theoretical unit for describing how speech conveys linguistic meaning. As such, the phonemes of a language comprise a minimal theoretical set of units that are sufficient to convey all meaning in the language. For American English, there are approximately 40 phonemes, which are made up of vowels and consonants. Each phoneme can be considered to be a code that consists of a unique set of articulatory gestures.

[0007] If speakers could exactly and consistently produce these phoneme sounds, speech would amount to a stream of underlying discrete codes. However, because of many different factors including, for example, agents, gender, and coarticulatory effects, every phoneme has a variety of acoustic manifestations in the course of flowing speech. Thus, from an acoustical point of view, the phoneme actually represents a class of sounds that convey the same meaning.

[0008] The variations in the way the phonemes are produced between people and even between utterances of the same person are referred to as prosody. Examples of prosody include tonal and rhythmic variations in speech, which provide a significant contribution to the formal linguistic structure of speech communication and are referred to as the prosodic features. The acoustic patterns of prosodic features are heard in changes in the duration, intensity, fundamental frequency, and spectral patterns of the individual phonemes that comprise the spoken message.

[0009] There are two distinctive components of prosody—i.e., linguistic components of prosody and paralinguistic components of prosody. The linguistic components of prosody are those that can change the meaning of a spoken phrase. In contrast, paralinguistic components of prosody are those that do not change the meaning of a series of spoken words. For example, when speaking the phrase “it's raining,” a rising intonation asks for a confirmation and, perhaps, conveys surprise or disbelief. On the other hand, a falling intonation may express confidence that the rain is indeed falling. The distinction between the rising and falling intonations is an example of varying a linguistic prosodic feature. By contrast, one could speak the phrase “it's raining” with a somewhat higher (or lower) overall pitch range, depending upon whether the listener is far away (or nearby), and this change in overall pitch range does not change the meaning of the spoken words. Such a change in pitch without altering meaning is an example of a paralinguistic prosodic feature.

[0010] The fundamental frequency contours of speech have been classified according to their communicative function. In English, a rising contour generally conveys to the listener that a question has been posed, that some response from the listener is required, or that more information is implied to follow within the current topic. Conversely, a falling contour generally conveys the opposite. Numerous subtle and not-so-subtle variations in the fundamental frequency contours signal other information to the listener as well, such as sarcasm, disbelief, excitement or anger. Unlike the phonemes, the prosodic features reflected in the acoustic patterns may not be discrete. In fact, it is often difficult or impossible to determine which features of prosody are discrete and which are not.

[0011] The human ear is extremely sensitive to minor changes in certain components of speech, and remarkably tolerant of other changes. For example, the tonal and rhythmic variations of speech are finely controlled by humans and, as noted above, convey considerable linguistic information. Thus, random variations in the pitch or duration of each phoneme, syllable or word of a spoken message can destructively interfere with the overall tonal and rhythmic pattern of the speech, i.e. the prosody. Even a 9-millisecond difference in the closure duration of an inter-vocal stop can shift the perception from voiced to voiceless, changing for example the word “rapid” into “rabid.” Therefore, simply changing the parameters for the timing of sound components may result in undesirable alterations in the prosodic features of the phonemes that comprise the speech and cannot be successfully applied to speech synthesis.

[0012] Another example of altering computer-generated sounds is disclosed in U.S. Pat. No. 5,007,095 to Nara et al., which describes a system for synthesizing speech having improved naturalness.

[0013] A method and apparatus for generating speech that sounds more natural using paralinguistic variation is described herein. According to one aspect of the present invention, a method for generating speech that sounds more natural comprises generating synthesized speech having certain prosodic features and applying a paralinguistic variation to the acoustic sequence representing the synthesized speech without altering the linguistic prosodic features. According to one aspect of the present invention, the application of the paralinguistic variation is correlated with a previous randomly applied paralinguistic variation to reflect a gradual change in the computer voice, while still maintaining a random quality. According to one aspect of the present invention, the application of the paralinguistic variation is correlated over time. According to one aspect of the present invention, the application of the paralinguistic variation is correlated with other paralinguistic variations, sometimes in accordance with a predetermined paragraph prosody.

[0014] According to one aspect of the present invention, a machine-accessible medium has stored thereon a plurality of instructions that, when executed by a processor, cause the processor to alter synthesized speech by applying a paralinguistic variation to the acoustic sequence representing the synthesized speech without altering the linguistic prosodic features. According to another aspect of the invention, the application of the paralinguistic variation is correlated with a previous randomly applied paralinguistic variation to reflect a gradual change in the computer voice, while still maintaining a random quality. According to one aspect of the present invention, the instructions cause the processor to correlate the application of the paralinguistic variation over time. According to one aspect of the present invention, the instructions cause the processor to correlate the paralinguistic variation with other paralinguistic variations, sometimes in accordance with a predetermined paragraph prosody.

[0015] According to one aspect of the present invention, an apparatus for applying a paralinguistic variation to an acoustic sequence representing synthesized speech without altering the prosodic features of the synthesized speech includes a speech synthesizer and a paralinguistic variation processor. The speech synthesizer generates synthesized speech having certain prosodic features and the paralinguistic variation processor applies paralinguistic variations to the acoustic sequence representing the synthesized speech without altering the prosodic features. According to one aspect of the present invention, the paralinguistic variation processor correlates the paralinguistic variations with a previous randomly applied paralinguistic variation to reflect a gradual change in the computer voice, while still maintaining a random quality. According to one aspect of the present invention, the paralinguistic variation processor correlates the application of the paralinguistic variation over time. According to one aspect of the present invention, the paralinguistic variation processor correlates the paralinguistic variation with other paralinguistic variations, sometimes in accordance with a predetermined paragraph prosody.

[0016] In yet another embodiment, an apparatus for applying a paralinguistic variation to an acoustic sequence representing synthesized speech without altering the prosodic features of the synthesized speech comprises analog circuitry.

[0025] A method and an apparatus for generating paralinguistic variations in a speech synthesis system to produce more natural sounding speech are provided. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

[0026] FIG. 1 is a block diagram illustrating one generalized embodiment of a speech synthesis system 100 incorporating the invention, and the operating environment in which certain aspects of the illustrated invention may be practiced. The speech synthesis system 100 receives a text input 104 and performs a text normalization 106 on the text input 104 using grammatical analysis 110 and word pronunciation 108 processes. For example if the text input 104 is the phrase “½,” the text is normalized to the phrase “one half,” pronounced as “wUHn hAHf.” In one embodiment, the speech synthesis system 100 performs prosodic generation 112 for the normalized text using a prosody model 114. The speech synthesis system 100 performs speech generation 116 to generate an acoustic phoneme sequence 120 for the normalized text that embodies the prosodic features representative of the received text 104 in accordance with a speech generation model 118.

[0027] FIG. 2 is a block diagram illustrating a generalized embodiment of the components of a prosody model 114 that may be used in speech synthesis system 100. A phoneme duration model 128 is used by the prosodic generation 112 to provide a duration for each of the initial set of phonemes generated for the normalized text, and a phoneme pitch model 130 is used to provide a pitch or pitch range. In one embodiment, the phoneme pitch model 130 also uses a set of intonation rules 132 to provide pitch information for the phonemes.

[0028] In one embodiment the prosodic generation 112 uses a paragraph prosody 134 in conjunction with the phoneme duration model 128 and the phoneme pitch model 130 to provide an overall prosodic pattern for a set of text inputs 104 that comprise a dialog, or other sequence of computer-generated speech. An overall prosodic pattern is beneficial because it can be used to guide the user to respond to the computer-generated speech in a certain way. For example, in a computer-user interface, a task may be automated using a series of voice commands, such as changing the desktop background. The task may involve generating multiple occurrences of speech that prompt the user to enter several commands before the task is completed. The paragraph prosody 134 is used to provide prosodic features to the phonemes that result in speech that helps to guide the user through the task. The overall tonal and rhythmic pattern of the generated speech, i.e. the prosodic features, can help a user to determine whether an additional input is required, whether they must make a choice among alternatives, or when the task is complete.

[0029] Referring again to FIG. 1, the speech synthesis system 100 performs the processing necessary to generate an acoustic phoneme sequence 120 for the normalized text that embodies the prosodic features representative of the received text 104. In one embodiment, the speech synthesis system 100 generates paralinguistic variations of the acoustic phoneme sequence 120 in accordance with a paralinguistic variation model 124 resulting in a naturalized acoustic phoneme sequence 126 that sounds more natural or less annoyingly mechanical than the acoustic phoneme sequence 120. The paralinguistic variation generation 122 varies the realization of the individual phonemes that comprise the acoustic phoneme sequence 120, i.e. how the phonemes are mapped onto the acoustic sequence 120, but retains the prosodic features representative of the received text input 104 that were generated using the prosody model 114.

[0030] FIG. 3 is a block diagram illustrating a generalized embodiment of a paralinguistic variation model 124. A paralinguistic variation may be any one or a combination of any one or more variations of paralinguistic parameters 136 that represent the non-phonemic properties of speech, such as the tonal contours, pitch, or rhythm of speech. Examples of some of the paralinguistic parameters 136 that may be employed in a speech synthesis system 100 incorporating an embodiment of the present invention are illustrated in FIG. 3 and may include the pitch range 138, the speaking rate 140, the volume 142, the spectral slope 144, the breathiness 146, the co-articulation 148, and the extremity of articulation 150, e.g. slurring or mumbling. During paralinguistic variation generation 122, one or more of the paralinguistic parameters 136 is applied to the acoustic phoneme sequence 120 to generate the naturalized acoustic phoneme sequence 126. The application of the paralinguistic parameter(s) 136 may be random or correlated or both as will be described more fully below.

[0031] The speech synthesis system 100 may be hosted on a processor, but is not so limited. For an alternate embodiment, the speech synthesis system 100 may comprise some combination of hardware and software that is hosted on a number of different processors. For another alternate embodiment, a number of the components of the speech synthesis system 100 may be hosted on a number of different processors. Another alternate embodiment has a number of different components of the speech synthesis system 100 hosted on a single processor.

[0032] In yet a another embodiment, the speech synthesis system 100 is implemented, at least in part, using analog circuitry. For example, the speech synthesis system 100 may be implemented as analog electronic circuits that produce a time-varying electric signal. In one embodiment, a voltage controlled oscillator (VCO) is coupled with one or more voltage controlled filters (VCFs), wherein the output of the VCO is provided to the VCFs. Control inputs to the VCFs can be used to produce different phonemes that represent a sentence that is to be spoken. A time-varying signal can be input to the VCO, and the pattern of voltage (as a function of time) represents the desired pitch contour for the spoken sentence. In such an embodiment, a second input could be provided to the VCO, this second input presenting a slowly-varying random value that is added to the pitch contour to change its overall pitch range in a paralinguistic manner. In a similar fashion, there may be slowly varying inputs to the VCFs that modify, for example, the center-frequency and/or bandwidths of the filter resonances to slightly vary the articulation in random ways.

[0033] In yet a further embodiment, various components of the speech synthesis system 100 may be implemented mechanically. For example, the pitch could be generated by a mechanical model of a human larynx, where air is forced through two stretched pieces of rubber. This can produce a pitched buzzing sound having a frequency that is determined by the tightness of the stretched rubber pieces. The buzzing sound could then be passed through a series of tubes whose diameters can be varied over the lengths of the tubes. The tubes, which would resonate at frequencies determined by their respective cross-sectional areas, can produce audible speech. In such an implementation, paralinguistic variations may be achieved using a mechanism that adjusts the tension in the stretched rubber pieces and/or by a mechanism that varies the diameters of the acoustic tubes.

[0034] FIG. 4 illustrates a computer system 400 hosting the speech synthesis system of one embodiment. The computer system 400 comprises, but is not limited to, a system bus 401 that allows for communication among a processor 402, a digital signal processor 408, a memory 404, and a mass storage device 407. The system bus 401 is also coupled to receive inputs from a keyboard 422, a pointing device 423, and a text input device 425, but is not so limited. The system bus 401 provides outputs to a display device 421 and a hard copy device 424, but is not so limited.

[0035] These elements 401-425 perform their conventional functions known in the art. Collectively, these elements are intended to represent a broad category of hardware systems, including but not limited to general purpose computer systems based on the PowerPC® processor family of processors available from Motorola, Inc. of Schaumburg, Ill., or the Pentium® processor family of processors available from Intel Corporation of Santa Clara, Calif.

[0036] It is to be appreciated that various components of hardware system 400 may be re-arranged, and that certain implementations of the present invention may not require nor include all of the above components. For example, a display device may not be included in system 400. Additionally, multiple buses (e.g., a standard I/O bus and a high performance I/O bus) may be included in system 400. Furthermore, additional components may be included in system 400, such, as additional processors (e.g., a digital signal processor), storage devices, memories, network/communication interfaces, etc.

[0037] In the illustrated embodiment of FIG. 4, the method and apparatus for speech synthesis using random paralinguistic variation according to the present invention as discussed above is implemented as a series of software routines run by hardware system 400. These software routines comprise a plurality or series of instructions to be executed by a processor in a hardware system, such as processor 402. Initially, the series of instructions are stored on a storage device of memory 404. It is to be appreciated that the series of instructions can be stored using any conventional storage medium, such as a diskette, CD-ROM, magnetic tape, DVD, ROM, Flash memory, etc. It is also to be appreciated that the series of instructions need not be stored locally, and could be received from a remote storage device, such as a server on a network, via a network/communication interface. The instructions are copied from the storage device, such as mass storage 407, into memory 404 and then accessed and executed by processor 402. In one implementation, these software routines are written in the C++ programming language. It is to be appreciated, however, that these routines may be implemented in any of a wide variety of programming languages.

[0038] FIG. 5 further illustrates the memory 404 of FIG. 4 in greater detail. The memory 404, which may include and/or be coupled with a memory controller, hosts the speech synthesis system of one embodiment. An input device (e.g., text input device 425) provides text input to a bus interface 440. The bus interface 440 allows for storage of the input text in the text input data memory component 502 in memory 404 via the system bus 401. The text is processed by the processor 402 and/or digital signal processor 408 using algorithms and data associated with the components 502-516 stored in the memory 404. As discussed herein, the components stored in memory 404 that provide the algorithms and data used in processing the text to generate synthetic speech comprise, but not limited to, text input data 502, speech synthesizer 504, speech synthesis model 506, speech randomizer 508, prosody rules 510, variation correlator 512, random variation rules 514, and prior applied variation data 516.

[0039] FIG. 6 illustrates a speech randomizer 508 and a variation correlator 512 of a speech synthesis system of one embodiment. An acoustic sequence 601 as generated by a speech synthesizer 504 is processed to apply a random variation 610 selected at random from the random variation rules 514 stored in memory 404. In some instances the random variation is correlated 620 with a prior applied variation 516 stored in memory 404 to reflect a gradual change in the computer voice. In one embodiment, the resulting randomized acoustic sequence 602 is then used to produce a spoken message as part of a talking computer-user interface.

[0040] FIG. 7 illustrates the random variation rules 514 stored on memory 404 in a speech synthesis system of one embodiment. An important aspect of the random variation rules is that their application to the acoustic sequence 601 of synthesized speech signals must not alter the linguistic prosodic features representative of the received text 104. There are two categories of random variation rules 514.

[0041] The first category is a slight random variation in the overall pitch range 710 within which the linguistically-motivated speech melody is mapped from its rule-generated symbolic transcription to the continuously-varying fundamental frequency values. The linguistically-motivated speech melody is a prosodic feature of the input text 104, and refers to the specific intonational tune of the spoken message, e.g. a question tune, a neutral declarative tune, an exclamation tune, and so on. The mapping of the rule-generated symbolic transcription to the continuously varying fundamental frequency values may include application of the prosody model 114 and, more specifically, the phoneme pitch model 130 and intonation rules 132 to provide pitch information for the phonemes that comprise the message. In one embodiment, a slight variation is achieved by raising the overall pitch range one semitone by applying a logarithmic transformation of log 12√{square root over (2)} to the acoustic sequence 601 of synthesized speech signals. The logarithmic transformation of the signal alters the sound of the synthesized speech while preserving the prosodic features representative of the text input 104 such as the linguistically-motivated speech melody. Other types of transformations to the overall pitch range that preserve the linguistic prosodic features of the synthesized speech may be employed without exceeding the scope of the present invention.

[0042] The second category is a random variation in the overall speaking rate 720 of the spoken message. The overall speaking rate of a spoken message can be modeled independently of the relative durations of the speech segments (e.g. phonemes) within that message. Moreover, it has been shown that listeners perceive the overall speaking rate independently of the relative durations of the speech segments within the message. Therefore, changes to the overall speaking rate of a spoken message may be achieved without altering the linguistic prosodic features of phoneme duration as generated according to the prosody model 114 and, more specifically, according to the phoneme duration model 128. In one embodiment a random variation is achieved by either slightly speeding up or slowing down the overall speaking rate of a spoken message by applying a mathematical transformation to the acoustic sequence 601 of synthesized speech signals. In one embodiment the mathematical transformation may be a linear transformation such as a factor of 1.25 to increase the speaking rate by 25 percent. The linear transformation of the signal alters the sound of the synthesized speech while preserving the prosodic features representative of the text input 104 such as the relative duration of the phonemes. Other types of transformations to the overall speaking rate that preserve the linguistic prosody components of the synthesized speech may be employed without exceeding the scope of the present invention.

[0043] FIG. 8 illustrates a flowchart of the processes of a speech randomizer 508 and variation correlator 512 of a speech synthesis system of one embodiment. At process block 810 the speech randomizer 508 receives the acoustic sequence 601 of synthesized speech signals that embody the prosodic features representative of the received text 104. At process block 820, the speech randomizer determines whether to correlate the variation to the acoustic sequence 601 according to a parameter or other pre-determined setting of the speech synthesis system or user interface in which the synthesized speech is being used. If the application of the variation is to be correlated, then at process block 830 the variation correlator 512 determines whether there was a prior applied variation 516 stored on memory 404. If so, referring to block 840, then the variation correlator 512 selects a random variation rule 514 that correlates with the prior applied variation 516 to reflect a gradual change in the computer voice of the synthesized speech. If there is no prior applied variation rule 516 stored on memory 404, then the variation correlator 512 defaults to process block 850, where the speech randomizer 508 selects a variation rule at random. In one embodiment, the selection of a variation rule at random may be controlled in part by a parameter or other external setting of the speech synthesis system or user interface, such as a user preference for pitch modulation instead of speaking rate modulation. Even then, however, the selection of the actual variation rule will be selected at random so as to avoid predictability in the variation of the computer voice of the synthesized speech. Once the variation to be applied is determined, the processing continues at process block 860 where the speech randomizer 508 applies the selected random variation rule to the acoustic sequence 601 of synthesized speech signals without altering the linguistic prosodic features representative of the received text 104.

[0044] Thus, a method and apparatus for a speech synthesis system using random paralinguistic variation has been described. Whereas many alterations and modifications of the present invention will be comprehended by a person skilled in the art after having read the foregoing description, it is to be understood that the particular embodiments shown and described by way of illustration are in no way intended to be considered limiting. References to details of particular embodiments are not intended to limit the scope of the claims.