Singing voice synthesis

[0002] This application claims priority under 35 USC § 119(e)(1) of provisional application No. 60/062,712, filed Oct. 22, 1997.

[0003] This invention relates to singing voice synthesis and more particularly to synthesis by concatenation of waveform segments.

[0004] Speech and singing differ significantly in terms of their production and perception by humans. In singing, for example, the intelligibility of the phonemic message is often secondary to the intonation and musical qualities of the voice. Vowels are often sustained much longer in singing than in speech, and precise, independent control of pitch and loudness over a large range is required. These requirements significantly differentiate synthesis of singing from speech synthesis.

[0005] Most previous approaches to synthesis of singing have relied on models that attempt to accurately characterize the human speech production mechanism. For example, the SPASM system developed by Cook (P. R. Cook, “SPASM, A Real Time Vocal Tract Physical Model Controller And Singer, The Companion Software Synthesis System,” Computer Music Journal, Vol. 17, pp. 30-43, Spring 1993.) employs an articulator-based tube representation of the vocal tract and a time-domain glottal pulse input. Formant synthesizers such as the CHANT system (Bennett, et al., “Synthesis of the Singing Voice,” in Current Directions in Computer Music Research, pp. 19-49, MIT Press 1989.) rely on direct representation and control of the resonances produced by the shape of the vocal tract. Each of these techniques relies, to a degree, on accurate modeling of the dynamic characteristics of the speech production process by an approximation to the articulartory system. Sinusoidal signal models are somewhat more general representations that are capable of high-quality modeling, modification, and synthesis of both speech and music signals. The success of previous work in speech and music synthesis motivates the application of sinusoidal modeling to the synthesis of singing voice.

[0006] In the article entitled, “Frequency Modulation Synthesis of the Singing Voice,” in Current Directions in Computer Research, (pp. 57-64, MIT Press, 1989) John Chowning has experimented with frequency modulation (FM) synthesis of the singing voice. This technique, which has been a popular method of music synthesis for over 20 years, relies on creating complex spectra with a small number of simple FM oscillators. Although this method offers a low-complexity method of producing rich spectra and musically interesting sounds, it has little or no correspondence to the acoustics of the voice, and seems difficult to control. The methods Chowning has devised resemble the “formant waveform” synthesis method of CHANT, where each formant waveform is created by an FM oscillator.

[0007] Mather and Beauchamp in an article entitled, “An Investigation of Vocal Vibrato for Synthesis,” in Applied Acoustics, (Vol. 30, pp. 219-245, 1990) have experimented with wavetable synthesis of singing voice. Wavetable synthesis is a low complexity method that involves filling a buffer with one period of a periodic waveform, and then cycling through this buffer to choose output samples. Pitch modification is made possible by cycling through the buffer at various rates. The waveform evolution is handled by updating samples of the buffer with new values as time evolves. Experiments were conducted to determine the perceptual necessity of the amplitude modulation which arises from frequency modulating a source that excites a fixed-formant filter—a more difficult effect to achieve in wavetable synthesis than in source/filter schemes. They found that this timbral/amplitude modulation was a critical component of naturalness, and should be included in the model.

[0008] In much previous singing synthesis work, the transitions from one phonetic segment to another have been represented by stylization of control parameter contours (e.g., formant tracks) through rules or interpolation schemes. Although many characteristics of the voice can be approximated with such techniques after painstaking hand-tuning of rules, very natural-sounding synthesis has remained an elusive goal.

[0009] In the speech synthesis field, many current systems back away from specification of such formant transition rules, and instead model phonetic transitions by concatenating segments from an inventory of collected speech data. For example, this is described by Macon, et al. in article in Proc. of International Conference on Acoustics, Speech and Signal Processing (Vol. 1, pp. 361-364, May 1996) entitled, “Speech Concatenation and Synthesis Using Overlap-Add Sinusoidal Model.”

[0010] For Patents see, E. Bryan George, et al. U.S. Pat. No. 5,327,518 entitled, “Audio Analysis/Synthesis System” and E. Bryan George, et al. U.S. Pat. No. 5,504,833 entitled, “Speech Approximation Using Successive Sinusoidal Overlap-Add Models and Pitch-Scale Modifications.” These patents are incorporated herein by reference.

[0011] In accordance with one embodiment of the present invention a singing voice synthesis is provided by providing a signal model and modifying said signal model using concatenated segments of singing voice units and musical control information to produce concatenated waveform segments.

[0012] These and other features of the invention will be apparent to those skilled in the art from the following detailed description of the invention, taken together with the accompanying drawings.

[0013] FIG. 1 is a block diagram of the system according to one embodiment of the present invention;

[0014] FIG. 2A and FIG. 2B is a catalog of variable-size units available to represent a given phoneme;

[0015] FIG. 3 illustrates a decision tree for context matching;

[0016] FIG. 4 illustrates decision tree for phonemes preceded by an already-chosen diphone or triphone;

[0017] FIG. 5 illustrates decision tree phonemes followed by an already-chosen diphone or triphone;

[0018] FIG. 6 is a transition matrix for all unit-unit combinations;

[0019] FIG. 7 illustrates concatenation of segments using sinusoidal model parameters;

[0020] FIG. 8A is the fundamental frequency,

[0021] FIG. 8B is the gain envelope plots for the phrase “ . . . sunshine shimmers . . . ” and

[0022] FIG. 8C is a plot of these two quantities against to each other;

[0023] FIG. 9 illustrates the voicing decision result, ω₀contour and phonetic annotation for the phrase “ . . . sunshine shimmers . . . ” using nearest neighbor clustering method;

[0024] FIG. 10 illustrates short-time energy smoothing;

[0025] FIG. 11 illustrates Cepstral envelope smoothing;

[0026] FIG. 12 illustrates pitch pulse alignment in absence of modification;

[0027] FIG. 13 illustrates pitch pulse alignment after modification;

[0028] FIG. 14 illustrates spectral tilt modification as a function of frequency and parameter T_in; and

[0029] FIG. 15 illustrates spectral characteristics of the glottal source in model (normal) and breathy speech wherein top is a vocal fold configuration, middle is time domain waveform and bottom is short-time spectrum.

[0030] The system 10 shown in FIG. 1 uses, for example, a commercially-available MIDI-based (Musical Instrument Digital Interface) music composition software as a user interface 13. The user specifies a musical score and phonetically-spelled lyrics, as well as other musically interesting control parameters such as vibrato and vocal effort from MIDI file 11. This control information is stored in a standard MIDI file format that contains all information necessary to synthesize the vocal passage. The MIDI file interpreter 13 provides separately the linguistic control information for the words and the musical control information such as vibrato, vocal effect and vocal tract length, etc.

[0031] Based on this input MIDI file, linguistic processor 17 of the system 10 selects synthesis model parameters from an inventory 15 of voice data that has been analyzed off-line by the sinusoidal model. Units are selected at linguistic processor 17 to represent segmental phonetic characteristics of the utterance, including coarticulation effects caused by the context of each phoneme. These units are applied to concatenator/smoother processor 19. At processor 19, algorithms as described in Macon, et al. “Speech Concatenation and Synthesis using Overlap-Add Sinusoidal Model” in Proc. of International Conference on Acoustics, Speech and Signal Processing (Vol. 1, pp.361-364, May 1996) are applied to the modeled segments to remove disfluencies in the signal at the joined boundaries. The sinusoidal model parameters are then used to modify the pitch, duration, and spectral characteristics of the concatenated voice units as specified by the musical score and MIDI control information. Finally, the output waveform is synthesized at signal model 20 using the ABS/OLA sinusoidal model. This output of model 20 is applied via a digital to analog converter 22 to the speaker 21. The MIDI file interpreter 13 and processor 17 can be part of a workstation PC 16 and processor 19 and signal model 20 can be part of a workstation or a Digital Signal Processing (DSP) 18. Separate MIDI files 13 can be coupled into the workstation 16. The interpreter 13 converts to machine information. The inventory 15 is also coupled to the workstation 16 as shown. The output from the model 20 may also be provided to files for later use.

[0032] The signal model 20 used is an extension of the Analysis-by-Synthesis/Overlap-Add (ABS/OLA) sinusoidal model of E. Bryan George, et al. in Journal of the Audio Engineering Society (Vol. 40, pp.497-516, June 1992) entitled, “An Analysis-by-Synthesis Approach to Sinusoidal Modeling Applied to the Analysis and Synthesis of Musical Tones.” In the ABS/OLA model, the input signal s[n] is represented by a sum of overlapping short-time signal frames sk[n]. $\begin{array}{cc}s\left[n\right]=\sigma \left[n\right]\sum _{k}w\left[n-k{N}_s\right]s_k\left[n\right]& \left(1\right)\end{array}$

[0033] where N_sis the frame length, w[n] is a window function, σ[n] is a slowly time-varying gain envelope, and S_k[n] represents the kth frame contribution to the synthesized signal. Each signal contribution S_k[n] consists of the sum of a small number of constant-frequency, constant-amplitude sinusoidal components. An interactive analysis-by-synthesis procedure is performed to find the optimal parameters to represent each signal frame. See U.S. Pat. No. 5,327,518 of E. Bryan George, et al. incorporated herein by reference.

[0034] Synthesis is performed by an overlap-add procedure that uses the inverse fast Fourier transform to compute each contribution S_k[n], rather than sets of oscillator functions. Time-scale modification of the signal is achieved by changing the synthesis frame duration and pitch modification is performed by altering the sinusoidal components such that the fundamental frequency is modified while the speech formant structure is maintained.

[0035] The flexibility of this synthesis model enables the incorporation of vocal qualities such as vibrato and spectral tilt variation, adding greatly to the musical expressiveness of the synthesizer output.

[0036] While the signal model of the present invention is the preferred ABS/OLA sinusoidal model, other sinusoidal models as well as sampler models, wavetable models, formant synthesis model and physical models such as waveguide model may also be used. Some of these models with referenced are discussed in the background. For more details on the ABS/OLA model, see E. Bryan George, et al. U.S. Pat. No. 5,327,518.

[0037] The synthesis system presented in this application relies on an inventory of recorded singing voice data 15 to represent the phonetic content of the sung passage. Hence an important step is the design of a corpus of singing voice data that adequately covers allophonic variations of phonemes in various contexts. As the number of “phonetic contexts” represented in the inventory increases, better synthesis results will be obtained, since more accurate modeling of coarticulatory effects will occur. This implies that the inventory should be made as large as possible. This goal, however, must be balanced with constraints of (a) the time and expense involved in collecting the inventory, (b) stamina of the vocalist, and (c) storage and memory constraints of the synthesis computer hardware. Other assumptions are:

[0038] a.) For any given voiced speech segment, re-synthesis with small pitch modifications produces the most natural-sounding result. Thus, using an inventory containing vowels sung at several pitches will result in better-sounding synthesis, since units close to the desired pitch will usually be found.

[0039] b.) Accurate modeling of transitions to and from silence contributes significantly to naturalness of the synthesized segments.

[0040] c.) Consonant clusters are difficult to model using concatenation, due to coarticulation and rapidly varying signal characteristics.

[0041] To make best use of available resources, the assumption can be made that the musical quality of the voice is more critical than intelligibility of the lyrics. Thus, the fidelity of sustained vowels is more important than that of consonants. Also, it can be assumed that, based on features such as place and manner of articulation and voicing, consonants can be grouped into “classes” that have somewhat similar coarticulatory effects on neighboring vowels.

[0042] Thus, a set of nonsense syllable tokens was designed with a focus on providing adequate coverage of vowels in a minimal amount of recording. All vowels V were presented within the contexts C_LV and VC_R, where C_Land C_Rare classes of consonants (e.g. voiced stops, unvoiced fricatives, etc.) located to the left and right of a vowel as listed in Table 1 of Appendix A. The actual phonemes selected from each class were chosen sequentially such that each consonant in a class appeared a roughly equal number of times across all tokens. These C_LV and VC_Runits were then paired arbitrarily to form C_LVC_Runits, then embedded in a “carrier” phonetic context to avoid word boundary effects.

[0043] This carrier context consisted of the neutral vowel /ax/ (in ARPAbet notation), resulting in units of the form /ax/C_LVC_R/ax/. Two nonsense word tokens for each /ax/C_LVC_R/ax/ unit were generated, and sung at high and low pitches within the vocalist's natural range.

[0044] Transitions of each phoneme to and from silence were generated as well.

[0045] For vowels, these units were sung at both high and low pitches. The affixes _/s/ and _/z/ were also generated in the context of all valid phonemes. The complete list of nonsense words is given in Tables 2 and 3 of Appendix A.

[0046] A set of 500 inventory tokens was sung by a classically-trained male vocalist to generate the inventory data. Half of these 500 units were sung at a pitch above the vocalist's normal pitch, and half at a lower pitch. This inventory was then phonetically annotated and trimmed of silences, mistakes, etc. using Entropic x-waves and a simple file cutting program resulting in about ten minutes of continuous singing data used as input to the off-line sinusoidal model analysis. (It should be noted that this is a rather small inventory size, in comparison to established practices in concatenative speech synthesis.)

[0047] Given this phonetically-annotated inventory of voice data, the task at hand during the online synthesis process is to select a set of units from this inventory to represent the input lyrics. This is done at processor 17. Although it is possible to formulate unit selection as a dynamic programming problem that finds an optimal path through a lattice of all possible units based on acoustic “costs,” (e.g., Hunt, et al. “Unit Selection in a Concatenative Speech Synthesis System Using a large Speech Database,” in Proc. of International Conference on Acoustics, Speech and Signal Processing, Vol. 1, pp. 373-376, 1996) the approach taken here is a simpler one designed with the constraints of the inventory in mind: best-context vowel units are selected first, and consonant units are selected in a second pass to complete the unit sequence.

[0048] The method used for choosing the each unit involves evaluating a “context decision tree” for each input phoneme. The terminal nodes of the tree specify variable-size concatenation units ranging from one to three phonemes in length. These units are each given a “context score” that orders them in terms of their agreement with the desired phonetic context, and the unit with the best context score is chosen as the unit to be concatenated. Since longer units generally result in improved speech quality at the output, the method places a priority on finding longer units that match the desired phonetic context. For example, if an exact match of a phoneme and its two neighbors is found, this triphone is used directly as a synthesis unit.

[0049] For a given ph oneme P in the input phone tic string and its left and right neighbors, P_Land P_R, the selection algorithm attempts to find P in a context most closely matched to P_LP P_R. When exact context matches are found, the algorithm extracts the matching adjacent phoneme(s) as well, to preserve the transition between these phonemes. Thus, each extracted unit consists of an instance of the target phoneme and one or both of its neighboring phonemes (i.e., it extracts a monophone, diphone, or triphone). FIG. 2 shows a catalog of all possible combinations of monophones, diphones, and triphones, including class ma tch properties, ordered by their preference for synthesis.

[0050] In addition to searching for phonemes in an exact phonemic context , however, the system also is capable of finding phonemes that have a context similar, but not identical, to the desired triphone context. For example, if a desired triphone cannot be found in the inventory, a diphone or monop hone taken from an acoustically similar context is used instead.

[0051] For example, if the algorithm is searching for lael in the context /d/-/ae/-/d/, but this triphone cannot be found in the inventory, the monophone /ae/ taken from the context /b/-/ae/-/b/ can be used instead, since /b/ and /d/ have a similar effect on the neighboring vowel. The notation of FIG. 2 indicates the resulting unit output, along with a description of the context rules satisfied by the units. In the notation of this figure, x_LP₁x_Rindicates a phoneme with an exact triphone context match (as /d/-/ae/-/d/ would be for the case described above). The label c_LP₁c_Rindicates a match of phoneme class on the left and right, as for /b/-/ae/-/b/ above. Labels with the symbol P₂indicate a second unit is used to provide the final output phonemic unit. For example, if /b/-/ae/-/k/ and /k/-/ae/-/b/ can be found, the two lael monophones can be joined to produce an /ae/ with the proper class context match on either side.

[0052] In order to find the unit with the most appropriate available context, a binary decision tree was used (shown in FIG. 3). Nodes in this tree indicate a test defined by the context label next to each node. The right branch out of each node indicates a “no” response; downward branches indicate “yes”. Terminal node numbers correspond to the outputs defined in FIG. 2. Diamonds on the node branches indicate storage arrays that must be maintained during the processing of each phoneme. Regions enclosed in dashed lines refer to a second search for phonemes with a desired right context to supplement the first choice (the case described at the end of the previous paragraph). The smaller tree at the bottom right of the diagram describes all tests that must be conducted to find this second phoneme. The storage locations here are computed once and used directly in the dashed boxes. To save computation at runtime, the first few tests in the decision tree are performed off-line and stored in a file. The results of the precomputed branches are represented by filled diamonds on the branches.

[0053] After the decision tree is evaluated for every instance of the target phoneme, the (nonempty) output node representing the lowest score in FIG. 2 is selected. All units residing in this output node are then ranked according to their closeness to the desired pitch (as input in the MIDI file). A rough pitch estimate is included in the phonetic labeling process for this purpose. Thus the unit with the best phonetic context match and the closest pitch to the desired unit is selected.

[0054] The decision to develop this method instead of implementing the dynamic programming method is based on the following rationale: Because the inventory was constructed with emphasis on providing a good coverage of the necessary vowel contexts, “target costs” of phonemes in dynamic programming should be biased such that units representing vowels will be chosen more or less independently of each other. Thus a slightly suboptimal, but equally effective, method is to choose units for all vowels first, then go back to choose the remaining units, leaving the already-specified units unchanged. Given this, three scenarios must be addressed to “fill in the blanks”:

[0055] 1. Diphones or triphones have been specified on both sides of the phoneme of interest. Result: a complete specification of the desired phoneme has already been found, and no units are necessary.

[0056] 2. A diphone or triphone has been specified on the left side of the phoneme of interest. Result: The pruned decision tree in FIG. 4 is used to specify the remaining portion of the phoneme.

[0057] 3. A diphone or triphone has been specified on the right side of the phoneme of interest. Result: The pruned decision tree in FIG. 5 is used to specify the remaining portion of the phoneme.

[0058] If no units have been specified on either side, or if monophone only have been specified, then the general decision tree in FIG. 3 can be used.

[0059] This inexact matching is incorporated into the context decision tree by looking for units that match the context in terms of phoneme class (as defined above). The nominal pitch of each unit is used as a secondary selection criterion when more than one “best-context” unit is available.

[0060] Once the sequence of units has been specified using the decision tree method described above, concatenation and smoothing of the units takes place.

[0061] Each pair of units is joined by either a cutting/smoothing operation or an “abutting” of one unit to another. The type of unit-to-unit transition uniquely specifies whether units are joined (cut and smoothed) or abutted. FIG. 6 shows a “transition matrix” of possible unit-unit sequences and their proper join method. It should be noted that the NULL unit has zero length—it serves as a mechanism for altering the type of join in certain situations.

[0062] The rest of this section will describe in greater detail the normalization, smoothing and prosody modification stages.

[0063] The ABS/OLA sinusoidal model analysis generates several quantities that represent each input signal frame, including (i) a set of quasi-harmonic sinusoidal parameters for each frame (with an implied fundamental frequency estimate), (ii) a slowly time-varying gain envelope, and (iii) a spectral envelope for each frame. Disjoint modeled speech segments can be concatenated by simply stringing together these sets of model parameters and re-synthesizing, as shown in FIG. 7. However, since the jointed segments are analyzed from disjoint utterances, substantial variations between the time- or frequency-domain characteristics of the signals may occur at the boundaries. These differences manifest themselves in the sinusoidal model parameters. Thus, the goal of the algorithms descibed here is to make discontinuities at the concatenation points inaudible by altering the sinusoidal model components in the neighborhood of the boundaries.

[0064] The units extracted from the inventory may vary in short-time signal energy, depending on the characteristics of the utterances from which they were extracted. This variation gives the output speech a very stilted, unnatural rhythm. For this reason, it is necessary to normalize the energy of the units. However, it is not straightforward to adjust units that contain a mix of voiced and unvoiced speech and/or silence, since the RMS energy of such segments varies considerably depending on the character of the unit.

[0065] The approach taken here is to normalize only the voiced sections of the synthesized speech. In the analysis process, a global RMS energy for all voiced sounds in the inventory is found. Using this global target value, voiced sections of the unit are multiplied by a gain term that modifies the RMS value of each section to match the target. This can be performed by operating directly on the sinusoidal model parameters for the unit. The average energy (power) of a single synthesized frame of length N_scan be written as $\begin{array}{cc}\begin{array}{c}{E}_{\mathrm{fr}}^2=\frac{1}{N_s}\sum _{n=0}^{N_s-1}{\left|s\left[n\right]\right|}^2\\ =\frac{1}{N_s}\sum _{n=0}^{N_s-1}{\left|\sigma \left[n\right]\sum _k{a}_k\cos \left({\omega }_{k}n+{\phi }_k\right)\right|}^2.\end{array}& \left(2\right)\end{array}$

[0066] Assuming that σ[n] is relatively constant over the duration of the frame, Equation (2) can be reduced to $\begin{array}{cc}\begin{array}{c}{E}_{\mathrm{fr}}^2=\frac{{\stackrel{\_}{\sigma }}^2}{1N_s}\sum _k{a}_k^2\sum _{n=0}^{N_s-1}{\left|\cos \left({\omega }_{k}n+{\phi }_k\right)\right|}^2\\ \approx \frac{1}{2}{\stackrel{\_}{\sigma }}^2\sum _k{a}_k^2,\end{array}& \left(3\right)\end{array}$

[0067] where {overscore (σ)}²is the square of the average of σ[n] over the fiame. This energy estimate can be found for the voiced sections of the unit, and a suitable gain adjustment can be easily found. In practice, the applied gain function is smoothed to avoid abrupt discontinuities in the synthesized signal energy.

[0068] In the energy normalization described above, only voiced segments are adjusted. This implies that a voiced/unvoiced decision must be incorporated into the analysis. Since several parameters of the sinusoidal model are already available as a byproduct of the analysis, it is reasonable to attempt to use these to make a voicing decision. For instance, the pitch detection algorithm of the ABS/OLA model (described in detail in cited article and patent of George, typically defaults to a low frequency estimate below the speaker's normal pitch range when applied to unvoiced speech. FIG. 8A shows fundamental frequency and FIG. 8B shows the gain contour plots for the phrase “sunshine shimmers,” spoken by a female, with a plot of the two against each other in FIG. 8C to the right. It is clear from this plot (and even the ω₀plot alone) that the voiced and unvoiced sections of the signal are quite discernible based on these values due to the clustering of data.

[0069] For this analyzed phrase, it is easy to choose thresholds of pitch or energy to discriminate between voiced and unvoiced frames, but it is difficult to choose global thresholds that will work for different talkers, sampling rates, etc. By taking advantage of the fact that this analysis is performed off-line, it is possible to choose automatically such thresholds for each utterance, and at the same time make the V/UV decision more robust (to pitch errors, etc.) by including more data in the V/UV classification.

[0070] This can be achieved by viewing the problem as a “nearest-neighbor” clustering of the data from each frame, where feature vectors consisting of ω₀estimates, frame energy, and other data are defined. The centroids of the clusters can be found by employing the K-means (or LBG) algorithm commonly used in vector quantization, with K=2 (a voiced class and an unvoiced class). This algorithm consists of two steps:

[0071] 1. Each of the feature vectors is clustered with one of the K centroids to which it is “closest,” as defined by a distance measure, d(v, c).

[0072] 2. The centroids are updated by choosing as the new centroid the vector that minimizes the average distortion between it and the other vectors in the cluster (e.g., the mean if a Euclidean distance is used).

[0073] These steps are repeated until the clusters/centroids no longer change. In this case the feature vector used in the voicing decision is

[0074] v=[ω₀{overscore (σ)}H_SNR]^T,  (4)

[0075] where ω₀is the fundamental frequency estimate for the frame, {overscore (σ)} is the average of the time envelope σ[n] over the frame, and H_NSRis the ratio of the signal energy to the energy in the difference between the “quasiharmonic” sinusoidal components in the model and the same components with frequencies forced to be harmonically related. This is a measure of the degree to which the components are harmonically related to each other. Since these quantities are not expressed in terms of units that have the same order of magnitude, a weighted distance measure is used:

[0076] d(v, c)=(v−c)^TC⁻¹(v−c),  (5)

[0077] where C is a diagonal matrix containing the variance of each element of v on its main diagonal.

[0078] This general framework for discrimination voiced and unvoiced frames has two benefits: (i) it eliminates the problem of manually setting thresholds that may or may not be valid across different talkers; and (ii) it adds robustness to the system, since several parameters are used in the V/UV discrimination. For instance, the inclusion of energy values in addition to fundamental frequency makes the method more robust to pitch estimation errors. The output of the voicing decision algorithm for an example phrase is shown in FIG. 9.

[0079] The unit normalization method described above removes much of the energy variation between adjacent segments extracted from the inventory. However, since this normalization is performed on a fairly macroscopic level, perceptually significant short-time signal energy mismatches across concatenation boundaries remain.

[0080] An algorithm for smoothing the energy mismatch at the boundary of disjoint speech segments is described as follows:

[0081] 1. The frame-by-frame energies of N_smoothframes (typically on the order of 50 ms) around the concatenation point are found using Equation (3).

[0082] 2. The average frame energies for the left and right segments, given by E_Land E_R, respectively, are found.

[0083] 3. A target value, E_target, for the energy at the concatenation point is determined. The average E_Land E_Rin the previous step is a reasonable assumption for such a target value.

[0084] 4. Gain corrections G_Land G_Rare found by $G_L=\sqrt{\frac{E_{\mathrm{target}}}{E_L}}{G}_R=\sqrt{\frac{E_{\mathrm{target}}}{E_R}}.$

[0085] 5. Linear gain correction functions that interpolate from a value of 1 and the ends of the smoothing region to G_Land G_Rat the respective concatenation points are created, as shown in FIG. 10. These functions are then factored into the gain envelopes σ_L[n] and σ_R[n].

[0086] It should be noted that incorporating these gain smoothing functions into σ_L[n] and σ_R[n] requires a slight change in methodology. In the original model, the gain envelope σ[n] is applied after the overlap-add of adjacent frames, i.e.,

[0087] x[n]=σ[n](w_s[n]s_L[n]+(1−w_s[n])s_R[n]),

[0088] where w_s[n] is the window function, and S_L[n] and S_R[n] are the left and right synthetic contributions, respectively. However, both σ_L[n] and σ_R[n] should be included in the equation for the disjoint segments case. This can be achieved by splitting σ[n] into 2 factors in the previous equation and then incorporating the left and right time-varying gain envelopes σ_L[n] and σ_R[n] as follows:

[0089] x[n]=w_s[n]σ_L[n]s_L[n]+(1−w_s[n])σ_R[n]s_R[n].

[0090] This algorithm is very effective for smoothing energy mismatches in vowels and sustained consonants. However, the smoothing effect is undesirable for concatenations that occur in the neighborhood of transient portions of the signal (e.g. plosive phonemes like /k/), since “burst” events are smoothed in time. This can be overcome by using phonetic label information available in the TTS system to vary N_smoothbased on the phonetic context of the unit concatenation point.

[0091] Another source of perceptible discontinuity in concatenated signal segments is mismatch in spectral shape across boundaries. The segments being joined are somewhat similar to each other in basic formant structure, due to matching of the phonetic context in unit selection. However, differences in spectral shape are often still present because of voice quality (e.g., spectral tilt) variation and other factors.

[0092] One input to the ABS/OLA pitch modification algorithm is a spectral envelope estimate represented as a set of low-order cepstral coefficients. This envelope is used to maintain formant locations and spectral shape while frequencies of sinusoids in the model are altered. An “excitation model” is computed by dividing the lth complex sinusoidal amplitude a_le^jφlby the complex spectral envelope estimate H(ω)evaluated at the sinusoid frequency ω_l. These excitation sinusoids are then shifted in frequency by a factor β, and the spectral envelope is re-multiplied by H(βω_l) to obtain the pitch-shifted signal. This operation also provides a mechanism for smoothing spectral differences over the concatenation boundary, since a different spectral envelope may be reintroduced after pitch-shifting the excitation sinusoids.

[0093] Spectral differences across concatenation points are smoothed by adding weighted versions of the cepstral feature vector from one segment boundary to cepstral feature vectors from the other segment, and vice-versa, to compute a new set of cepstral feature vectors. Assuming that cepstral features for the left-side segment { . . . , L2, L₁, L₀} and features for the right-side segment {R₀, R₁, R₂. . . } are to be concatenated as shown in FIG. 11, smoothed cepstral features L_k^sfor the left segment and R_k^sfor the right segment are found by:

[0094] L_k^s=w_kL_k+(1−w_k)R₀  (7)

[0095] R_k^s=w_kR_k+(1−w_k)L₀  (8)

[0096] where $w_k=0.5+\frac{k}{2N_{\mathrm{smooth}}},$

[0097] k=1,2, . . . , N_smoothand where N_smoothframes to the left and right of the boundary are incorporated into the smoothing. It can be shown that this linear interpolation of cepstral features is equivalent to linear interpolation of log spectral magnitudes.

[0098] Once L_k^sand R_k^sare generated, they are input to the synthesis routine as an auxiliary set of cepstral feature vectors. Sets of spectral envelopes H_k(ω) and H_k^s(ω) are generated from {L_k, R_k} and {L_k^s,R_k^s}, respectively. After the sinusoidal excitation components have been pitch-modified, the sinusoidal components are multiplied by H_k^s(ω) for each frame k to impart the spectral shape derived from the smoothed cepstral features.

[0099] One of the most important functions of the sinusoidal model in this synthesis method is a means of performing prosody modification on the speech units.

[0100] It is assumed that higher levels of the system have provided the following inputs: a sequence of concatenationed, sinusoidal-modeled speech units; a desired pitch contour; and desired segmental durations (e.g., phone durations).

[0101] Given these inputs, a sequence of pitch modification factors {β_k} for each frame can be found by simply computing the ratio of the desired fundamental frequency to the fundamental frequency of the concatenated unit. Similarly, time scale modification factors {ρ_k} can be found by using the ratio of the desired duration of each phone (based on phonetic annotations in the inventory) to the unit duration.

[0102] The set of pitch modification factors generated in this manner will generally have discontinuities at the concatenated unit boundaries. However, when these-pitch modification factors are applied to the sinusoidal model frames, the resulting pitch contour will be continuous across the boundaries.

[0103] Proper alignment of adjacent frames is essential to producing high quality synthesized speech or singing. If the pitch pulses of adjacent frames do not add coherently in the overlap-add process a “garbled” character is clearly perceivable in the re-synthesized speech or singing. There are two tasks involved in properly aligning the pitch pulses: (i) finding points of reference in the adjacent synthesized frames, and (ii) shifting frames to properly align pitch pulses, based on these points of reference.

[0104] The first of these requirements is fulfilled by the pitch pulse onset time estimation algorithm described in E. Bryan George, et al. U.S. Pat. No. 5,327,518. This algorithm attempts to find the time at which a pitch pulse occurs in the analyzed frame. The second requirement, aligning the pitch pulse onset times, must be viewed differently depending on whether the frames to be aligned come from continuous speech or concatenated disjoint utterances. The time shift equation for continuous speech will be now be briefly reviewed in order to set up the problem for the concatenated voice case.

[0105] The diagrams in FIGS. 12 and 13 depict the locations of pitch pulses involved in the overlap-add synthesis of one frame. Analysis frames k and k +1 each contribute to the synthesized frame, which runs from 0 to N_s−1. The pitch pulse onset times τ_kand τ_k+1 describe the locations of the pitch pulse closest to the center of analysis frames k and k+1, respectively. In FIG. 13, the time-scale modification factor ρ is incorporated by changing the length of the synthesis frame to ρN_s, while pitch modification factors β_kand β_k+1are applied to change the pitch of each of the analysis frame contributions. A time shift δ is also applied to each analysis frame. We assume that time shift δ_khas already been applied, and the goal is to find δ_k+1to shift the pitch pulses such that they coherently sum in the overlap-add process.

[0106] From the schematic representation in FIG. 12, an equation for the time location of the pitch pulses in the original, unmodified frames k and k+1 can be written as follows:

[0107] t_k[i]=τ_k+iT₀^kt_k+1[i]=τ_k+1+iT₀^k+1,  (9)

[0108] while the indices I that refer to the pitch pulses closet to the center of the frame are given by: $\begin{array}{cc}{\hat{l}}_k=\lfloor \frac{{\tau }_k+\frac{N_s}{2}}{T_0^k}\rfloor & \left(10\right)\\ {\hat{l}}_{k+1}=-\lfloor \frac{{\tau }_{k+1}+\frac{N_s}{2}}{T_0^{k+1}}\rfloor & \end{array}$

[0109] Thus t_k[{circumflex over (l)}_k] and t_k+1[{circumflex over (l)}_k+1] are the time locations of the pitch pulses adjacent to the center of the synthesis frame.

[0110] Referring to FIG. 13, equations for these same quantities can be found for the case where the time-scale/pitch modifications are applied: $\begin{array}{cc}{t}_k\left[i\right]=\frac{{\tau }_k}{{\beta }_k}-{\delta }_k+i\left(\frac{T_0^k}{{\beta }_0}\right)& \left(11\right)\\ t_{k+1}\left[i\right]=\frac{{\tau }_{k+1}}{{\beta }_{k+1}}-{\delta }_{k+1}+i\left(\frac{T_0^{k+1}}{{\beta }_{k+1}}\right)& \left(12\right)\\ {\hat{l}}_k=\lfloor \frac{-{\tau }_k+{\beta }_k\left({\delta }_k+\frac{\rho N_s}{2}\right)}{T_0^k}\rfloor & \left(13\right)\\ {\hat{l}}_{k+1}=-\lambda \lfloor \frac{{\tau }_{k+1}+{\mathrm{\rho \beta }}_{k+1}\frac{N_s}{2}}{T_0^{k+1}}\rfloor & \left(14\right)\end{array}$

[0111] Since the analysis frames k and k+1 were analyzed from continuous speech, we can assume that the pitch pulses will naturally line up coherently when β=ρ=1. Thus the time difference Δ in FIG. 13 will be approximately the average of the pitch periods T₀^kand T₀^k+1. To find δ_k+1after modification, then, it is reasonable to assume that this time shift should become {circumflex over (Δ)}=Δ/β_av, where β_avis the average of β_kand β_k+1.

[0112] Letting {circumflex over (Δ)}=Δ/β_avand using Equations (11) through (14) to solve for δ_k+1results in the time shift equation. $\begin{array}{cc}{\delta }_{k+1}={\delta }_k+\left({\rho }_k-1/{\beta }_{\mathrm{av}}\right)N_s+\frac{{\beta }_k-{\beta }_{k+1}}{2{\beta }_{\mathrm{av}}}\left(\frac{{\tau }_k}{{\beta }_k}+\frac{{\tau }_{k+1}}{{\beta }_{k+1}}\right)-\frac{{\hat{l}}_k}{{\beta }_k}{T}_0^k+\left(i_k{T}_0^k-i_{k+1}{T}_0^{k+1}\right)/{\beta }_{\mathrm{av}}.& \left(15\right)\end{array}$

[0113] It can easily be verified that Equation (15) results in δ_k+1=δ_kfor the case ρ=β_k=β_k+1=1. In other words, the frames will naturally line up correctly in the no-modification case since they are overlapped and added in a manner equivalent oto that of the analysis method. This behavior is advantageous, since it implies that even if the pitch pulse onset time estimate is in error, the speech will not be significantly affected when the modification factors ρ, β_k, and β_k+1are close to 1.

[0114] The approach to finding δ_k+1given above is not valid, however, when finding the time shift necessary for the frame occurring just after a concatenation point, since even the condition ρ=β_k=β_k+1=1 (no modifications) does not assure that the adjacent frames will naturally overlap correctly. This is, again, due to the fact that the locations of pitch pulses (hence, onset times) of the adjacent frames across the boundary are essentially unrelated. In this case, a new derivation is necessary.

[0115] The goal of the frame alignment process is to shift frame k+1 such that the pitch pulses of the two frames line up and the waveforms add coherently. A reasonable way to achieve this is to force the time difference Δ between the pitch pulses adjacent to the frame center to be the average of the modified pitch periods in the two frames. It should be noted that this approach, unlike that above, makes no assumptions about the coherence of the pulses prior to modification. Typically, the modified pitch periods T₀^k/β_kand T₀^k+1/β_k+1will be approximately equal, thus,

[0116] Δ={tilde over (T)}₀^avg=t_k+1[{circumflex over (l)}_k+1]+ρN_s−t_k[{circumflex over (l)}_k],  (16)

[0117] where ${\tilde{T}}_0^{\mathrm{avg}}=\left(\frac{T_0^k}{{\beta }_k}+\frac{T_0^{k+1}}{{\beta }_{k+1}}\right)/2.$

[0118] Substituting Equations (11) through (14) into Equation (16) and solving for δ_k+1, we obtain $\begin{array}{cc}{\delta }_{k+1}={\delta }_k+\frac{{\tau }_{k+1}}{{\beta }_{k+1}}-\frac{{\tau }_k}{{\beta }_k}+{\hat{l}}_{k+1}\left(\frac{T_0^{k+1}}{{\beta }_{k+1}}\right)-{\hat{l}}_k\left(\frac{T_0^k}{{\beta }_k}\right)+\rho N_s-{\tilde{T}}_0^{\mathrm{avg}}.& \left(17\right)\end{array}$

[0119] This gives an expression for the time shift of the sinusoidal components in frame k+1. This time shift (which need not be an integer) can be implemented directly in the frequency domain by modifying the sinusoid phases φ_iprior to re-synthesis:

[0120] {tilde over (φ)}_i=φ_i+iβω₀δ.  (18)

[0121] It has been confirmed experimentally that applying Equation (17) does indeed result in coherent overlap of pitch pulses at the concatenation boundaries in speech synthesis. However, it should be noted that this method is critically dependent on the pitch pulse onset time estimates τ_kand τ_k+1. If either of these estimates is in error, the pitch pulses will not overlap correctly, distorting the output waveform. This underscores the importance of the onset estimation algorithm described in E. Bryan George, et al. U.S. Pat. No. 5,327,518. For modification of continuous speech, the onset time accuracy is less important, since poor frame overlap only occurs due to an onset time error when β is not close to 1.0, and only when the difference between two onset time estimates is not an integer multiple of a pitch pulse. However, in the concatenation case, onset errors nearly always result in audible distortion, since Equation (17) is completely reliant on the correct estimation of pitch pulse-onset times to either side of the concatenation point.

[0122] Pitchrmarks derived from an electroglottograph can be used as initial estimates of the pitch onset time. Instead of relying on the onset time estimator to search over the entire range [−T₀/2, T₀/2], the pitchmark closest to each frame center can be used to derive a rough estimate of the onset time, which can then be refined using the estimator function described earlier. The electroglottograph produces a measurement of glottal activity that can be used to find instants of glottal closure. This rough estimate dramatically improves the performance of the onset estimator and the output voice quality.

[0123] The musical control information such as vibrato, pitch, vocal effect scaling, and vocal tract scaling is provided from the MIDI file 11 via the MIDI file interpreter 13 to the concatenator/smoother 19 in FIG. 1 to perform modification to the units from the inventory.

[0124] Since the prosody modification step in the sinusoidal synthesis algorithm transforms the pitch of every frame to match a target, the result is a signal that does not exhibit the natural pitch fluctuations of the human voice.

[0125] In an article by Klatt, et al., entitled, “Analysis, Synthesis, and Perception of Voice Quality Variations Among Female and Male Talkers,” Journal of the Acoustical Society of America (Vol. 87, pp. 820-857, February 1990), a simple equation for “quasirandom” pitch fluctuations in speech is proposed: $\begin{array}{cc}\Delta F_0=\frac{F_0}{100}\left(\sin \left(12.7\pi t\right)+\sin \left(7.1\pi t\right)+\sin \left(4.7\pi t\right)\right)/3.& \left(19\right)\end{array}$

[0126] The addition of this fluctuation to the desired pitch contour gives the voice a more “human” feel, since a slight wavering is present in the voice. A global scaling of ΔF₀is incorporated as a controllable parameter to the user, so that more or less fluctuation can be synthesized.

[0127] Abrupt transitions of one note to another at a different pitch are not a natural phenomena. Rather, singers tend to transition somewhat gradually from one note to another. This effect can be modeled by applying a smoothing at note-to-note transitions in the target pitch contour. Timing of the pitch change by human vocalists is usually such that the transition between two notes takes place before the onset of the second note, rather than dividing evenly between the two notes.

[0128] The natural “quantal unit” of rhythm in vocal music is the syllable. Each syllable of lyric is associated with one or more notes of the melody. However, it is easily demonstrated that vocalists do not execute the onsets of notes at the beginnings of the leading consonant in a syllable, but rather at the beginning of the vowel. This effect has been cited in the study of rhythmic characteristics of singing and speech. Applicants' system 10 employs rules that align the beginning of the first note in a syllable with the onset of the vowel in that syllable.

[0129] In this work, a simple model for scaling durations of syllables is used. First an average time scaling factor ρ_syllis computed: $\begin{array}{cc}{\rho }_{\mathrm{syll}}=\frac{\sum _{n=1}^{N_{\mathrm{notes}}}{D}_n}{\sum _{m=1}^{N_{\mathrm{phon}}}{D}_m},& \left(20\right)\end{array}$

[0130] where the values D_nare the desired durations of the N_notesnotes associated with the syllable and D_mare the durations of the N_phonphonemes extracted from the inventory to compose the desired syllable. If ρ_syll>1, then the vowel in the syllable is looped by repeating a set of frames extracted from the stationary portion of the vowel, until ρ_syll≈1. This preserves the duration of the consonants, and avoids unnatural time-stretching effects. If ρsyll<1, the entire syllable is compressed in time by setting the time-scale modification factor ρ for all frames in the syllable equal to ρ_syll.

[0131] A more sophisticated approach to the problem involves phoneme-and context-dependent rules for scaling phoneme durations in each syllable to more accurately represent the manner in which humans perform this adjustment.

[0132] The physiological mechanism of the pitch, amplitude, and timbral variation referred to as vibrato is somewhat in debate. However, frequency modulation of the glottal source waveform is capable of producing many of the observed effects of vibrato. As the source harmonics are swept across the vocal tract resonances, timbre and amplitude modulations as well as frequency modulation take place. These modulations can be implemented quite effectively via the sinusoidal model synthesis by modulating the fundamental frequency of the components after removing the spectral envelope shape due to the vocal tract (an inherent part of the pitch modification process).

[0133] Most trained vocalists produce a 5-6 Hz near-sinusoidal vibrato. As mentioned, pure frequency modulation of the glottal source can represent many of the observed effects of vibrato, since amplitude modulation will automatically occur as the partials “sweep by” the formant resonances. This effect is also easily implemented within the sinusoidal model framework by adding a sinusoidal modulation to the target pitch of each note. Vocalists usually are not able to vary the rate of vibrato, but rather modify the modulation depth to create expressive changes in the voice.

[0134] Using the graphical MIDI-based input to the system, users can draw contours that control vibrato depth over the course of the musical phrase, thus providing a mechanism for adding expressiveness to the vocal passage. A global setting of the vibrato rate is also possible.

[0135] In synthesis of bass voices using a voice inventory recorded from a baritone male vocalist, it was found that the voice took on an artificial-sounding “buzzy” quality, caused by extreme lowering of the fundamental frequency. Through analysis of a simple tube model of the human vocal tract, it can be shown that the nominal formant frequencies associated with a longer vocal tract are lower than those associated with a shorter vocal tract. Because of this, larger people usually have voices with a “deeper” quality; bass vocalists are typically males with vocal tracts possessing this characteristic.

[0136] To approximate the differences in vocal tract configuration between the recorded and “desired” vocalists, a frequency-scale warping of the spectral envelope (fit to the set of sinusoidal amplitudes in each frame) was performed, such that

[0137] H(ω)=H(ω/μ),

[0138] where H(ω) is the spectral envelope and μ is a global frequency scaling factor dependent on the average pitch modification factor. The factor μ typically lies in the range 0.75<μ<1.0. This frequency warping has the added benefit of slightly narrowing the bandwidths of the formant resonances, mitigating the buzzy character of pitch-lowered sounds. Values of μ>1.0 can be used to simulate a more child-like voice, as well. In tests of this method, it was found that this frequency warping gives the synthesized bass voice a much more rich-sounding, realistic character.

[0139] Another important attribute of the vocal source in singing is the variation of spectral tilt with loudness. Crescendo of the voice is accompanied by a leveling of the usual downward tilt of the source spectrum. Since the sinusoidal model is a frequency-domain representation, spectral tilt changes can be quite easily implemented by adjusting the slope of the sinusoidal amplitudes. Breathiness, which manifests itself as high-frequency noise in the speech spectrum, is another acoustic correlate of vocal intensity. This frequency-dependent noise energy can be generated within the ABS/OLA model framework by employing a phase modulation technique during synthesis.

[0140] Simply scaling the overall amplitude of the signal to produce changes in loudness has the same perceptual effect as turning the “volume knob” of an amplifier; it is quite different from a change in vocal effort by the vocalist. Nearly all studies of singing mention the fact that the downward tilt of the vocal spectrum increases as the voice becomes softer. This effect is conveniently implemented in a frequency-domain representation such as the sinusoidal model, since scaling of the sinusoid amplitudes can be performed. In the present system, an amplitude scaling function based on the work of Bennett, et al. in Current Directions in Computer Research (pp. 19-44) MIT Press, entitled, “Synthesis of the Singing Voice” is used: $\begin{array}{cc}{G}_{\mathrm{dB}}=\frac{T_{in}{\log }_{10}\left(F_l/500\right)}{{\log }_{10}\left(3000/500\right)},& \left(21\right)\end{array}$

[0141] where F_lis the frequency of the lth sinusoidal component and T_inis a spectral tilt parameter controlled by a MIDI “vocal effort” control function input by the user. This function produces a frequency-dependent gain scaling function parameterized by T_inas shown in FIG. 14

[0142] In studies of acoustic correlates of perceived voice qualities, it has been shown that utterances perceived as “soft” and “breathy” also exhibit a higher level of high frequency aspiration noise than fully phonated utterances, especially in females. This effect on glottal pulse shape and spectrum is shown in FIG. 15. It is possible to introduce a frequency-dependent noise-like character to the signal by employing the subframe phase randomization method. In this system, this capability has been used to model aspiration noise. The degree to which the spectrum is made noise-like is controlled by a mapping from the MIDI-controlled vocal effort parameter to the amount of phase dithering introduced.

[0143] Informal experiments with mapping the amount of randomization to (i) a cut-off frequency above which phases are dithered, and (ii) the scaling of the amount of dithering within a fixed band, have been performed. Employing either of these strategies results in a more natural, breathy, soft voice, although careful adjustment of the model parameters is necessary to avoid an unnaturally noisy quality in the output. A refined model that more closely represents the acoustics of loudness scaling and breathiness in singing is a topic for more extensive study in the future.

[0144] Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.