US 6,336,092 B1Grant
Targeted vocal transformation
Issue Date:2002-01-01
•39 Claims
•8 Drawing Sheets
Abstract
The invention is a method for transforming a source individual's voice so as to adopt the characteristics of a target individual's voice. The excitation signal component of the target individual's voice is extracted and the spectral envelope of the source individual's voice is extracted. The transformed voice is synthesized by applying the spectral envelope of the source individual to the excitation signal component of the voice of the target individual. A higher quality transformation is achieved using an enhanced excitation signal created by replacing unvoiced regions of the signal with interpolated data from adjacent voiced regions. Various methods of transforming the spectral characteristics of the source individual's voice are also disclosed.
Metadata
Assignee
- Ivl Technologies Ltd
Inventors
- Brian Charles Gibson
- Peter Ronald Lupini
- Dale John Shpak
Application Information
Application Number:US 08/848,050
Filing Date:1997-04-28
Priority Date:1997-04-28
Art Unit:7
Classifications
IPC:
G10L 1306G10L 1104
Field of Search:
7042767042077042097042787042657042007042037042057042087042137042147042707042667042677042687042698460284603
Patent Drawings (8 sheets)
Description
Field of the Invention
[0002] This invention relates to the transformation of a person's voice according to a target voice. More particularly, this invention relates to a transformation system where recorded information of the target voice can be used to guide the transformation process. It further relates to the transformation of a singer's voice to adopt certain characteristics of a target singer's voice, such as pitch and other prosodic factors.
Background of the Invention
[0003] There are a number of applications where it may be desirable to transform a person's voice (the source vocal signal) into a different person's voice (the target vocal signal). This invention performs such a transformation and is suited to applications where a recording of the target voice is available for use in the transformation process. Such applications include Automatic Dialogue Replacement (ADR) and Karaoke. We have chosen to describe the karaoke application because of the additional demands for accurate pitch processing in such a system but the same principles apply for a spoken-word system.
[0004] Karaoke allows the participants to sing songs made popular by other artists. The songs produced for karaoke have the vocal track removed leaving behind only the musical accompaniment. In Japan, karaoke is the second largest leisure activity, after dining out. Some people, however, cannot participate in the karaoke experience because they are unable to sing in the correct pitch.
[0005] Often, as part of the karaoke experience, the singer tries to mimic the style and sound of the artist who originally made the recording. This desire for voice transformation is not limited to karaoke but is also important for impersonators who might mimic, for example, Elvis Presley performing one of his songs.
[0006] Most of the research in voice transformation has related to the spoken voice as opposed to the sung voice. H. Kuwabara and Y. Sagisaka, Acoustic characteristics of speaker individuality: Control and conversion, Speech Communication,vol. 16, 1995 separated the factors responsible for voice individuality into two categories:
[0007] physiological factors (e.g. length of the vocal tract, glottal pulse shape, and position and bandwidth of the formants), and
[0008] socio-linguistic and psychological factors, or prosodic factors (e.g. pitch contour, duration of words, timing and rhythm)
[0009] The bulk of the research into voice transformation has focused on the direct conversion of the physiological factors, particularly vocal tract length compensation and formant position/bandwidth transformation. Although it appears to be recognized that the most important factors for voice individuality are the prosodic factors, current speech technologies have not allowed useful extraction and manipulation of the prosodic features and have instead focused on direct mapping of vocal characteristics.
[0010] The inventors have found that the important characterizing parameters for successful voice conversion to a specified target depend on the target singer. For some singers, the pitch contour at the onset of notes (for example the “scooping” style of Elvis Presley) is critical. Other singers may be recognized more for the “growl” in their voice (e.g. Louis Armstrong). The style of vibrato is another important factor of voice individuality. These examples all involve prosodic factors as the key characterizing features. While physiological factors are also important, we have found that the transformation of physiological parameters need not be exact in order to achieve a convincing identity transformation. For example it may be enough to transform the perceived vocal-tract length without having to transform the individual formant locations and bandwidths.
Summary of the Invention
[0011] The present invention provides a method and apparatus for transforming the vocal characteristics of a source singer into those of a target singer. The invention relies on the decomposition of a signal from a source singer into excitation and vocal tract resonance components. It further relies on the replacement of the excitation signal of the source singer with an excitation signal derived from a target singer. This disclosure also presents methods of shifting the timbre of the source singer into that of the target singer by modifying the vocal tract resonance model. Additionally, pitch-shifting methods may be used to modify the pitch contour to better track the pitch of the source singer.
[0012] According to the invention, the excitation component and pitch contour of the vocal signal of the target singer are first obtained. This is done by essentially extracting the excitation signal and pitch data from the target singer's voice and storing them for use in the vocal transformer.
[0013] The invention allows the transformation of voice either with or without pitch correction to match the pitch of the target singer. When used to transform voice with pitch correction, the source singer's vocal signal is converted from analog to digital data, and then separated into segments. For each segment, a voicing detector is used to determine whether the signal contains voiced or unvoiced data. If the signal contains unvoiced data, the signal is sent to the digital to analog converter to be played on the speaker. If the segment contains voiced data, the signal is analyzed to determine the shape of the spectral envelope which is then used to produce a time-varying synthesis filter. If timbre and/or gender shifting or other vocal transformations are also desired, or in cases where doing so will improve the results (e.g., where the spectral shapes of the source and target voices are very different) the spectral envelope may first be transformed, then used to create the time-varying synthesis filter. The transformed vocal signal is then created by passing the target excitation signal through the synthesis filter. Finally, the amplitude envelope of the untransformed source vocal signal is used to shape the amplitude envelope of the transformed source vocal.
[0014] When used as a voice transformer without pitch correction, two extra steps are performed. First the pitch of the source vocal is extracted. Then the pitch of the target excitation is shifted using a pitch shifting algorithm so that the target excitation pitch is made to track the pitch of the source vocal.
Brief Description of the Drawings
[0015] The invention may be more fully appreciated by reference to the following description of the preferred embodiments thereof in conjunction with the drawings wherein:
[0016] FIG. 1 is a block diagram of a processor used to create a target excitation signal.
[0017] FIG. 2 is a block diagram of a processor used to create an enhanced target excitation signal.
[0018] FIG. 3 is a block diagram of a vocal transformer with pitch correction.
[0019] FIG. 4 is a block diagram of a vocal transformer without pitch correction (i.e. the pitch is controlled by the source singer).
[0020] FIG. 5 is a graph illustrating the effect of conformal mapping on a spectral envelope.
[0021] FIG. 6 is a graph illustrating the different spectral envelopes for voicing at different pitches.
[0022] FIG. 7 is a block diagram illustrating separate modifications of the low frequency and high frequency components of the spectral envelope.
[0023] FIG. 8 is a block diagram illustrating the processing of only the voice-band portion of a signal having a high sampling rate.
Detailed Description of the Best Mode and the Preferred Embodiments
[0024] Referring to the block diagram of FIG. 1, a target vocal signal is first converted to digital data. This step is, of course, not required if the input signal is already presented in digital format.
[0025] The first step is to perform spectral analysis on the target vocal signal. The spectral envelope is determined and used to create a time-varying filter for the purpose of flattening the spectral envelope of the target vocal signal. The method used for performing spectral analysis could employ various techniques from the prior art for generating a spectral model. These spectral analysis techniques include all-pole modeling methods such as linear prediction (see for example, P. Strobach, “Linear Prediction Theory”, Springer-Verlag, 1990), adaptive filtering (see J. I. Makhoul and L. K. Cosell, “Adaptive Lattice Analysis of Speech,” IEEE Trans. Acoustics, Speech, Signal Processing, vol. 29, pp. 654-659, June 1981), methods for pole-zero modeling such as the Steiglitz-McBride algorithm (see K. Steiglitz and L. McBride, “A technique for the identification of linear systems”, IEEE Trans. Automatic Control, vol. AC-10, pp. 461-464, 1965), or transform-based methods including multi-band excitation (D. Griffin and J. Lim, “Multiband excitation vocoder”, IEEE Trans. Acoustics, Speech, Signal Process., vol. 36, pp. 1223-1235, August 1988) and cepstral-based methods (A. Oppenheim and R. Schafer, “Homomorphic analysis of speech”, IEEE Trans. Audio Electroacoust., vol. 16, June 1968). The all-pole or pole-zero models are typically used to generate either lattice or direct-form digital filters. The amplitude of the frequency spectrum of the digital filter is chosen to match the amplitude of the spectral envelope obtained from the analysis.
[0026] The preferred embodiment uses the autocorrelation method of linear prediction because of its computational simplicity and stability properties. The target voice signal is first separated into analysis segments. The autocorrelation method generates P reflection coefficients ki. These reflection coefficients can be used directly in either an all-pole synthesis digital lattice filter or an all-zero analysis digital lattice filter. The order of the spectral analysis P depends on the sample rate and other parameters as described in J. Markel and A. H. Gray Jr., Linear Prediction of Speech.Springer-Verlag, 1976.
[0027] The alternative direct-form implementation for this all-pole method has a time-domain difference equation of the form:
[0028] where y(k) is the current filter output sample value, x(k) is the current input sample value, and the a(i)'s are the coefficients of the direct-form filter. These coefficients a(i) are computed from the values of the reflection coefficients ki. The corresponding z-domain transfer function for the all-pole synthesis is:
[0029] The complementary all-zero analysis filter has a difference equation given by:
[0030] and a z-domain transfer function given by:
[0031] Whether using a lattice, direct-form, or other digital filter implementation, the target vocal signal is processed by an analysis filter to compute an excitation signal having a flattened spectrum which is suitable for vocal transformation applications. For use by a vocal transformer, this excitation signal can either be computed in real time or it can be computed beforehand and stored for later use. The excitation signal derived from the target may be stored in a compressed form where only the information essential to reproducing the character of the target singer are stored.
[0032] As an enhancement to the vocal transformer, it is possible to further process the target excitation signal in order to make the system more forgiving of timing errors made by the source singer. For example, when the source singer sings a particular song his phrasing may be slightly different from the target singer's phrasing of that song. If the source singer begins singing a word slightly before the target singer did in his recording of the song there would be no excitation signal available to generate the output until the point where the target singer began the word. The source singer would perceive that the system is unresponsive and would find the delay annoying. Even if the alignment of the words is accurate it is unlikely that the unvoiced segments from the source singer will line up exactly with the unvoiced segments for the target singer. In this case the output would sound quite unnatural if the excitation from an unvoiced portion of the target singer's signal was applied to generate a voiced segment in the output. The goal of this enhanced processing is to extend the excitation signal into the silent region before and after each word in the song and to identify unvoiced regions within the words and provide voiced excitation for those segments.
[0033] The enhanced excitation processing system is shown in FIG. 2. The target excitation signal is separated into segments which are classified as being either voiced or unvoiced. In the preferred embodiment, voicing detection is accomplished by examining the following parameters: average segment power, average low-band segment power, and zero crossings per segment. If the total average power for a segment is less than a 60 db below the recent maximum average power level, the segment is declared silent. If the number of zero crossings exceeds 8/ms, the segment is declared unvoiced. If the number of zero crossings are less than 5/ms, the segment is declared voiced. Finally, if the ratio of low-band average power to total band average power is less than 0.25, the segment is declared unvoiced. Otherwise it is declared voiced.
[0034] For voiced segments, the pitch is extracted. For unvoiced or silent segments, the pitch is set to 0 and the unvoiced data is replaced with silence. The target excitation signal is then analyzed for gaps which are left due to non-voiced regions. The gaps are then filled in with interpolated voiced data from previous and subsequent voiced regions.
[0035] There are several ways in which the interpolation can be accomplished. In all cases, the goal is to create an interpolated voiced signal having a pitch contour which blends with the bounding pitch contour in a meaningful way (for example, for singing, the interpolated notes should sound good with the background music). For some applications, the interpolated pitch contour may be calculated automatically, using, for example, cubic spline interpolation. In the preferred embodiment, the pitch contour is first computed using spline interpolation, and then any portions which are deemed unsatisfactory are fixed manually by an operator.
[0036] Once a suitable pitch contour is obtained, the gaps in the waveform left due to removal of unvoiced regions must be filled in at the interpolated pitch value. There are several methods for doing this. In one method, the samples from voiced segments prior to the gap are copied across the gap and then pitch shifted using the interpolated pitch contour. In the preferred embodiment, sinusoidal synthesis is used to morph between the waveforms on either side of the gap. Sinusoidal synthesis has been used extensively in fields such as speech compression (see, for example, D. W. Griffin and J. S. Lim, “Multiband excitation vocoder,” IEEE Trans. Acoustics, Speech, and Signal Processing, vol. 36, pp. 1223-1235, August, 1988). In speech compression, sinusoidal synthesis is used to reduce the number of bits required to represent a signal segment. For these applications, the pitch contour over a segment is usually interpolated using quadratic or cubic interpolation. For our application, however, the goal is not one of compression, but rather the “morphing” of one sound into another following a pitch contour which is pre-defined (possibly even manually generated by an operator), therefore a new technique has been developed for the preferred embodiment (note that the equations are shown in the continuous time domain for simplicity) as set out below.
[0037] Assume that a gap between times t1and t2must be filled in via sinusoidal interpolation. First, the pitch contour, w(n), is determined (automatically or manually by an operator). Then spectral analysis using the Fast Fourier Transform (FFT) with peak picking (see, for example, R. J. McAulay and T. F. Quatieri, “Sinusoidal Coding”, in Speech Coding and Synthesis, Elsevier Science B.V, 1995) is performed at t1and t2to obtain the spectral magnitudes Ak(t1) and Ak(t2), and phases φk(t1) and φk(t2), where the subscript k refers to the harmonic number. The synthesized signal segment, y(t), can then be computed as:
[0038] where K is the number of harmonics in the segment (set to half the length of the number of samples in the longest pitch period of the segment). The model we use for the time varying phase for t1≦t≦t2is given by:
[0039] where rk(t) is a random pitch component used to reduce the correlation between harmonic phases and thus reduce perceived buzziness, and dkis a linear pitch correction term used to match the phases at the start and end of the synthesis segment. Using the fact that we want θk(t1)=φ(t1) and θk(t2)=φ(t2) in order to avoid discontinuous phase at the segment boundaries, it can be shown that the smallest possible value for dkwhich satisfies this constraint is given by:
[0040] where T=(t2−t1), and
[0041] The random pitch component, rk(t), is obtained by sampling a random variable having a variance which is determined for each harmonic by computing the difference between the predicted phase and measured phase for signal segments adjacent to the gap to be synthesized, and setting the variance proportional to this value.
[0042] Finally as with the unenhanced excitation extraction described earlier, the amplitude envelope of the target excitation signal is flattened using automatic gain compensation.
[0043] The excitation signal can also be a composite signal which is generated from a plurality of target vocal signals. In this manner, the excitation signal could contain harmony, duet, or accompaniment parts. For example, excitation signals from a male singer and a female singer singing a duet in harmony could each be processed as described above. The excitation signal which is used by the apparatus would then be the sum of these excitation signals. The transformed vocal signal which is generated by the apparatus would therefore contain both harmony parts with each part having characteristics (e.g., pitch, vibrato, and breathiness) derived from the respective target vocal signals.
[0044] The resulting basic or enhanced target excitation signal and pitch data are then typically stored, usually for later use in a vocal transformer. Alternatively, the unprocessed target vocal signal may be stored and the target excitation signal generated when needed. The enhancement of the excitation could be entirely rule-based or the pitch contour and other controls for generating the excitation signal during silent and unvoiced segments could be stored along with the unprocessed target vocal signal.
[0045] The block diagram of FIG. 3 will now be described.
[0046] A block of source vocal signal samples is analyzed to determine whether they are voiced or unvoiced. The number of samples contained in this block would typically correspond to a time span of approximately 20 milliseconds. e.g., for a sample rate of 40 kHz, a 20 ms block would contain 800 samples. This analysis is repeated on a periodic or pitch-synchronous basis to obtain a current estimate of the time-varying spectral envelope. This repetition period may be of lesser time duration than the temporal extent of the block of samples, implying that successive analyses would use overlapping blocks of vocal samples.
[0047] If the block of samples are determined to represent unvoiced input, the block is not further processed and is presented to the digital to analog converter for presentation to the output speaker. If the block of samples is determined to represent voiced input, a spectral analysis is performed to obtain an estimate of the envelope of the frequency spectrum of the vocal signal.
[0048] It may be desirable or even necessary to modify the shape of the spectral envelope in some voice conversions. For example where the source and target vocal signals are of different genders, it may be desirable to shift the timbre of the source's voice by scaling the spectral envelope to more closely match the timbre of the target vocal signal. In the preferred embodiment, the optional section for modification of the spectral envelope (entitled “Modify Spectral Envelope” in FIG. 3) alters the frequency spectrum of the envelope obtained from the Spectral Analysis block. Five methods for spectral modification are contemplated.
[0049] A first method is to modify the original spectral envelope by applying a conformal mapping to the z-domain transfer function in equation (2). Conformal mapping modifies the transfer function, resulting in a new transfer function of the form:
[0050] Applying conformal mapping results in a modified spectral envelope, as shown in FIG. 5. Details of the technique of applying a conformal mapping to a digital filter can be found in A. Constantinides, “Spectral transformations for digital filters,” Proceedings of the IEEE, vol. 117, pp. 1585-1590, August 1970. The advantage of this method is that it is unnecessary to compute the singularities of the transfer function.
[0051] A second method is to find the singularities (i.e., poles and zeros) of the digital filter transfer function, to then modify the location of any or all of these singularities, and then to use these new singularities to generate a new digital filter having the desired spectral characteristics. This second method applied to vocal signal modifications is known in the prior art.
[0052] A third method for modifying the spectral envelope, which obviates the need for a separate Modify Spectral Envelope step, is to modify the temporal extent of the blocks of vocal signals prior to the spectral analysis. This results in the spectral envelope obtained as a result of the spectral analysis being a frequency-scaled version of the unmodified spectral envelope. The relationship between time scaling and frequency scaling is described mathematically by the following property of the Fourier transform:
[0053] where the left side of the equation is the time-scaled signal and the right side of the equation is the resulting frequency-scaled spectrum. For example, if the existing analysis block is 800 samples in length (representing 20 ms of the signal), an interpolation method could be used to generate 880 samples from these samples. Since the sampling rate is unchanged, this time-scales the block such that it now represents a longer time period (22 ms). By making the temporal extent longer by 10 percent, the features in the resulting spectral envelope will be reduced in frequency by 10 percent. Of the methods for modifying the spectral envelope, this method requires the least amount of computation.
[0054] A fourth method would involve manipulating a frequency-transformed representation of the signal as described in S. Seneff, System to independently modify excitation and/or spectrum of speech waveform without explicit pitch extractions, IEEE Trans. Acoustics, Speech, Signal Processing, Vol. 30, August 1982.
[0055] A fifth method is to decompose the digital filter transfer function (which may have a high order) into a number of lower-order sections. Any of these lower-order sections could then be modified using the previously-described methods.
[0056] A particular problem arises when the pitch of the target singer and the source singer differ by an appreciable amount, e.g. an octave, in that their respective spectral envelopes will have significant differences, especially in the low-frequency region below about 1 kHz. For example, in FIG. 6, low-pitched voicing results in a low-frequency resonance near 200 Hz whereas high-pitched voicing results in a higher-frequency resonance near 400 Hz. These differences can cause two problems:
[0057] a reduction in low-frequency power in the transformed vocal signal; and
[0058] amplification of system noise by a spectral peak that does not have a frequency near a harmonic of the output pitch.
[0059] These problems can be alleviated by modifying the low-frequency portion of the spectral envelope which can be accomplished by employing the aforementioned methods for modifying the spectral envelope. The low-frequency portion of the spectral envelope can be modified directly by using methods two or four.
[0060] Methods one and three can also be used for this purpose if the target vocal signal is split into a low-frequency component (e.g., less than or equal to 1.5 kHz) and a high-frequency component (e.g., greater than 1.5 kHz). A separate spectral analysis can then be undertaken for both components as shown in FIG. 7. The spectral envelope from the lower-frequency analysis would then be modified in accordance to the difference in pitches or difference in the location of the spectral peaks. For example, if the target singer's pitch was 200 Hz and the source singer's pitch was 400 Hz, the unmodified source spectral envelope may have a peak near 400 Hz and, without a peak near 200 Hz, there would be a smaller gain near 200 Hz, resulting in the first problem noted above. We would therefore modify the lower-frequency envelope to move the spectral peak from 400 Hz toward 200 Hz.
[0061] The preferred embodiment modifies the low-frequency portion of the spectral envelope in the following manner:
[0062] 1. The source vocal signal S(t) is lowpass filtered to create a bandlimited signal SL(t) containing only frequencies below about 1.5 kHz.
[0063] 2. This bandlimited signal SL(t) is then re-sampled at about 3 kHz to create a lower-rate signal SD(t)
[0064] A low-order spectral analysis (e.g., P=4) is performed on SD(t) and the direct-form filter coefficients aD(i) are computed.
[0065] 3. These coefficients are modified using the conformal-mapping method to scale the spectrum in proportion to the ratio between the pitch of the target vocal signal and pitch of the source vocal signal.
[0066] 4. The resulting filter is applied to the signal SL(t) (having the original sampling rate) using the technique of interpolated filtering.
[0067] Using this technique, the low-frequency and high-frequency portions of the signal are processed separately and then summed to form the output signal, as shown in FIG. 7. With reference to FIG. 7, the apparatus can be used to modify only the low-frequency spectral envelope or only the high-frequency spectral envelope. In this way, it can modify the low-frequency resonances without affecting the timbre of the high-frequency resonances or it can change only the timbre of the high-frequency resonances. It is also possible to modify both of these spectral envelopes concurrently.
[0068] Another method which can be used to alleviate the aforementioned problems regarding the low-frequency region of the spectral envelope is to increase the bandwidth of the spectral peaks. This can be accomplished by applying techniques from prior art such as:
[0069] bandwidth expansion
[0070] modifying the radius of selected poles
[0071] windowing the autocorrelation vector prior to computing the filter coefficients
[0072] High-fidelity digital audio systems typically employ higher sampling rates than are used in speech analysis or coding systems. This is because, with speech, most of the dominant spectral components have frequencies less than 10 kHz. When using a high sampling rate with a high-fidelity system, the aforementioned order of the spectral analysis P can be reduced if the signal is split into high-frequency (e.g., greater than 10 kHz) and low-frequency (e.g. less than or equal to 10 kHz) signals by using digital filters. This low-frequency signal can then be down-sampled to a lower sampling rate before the spectral analysis and will therefore require a lower order of analysis.
[0073] The lower sampling rate and the lower order of analysis both result in reduced computational requirements. In the preferred embodiment, the input vocal signal is sampled at a high rate of over 40 kHz. The signal is then split into two equal-width frequency bands, as shown in FIG. 8. The low-frequency portion is decimated and then analyzed in order to generate the reflection coefficients ki. The excitation signal is also sampled at this high rate and then filtered using an interpolated lattice filter (i.e., a lattice filter where the unit delays are replaced by two unit delays). This signal is then post-filtered by a lowpass filter to remove the spectral image of the interpolated lattice filter and gain compensation is applied. The resulting signal is the low-frequency component of the transformed vocal signal. The interpolated filtering technique is used rather than the more conventional downsample-filter-upsample method since it completely eliminates distortion due to aliasing in the resampling process. The need for an interpolated lattice filter would be obviated if the excitation signal was sampled at a lower rate matching the decimated rate. Preferably, the invention would use two different sampling rates concurrently thereby reducing the computational demands.
[0074] The final output signal is obtained by summing a gain-compensated high-frequency signal and the transformed low-frequency component. This method can be applied in conjunction with the method illustrated in FIG. 7.
[0075] The spectral envelope can therefore be modified by a plurality of methods and also through combinations of these methods. The modified spectral envelope is then used to generate a time-varying synthesis digital filter having the corresponding frequency response. In the block entitled Apply Spectral Envelope, this digital filter is applied to the target excitation signal which was generated as a result of the excitation signal extraction processing step. The preferred embodiment implements this filter using a lattice digital filter. The output of this filter is the discrete-time representation of the desired transformed vocal signal.
[0076] The purpose of the block in FIG. 3 entitled Apply Amplitude Envelope is to make the amplitude of the transformed vocal signal track the amplitude of the source vocal. This block requires a number of subsidiary computations:
[0077] The level of the digitized source vocal signal Ls.
[0078] The level of the digitized target excitation signal Le.
[0079] The level of the signal after applying the spectral envelope Lt.
[0080] These levels are used to compute an output amplitude level which is applied to the original signal after it has passed through the synthesis filter.
[0081] In the preferred embodiment, each level is computed using the following recursive algorithm:
[0082] The frame level Lf(i) for the ith frame of 32 samples is computed as the maximum of the absolute values of the samples within the frame.
[0083] A decayed previous level is computed as Ld(i)=0.99 L(i−1).
[0084] The level is computed as L(i)=max { Lf(i),Ld(i)}.
[0085] The amplitude envelope to be applied to the current output frame is also computed using a recursive algorithm:
[0086] Compute the unsmoothed amplitude correction Ar(i)=LsLe/Lt.
[0087] Compute the smoothed amplitude correction As(i)=0.9As(i−1)+0.1Ar(i)
[0088] This algorithm uses delayed values of Lsand Leto compensate for processing delays within the system.
[0089] The frame-to-frame values of Asare linearly interpolated across the frames to generate a smoothly-varying amplitude envelope. Each sample from the Apply Spectral Envelope block is multiplied by this time-varying envelope.
[0090] FIG. 4 illustrates the case where the pitch of the source vocal signal is to be retained. In such a case, the pitch of the source vocal signal is determined. A method for doing so is disclosed in Gibson, et al., U.S. Pat. No. 4,688,464, the contents of which are incorporated herein by reference. The target excitation signal is then pitch shifted by the amount required to track the pitch of the source vocal signal before applying the modified or unmodified source spectral envelope to the excitation signal. A method of pitch shifting suitable for this purpose is disclosed in Gibson et al., U.S. Pat. No. 5,567,901, the contents of which are incorporated herein by reference. Note that while this mode of operation gives the source singer more control over the output, it can also significantly reduce the effectiveness of the transformation in cases where the character of the target singer is identified by fast varying pitch changes such as vibrato or pitch scooping. To prevent the loss of characteristic rapid pitch changes, the pitch detection process may also use long-term averaging when computing pitch shift amounts. Pitch data is averaged over ranges between 50 ms and 500 ms depending on the characteristics of the target singer. The averaging calculation is reset whenever a new note is detected. In some applications the pitch of the target excitation is shifted by a fixed amount, to accomplish a key change, and the pitch of the source singer is ignored.
[0091] It will be appreciated by those skilled in the art that variations of the preferred embodiment may also be practised without departing from the scope of the invention. It will also be appreciated that the approaches of the invention are not limited to singing voices but may equally be applied to speech.
Claims
What is claimed is:
1. A method of transforming the voice of a source individual so as to adopt characteristics of a target individual, comprising:
providing a spectral envelope derived from the voice of the source individual;
providing an excitation signal component derived from the voice of the target individual; and
applying the spectral envelope from the source individual to the excitation signal component from the target individual.
2. The method according to claim 1 further comprising the step of extracting and storing the excitation signal component from the voice of the target individual.
3. The method according to claim 2 wherein the step of extracting the excitation signal is performed by flattening the spectral envelope of the target vocal signal.
4. The method according to claim 2 further comprising the step of storing said extracted excitation signal.
5. The method of claim 4 wherein said step of storing comprises storing said extracted excitation signal in compressed form.
6. The method according to claim 2 wherein the step of extracting the excitation signal comprises the steps of:
performing spectral analysis on the target vocal signal to determine the time-varying spectral envelop thereof;
using said spectral envelope to produce a time-varying filter; and
using said time-varying filter to flatten said spectral envelopes.
7. The method according to claim 6 further comprising the steps of identifying voiced and unvoiced signal segments in the excitation signal component and replacing unvoiced signal segments with interpolated data from the voiced signal segments.
8. The method according to claim 7 wherein unvoiced segments in the signal are identified by comparing the parameters of the segments to thresholds selected from among the group of parameters comprising: average segment power, average low-band segment power, zero crossings per segment.
9. The method according to claim 7 wherein said step of replacing with interpolated data comprises using sinusoidal synthesis to morph between the edges of the voiced signals adjacent said silence portions.
10. The method according to claim 1 further comprising the steps of storing said excitation signal; and
performing spectral analysis on a vocal signal representative of the voice of the source individual so as to determine the spectral envelope of said vocal signal.
12. The method according to claim 10 further comprising the steps of:
obtaining a digital transfer function corresponding to the spectral envelope of said vocal signal;
decomposing said digital transfer function into a plurality of lower order sections; and,
modifying the spectral characteristics of at least one of said lower-order sections.
13. The method according to claim 10 further comprising the step of transforming the spectral envelope by applying conformal mapping to the difference equation of the time-varying synthesis filter.
14. The method according to claim 13 wherein said vocal signal represents singing.
16. The method according to claim 10 wherein at least one of the source individual and the target individual is a singer and further comprising the step of applying conformal mapping to the difference equation of the time-varying synthesis filter.
17. The method according to claim 1 further comprising the step of determining the pitch of the vocal signal representative of the target individual.
18. The method according to claim 17 further comprising the step of transforming the pitch of the target excitation signal to match the pitch of the source vocal signal.
19. The method according to claim 18 further comprising the step of determining the average pitch of the vocal signal of the source individual over periods of at least 50 milliseconds.
20. The method according to claim 1 further comprising the steps of:
segmenting a signal representative of the voice of said source individual into voiced and non-voiced regions;
if a given region represents voiced input, generating output by applying a spectral envelope derived from said region to said excitation signal component; and,
if said given region represents unvoiced input, generating output based on said region without reference to said excitation signal component.
21. The method according to claim 1 further comprising the steps of:
transforming the spectral envelope of said second signal prior to applying said spectral envelope of said second signal to said excitation signal;
determining the amplitude envelope of the source vocal signal; and,
applying said amplitude envelope to an output signal resulting from applying the spectral envelope of the voice of the source individual to an excitation signal derived from the voice of the target individual.
22. The method according to claim 1 wherein said source individual and said target individual are singers.
23. The method according to claim 1 further comprising the step of transforming the spectral envelope of said second vocal signal prior to applying said spectral envelope of said vocal signal to said excitation signal and wherein said step of transforming comprises modifying the temporal extent of a block of samples of vocal signals representative of the voice of the source individual prior to the step of performing spectral analysis.
25. The method according to claim 24 further comprising the steps of:
decimating the low frequency portion;
analyzing the low frequency portion and generating reflection coefficients ki;
sampling the excitation signal at the same rate as a rate at which the source vocal signal is sampled;
filtering the sampled excitation signal using an interpolated lattice filter;
post-filtering the excitation signal by a lowpass filter to remove the spectral image of the interpolated lattice filter; and,
applying gain compensation.
26. The method according to claim 24 further comprising the steps of:
decimating the low frequency portion;
analyzing the low frequency portion and generating reflection coefficients ki;
sampling the excitation signal at a rate matching the decimated rate of the low frequency portion; and,
applying gain compensation.
27. The method according to claim 1 wherein said step of applying a spectral envelope derived from the voice of a source individual comprises the steps of splitting said vocal signal into plurality of frequency bands, independently transforming the spectral envelopes corresponding to said bands and applying said transformed spectral envelopes to said bands.
28. The method according to claim 27 where the steps of transforming and applying the spectral envelope in any band comprises the following steps:
resampling said signal in said band to create a resampled signal SD(t) with a lower effective sampling rate;
performing a low-order spectral analysis on SD(t) and computing the direct-form filter coefficients aD(i);
modifying the coefficients aD(i) using conformal-mapping to scale the spectrum in proportion to the ratio between the pitch of the target vocal signal and pitch of the source vocal signal; and,
applying the resulting filter to the target excitation signal.
29. The method according to claim 27 where the steps of transforming and applying the spectral envelope in any band comprises the following steps:
resampling said signal in said band to create a resampled signal SD(t) with a lower effective sampling rate;
performing a temporal scaling of the said signal in said band;
performing a low-order spectral analysis on SD(t); and,
applying the resulting filter to the target excitation signal.
30. The method according to claim 1 further comprising the step of extracting and storing the excitation signal component from the voice of the target individual and wherein unvoiced regions of said excitation signal component are replaced with interpolated voiced data.
31. The method according to claim 30 further comprising the step of determining a pitch contour for the excitation signal.
32. The method according to claim 30 further comprising the steps of:
segmenting the excitation signal component into analysis segments; and,
determining whether each of said analysis segments represents voiced or unvoiced signal by comparing parameters of the segments to thresholds selected from among the group of parameters comprising: average segment power, average low-band segment power, zero crossings per segment.
33. The method according to claim 30 wherein said step of replacing unvoiced regions with interpolated voiced data comprises using sinusoidal synthesis to morph between the edges of voiced signal portions adjacent unvoiced regions.
34. The method according to claim 33 further comprising the use of a random pitch component.
35. The method according to claim 33 further comprising the step of storing parameters characterizing said excitation signal component, said parameters being selected from among the group comprising pitch contour and location of unvoiced regions and using said parameters in performing said step of replacing with interpolated voiced data.
36. A method of transforming the voice of a source individual so as to adopt characterstics of a target individual, comprising:
providing a vocal signal representative of the voice of a target individual;
extracting an excitation signal component of said vocal signal;
storing the excitation signal component of said vocal signal; and
applying the excitation signal component of said vocal signal to a signal derived from the voice of the source individual.
37. The method according to claim 36 further comprising the step of storing said extracted excitation signal.
38. A method of transforming the voice of a source individual so as to adopt characteristics of the voices of at least two target individuals comprising:
providing a spectral envelope derived from the voice of the source individual;
providing a combined excitation signal derived from the voices of the at least two target individuals; and
applying the spectral envelope from the source individual to the combined excitation signal from the at least two target individuals.
39. The method according to claim 38 further comprising the steps of:
extracting the excitation signal components from the voices of each of the target individuals;
combining the extracted excitation signal components from the voices of each of the target individuals into a combined excitation signal; and,
performing spectral analysis on a vocal signal representative of the voice of the source individual so as to determine the spectral envelope of said vocal signal.
Patent Citations (51)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US3539701(A) | 1970-11-01 | Milde | Applicant |
| US3600516(A) | 1971-08-01 | King | Examiner |
| US3929051(A) | 1975-12-01 | Moore | Applicant |
| US3986423(A) | 1976-10-01 | Rossum | Applicant |
| US3999456(A) | 1976-12-01 | Tsunoo et al. | Applicant |
| US4004096(A) | 1977-01-01 | Bauer et al. | Examiner |
| US4076960(A) | 1978-02-01 | Buss et al. | Applicant |
| US4081607(A) | 1978-03-01 | Vitols et al. | Applicant |
| US4142066(A) | 1979-02-01 | Ahamed | Applicant |
| US4279185(A) | 1981-07-01 | Alonso | Applicant |
| US4311076(A) | 1982-01-01 | Rucktenwald et al. | Applicant |
| US4387618(A) | 1983-06-01 | Simmons, Jr. | Applicant |
| US4464784(A) | 1984-08-01 | Agnello | Applicant |
| US4508002(A) | 1985-04-01 | Hall et al. | Applicant |
| US4519008(A) | 1985-05-01 | Takenouchi et al. | Applicant |
| US4561102(A) | 1985-12-01 | Prezas | Examiner |
| US4596032(A) | 1986-06-01 | Sakurai | Applicant |
| US4688464(A) | 1987-08-01 | Gibson et al. | Applicant |
| US4771671(A) | 1988-09-01 | Hoff, Jr. | Applicant |
| US4802223(A) | 1989-01-01 | Lin et al. | Applicant |
| US4915001(A) | 1990-04-01 | Dillard | Applicant |
| US4991218(A) | 1991-02-01 | Kramer | Applicant |
| US4991484(A) | 1991-02-01 | Kawashima | Applicant |
| US4995026(A) | 1991-02-01 | Makabe et al. | Applicant |
| US5005204(A) | 1991-04-01 | Deaett | Applicant |
| US5048390(A) | 1991-09-01 | Adachi et al. | Applicant |
| US5054360(A) | 1991-10-01 | Lisle et al. | Applicant |
| US5056150(A) | 1991-10-01 | Yu et al. | Examiner |
| US5092216(A) | 1992-03-01 | Wadhams | Examiner |
| US5131042(A) | 1992-07-01 | Oda | Examiner |
| US5194681(A) | 1993-03-01 | Kudo | Examiner |
| US5231671(A) | 1993-07-01 | Gibson et al. | Examiner |
| US5301259(A) | 1994-04-01 | Gibson et al. | Examiner |
| US5307442(A) | 1994-04-01 | Abe et al. | Examiner |
| US5327521(A) | 1994-07-01 | Savic et al. | Examiner |
| US5369725(A) | 1994-11-01 | Iizuka et al. | Examiner |
| US5428708(A) | 1995-06-01 | Gibson et al. | Examiner |
| US5536902(A) | 1996-07-01 | Serra et al. | Applicant |
| US5567901(A) | 1996-10-01 | Gibson et al. | Examiner |
| US5641926(A) | 1997-06-01 | Gibson et al. | Examiner |
| US5644677(A) | 1997-07-01 | Park et al. | Applicant |
| US5750912(A) | 1998-05-01 | Matsumoto | Applicant |
| US5765127(A) | 1998-06-01 | Nishiguchi et al. | Examiner |
| EP504684(A3) | 1992-09-01 | Applicant | |
| GB2087123(A) | 1982-05-01 | Applicant | |
| GB2094053(A) | 1982-09-01 | Applicant | |
| JP403007995 | 1989-06-01 | Examiner | |
| JP406250695 | 1993-02-01 | Examiner | |
| WO90/03640 | 1990-04-01 | Applicant | |
| WO90/13887 | 1990-11-01 | Applicant | |
| WO93/18505 | 1993-09-01 | Applicant |
Non-Patent Literature (23)
- Robert Bristow-Johnson, “A Detailed Analysis of a Time-Domain Formant-Corrected Pitch-Shifting Algorithm,” Fostex Research and Development, Inc., J. Audio Eng. So., vol. 43, No. 5, May 1995, pp. 340-352.Applicant
- Lawrence R. Rabixer et al., “A Comparative Performance Study of Several Pitch Detection Algorithms,” IEEE Transactions on Acoustics, Speech and Signal Processing, vol. ASSP-24, No. 5, Oct. 1976, pp. 399-418.Applicant
- Warren Tucker et al., “A Pitch Estimation Algorithm for Speech and Music,” IEEE Transactions on Acoustics, Speech and Signal Processing, vol. ASSP-26, No. 6, Dec. 1978, pp. 597-604.Applicant
- Affidavit of Robert Bristow-Johnson dated Feb. 24, 1997.Applicant
- Letter from Robert Bristow-Johnson to Mark Wachsler dated Feb. 10, 1997.Applicant
- Affidavit of Russell Pinkston dated Mar. 3, 1997.Applicant
- Letter from Russell Pinkston to Mr. Mark Wachsler dated Feb. 11, 1997.Applicant
- Affidavit of Keith Lent dated Mar. 10, 1997.Applicant
- Letter from Keith Lent to Mark Wachsler dated Feb. 27, 1997.Applicant
- Letter from Prof. Giovanni De Poli to Mark Wachsler dated Feb. 14, 1997.Applicant
- Keith Lent et al., “Accelerando: A Real-Time, General Purpose Computer Music System,” Computer Music Journal, vol. 13, No. 4, Winter 1989, pp. 54-64.Applicant
- K. Nakata, A. Ichikawa, “Speech synthesis for an unlimited vocabulary,” Proc. Speech Communication Seminar, vol. 2, 261-266, 1974.Applicant
- M. Mezzalama, E. Rusconi, “Intonation in speech synthesis: a preliminary study for the Italian language,” idem, pp. 315-325.Applicant
- W. Endres, E. Grossman, “Manipulation of the time functions of vowels for reducing the number of elements needed for speech synthesis,” idem, pp. 267-275.Applicant
- Lent, K., “An Efficient Method for Pitch Shifting Digitally Sampled Sounds,” Computer Music Journal, 13: 65-71, No. 1 (Winter 1989).Applicant
- R. C. Nieberle et al., “CAMP: Computer-Aided Music Processing,” Computer Music Journal, 15: 33-40, No. 2 (Summer 1991).Applicant
- W.F. McGee et al., “A Real-Time Logarithmic-Frequency Phase Vocoder,” Computer Music Journal, 15: 20-27, No. 1 (Spring 1991).Applicant
- The Vocalist Vocal Harmony Processor, product manual of DigiTech, A Harman International Company, DOD Electronics Corporation (1991).Applicant
- Vocalist II Vocal Harmony Processor, product manuel of DigiTech, A Harman International Company, DOD Electronics Corporation (1992).Applicant
- R. Bristow-Johnson, “A Detailed Analysis of a Time-Domain Formant Correct Pitch Shifting Algorithm,” presented at 95th Convention of the AES in New York, 3718 (A1-AM-5): 1-14; Figures 1-9 (Oct. 7-10, 1993).Applicant
- S. Seneff, “System to Independently Modify Excitation and/or Spectrum of Speech Waveform Without Explicit Pitch Extraction,” IEEE Trns on Acoustics, Speech & Signal Processing, ASSP-30: 566-578, #4, 8/82.Applicant
- Mizuno et al., “Voice Conversion Based on Piecewise Linear Conversion Rules of Formant Frequency and Spectrum Tilt,” Pro. of ICASSP, Speech Processing 1. Adelaide, Apr. 19-22, 1994, vol. 1, pp. I-469-472, IEEE XP000529420.Applicant
- G. De Poli et al., “An Effective Software Tool for Digital Filter Design,” IEEE, Via Gradenigo 6/A, 35131 Padova—Italy, 1986, pp. 237-243.Applicant