Method for Performing Time-Scale Modification of Speech Information or Speech Signals

This invention relates to digital signal processing and more particularly to time domain digital speech processing in order to vary the rate of reproduction of speech without changing pitch.

In recent years various techniques have been developed for achieving time compression/expansion of audio information, particularly speech information. In order to utilize time compression or expansion effectively, where the compression or expansion factor is significant, some mechanism is necessary to correct for changes in pitch which would normally follow a direct application of acceleration or deceleration techniques. Acceleration or deceleration of recorded speech is easily achieved by speeding or slowing the rate of reproduction, which in turn raises or lowers pitch, as is expected.

Time compression and expansion of speech is useful in many applications. Time compression allows matching of speech information to a desired playback time. Time expansion is particularly useful for example, in dictation equipment to speed up playback or in foreign language learning situations to slow down playback to improve comprehension, which may be difficult or otherwise impaired.

Numerous techniques have been developed to achieve time compression and/or expansion, particularly techniques which manipulate analog signal representations. Of the various prior art techniques, the following patents or publications are representative:

Roucos and Wilgus, "High Quality Time-Scale Modification for Speech," ICASSP 85. Proceedings of the IEEE International Conference of Acoustics, Speech, and Signal Processing, pp. 493-6, Volume 2, 1985 (26-29 March 1985), IEEE. This relatively recent paper represents a development in the algorithms for reproducing speech using digital techniques. The research group is Bolt, Beranek & Newman Inc. of Cambridge, Mass.

Makhoul, J. and El-Jaroudi, "Time-Scale Modification in Medium to Low Rate Speech Coding," ICASSP 86. Proceedings of the IEEE International Conference of Acoustics, Speech, and Signal Processing pp. 1705-1708, Volume 3, 1986, (Apr. 7-11, 1986), IEEE. This paper produced by the same research group related to the foregoing describes further development in digital signal processing techniques for rate modifying speech.

These two papers relate to description and implementation of the synchronous-overlap-and-add method of time-scale modification. The algorithm described therein allows arbitrary linear or nonlinear scaling of the time axis using a modified overlap-and-add procedure operating on the time domain waveform. The Makhoul paper describes the implementation of a technique involving generalized cross-correlaton between a normalized source signal (y(n)) and a normalized derived signal (x(n)). The technique was originally described in the Roucos paper.

Asada et al., U.S. Pat. No. 4,435,832 issued Mar. 6, 1984, to Hitachi, describes a speech synthesizer wherein LPC (linear predictive coding) techniques are employed to synthesize speech. Control is exercised over the rate of speech by lengthening or shortening the time interval of interpolation between the fetching of each of the LPC parameters to synthesize the speech. This technology is essentially unrelated to the present invention, since the present invention is unrelated to synthesized speech or parametrically-defined speech.

Klasco et al., U.S. Pat. No. 4,406,001 issued Sept. 20, 1983, to The Variable Speech Control Company of San Francisco, describes a time compression/expansion audio reproduction system of the type which relies on analog circuitry. It provides speech correction by repetitive variable time delay achieved by separating the reproduced signal from a recording into components which are separately delayed. The signal is separated into contiguous frequency bands, each of which is delayed synchronously. The signal is then recombined after delay, and low-pass filtering techniques are employed to remove high-frequency components introduced into the speech components by the signal processing technique. This technology is readily distinguishable from the present invention for at least two reasons. First, this technology relies on analog methods, whereas the present invention is digital in nature. Second, the present invention does not require filtering of speech components. Other distinctions will also be apparent to those of ordinary skill in this art.

Brantingham et al., U.S. Pat. No. 4,209,844, issued June 24, 1980, to Texas Instruments, describes a digital filter technique using a form of linear predictive coding (LPC). Specifically, the patent describes an invention embodied in a device implementing a lattice-type filter for generating complex waveforms suitable for implementation in semiconductor device technology. The invention appears to be unsuited to time-domain speech processing and further is not applicable to time scale modification in the time domain.

Kohut et al., U.S. Pat. No. 4,022,974, issued May 10, 1987, to Bell Telephone Laboratories, describes a predictive speech synthesizer having the capability of varying speech without changing pitch. The Bell technique is substantially unrelated to the present invention, since it relates primarily to parametric speech and does not deal with a actual time domain speech signal.

What is needed is a simple yet effective digital technique for providing time scale modification of real time or near real time speech signals.

According to the invention, method and apparatus are provided to process time domain speech signal containing speech information, the rate of reproduction of which is to be varied without changing pitch. The basic process comprises superimposing partially overlapping blocks of speech samples in a manner such that the pitch periodicity is maintained. The extent of superimposition is a function of the desired increase or decrease , or variance, in the time scale of the speech. In accordance with a preferred embodiment of the invention, maintenance of speech periodicity is achieved by fixing the precise superimposition in the time domain such that the superimposed waveforms achieve a best match using a technique which does not require multiplication or division.

Relatively smooth transition between superimposed speech signals are realized by applying a graduated weighting thereto.

In accordance with a preferred embodiment of the invention, if the extent of superimposition exceeds the amount of overlap, an accelerated speech output is provided, and if the extent of superimposition is less than the amount of overlap, a decelerated speech output is provided.

To minimize required computational load, the search range, that is, the range over which superimposition is varied in order to achieve a best match between speech segments, is selected as a function of pitch, thus ensuring that a sufficient number of samples are taken to assure that pitch pulses are contained in a sample set without requiring superfluous computations.

A specific embodiment of the invention allows for speech expansion of up to 150% and speech compression to as little as 40% of the duration of the source.

The method according to the invention may be incorporated into an embodiment using programmable digital signal processing hardware, such as a Texas Instruments TMS 320 Series device. Therefore it is not necessary to describe such devices in detail, since the combination of such components with programs in general are known to those of skill in the art. The application of such devices in accordance with the invention is nevertheless not apparent from the devices.

The method in accordance with the invention is substantially simpler, faster and more efficient than other methods which might be considered for purposes similar to the intended application. As one consequence, the method in accordance with the invention is more easily adapted to implementation in Very Large Scale Integration (VLSI) technology.

The method in accordance with the invention makes use of a waveform-segments-matching technique which takes advantage of the periodic nature of the signals produced by speech, and more specifically the existence of pitch pulses within a speech signal. Hence, in accordance with the invention, use is made of the maximum value of the pitch period of the input speech to reduce complexity, a technique not used heretofore.

The invention will be better understood by reference to the following detailed description in connection with the accompanying drawings.

FIG. 1 is a block diagram of a device which operates in accordance with the invention.

FIG. 2 is a flow chart of a method in accordance with the invention.

FIGS. 3A through 3D are illustrations showing operation of the method and apparatus according to the invention.

Referring to FIG. 1, a block diagram is shown of a signal processing apparatus 10 illustrating a typical environment of apparatus in accordance with the invention. Many variations will be apparent to those of ordinary skill in this art, including such variations as to the type of input devices and output components.

In the illustrative embodiment, the signal processing apparatus 10 includes a time-domain speech sampling means 12, the input port 11 of which receives live real-time or substantially real-time analog speech signals, and the output port 13 of which is coupled to digital storage means 14, such as a computer memory or set of digital storage registers. The digital storage means 14 has a digital signal output which is coupled to a digital signal processing means 16, such as a microcomputer constructed around a programmable microprocessor or special purpose digital signal processing device.

A suitable microprocessor is a Motorola 68000 series microprocessor or a Texas Instruments TMS 32020 DSP Chip preprogrammed to receive digital input data temporarily stored in the digital storage means 16, to process the digital input data in accordance with the method of the invention and to provide as a digital output signal digital output data to an output means such as a digital-to-analog converter means 18.

The digital-to-analog converter means 18 reconstructs an analog signal for audio reproduction and therefore has an output terminal which is coupled to an audio amplifier means 20 or the like, such as an analog recorder. In addition, output of the digital signal processor 16 is provided to interim storage means 22 which provides a second input to the digital signal processing means 16 for use in comparing the resultant digital output with subsequently received speech segments (frames or portions of frames) as explained hereinbelow.

Referring to FIG. 2, there is shown a flow chart for the relevant portion of a computer program for processing digitized input speech information in accordance with the invention. FIGS. 3A-3D, which are to be viewed as one diagram in connection with FIG. 2, illustrate the time relationship among block of speech samples. These blocks may represent the content of registers or temporary storage locations, each element of which contains data representing the amplitude of a given speech sample.

Phase information is for the most part ignored or otherwise only indirectly accounted for by the method according to the invention. It is known that the human ear is substantially immune to inaccuracies in phase information in speech.

In accordance with the invention, incoming speech is sampled at a selected sampling rate, and the samples are combined into blocks, herein termed "input blocks," the samples in each input block representing the amplitude of the speech i.sctn. signal for such sample. Each input block overlaps the preceding input block by a predetermined number of samples. The number of samples by which each successive input block exceeds or extends beyond the preceding input block is termed the overlap value or OV and is a function of the sampling rate and of the number of samples contained in an input block.

Normally, the sample values are normalized to a range suitable for subsequent processing. (Automatic gain control may be employed independently of the normalized values.) In a specific embodiment, a maximum pitch period of no more than 17 ms is assumed, and each input block contains a uniform number of samples, selected to be between 80 and 120, representing a nominal 10-15 ms segment of speech information. A 10 ms segment is considered time invariant for the purpose of speech, which has a nominal spectrum of information of 200 Hz to 4000 Hz.

The method of the invention normally begins with initializing of variables and memory locations, which are set in accordance with preselected initializing values (Step A). The values to be initialized include user-selectable parameters, such as the number of samples which will be contained in each input block, the value of overlap value OV and the speed control value SCV, which indicates the amount by which it is desired to speed up or slow down speech (Step B).

The speed control value SCV is typically expressed as a number of samples. If the SCV is selected to exceed the overlap value OV, the output signal will be slowed relative to the input signal. If the SCV is selected to be less than the OV value, the output signal will be speeded up relative to the input signal.

FIG. 3A illustrates three successive input blocks on a continuing time scale, illustrating the overlapping thereof. In accordance with the present invention, an output block is defined and typically comprises an input block of speech samples which is stored in storage means 22. A superimposition reference pointer P is placed at a location along the output block in accordance with the SCV value (Step C).

FIG. 3B illustrates the pointer P at a location on an output block which produces speeding up of the output speech. Were the pointer P at the OV line, the output speech would be provided at exactly the same speed as the input speech.

A search range of a selected number of samples SR to either side of the pointer is selected as a function of the pitch frequency of the speech (Step D). The search range is requited to be approximately equal to the maximum pitch frequency. The selection of a search range is a particular feature of the present invention, as it enables preservation of pitch without requiring superfluous computations which require excess computing capability and computation time.

An input block, such as input block I, is defined (Step E). The first N samples of the input block (FIG. 3A) then undergo best fit matching to the portion of the output block within the above-defined search range, preferably by means of an Average Magnitude Difference Function (AMDF) adapted to the present invention, in order that the pitch pulses of the input block and the output block match as nearly as possible. Once the desired match has been found the input and output blocks are superimposed (FIG. 3C) at the location providing the best match, thereby preserving the pitch without creating undesired discontinuity between output blocks (Step F). In accordance with a preferred embodiment of the invention, the AMDF calculates the absolute value of the difference between the input block and the output block for each of a plurality of different possible superimpositions within the predetermined search range, thus identifying the superimposition having the lowest difference so that it may be selected for use in the subsequent processes. Use of the AMDF is a particular feature of the invention which represents a significant advance over the art and a departure from the prior art which employs cross-correlation functions. Such prior art functions involve multiplications which require substantial computation capabilities and computation time. Use of the AMDF increases capabilities without sacrificing computation power, which for example gives the method according to the invention an inherent bandwidth advantage over the prior art. A description of an Average Magnitude Difference Function suitable for implementation in the present invention is found in Digital Processing of Speech Signals, by L. R. Rabiner and R. W. Schafer, pp. 149-150 (Prentice-Hall, 1978), the content of which is incorporated herein by reference.

The superimposed portions of the output block and the input block are combined by a desired weighting arrangement or factor W (FIG. 3C) so as to provide a smooth transition from the sample values of the output block to those of the input block (Steps G and H). A substantially linear ramp is a suitable weighting factor, as illustrated in FIG. 3C.

The weighted combination of the input block with the overlapping portion of the output block becomes a new or next output block, herein indicated as output block II and shown in FIG. 3D. Output block II is stored in storage means 22.

According to the invention, that portion of the output block I which did not overlap the input block is output for the DAC 18 (FIG. 1) (Step I).

It is to be appreciated that the difference between the location of the pointer and the location at which superimposition begins is a potential source of distortions if combined over several output blocks. Accordingly, signal processor 16 operates to store the information on this difference (Step J) and to position the pointer on the subsequent output block so as to compensate for this difference.

Reference is made to the Appendix for a detailed technical description illustrating a specific embodiment of the invention.

The invention has now been explained with reference to specific embodiments. Other embodiments will be apparent to those of ordinary skill in the relevant art. It is therefore not intended that the invention be limited, except as indicated by the appended claims. ##SPC1##