Speech analysis using multiple noise compensation

[0002] This invention relates to a speech analysis system for processing speech which is subject to different forms of distortion. It is particularly (although not exclusively) relevant to recognition of words, languages or speakers in two way telephone conversations.

[0003] The problem to which the invention is addressed may be illustrated in one aspect by automatic speech recognition technology as used in telephone systems. Here the system's performance is often severely degraded by changes in a speech signal due to the position of the telephone handset or by the characteristics of the handset, telephone line and exchange. Attempts may be made to compensate for the problem by using some form of automatic gain control (AGC). Unfortunately this may be difficult to implement. For example, in two way telephone conversations in which the apparatus is connected using a two wire configuration, there are often substantial differences between the intensity levels of the speech signals of the persons speaking to one another. Using more sophisticated technology it is possible to intercept a call at a local exchange and to obtain separate signals from each telephone instrument. While this offers some improvement it does not address the difficult problem of reverse channel echo, which arises from contamination of the speech of one party to the conversation with that of the other.

[0004] The problem is not limited to differences in speech level. Many speech recognition systems attempt to adapt in some manner to the characteristics of the individual speaker or microphone. If speaker characteristics change frequently, compensation becomes very difficult.

[0005] Various methods are known for improving recognition performance by compensating for distortion or speaker characteristics. Current speech recognition systems convert the input signal from a waveform in the time domain into successive vectors in the frequency domain during a process sometimes known as “filterbank analysis”. These vectors are then matched to models of the speech signal. In some systems the vectors undergo a transformation prior to matching to speech models. It is possible to counteract signal distortion and speaker effects by applying some form of compensation to the vectors before transformation and matching. There are a number of known methods for determining the appropriate compensation. One such method is disclosed by Sadaoki Furui, “Cepstral Analysis Technique for Automatic Speaker Verification”, IEEE Trans Acoustics, Speech and Signal processing, 29(2):254-272, April 1981. It involves averaging data obtained by filterbank analysis over an entire conversation to obtain the long term spectral characteristics of a signal and applying a compensation for distortions during a second pass over the data. The compensated data is then passed to a speech recognition device for matching to speech models.

[0006] There are two main problems with this approach when applied to multi-speaker speech signals or single speaker speech signals where the form of distortion changes. First, since a single correction is applied for the entire conversation it is poorly suited to conversations in which the speaker characteristics change frequently. This may happen during telephone conversations or other dialogues. Secondly, it is necessary to process the entire conversation to obtain the appropriate correction before recognition commences, which makes it unsuitable for real time applications.

[0007] A preferable approach is to use a technique sometimes known as spectral shape adaptation (SSA). A recognition system using this technique provides information on the expected spectral characteristics of the signal to be recognised at each time instant, and this is compared to the equivalent actually present in that signal to provide a difference term. The difference term is then averaged over a number of successive signals (time averaging) to provide a correction term.

[0008] A system of this kind has been described by Yunxin Zhao, “Iterative Self-Learning Speaker and Channel Adaptation under Various Initial Conditions”, Proc IEEE ICASSP [11] pages 712-715. Here data is processed on a sentence by sentence basis. An input signal undergoes filterbank analysis to create successive vectors each indicating the variation in signal energy over a number of frequency bands. The vectors are processed by matching to speech model states. The parameters of the model state to which a vector has been matched are used to predict a value for that vector which would be expected according to the model. The difference between the vector and the predicted value is computed and time averaged with difference values obtained for earlier vectors from the sentence to determine the average distortion suffered by each sentence. The SSA parameters determined for one sentence are then used to process the next sentence.

[0009] Zhao's approach is unfortunately not appropriate where there are two or more speakers or forms of distortion because it can result in SSA parameters derived from speech of one speaker or subject to a particular form of distortion being applied in connection with a different speaker or form of distortion.

[0010] It is an object of the invention to provide a speech analysis system arranged to counteract multiple forms of distortion.

[0011] The present invention provides a speech analysis system for processing speech which has undergone distortion, and including compensating means for modifying data vectors obtained from speech to compensate for distortion, matching means for matching modified data vectors to models, and deriving means for deriving distortion compensation from data vectors for use by the compensating means; characterised in that:

[0012] a) the compensating means is arranged to compensate for a plurality of forms of distortion by modifying each data vector with a plurality of compensations to provide a respective set of modified data vectors compensated for respective forms of distortion,

[0013] b) the matching means is arranged to indicate the modified data vector in each set exhibiting the greatest matching probability and the form of distortion for which it has been compensated, and

[0014] c) the deriving means is arranged to derive compensation on the basis of the modified data vector in each set exhibiting greatest matching probability for use by the compensating means in compensating for the form of distortion for which that modified data vector was compensated.

[0015] The invention provides the advantage that compensation differentiates between forms of distortion so that the likelihood of correct speech analysis is improved.

[0016] The invention may be arranged to analyse speech from a plurality of speech sources each associated with a respective form of distortion, and wherein:

[0017] a) the compensating means is arranged to provide modified data vectors in each set compensated for distortion associated with respective speech sources,

[0018] b) the matching means is arranged to implement models divided into classes associated with speech and non-speech, and to indicate the model class associated with the modified data vector in each set exhibiting the greatest matching probability, and

[0019] c) the deriving means is arranged to derive a compensation from modified data vectors associated with speech class models.

[0020] The system of the invention may be arranged to update non-speech models within the matching means. The matching means may be arranged to identify the modified data vector in each set exhibiting the greatest matching probability taking into account earlier matching and speech recognition constraints, in order to assess matching probability over a sequence of data vectors.

[0021] The deriving means may be arranged to derive a compensation by averaging over a contribution from the modified data vector in each set exhibiting the greatest matching probability and the model with which it is matched and preceding contributions of like kind. Averaging may be carried out with by infinite impulse response filtering means.

[0022] The matching means may be arranged to implement hidden Markov model matching based on speech models with states having matching probability distributions and associated estimation values for vectors matching therewith; the estimation values may be mean values of respective probability distributions; the deriving means may be arranged to employ estimation values to derive compensation. Each model may have one or more states.

[0023] The deriving means and the compensating means may be arranged in combination to avoid implementing compensation not associated with a speech source. The matching means may employ models in different classes associated with respective types of acoustic data source, such as speech and noise sources, and may indicate that compensation is not to be derived in response to matching to a noise source. It may be arranged to adapt speech models to increase conformity with data vectors.

[0024] In one embodiment, the system of the invention includes means for generating data vectors having elements representing logarithmically expressed averages over respective frequency intervals, and wherein:

[0025] a) the compensating means is arranged to provide a set of modified data vectors by adding to each data vector a set of compensation vectors associated with respective forms of distortion, and

[0026] b) the deriving means is arranged to derive an updated compensation vector from a first contribution from the modified data vector in each set exhibiting the greatest matching probability and an estimation vector from the model with which it is matched, together with earlier like contributions associated with the same model class.

[0027] The invention may include a respective channel for transfer of each modified data vector to the matching means.

[0028] In another aspect, the invention provides a method for analysing speech which has undergone distortion including the steps of:

[0029] a) modifying speech data vectors to compensate for distortion,

[0030] b) matching modified data vectors to models, and

[0031] c) deriving and applying distortion compensation, characterised in that:

[0032] i) step (a) comprises applying a plurality of compensations to each data vector to provide a respective set of modified data vectors compensated for respective forms of distortion,

[0033] ii) step (b) comprises identifying the modified data vector in each set exhibiting the greatest matching probability and the form of distortion for which it was compensated, and

[0034] iii) step (c) includes deriving a compensation from the modified data vector in each set exhibiting the greatest matching probability for use in compensating for the form of distortion for which that vector was compensated.

[0035] The system of the invention may be employed for speech recognition, or alternatively for other analysis purposes such as language identification, recognition or assessment of a speaker's age, gender or other attributes. It may be used to detect which of a variety of speakers is talking at a given instant.

[0036] In order that the invention might be more fully understood, embodiments thereof will now be described, by way of example only, with reference to the accompany drawings, in which:

[0037] FIG. 1 is a block diagram of a speech analysis system of the invention; and

[0038] FIG. 2 graphically illustrates speech model states employed in the system of FIG. 1;

[0039] Referring to FIG. 1, a speech analysis system of the invention is indicated generally by 10. The system 10 incorporates two telephones 12A and 12B multiplexed together by connection to a line 14. The line 14 is connected to an analogue to digital (A/D) converter 16, which is itself connected to a filterbank analyser 18. The analyser 18 implements a four hundred point discrete Fourier transform (DFT) using a 20 mS Hamming window to deal with the usual problem of non-infinite signal samples. It combines the Fourier components so obtained into twenty-six subsets over which average values are obtained for each of twenty-six frequency intervals or bins on an approximately mel (ie perceptually linear) scale, and produces logarithmic amplitude values. It therefore simulates a bank of bandpass filters with logarithmic outputs in twenty-six channels. Output from the analyser 18 passes to a buffer memory 20 and to a parallel arrangement of first and second compensation units 22A and 22B each providing spectral shape adaptation (SSA). The SSA compensation units 22A and 22B are digital adders. They are connected to a model matching computer 24 providing a system output at 26.

[0040] The matching computer 24 is also connected to a difference calculator 28; this calculator receives a second input from the buffer memory 20, which itself receives timing information from the matching computer 24 via a line 30. Output from the difference calculator 28 passes via a line 32 as a first input to a first demultiplexer 34, which receives a second input of a model class indication from the matching computer 24 via a line 36. The demultiplexer 34 has two outputs 38 and 40, of which the first output 38 is connected to a second demultiplexer 42 and the second output 40 is unconnected. The second demultiplexer 42 receives a second input of matched model identity from the computer 24 via a line 44. It has first and second outputs 46A and 46B. The first output 46A is connected to a first filter bank 48A in series with a first SSA parameter store 50A and the first compensation unit 22A. Similarly, the second output 46B is connected to a second filter bank 48B in series with a second SSA parameter store 50B and the second compensation unit 22B. Each of the filter banks 48A and 48B is an array of twenty-six infinite impulse response (IIR) filters.

[0041] The recognition system 10 operates as follows. When two speakers are conversing by means of the two telephones 12A and 12B, an analogue speech signal is present on the line 14. This signal is digitised by the A/D converter 16, which samples the signal and provides digital output signals at the rate of 20 KHz each accompanied by a time frame number i indicating the time t_iat which it was sampled. At any instant, the signal may have originated from one or the other or both of the telephones 12A and 12B, or it may consist of noise in an interval between speech signals. Digitised signals pass from the converter 16 to the filterbank analyser 18, where they are used in calculating successive four hundred point discrete Fourier transforms (DFT). Each DFT is a set of four hundred points, and each point represents a spectral amplitude at a respective frequency of the four hundred frequencies for which the DFT is calculated. Each set of points is “binned” or separated into twenty-six sub-groups, added to provide a sum for each sub-group, and the logarithm of each sum is taken to provide average energies on a logarithmic scale for twenty-six frequency intervals. The analyser 18 is the digital equivalent of a bank of twenty-six analogue bandpass filters. Each output transform from the analyser 18 is a vector having twenty-six components, and each component represents the logarithm of the energy in a respective frequency interval obtained by averaging over associated Fourier transform coefficients.

[0042] Individual data vectors are output from the analyser 18 in succession every 10 mS. The i^thdata vector output from the analyser 18 at time t_iis defined as O_i. As has been said, it may have originated from one or the other of the telephones 12A or 12B, or from a combination of both, or from noise. The data vectors are passed to the buffer memory 20, which stores the five hundred most recently obtained vectors each at a respective address number corresponding to its time of production t_i. The address numbers are the time frame numbers i expressed in modulo 500, ie address 500+i=address i, so that the oldest entry in the buffer memory 20 becomes overwritten by the latest entry when the time frame number exceeds 500.

[0043] Each data vector O_iis also passed to both compensation units 22A and 22B, which apply a transformation using respective sets of twenty-six parameters p_Ajand p_Bj(j=0 to 25) stored in logarithmic form in respective parameter stores 50A and 50B. The production of these parameters will be explained later. The compensation units 22A and 22B add the contents of respective stores 50A and 50B to the current vector data O_ito produce respective compensated data vectors O_i^Aand O_i^B. This addition is of logarithmic quantities, and corresponds to formation of a scalar product of two vectors. Compensation units 22A and 22B and their respective parameter stores 50A and 50B define two separate channels for passing modified data vectors associated with different data sources (telephones 12A and 12B) to the matching computer 24. It is also possible to use a single channel in which modified data vectors associated with different data sources are multiplexed for later separation in the computer 24. Each set of parameters p_Ajor p_Bjin a store 50A or 50B corresponds to the current estimate of the compensation appropriate to counteract the distortion associated with the speech signal from a respective telephone 12A or 12B. The two stores 50A and 50B normally contain different parameter sets, and the compensated vectors O_i^Aand O_i^Btherefore generally differ despite being derived from the same filterbank output.

[0044] After compensation, both compensated data vectors O_i^Aand O_i^Bare passed to the model matching computer 24 together with their time frame number i. The computer 24 implements a procedure to be described with reference to FIG. 2, which (to permit illustration) shows a much simpler model space of two dimensions (as opposed to twenty-six in the present example). The model matching computer 24 implements a conventional hidden Markov model matching algorithm of the kind well known in the prior art. See for example the standard text on speech recognition, “Speech Synthesis and Recognition”, J N Holmes, Van Nostrand Reinhold (UK) 1988, Chapters 7 and 8 in particular. The computer 24 employs models which are conventional hidden Markov models (HMM) of telephone speech. Each model corresponds to a word or part of a word (sub-word); each model has a number of states and corresponds to a sequence of sounds. For example, in a digit recogniser for the numbers 0 to 9, ten states per model may be used. A state is represented by a single component gaussian probability distribution having the same dimensionality as that of the model space; ie the distribution has the form:

[0045] C_sexp(−((x₀−μ_s,0)/2σ_s,0)². . . −((x_k−μ_s,k)/2_s,k)². . . ((x₂₅−μ_s,25)/2σ_s,25)²)  (2)

[0046] where C_Sis a constant for the model state, μ_s,kand σ_S,k(k=0 . . . 25) are the twenty-six components of the mean and standard deviation of the probability distribution for model state S in a twenty-six dimensional model space, and x_kare variables defining that space. Each model state is also associated with a class which may be either speech or non-speech according to whether the model is a model for a human speech sound or for noise originating from some other source (eg doors slamming, noises introduced by telephone apparatus etc).

[0047] FIG. 2 is a two-dimensional illustration in which features equivalent to earlier twenty-six-dimensional elements are indicated by “(2)” suffixes. It shows three two-dimensional model states, S1(2), S2(2) and S3(2), each of which is a probability distribution represented by contour lines such as 60; the model states have most probable or estimation values μ_S1(2), μ_S2(2)and μ_S3(2)respectively, each of which is the peak value of the associated distribution defined by its set of contours. Also shown are O_i^A(2)and O_i^B(2), two dimensional equivalents of the twenty-six dimensional compensated data vectors O_i^Aand O_i^B. The probability of a match between a compensated data vector and a model state is indicated by the magnitude of the probability distribution of the model state at the position of the vector subject to any constraints imposed by earlier matching history. This probability is a maximum when a compensated data vector coincides with the peak value of a model state, unless constraints dictate otherwise. In the simplified two dimensional situation shown in FIG. 2, probability considerations may indicate that a data vector O_i^A(2)should be matched to model state S1(2) and vector O_i^B(2)should be matched to model state S3(2), if this is not contra-indicated by matching of earlier vectors O_i−m^A(2), where m=1, 2, . . . .

[0048] A comparison is then made between the quality of these two matches; ie, indicating twenty-six dimensional quantities by absence of “(2)” suffixes once more, is the match between O_i^Aand S1 better or worse than that between O_i^Band S3? In other words, is the magnitude of the probability distribution of the model state S1 at the position of O_i^Agreater than that of the model state S3 at the position of O_i^Bsubject to any constraints imposed by earlier matching history. Only the better of the two matches is accepted, which will be taken to be the latter (ie S3) in the following discussion. (In a prior art HMM based recognition system only one compensated data vector would be processed for any individual data vector O_iproduced by the analyser 18, and so no comparison of the quality of the best matches would be made).

[0049] Once the computer 24 has identified which of vectors O_i^Aor O_i^Bprovides a best match to a model state, it computes an estimation vector Ô_i, this being the highest probability value of that model state. For single component gaussian models the vector Ô_iis equal to the mean value of the matched model state.

[0050] The computer 24 outputs the following information:

[0051] a) the identity of the best matched model state (S3 in this example), which is output at 26;

[0052] b) the 26 dimensional vector Ô_i(μ_S3in this example for the model state S3), which is the estimation value according to the matched model; ie it corresponds to that data vector having the highest probability of being a match to the matched model state; it is passed to the difference calculator 26;

[0053] c) the address of the i^thdata vector O_istored in the buffer memory 20 (ie the time frame number i expressed in modulo 500);

[0054] d) the class of the best matched model state, i.e. non-speech or speech; a digital 0 or 1 is output to indicate respectively that the computer 24 has matched to non-speech or speech, and this is received by the first demultiplexer 34; and

[0055] e) the identity of the best matched vector, ie an indication of data vector distortion; when the computer 24 matches to speech, it outputs a digital 0 or 1 to indicate respectively that the better match is to compensated vector O_i^Aor to compensated vector O_i^B, which in turn indicates the system has deduced that the vector distortion is associated with telephone A or telephone B respectively; this indication is received by the second demultiplexer 42.

[0056] The vector Ô_iis passed to the difference calculator 28. At the same time, the buffer memory 20 receives from the computer 24 the memory address of the data vector O_ifrom which the compensated vectors O_i^Aand O_i^Bwere derived. In response to this address it outputs O_ito the difference calculator 28 for comparison with Ô_i. The difference calculator 28 computes the difference between these model and data vectors to generate a difference vector D_i, which represents an instantaneous estimate of the distortion process which the speech signal has undergone at time t_i.

[0057] The difference vector D_iis passed to the first demultiplexer 34, which at the same time receives an indication of model class in the form of a digital 1 or 0; of these digits, 1 indicates that the model matching computer 24 has recognised a speech signal and 0 that it has recognised a non-speech signal, ie silence or noise. If the model class digit is 1, the first demultiplexer 28 routes the difference vector D_ifrom the calculator 28 to the second demultiplexer 42. If it is 0, the difference vector D_iappears at the second demultiplexer output 40, which is unconnected. It provides for information corresponding to non-speech signals (which is unwanted in this example) to be removed from the distortion correction process. Such information could of course be used for other purposes if required, as will be described later.

[0058] The second demultiplexer 42 receives an input digit 0 or 1 indicating the identity of the source of the signal which is associated with the model vector; the digits 0 and 1 indicate matching to compensated data vectors O_i^Aand O_i^Brespectively. On receipt of digit 0, the second demultiplexer 42 routes the difference vector D_ito the first filter bank 48A; in response to digit 1, the difference vector D_iis routed to the second filter bank 48B.

[0059] Each of the filter banks 48A and 48B is an array of twenty-six infinite impulse response filters. The difference vector D_iis derived in response to a single data vector O_i, and it represents an instantaneous estimate of the correctable distortion at time t_idue to microphone and line characteristics combined with differences due to the random nature of speech-like sounds. The first demultiplexer 34 provides for difference vectors computed from non-speech sounds to be removed from the correction process before they reach the filter banks 48A and 48B, and the second demultiplexer 42 routes each difference vector to a filter bank 48A and 48B associated with the telephone 12A or 12B to which the relevant identified model vector corresponds.

[0060] The filter banks 48A and 48B have like time constants of 0.5 seconds. They provide short term averaging over several words (eg about three words or 1.5 seconds). This counteracts the effects of random variations and errors of word recognition or simultaneous speech into both telephones 12A and 12B while tracking longer term variations due to correctable distortions. In response to receipt of each input difference vector D_i, an output compensation vector C_iis produced which is an average over D_iand contributions from earlier difference vectors diminishing exponentially with time; the contributions to the compensation vector C_iarise predominantly from the most recent fifty difference vectors D₋₄₉to D_i.

[0061] Compensation vectors output from the first filter bank 48A pass to the first parameter store 50A and those from the second filter bank 48B are passed to the second parameter store 50B, replacing the existing contents in each case to provide updating. The elements of each compensation vector provide the set of parameters which are added to data vectors in one or other of the SSA compensation units 22A and 22B to provide spectral shape adaptation. Each of the parameters stores 50A and 50B maintains its contents until they are updated in response to identification of a speech signal from the respective telephone 12A or 12B.

[0062] There is a short delay in producing the match. In one embodiment of the invention in which elements 18 to 50B were implemented collectively by a computer, the time interval required to obtain updated parameters from the speech signal was 0.5 seconds, or about the duration of an average word. In consequence, it compensates for distortions whose effects vary from word to word. Typical prior art systems are much slower to provide compensation than this, and cannot cope with relatively short term distortion. The Zhao reference previously mentioned requires a whole sentence for analysis. In the present example, the transformation employed provides compensation for changes to the mean signal level and linear distortions introduced by signal channel or microphone characteristics.

[0063] The invention may be adapted to accommodate applications such as conference calls, where there are more than two telephones or speakers. This would involve the system 10 being extended by additional units equivalent to 22A, 48A and 50A and increasing demultiplexer capacity. A three way conference call would require three compensation units, 22A, 22B and 22C, three parameter stores 50A, 50B and 50C and three filters 48A, 48B and 48C. Each of the compensation units 22A to 22C would pass its output to the model matching computer 24, which would be reprogrammed to decide which of three compensated vectors O_i^A, O_i^Band O_i^Cbest matches one of the model states. Output 44 from the computer 24 would be one of three levels, 0, 1 or 2 and demultiplexing apparatus would be used to select one of three outputs 46A, 46B or 46C accordingly.

[0064] The invention may incorporate provision for adaptation for non-speech noise originating (or picked up by) telephone 12A or 12B. In this case the matching computer 24 includes two model states for tracking background noise. These model states are associated with noise from telephones 12A and 12B respectively. They have parameters which are adapted according to estimated current noise as follows. Non-speech (ie noise) difference vectors output from the demultiplexer 34 are used by the computer 24 to estimate background noise levels for telephones 12A and 12B. The computer 24 differentiates between noise originating in telephone 12A and telephone 12B. From the non-speech difference vectors, the computer 24 calculates changes to the parameters of its non-speech models for respective telephones 12A and 12B. The current estimate of the noise for each telephone produced in this way is averaged with 50 previous estimates for that telephone to provide insensitivity to matching errors arising from the algorithms implemented by the computer 24.