Noise Imaging Protection for Multi-Channel Audio Signals

The present invention relates to the coding of audio signals. More specifically, the present invention relates to encoding multichannel audio signals such as stereo signals.

Sometimes when we play a digital recording on a stereo the output sounds noisy. There are numerous factors that may make the sound emanating from our stereos more or less noisy.

Quantization of an audio signal to produce a digital recording introduces noise into the audio signal. The smaller the quantization step size, the less noise introduced. However, the smaller the quantization step size, the greater the number of bits required to store the digital recording.

Known techniques select quantization step sizes in such a manner so as to make the noise introduced by the quantization step size just imperceiveable to the human auditory system. Such a fundamental idea is discussed in U.S. Pat. No. 5,341,457 ("the '457 patent"), assigned to the assignee of the present invention and incorporated by reference herein as if set forth in its entirety. A technique described in the '457 patent determines a noise masking threshold (e.g., quantization step size) that, if used to encode, will introduce quantization noise that is just imperceiveable.

However, when the above technique is used for a multichannel, e.g., stereo signal, there are shortcomings. For example, in a stereo environment, the noise masking thresholds do not account for the effect on a listener of what is known as "noise unmasking." In stereo environment, "noise unmasking" occurs when noise introduced during quantization appears to the listener to emanate from a different direction than the sound the listener desires to hear. When this occurs, the quantization noise is said to be unmasked.

In attempting to solve the problem of noise unmasking, another technique was developed. Essentially, this technique describes adjusting noise masking thresholds of a sum channel signal ("S") and a difference channel signal ("D"). The adjusted noise masking thresholds provide additional protection against noise unmasking and, when selectively used, prove advantageous. However, the technique only adjusts noise masking thresholds for S and D (not for a left channel signal ("L") and a right channel signal ("R"). Thus, for encoders that do not use S and D to encode (i.e., encoders that encode always based upon L and R), this technique does not help. Examples of such encoders are encoders made in accordance with the standards referred to as MPEG Layers I and II. Further, even when used with S and D encoders, the technique may result in overcoding. This is because the technique does not always identify signals for which additional noise unmasking protection is unnecessary. Instead, it merely adds additional noise masking protection, based upon adjusted S and D masking thresholds, if the L masking threshold and the R masking threshold are similar.

We have recognized that the L masking threshold and the R masking threshold may be advantageously adjusted to reduce the effect of noise unmasking. More specifically, a given time block of a stereo audio signal comprises a left channel signal and a right channel signal, each of which are represented in a frequency domain by a first plurality of frequency partitions and a second plurality of frequency partitions, respectively. For a frequency partition from the first plurality of frequency partitions and a corresponding frequency partition from the second plurality of frequency partitions, a technique executed in accordance with the invention first calculates a set of masking thresholds comprising a left channel masking threshold and a right channel masking threshold. Next, based upon the set of masking thresholds, an adjusted left channel masking threshold and an adjusted right channel masking threshold is calculated. These adjusted thresholds are adjusted to reduce effects of noise unmasking. Finally, based upon the adjusted thresholds, a portion of the stereo audio signal may be encoded.

Advantageously, audio signals may be encoded with adjusted L and R that were generated based upon L and R thresholds wherein the adjusted L and R thresholds reduce the effects of noise unmasking.

Also advantageously, adjusted L and R thresholds may be generated based upon S and D thresholds, in addition to L and R thresholds, in order to identify signals that do not need additional noise unmasking protection, and thus, prevent unnecessary overcoding.

Other advantages of the present invention will be apparent from the detailed description and associated figures.

FIG. 1 shows an encoder for perceptually encoding a stereophonic audio signal;

FIG. 2 shows a flowchart of a known method for adjusting S and D thresholds to account for noise unmasking when used within the environment of FIG. 1;

FIG. 3 shows a flowchart of the present invention for adjusting L, R, S, and D thresholds to account for noise unmasking thresholds when used within the environment of FIG. 1;

FIG. 4 is a detailed block diagram of an encoder capable of calculating noise masking thresholds in accordance with the steps shown in the flowchart of FIG. 2; and

FIG. 5 is a block diagram of a portion of an encoder capable of calculating noise masking thresholds in accordance with the steps shown in the flowchart of FIG. 3.

Referring to FIG. 1, a flint microphone 102 and a second microphone 104 detect audio signals and pass the audio signals to an encoder 106. The encoder comprises a means for converting the audio signals to a series of short term frequency spectra 108 ("means for converting 108"), a means for calculating noise thresholds 110 ("means for calculating 110"), and a means for encoding based upon the noise thresholds 112 ("means for encoding 112"), all connected as shown. The means for calculating 110 is coupled to the means for converting 108 and to the means for encoding 112 via a first bus 114 and a second bus 116, respectively.

As is well known in the art, the means for converting 108 may be accomplished with methods such as FFTs and others. See, e.g., Malvar, Henrique, "Signal Processing With Lapped Transforms", Artech House (1992). There is a particular odd frequency FFT whose real part is equivalent to an MDCT. With reference to the text below, we will use the term MDCT to refer to the real part of the particular odd frequency FFT. The common thread amongst all of these techniques, as used in the present environment, is that they all output complex spectra for both the left and right channels. Thus, for a given time block, L and R have a frequency spectrum.

Referring to FIG. 2, a known method that can be used by the means for calculating 110 is described. First, as mentioned above, bus 114 inputs complex spectra representing L and R into the means for calculating 110. This is shown as function block 202. We will now describe what happens to a left complex spectrum and a corresponding right complex spectrum.

Again referring to FIG. 2, as shown in function block 204, for the left complex spectrum and the corresponding right complex spectrum, a sum complex spectrum and a difference complex spectrum are calculated. Next, as shown in function block 206, a left measure of tonality, a right measure of tonality, a sum measure of tonality, and a difference measure of tonality are calculated based upon the left complex spectrum, right complex spectrum, the sum complex spectrum, and the difference complex spectrum, respectively. Examples of how to calculate measures of tonality may be found in the U.S. Pat. No. 5,040,217 ("the '217 patent") which is assigned to the assignee of the present invention and incorporated herein as if set forth in its entirety. Next, as shown in function box 208, based upon the left complex spectrum, right complex spectrum, the sum complex spectrum, and the difference complex spectrum, and their respective measures of tonality, a set of masking thresholds is calculated. The set of masking thresholds comprises a left channel masking threshold, TLC, a right channel masking threshold, TRC, a sum channel masking threshold, TSC, and a difference channel masking threshold TDC. The manner in which TLC, TRC, TSC, and TDC are calculated is described in U.S. Pat. No. 5,285,498 ("the '498 patent") which is incorporated by reference as if set forth in its entirety. However, these masking thresholds, TLC, TRC, TSC, and TDC, are not adjusted to reduce the effects of noise unmasking. Also calculated, as shown in function box 208, is a sum channel protected masking threshold, TSP, and a difference channel protected masking threshold, TDP. These masking thresholds, TSP and TDP, represent the smallest quantization step size that may be required to account for noise unmasking under the conditions that TLC is approximately equal to TRC and encoding will be done using S and D (as opposed to L and R). The manner in which it is determined that TLC is approximately equal to TRC and the subsequent manner in which to calculate TSP and TDP is described in the '498 patent. In the '498 patent, thresholds TLC, TRC, TSC, TDC, TSP, and TDP are referred to as THR.sub.l, THR.sub.r, BTHR.sub.m, BTHR.sub.s, MLD.sub.m, and MLD.sub.s, respectively.

Still referring to FIG. 2, alter calculating the set of masking thresholds (TLC, TRC, TSC, TDC, TSP, and TDP), as shown in box 210, an adjusted sum channel masking threshold, TSA and an adjusted difference channel masking threshold, TDA are calculated as described in the '498 patent. In the '498 patent TSA and TDA are referred to as THR.sub.m and THR.sub.s, respectively.

Referring to FIG. 3, a method in accordance with the present invention that can be used by the means for calculating 110 is described. It should be noted that the solid arrows in FIG. 3 indicate the order of steps. The dashed arrows going from function block to function block indicate that information from one function block is needed as an input to another function block.

Again referring to FIG. 3, a first bus 114 inputs complex spectra representing both L and R into the means for calculating 110. This is shown as function block 302. We will now describe what happens to a left complex spectrum (e.g., a first plurality of frequency partitions) and a corresponding right complex spectrum (e.g., a second plurality of frequency partitions). The first plurality of frequency partitions and the second plurality of frequency partitions are partitioned as shown in Table I. Table I shows 857 points in the left complex

spectrum and how the spectral points are divided into the first plurality of frequency partitions. Preferably, at a sampling rate of 48 kHz, the spectral points are evenly spaced and occur every 24,000/1024 Hz (approximately 24 Hz). More specifically, we will now describe calculating a set of adjusted noise masking thresholds for a frequency partition from the first plurality of frequency partitions and a corresponding frequency partition from the second plurality of frequency partitions.

Yet again referring to FIG. 3, as shown in function block 304, for the left complex spectrum and the corresponding right complex spectrum a sum complex spectrum and a difference complex spectrum are calculated. Next, as shown in function block 306, a left measure of tonality and a right measure of tonality are calculated based upon the left complex spectrum and the right complex spectrum, respectively. An examples of how to calculate measure of tonality may be found in the '217 patent. Next, as shown in function block 308, the measure of tonality for the left complex spectrum and right complex spectrum are compared and the minimum of the two values is chosen. As shown in function box 310, the minimum is used, in conjunction with the sum complex spectrum to calculate TSC and TSP. This is a first portion of a set of masking thresholds. Of course, the TSC and TSP calculated in box 310 will differ from the TSC and TSP calculated with reference to FIG. 2 because the measure of tonality used is different. As also shown in function box 310, the minimum is used, in conjunction with the difference complex spectrum to calculate TDC and TDP. This is a second portion of the set of masking thresholds. Of course, the TDC and TDP calculated in box 310 will differ from the TDC and TDP calculated with reference to FIG. 2 because the measure of tonality used is different.

Still referring to FIG. 3, as shown in box 312, a third portion and a fourth portion of the set of masking thresholds is calculated. TLC and TLP are calculated from the left complex spectrum and the measure of tonality for the left complex spectrum. Likewise, TRC and TRP are calculated from the right complex spectrum and the measure of tonality for the right complex spectrum.

Still referring to FIG. 3, the set of masking thresholds comprises the first portion, the second portion, the third portion, and the fourth portion which are, TSC and TSP, TDC and TDP, TLC and TLP, and TRC and TRP, respectively. As shown in function box 314, from these eight values which comprise the set of masking thresholds, the set of adjusted noise masking thresholds is calculated according to the following equations:

Referring to FIG. 4, a detailed block diagram of an encoder capable of calculating noise masking thresholds in accordance with the steps shown in the flowchart of FIG. 2 is shown. The detailed block diagram of the means for calculating 110 comprises tonality calculators 402, 404, 406, and 408, summer 410, subtractor 412, threshold calculators 414, 416, 418, and 420, adjusters 422 and 424, comparator 426, and multiplexer 428 all connected as shown. Essentially, the means for calculating 110 uses R and L as inputs, via bus 114, and outputs TLC, TRC, TSA, and TDA, via multiplexer 428, onto output bus 116. Note that the comparator 426 compares TRC and TLC to determine if they are "similar." This produces a control signal used to control adjusters 422 and 424. Calculation of the measure of similarity is discussed in the '498 patent.

Referring to FIG. 5, a block diagram of a portion of an encoder capable of calculating noise masking thresholds in accordance with the steps shown in the flowchart of FIG. 3 is shown. The detailed block diagram of the means for calculating 110 comprises tonality calculators 502 and 504, adder 506, subtractor 508, threshold calculators 510, 512, 514, and 516, a selector 518, an adjuster 520, and a multiplexer 522 all connected as shown. Essentially, the means for calculating 110 uses R and L as inputs, via bus 114, and outputs TLA, TRA, TSA, and TDA, via multiplexer 522, onto output bus 116. Note that the selector 518 simply selects the minimum of the two tonality values that are input into it. Also, note that the adjuster 520 implements equations (1) through (5) above.

With reference to FIGS. 3 and 5, the explanations above describe how to calculate TLA, TRA, TSA, and TDA for a frequency partition from the first plurality of frequency partitions and a corresponding frequency partition from the second plurality of frequency partitions. However, additional calculations are necessary to encode, e.g., a time block, e.g., 20 msec. in length, of the audio signal. First, TLA, TRA, TSA, and TDA must be calculated for a set of frequency partitions wherein a first frequency partition in the set of frequency partitions is from the first plurality of frequency partitions and a corresponding second frequency partition in the set of frequency partitions is from the second plurality of frequency partitions. When this is complete, the result will be N sets of adjusted noise masking thresholds (TLA.sub.I, TRA.sub.I, TSA.sub.I, and TDA.sub.I, wherein N is equal to the number of frequency partitions in the first set of frequency partitions (and, consequently, also equal to the number of frequency partitions in the second set of frequency partitions) and wherein "I" is an index to the set of frequency partitions within the left complex spectrum and right complex spectrum. After this is done, the process may be repeated for each time block of the audio signal that is to be encoded.

In order to encode a time block, the left complex spectrum, right complex spectrum, TLA.sub.I, TRA.sub.I, TSA.sub.I, and TDA.sub.I, for each "I," are operated upon as follows:

(a) The real part of the left complex spectrum ("left MDCT") and the real part of the right complex spectrum ("right MDCT") are defined into M coder bands (as opposed to N partitions) in accordance with Table II. Table II shows 1024 points in the left MDCT and how the MDCT points are divided into coder bands. Preferably, at a sampling rate of 48 kHz the MDCT points are evenly spaced and occur every 24,000/1024 Hz (approximately 24 Hz).

(b) A further adjusted set of thresholds (i.e., final thresholds: TLF, TRF, TSF, and TDF) for each of the forty nine coder bands shown in Table II is calculated in accordance with the '498 patent at column 17 line 54 through column 18, line 50. However, it should be noted that in the '498 patent, the final set of thresholds is calculated based upon the equivalent of TLC, TRC, TSA, and TDA which are referred to in the '498 patent as THR.sub.l, THR.sub.r, THR.sub.s, and THR.sub.d, respectively. The '498 patent details how to convert from one threshold, e.g., TLA, to another threshold, TLF, for an individual threshold. Thus, for example, in converting from TLA to TLF, one would use the value for TLA as the value for nb.sub.b in column 17, line 63 of the '498 patent. This would result in TLF being output as nband.sub.w as shown in the equation on column 18, lines 28-29 of the '498 patent.

(c) Each coder band in the left MDCT, the right MDCT, the sum MDCT, and the difference MDCT is encoded with either TLF and TRF as applied to the left MDCT and the right MDCT, respectively or TSF and TDF as applied to the sum MDCT and difference MDCT, respectively, depending upon which method of encoding requires fewer bits. Preferably, after the decision between left/right or sum/difference coding has been made for each coder band, the bit rate adjustment process as described in the '498 patent at column 21, line 54 through column 24, line 46 is implemented.

The present invention has been described in the environment of a stereo encoder. However, those skilled in the art will realize that there are many modifications that may be made to the above described invention. For example, the invention may be used in any multi-channel (e.g., five channel) encoding system. Additionally, the invention may be used to manufacture a digital storage medium (e.g., compact disk, CD-ROM, Digital Audio Tape, etc . . . ). Additionally, the invention may be used in noise measuring instrumentation. Still further, the invention may be used in a transmission environment (e.g., digital audio broadcast) rather than a recording enviromnent. Thus, the invention is defined by the appended claims.