Subband Coding Method With Wavelet Transform for High Efficiency Video Signal Compression

1. Field of the Invention

The present invention relates to an image data compressing technique and, more particularly, to a subband coding method.

2. Description of Related Art

In image data compressing techniques such as MPEG, an image is divided into blocks each of which is subjected to orthogonal transform like DCT (Discrete Cosine Transform) to concentrate the power of the image to specific frequency components and is then quantized and coded, thus accomplishing image data compression. Since this image data compressing technique involves the blocking of an image, it is likely to suffer the deterioration of the quality of an image such as blocking effects which make the boundary between adjoining blocks in a restored image discontinuous.

Recently, a subband coding method using wavelet transform as a solution to this shortcoming has been considered. This method divides a video signal into a plurality of frequency component signals based on filtering, not block transform, for the concentration of the power of the signal to specific frequency components, thereby acquiring a plurality of subband signals, and then quantizes and encodes the subband signals to accomplish image data compression. An image data compressing apparatus which employs such subband coding is disclosed in, for example, Japanese Unexamined Patent Publication No. 5-276499. The frequency regions of the individual frequency component signals obtained by dividing a video signal are called "subbands".

It was not however known that what kind of coding should be performed on a plurality of subband signals to ensure high compression efficiency and provide images with less quality deterioration in the subband coding using wavelet transform.

Accordingly, it is an objective of the present invention to provide a subband coding method with wavelet transform capable of compressing a video signal with a high compression efficiency and with less quality deterioration of a restored image.

According to one aspect of this invention, there is provided a subband coding method for dividing a digital video signal into a plurality of frequency bands in vertical and horizontal directions in a spatial frequency region by executing two-dimensional multilayer (n-layer) wavelet transform on each processing unit (field) of the digital video signal, then extracting data from the plurality of frequency bands in proportion to the number of data in each frequency band and rearranging the data to a plurality of data blocks each consisting of 2.sup.n .times.2.sup.n pieces of data, and scanning data in the data blocks in a predetermined order to generate a sequence of data, and accomplishing run-length coding on the sequence of data. Scanning of data in the data block is performed in an order from data extracted from a layer consisting of the lowest frequency bands to data extracted from a layer consisting of the highest frequency bands. For data extracted from the same layer, scanning is done in an order from data extracted from an LH band consisting of high frequency components in a vertical direction and low frequency components in a horizontal direction, data extracted from an HL band consisting of low frequency components in the vertical direction and high frequency components in the horizontal direction, and data extracted from an HH band consisting of high frequency components in the vertical and horizontal directions. For data extracted from the same band, scanning is done in the horizontal direction for data extracted from the LH band and in the vertical direction for data extracted from the HL band.

FIG. 1 is a block diagram showing an image data compressing apparatus using wavelet transform to which the present invention is adapted;

FIG. 2 is a block diagram showing the structure of a band divider in the apparatus in FIG. 1;

FIG. 3 is a diagram showing divided subbands in a spatial frequency region;

FIG. 4 is a block diagram showing the structures of a quantizing section and a coding section in the apparatus in FIG. 1;

FIG. 5 is a diagram showing the ratio of quantization step size for a luminance signal;

FIG. 6 is a diagram showing the ratio of quantization step size for a color difference signal;

FIG. 7 is a diagram showing the spectrum of the vertical frequency and time domain of a luminance signal;

FIG. 8 is a diagram exemplifying how the quantization step sizes of the individual subbands of a luminance signal are set;

FIG. 9 is a diagram of another example showing how the quantization step sizes of the individual subbands of a luminance signal are set;

FIG. 10 is a diagram exemplifying how the quantization step sizes of the individual subbands of a color difference signal are set;

FIGS. 11A through 11E are diagrams for explaining wavelet transform for one frame;

FIGS. 12A and 12B are diagrams for explaining blocking;

FIG. 13 is a diagram showing 64 pieces of data in a block; and

FIG. 14 is a diagram showing the scanning and output order in a block.

An embodiment of the present invention will now be described with reference to the accompanying drawings.

FIG. 1 shows an image data compressing apparatus embodying this invention. In this image data compressing apparatus, an input digital video signal is supplied to a band divider 1. The band divider 1, which will be discussed in detail later, performs 3-layer two-dimensional wavelet transform on the digital video signal in the spatial domain to divide the digital video signal into ten subbands.

The video signals in the ten subbands are supplied to a quantizing section 2. The quantizing section 2 performs linear quantization to further reduce the transmission rate of the signal whose redundancy has been eliminated. In high-efficiency coding, quality of a compressed image is mostly determined by the quantization. The quantized signals are supplied to a coding section 3 which in turn encodes each block of signals (which will be discussed later) to further reduce the transmission rate. Since run-length coding is used in this coding process, the consecutiveness of 0's determines the coding gain.

FIG. 2 specifically shows the structure of the band divider 1. An input digital video signal is a component digital signal having a 4:2:2 structure in a so-called D-1 format defined in the CCIR Rec.601. More specifically, this is the image data format in which luminance signal data Y has a sampling frequency of 13.5 MHz, and two color difference signal data Cr and Cb have sampling frequencies of 6.75 MHz. The input digital video signal is supplied to an LPF (Low-Pass Filter) 11 and HPF (High-Pass Filter) 12 for horizontal frequency division. Connected to the LPF 11 is a subsampler (2.dwnarw.1) 13 which thins a low-band signal by the ratio of 2:1. Likewise, a subsampler 14 which thins a high-band signal by the ratio of 2:1 is connected to the HPF 12. The low-band signal output from the subsampler 13 is supplied to an LPF 15 and HPF 16 for vertical frequency division. The high-band signal output from the subsampler 14 is supplied to an LPF 17 and HPF 18 for vertical frequency division. Subsamplers 19 to 22 are respectively connected to the LPF 15, HPF 16, LPF 17 and HPF 18. The output signal of the subsampler 20 is a subband signal LH3 as one of ten divided signals, the output signal of the subsampler 21 is a subband signal HL3 and the output signal of the subsampler 22 is a subband signal HH3.

The portion indicated by those numerals "11" to "22" forms the first layer. Connected to the output of the subsampler 19 that generates the signal LL3 of the lowest frequency component in the first layer is the second layer, which comprises LPFs 23, 27 and 29, HPFs 24, 28 and 30 and subsamplers 25, 26 and 31 to 34 in the same manner as the first layer. The output signal of the subsampler 32 is a subband signal LH2, the output signal of the subsampler 33 is a subband signal HL2 and the output signal of the subsampler 34 is a subband signal HH2. Further, the third layer, which comprises LPFs 35, 39 and 41, HPFs 36, 40 and 42 and subsamplers 37, 38 and 43 to 46 in the same manner as the first layer or the second layer, is connected to the output of the subsampler 31 that generates the signal LL2 of the lowest frequency component in the second layer. The output signal of the subsampler 43 is a subband signal LL1, the output signal of the subsampler 44 is a subband signal LH1, the output signal of the subsampler 45 is a subband signal HL1 and the output signal of the subsampler 46 is a subband signal HH1.

FIG. 3 shows the bands of the signals LL1 to HH3 obtained by dividing the input into ten subbands, in a two-dimensional frequency region. In FIG. 3, the vertical scale represents the frequency in the vertical direction, and the horizontal scale represents the frequency in the horizontal direction. The ratio of the areas of the individual regions in FIG. 3 matches with the ratio of the number of pieces of data in the individual subband signals.

FIG. 4 specifically shows the structures of the quantizing section 2 and the coding section 3. The ten subband signals, or two-dimensional wavelet transform coefficient data output from the band divider 1 are supplied to a quantizer 51 in the quantizing section 2. The quantizer 51 linearly quantizes the transform coefficient data in accordance with a quantization step size output from a quantization characteristic selector 52, and sends the quantized data to the aforementioned coding section 3. Since a memory 53 having previously stored the ratio of the quantization step sizes for each subband is connected to the selector 52, the selector 52 selects a ratio of the quantization step size from the memory 53 in accordance with the frequency band of the supplied transform coefficient data to set the quantization step size.

In the coding section 3, the quantized wavelet transform coefficient data from the quantizer 51 is supplied to a blocking section 54. The blocking section 54 rearranges the quantized wavelet transform coefficient data into blocks. Each block contains 64 wavelet transform coefficient data picked up from ten subbands. The rearranging method will be discussed later. Connected to the blocking section 54 is a data output processor 55 which outputs the data in a predetermined order block by block. The output order for the 64 transform coefficient data is set by a data output setting section 56. The setting section 56 includes, for example, a memory which stores a predetermined data output order.

A run-length coder 58 is connected to the output of the data output processor 55. The run-length coder 58 performs run-length coding on a data train output from the data output processor 55. The run-length coded data train is compressed video signal data.

In the thus constituted quantizing section 2 and coding section 3, the transform coefficient data output from the band divider 1 is linearly quantized by the quantizer 51 in accordance with the quantization step size output from the quantization characteristic selector 52. An example of the quantization step size ratios of the individual subbands to the subband LL1 of luminance signal is shown in FIGS. 5 and 6. FIG. 5 shows the ratios of quantization step sizes for a luminance signal, and FIG. 6 shows the ratios of quantization step sizes for a color difference signal. When the transform coefficient data is luminance signal data, the quantization characteristic selector 52 sets the quantization step sizes of subbands according to the ratios shown in FIG. 5 and outputs them to the quantizer 51. When the transform coefficient data is color difference signal data, the quantization characteristic selector 52 sets the quantization step sizes of subbands according to the ratios shown in FIG. 6 and outputs them to the quantizer 51. The quantization is carried out by, for example, dividing the transform coefficient data by the set quantization step size.

When one frame of a luminance signal in an interlaced image has a size of 704 pixels by 480 lines as defined in the CCIR Rec.601, the spectrum of the vertical and time domain in the three-dimensional spectrum of the luminance signal become as shown in FIG. 7. The vertical sampling frequency is 480 c/h with respect to one screen height, and the temporal sampling frequency along the time is 60 Hz. In interlace scanning, sampling points of the first field and second field have offset. Therefore, one of carriers (see circles in FIG. 7) occurs at the vertical spatial frequency v=240 c/h and the temporal frequency f=30 Hz. These carriers arise from an interlaced structure. A frequency range of the interlaced image is restricted to the region bounded by a lozenge in FIG. 7.

When the luminance signal is separately coded for first and second fields, each field has a size of 704 pixels by 240 lines. The intra-field coding can accomplish high-efficiency coding without interfering with image movement. Since the vertical frequency component is folded in this case, the distribution of the vertical spectrum of each field greatly differs from the distribution of the vertical spectrum of a frame. The highest frequency component and intermediate frequency components in a frame in the vertical direction are respectively aliased into DC component and vertical high frequency components in the field processing. As a result of aliasing, the intra-field coded signal has a large amount of high vertical frequency components. Taking this fact into consideration, the quantization step sizes of subbands in vertical direction are set finer than those in horizontal direction.

In order to estimate relative importance of each subband, quality and signal-to-noise ratio (SNR) of restored image, in which all transform coefficients in a specific subband were eliminated and coefficients in the other subbands were not quantized, were evaluated. The original image used in the experiment is the "mobile & calendar" which is a standard MPEG image.

The following are the experimental results for luminance signal. The numerals in parentheses indicate the average SNR.

(1) The transform coefficients in HH3 are eliminated. (31.4 dB)

Quality: Slight disturbance on oblique lines which is hardly significant.

(2) The transform coefficients in LH3 are eliminated. (22.1 dB)

Quality: Shadowy over the entire image.

(3) The transform coefficients in HL3 are eliminated. (28.2 dB)

Quality: Disturbance on vertical lines.

(4) The transform coefficients in HH2 are eliminated. (30.0 dB)

Quality: Oblique deterioration occurred.

(5) The transform coefficients in LH2 are eliminated. (22.8 dB)

Quality: All edge portions blurred.

(6) The transform coefficients in HL2 are eliminated. (27.4 dB)

Quality: Disturbance on the vertical edge portions.

(7) The transform coefficients in HH1 are eliminated. (31.2 dB)

Quality: Oblique waves occurred on the entire image.

(8) The transform coefficients in LH1 are eliminated. (23.8 dB)

Quality: Blurring over the entire image.

(9) The transform coefficients in HL1 are eliminated. (29.3 dB)

Quality: Vertical waves occurred on the entire image.

It is apparent from the results of the experiment on the luminance signal that the overall image becomes unclear and the SNR becomes significantly poor for the images obtained by restoring compressed image signals from which the vertical frequency components, such as subband signal LH1, LH2 or LH3, have been completely removed. Therefore, the quantization step size of LH3 should be set smaller than that of HL3 and the quantization step size of HH3 can be set large. In the experiment, the restored image had a good quality with less deterioration when the quantization step size for each subband of the luminance signal was set to the level shown in FIG. 8 or FIG. 9. The quantization step size of HH3 in FIG. 9 is 10000.0 which indicates that data has been eliminated completely.

The following are the results of quality assessment on restored image in which all transform coefficients of color difference signals in a specific subband were eliminated and coefficients in the other subbands were not quantized.

(1) The transform coefficients in HH3 are eliminated.

Quality Slight disturbance in the oblique direction.

(2) The transform coefficients in LH3 are eliminated.

Quality: Color changed and blurred at the edge portions.

(3) The transform coefficients in HL3 are eliminated.

Quality: Vertical deterioration occurred.

(4) The transform coefficients in HH2 are eliminated.

Quality: Oblique deterioration occurred.

(5) The transform coefficients in LH2 are eliminated.

Quality: Disturbance occurred at the edge portions.

(6) The transform coefficients in HL2 are eliminated.

Quality: Disturbance occurred on the vertical edge portions.

(7) The transform coefficients in HH1 are eliminated.

Quality: Oblique waves occurred on the entire image.

(8) The transform coefficients in LH1 are eliminated.

Quality: Blurring occurred over the entire image.

(9) The transform coefficients in HL1 are eliminated.

Quality: Vertical waves occurred on the entire image.

It is apparent from the results of the experiment on the color difference signals that the quantization step size of LH3 can be set greater than that of the signal HL3. In the experiment, the restored images had an excellent quality when the quantization step size of the color difference signal for each subband was set to the level in FIG. 10. The quantization step size of HH3 in FIG. 10, like that in FIG. 9, is set to 10000.0 which likewise indicates the complete elimination of data.

The blocking section 54 operates as follows: One piece of data in each of LL1, LH1, HL1 and HH1 which have the same size in spatial frequency regions is assigned to one block. With regard to each of LH2, HL2 and HH2, four pieces of data are assigned to one block. With regard to each of LH3, HL3 and HH3, sixteen pieces of data are assigned to one block. The single block contains 64 pieces of data. When those 64 pieces of data are extracted from the output signal of the quantizer 51, a single blocking process has been completed.

This operation will be described more specifically. Suppose that one frame of input digital video signals has an interlaced structure of M pixels by N lines as shown in FIG. 11A. Then, wavelet transform is performed separately for the first and second fields having M pixels by N/2 lines, respectively as shown in FIGS. 11B and 11C. When 3-layer wavelet transform is performed on one field of an image, ten subbands are produced as mentioned earlier, and the sum of the numbers of data points of the ten subbands is equal to M.times.N/2 which is the number of data points for one field as shown in FIGS. 11D and 11E. Since the two-dimensional frequency region is divided into the ten subbands in an octave scale by wavelet transform, not all the numbers of data points of ten subbands are the same. As the sampling interval of data is inversely proportional to the band width of the subband, the numbers of data points in the subbands form a quad tree structure from the low frequency to the high frequency. That is, the number of pieces of data is (M/8).times.(N/2/8) in LL1, HL1, LH1 and HH1, the number of pieces of data is (M/4).times.(N/2/4) in HL2, LH2 and HH2, and the number of pieces of data is (M/2).times.(N/2/2) in HL3, LH3 and HH3.

Assuming that a block consists of 8.times.8 pieces of data, and data picked up from ten subbands are rearranged in the block. This block is called "overlap macroblock". FIGS. 12A and 12B show a process of construction of overlap macroblocks from ten subbands in a field. In proportion to the number of pieces of data of each of the subbands, one piece of data is extracted from each of LL1, HL1, LH1 and HH1, four pieces of data are extracted from each of HL2, LH2 and HH2, and sixteen pieces of data are extracted from each of HL3, LH3 and HH3, as shown in FIG. 12A. Those pieces of data are allocated at the associated positions as shown in FIG. 12B, thus forming an overlap macroblock. As a result, (M/8).times.(N/16) data blocks are produced per field. This process is simple data rearrangement, and quantization is executed subband by subband and does not thus cause any blocking effects.

Each overlap macroblock contains 64 pieces of data numbered from 0 to 63 as shown in FIG. 13. That is, the data 0 is the data picked up from LL1, the data 1 is the data picked up from HL1, the data 8 is the data picked up from LH1, the data 9 is the data picked up from HH1, the data 2, 3, 10 and 11 are the data picked up from HL2, the data 16, 17, 24 and 25 are the data picked up from LH2, the data 18, 19, 26 and 27 are the data picked up from HH2, the data 4 to 7, 12 to 15, 20 to 23 and 28 to 31 are the data picked up from HL3, the data 32 to 35, 40 to 43, 48 to 51 and 56 to 59 are the data picked up from LH3, and the data 36 to 39, 44 to 47, 52 to 65 and 60 to 63 are the data picked up from HH3.

The upper left of the thus obtained overlap macroblock indicates the lowest frequency component in both the vertical and horizontal directions and the lower right indicates the highest frequency component in the oblique direction. The overlap macroblock, like the block obtained by DCT, represents the power distribution of the frequency components from the low frequency to the high frequency in the direction from the upper left to the lower right. There are ten types of transform coefficients in this case whereas there are 64 types for the DCT. In other words, an overlap macroblock contains a plurality of transform coefficients belonging to the same types except for the coefficients picked up from LL1, HL1, LH1 and HH1.

Sixty-four pieces of data are output from the data output processor 55 in a one-dimensional order determined by the data output setting section 56. The setting section 56 stores the data order to scan and output data in the following order. The numerals below are associated with data numbers shown in FIG. 13.

b 0.fwdarw.8.fwdarw.1.fwdarw.9.fwdarw.16.fwdarw.17.fwdarw.24.fwdarw.25.fwdar w.2.fwdarw.10.fwdarw.3.fwdarw.11.fwdarw.26.fwdarw.18.fwdarw.27.fwdarw.19.fw darw.32.fwdarw.33.fwdarw.34.fwdarw.35.fwdarw.40.fwdarw.41.fwdarw.42.fwdarw. 43.fwdarw.48 .fwdarw.49.fwdarw.50.fwdarw.51.fwdarw.56.fwdarw.57.fwdarw.58.fwdarw.59.fwd arw.4.fwdarw.12.fwdarw.20.fwdarw.28.fwdarw.5 .fwdarw.13.fwdarw.21.fwdarw.29.fwdarw.6.fwdarw.14.fwdarw.22.fwdarw.30.fwda rw.7.fwdarw.15.fwdarw.23.fwdarw.31.fwdarw.60.fwdarw.52.fwdarw.61.fwdarw.44. fwdarw.53.fwdarw.62.fwdarw.36.fwdarw.45.fwdarw.54.fwdarw.63.fwdarw.37.fwdar w.46.fwdarw.45.fwdarw.38.fwdarw.47.fwdarw.39

The scanning and outputting order in a block becomes as indicated by the arrows in FIG. 14. When one image (one field) consists of MN/128 data blocks from the first block to the (MN/128)-th block as in the above case, MN/128 blocks are output.

This data scanning order is determined by taking the following three points in consideration.

(1) The scanning and outputting direction in a block is changed in accordance with the subbands.

(2) For subbands belonging to the same layer, the data are scanned and output in the order of LH.fwdarw.HL.fwdarw.HH.

(3) The scanning and outputting are performed toward higher layers from a lower layer.

With regard to the point (1), the data are scanned and output in the vertical direction for the subbands, such as LH2 and LH3, which have a low frequency in the horizontal direction and a high frequency in the vertical direction. Consequently, zero-runs (consecutive 0's) are likely to occur. With regard to the point (2), in the same layer, since LH contains aliased components by interlace, zero-runs are most unlikely to occur, while HH contains an oblique frequency components, they are coarsely quantized so that zero-runs are likely to occur. In this respect, if the data are scanned and output in the order of LH.fwdarw.HL.fwdarw.HH and an EOB (End Of Block) code, which indicates that there are no non-zero data after that, is affixed in the run-length coding, data following that code can be cut off. With regard to the point (3), power of a higher layer is lower than that of a lower layer and there are more 0's for the higher layer, so that data following the EOB code can be cut off in the run-length coding.

When a one-dimensional sequence of 64 pieces of data is obtained for each overlap macroblock consisting of 8.times.8 (=64) pieces of transform coefficient data as explained above, the coding section 3 performs variable length coding on the zero-runs and levels of the one-dimensional sequence of 64 pieces of data by the run-length coder 58. The run-length coder codes combination of zero-run and level of data, which terminates the zero-run, with Haffman code. If there are no non-zero data after the last non-zero data in a block, an EOB (End Of Block) code is added and the coding on the block is terminated.

If transform coefficient data of the subband HH3 are cut off by the quantization in the quantizing section 2, a one-dimensional sequence of 48 pieces of data is obtained from a single overlap macroblock consisting of 8.times.8=64 pieces of transform coefficient data. In this case, the coding section 3 performs variable length coding on the zero-runs and levels of the one-dimensional sequence of 48 pieces of data.

Although a component digital signal having a 4:2:2 structure is used as an input digital video signal in the above-described embodiment, a component digital signal having a 4:1:1 structure as defined in the CCIR Rec.601 may also be used.

Although 3-layer wavelet transform is executed in this embodiment, this invention is not limited to this particular type and other types of multilayer wavelet transform such as 2-layer wavelet transform and 4-layer wavelet transform may also be performed.

According to the subband coding method of the present invention, a digital video signal is divided into a plurality of frequency bands in vertical and horizontal directions in a spatial frequency region by executing two dimensional multilayer wavelet transform on each processing unit (field) of the digital video signal. Then a specific number of data proportional to the number of data in each subband, for example 1, 4, and 16, are extracted from subbands, the extracted data are rearranged to a data block consisting of, for example, 8.times.8 pieces of data, and data in the data block are scanned in a predetermined order to generate a sequence of data. The data scanning is performed in the order from data extracted from a layer consisting of the lowest frequency bands to data extracted from a layer consisting of the highest frequency bands. For data extracted from the same layer, scanning is done in the order from data extracted from the LH band consisting of high frequency components in the vertical direction and low frequency components in the horizontal direction, data extracted from the HL band consisting of low frequency components in the vertical direction and high frequency components in the horizontal direction, and data extracted from the HH band consisting of high frequency components in the vertical and horizontal directions. For data extracted from the same layer, scanning is done in the horizontal direction for data extracted from the LH band, and in the vertical direction for data extracted from the HL band. Consequently, the occurrence of zero-runs increases, thus ensuring efficient run-length coding.

For example, the SNR is 30.441 dB, the amount of data per frame is 1,286,596 bits, and the bit rate is 38.60 Mbps according to the coding method which involves zigzag scanning as done in the conventional orthogonal-transform coding, whereas the SNR is 30.441 dB, the amount of data per frame is 1,194,033 bits, and the bit rate is 35.82 Mbps in the subband coding method of the present invention.

It is apparent from the above that the present invention can compress a video signal with a high compression efficiency and with less quality deterioration of a restored image.