Enhancing image compression performance by morphological processing

[0002] 1. Field of the Invention

[0003] The invention is related to image processing. More specifically, the invention relates to the enhancement of the image quality by reducing artifacts due to heavy quantization for compression.

[0004] 2. Description of Related Art

[0005] One crucial aspect of imaging is the process of image compression. In digital systems such as digital cameras image compression is utilized to store and transmit a large amount of image information in fewest number of bits possible while still maintaining an image quality that is acceptable. One example of a popular image compression scheme is JPEG (Joint Photographic Experts Group) which is based upon the Discrete Cosine Transform (DCT). Recently, new image compression schemes have been developed utilizing the Discrete Wavelet Transform (DWT). Unlike the DCT which is periodic in nature, the DWT is better suited and often more efficient in representing sharp discontinuities in data such as that present in image edge features. The DWT decomposes an input image into a number of “sub-bands” in multiple levels which are shown in FIG. 1. FIG. 1 shows the results of iteratively applying a 2-dimensional (2-D) DWT to an image.

[0006] The first iteration of the 2-D DWT decomposes an image I into four sub-bands S0, S1, S2 and S3. The sub-band S0 is also referred to as the “LL” sub-band, based on the double low-pass filtering-used to generate it. S0 (LL) is essentially a scaled approximation of the original image I, and contains the most salient spatial information to the original image. The sub-bands S1, S2 and S3 contain edge information and when the input image is noisy, also a considerable amount of that noise. The sub-bands S1, S2, and S3 are also referred to as HL, LH and HH sub-bands, respectively, due to the various low-pass and high-pass filtering used to generate them. The level 1 sub-bands S0, S1, S2, and S3 result from applying the 2-D DWT once to the image I. Since LL sub-band is the scaled version of the original image, we can apply the same decomposition in it. If the 2-D DWT is applied again, to the sub-band S0, a level 2 2-D DWT is being performed. This would result in the generation of four new sub-bands S10, S11, S12, and S13 after decomposition to S0. After such decomposition, a mechanism known as quantization, which is the mapping of one range of values to a smaller range, is employed to yield compression. One such quantization technique, known as uniform scalar quantization, divides the original data points by a threshold number T (or quantized parameter) greater than or equal to 1, in order to achieve the mapping.

[0007] Since the sub-bands S2, S3 and S1 are perceptually (from a visual standpoint) less significant than the S0 sub-band, these sub-bands may be more “roughly” quantized (i.e., assigned a higher quantization threshold T) so that the values therein are compressed greater. Such scalar quantization may use one quantization threshold T_ithroughout one sub-band S_iand then different a T_jin another sub-band S_j. Accordingly, FIG. 1 shows a threshold T₀applicable to sub-band S₃, a threshold T₁applicable to both sub-bands S₁and S₂, T₃applicable to S₁₃, T₂applicable S₁₁and S₁₂and T₄applicable to sub-band S₁₀. These quantization thresholds maybe selected depending upon the desired level of compression by introducing quantization loss. The higher the quantization threshold, the more compression and hence more possible loss when the compressed image is reconstructed.

[0008] Up to a certain level of quantization, the loss incurred may not be visible by the human eyes. Such compression is referred to as visually lossless compression. For the application which warrants more compression beyond a visually lossless scheme, further compression can be achieved by using a higher quantization threshold than that used for visually lossless compression. As a result, however, visible artifacts will occur in the reconstructed image. We can enhance the compression performance, by reducing these visible artifacts.

[0009] When DWT coefficients (from applying a DWT upon an image) are then heavily quantized, “Ringing Artifacts” are often produced in the decompressed (reconstructed) image. Due to the well-known Gibbs phenomenon, the Ringing Effect (which leads to Ringing Artifacts) places rings around a homogenous area of an image such as a clear sky or background. To reduce this effect, several digital filtering techniques have been devised, but these, when applied, have the additional effect of introducing smoothness or blurring. By reducing the sharpness of an image, such filtering, used to remove the Ringing Effect, also degrades image quality. It would be desirable to remove ringing artifacts while preserving the sharp edges which are the most perceived features by the human visual system.

[0010] What is disclosed is a method that includes defining a morphological filtering operator designed to remove ringing artifacts in a decompressed image, and then applying that operator upon pixels of the decompressed image that belong to non-texture image regions.

[0011] FIG. 1 illustrates the decomposition of an image using a Discrete Wavelet Transform.

[0012] FIG. 2 is a flow diagram of the basic methodology of the invention.

[0013] FIG. 3 is a flow diagram of one embodiment of the invention.

[0014] FIG. 4 is an example of performing texture detection according to at least one embodiment of the invention.

[0015] FIG. 5 is a system diagram of one embodiment of the invention.

[0016] The exemplary embodiments described herein are provided merely to illustrate the principles of the invention and should not be construed as limiting the scope of the invention. Rather, the principles of the invention may be applied to a wide range of systems to achieve the advantages described herein and to achieve other advantages or to satisfy other objectives as well.

[0017] One way to enhancing the image compression performance (higher compression) is to use coarser quantization (which introduces artifacts in the decompressed image) and reduce the resulting artifact by a suitable post-processing technique. In order to achieve reduction of the Ringing, a post image processing technique is described that identifies textures and applies a morphological filtering technique on the non-texture pixels. The appearance of ringing artifacts are more prominent in homogenous areas of an image compared to its appearance in texture regions. Morphological filters, when properly employed, can have the effect of preserving sharp edge features which are most readily perceived by the human visual system. Morphology, which is characterization of the form and structure of an object, can be based upon two operations known as dilation and erosion. Dilation of an object or sub-image causes that object or sub-image to grow in relation to a defined background neighborhood (also called a structuring element). Erosion, on the other hand, causes the object or sub-image to be reduced or shrink in relation to a background neighborhood. Dilation can make that object or sub-image more prominent by fitting into the region or background, while erosion can obscure it with the background. The neighborhood chosen may be either geometric (following a set pattern) or adaptive (modified depending upon certain characteristics).

[0018] The operations of dilation and erosion is not static and may be defined as befits the application. If we let the neighborhood about and including a pixel x be designated as G(x), then dilation and erosion are defined, in one embodiment of the invention as:

[0019] dilation(x)=maximum{G(x)}

[0020] erosion(x)=minimum{G(x)}.

[0021] Thus, a dilation of a pixel would yield a set of all the maximums of a G neighboring surrounding each of the pixels in the given neighborhood G(x) while an erosion would yield a set of the minimums of a G neighborhood surrounding each of the pixels in the G(x) neighborhood. By convention and definition, dilation and erosion are not inverse operations of each other. Two other morphological operations based on erosion and dilation may be defined as:

[0022] closing(x)=erosion(dilation(x));

[0023] and

[0024] opening(x)=dilation(erosion(x)).

[0025] Thus, according to the non-inversibility of the dilation and erosion operations with respect to one another, generally:

[0026] closing(x)<>opening(x), except in certain degenerate cases.

[0027] Utilizing these morphological operations, a filtering operation can be defined and selectively applied to an image as described below. Practically, the “opening” operations has a “low-pass” filtering in an image which reduces the noise or irregularities in the structuring element. “Closing” is considered to be a dual operation to “opening” in that such features in the structuring element are highlighted.

[0028] FIG. 2 is a flow diagram of the basic methodology of the invention.

[0029] The methodology for removing ringing artifacts includes a morphological filtering performed on a post-capture and compression basis. First, a filter operation should be selected that will be applied to the image space (block 210). This filtering operation is based upon morphological operations such as those given above. In one embodiment of the invention, a filtering operation f(x) is selected to be:

[0030] f(x)={opening(x)+closing(x)}/2.

[0031] This filter is to be applied selectively to the image in its non-texture or homogenous regions. In this processing technique, the image should be operated upon when in decompressed form. For instance, consider an image captured by a digital camera or other such device. As described in greater detail above, this image is often compressed by techniques utilizing such constructs as DWT (Discrete Wavelet Transform) and are simultaneously or subsequently quantized throughout the various image sub-bands. Such a compressed image may show ringing artifacts upon decompression if coarser quantization is applied. Since each DWT sub-band may have less severe or more sever quantization (i.e., a higher or lower quantization threshold) than others, the methodology can take this differentiation into account. Once the image is decompressed (block 220), the morphological filtering operation defined at block 210 may then be employed. Once set the filtering operation can be repeatedly used such that it need not be changed for each image.

[0032] The methodology operates repeatedly upon each pixel in the image (or sub-image) and thus checks whether any pixels remain to be processed (block 230). If so, the pixel is fetched (i.e., its intensity value is read out, along with its coordinates in the image map) (block 240). According to an embodiment of the invention, the morphological filtering operation is applied only to non-texture image regions. Thus, it must be determined whether or not the fetched pixel belongs to a texture region or a more homogenous non-texture region (block 250). A procedure for doing so in accordance with at least one embodiment of the invention is described in greater detail below. If the pixel does not belong to a texture region of the image, then the pixel is filtered using the defined filtering operation (block 260), which is morphological, rather than linear, in nature. If the pixel does belong to a texture region, no filtering is applied to it. This has the intended advantage of not obscuring or smoothing edge features in the image since the filter is applied to pixels belonging to homogenous rather than texture regions. Further, since the technique is a post-processing methodology, an image may be more highly compressed, at least in as much as such compression was conventionally limited when having to consider the ringing effect.

[0033] The methodology shown in blocks 230 through 260 are repeated for each pixel in the image until all such pixels are subjected to the selectively applied morphological filtering.

[0034] FIG. 3 is a flow diagram of one embodiment of the invention.

[0035] As mentioned above, the embodied methodology is designed to work on a pot-processing basis. Thus, the image is first decompressed (block 310). The methodology operates repeatedly upon each pixel in the image (or sub-image) and thus checks whether any pixels remain to be processed (block 320). If so, the pixel x is fetched (i.e., its intensity value is read out, along with its coordinates in the image map) (block 330).

[0036] A pre-requisite for a morphological filtering operation is the definition of a neighborhood (called a structuring element) over which it is evaluated. When applied to image processing, the structuring element is a neighborhood about the pixel (or sub-image) under consideration. Thus, about the fetched pixel x, a neighborhood G(x) should be defined (block 340). The neighborhood may be geometric, fitting a set pattern about and including the pixel x, such as a three pixels by three pixels square area. Also, the neighborhood may be adaptive, and depending upon the location (or sub-band association) of the pixel, a differing neighborhood. Where color planes have been separated, such as in an RGB image originating pre-compression in a Bayer pattern form, the ringing effect reduction is applied separately to each color plane. In such instance so too the chosen neighborhood includes pixels belonging only to color plane associated with the pixel x.

[0037] Once a neighborhood is chosen, a threshold is also set (block 350). This threshold is used in comparing with a variance of the neighborhood such that the determination may be made whether the pixel x belongs to a texture or not. The threshold T may be uniform throughout the entire image or vary according tot the sub-band association, color plane association or other factors of the pixel x. If the threshold T is uniform throughout, it need not be chosen for each pixel, unlike what is exemplified in the flow of FIG. 3.

[0038] According to one embodiment of the invention, the primary metric utilized in discriminating texture from non-texture regions is the local variance V(x) over the neighborhood G(x). In general, the variance for N sample points Y₁, Y₂, . . . Y_Nis given by: $V\left(x\right)=\sqrt{\sum _{i=1}^N{\left(\stackrel{\_}{Y}-Y_i\right)}^2/N}$

[0039] As given above, the local variance V(x) with sample points consisting of the pixel intensities within the neighborhood G(x) is then computed (block 360). The disturbance due to ringing artifacts is more pronounced and prominent in homogenous regions of the image than in texture regions. Whether a particular pixel belongs to a texture or not can be ascertained by comparing the variance V(x) with the chosen threshold T (block 370). If the computed variance V(x) is less than the chosen threshold T, then the morphological filtering operation is performed. In such an instance, first, the filter operator f(x) is computed (block 380). As described above, one embodiment of the invention uses a filter operator f(x)={opening(x)+closing(x)}/2. The filtering is performed by replacing the fetched pixel x with the value resulting from the computed filter operator f(x) (block 390).

[0040] By discriminating pixels belonging to textures and non-textures, the morphological filtering can be selectively performed on those pixels belonging to non-textures. Thus, if the variance V(x) is not less than the threshold T, no filtering operation is performed. Alternatively, it may be desirable to perform one type of filtering operation on non-texture pixels and yet another operation (whether filtering or some other image enhancement) on texture pixels. The processing shown in blocks 320 through 390 are repeated for all pixels in the image (or sub-image) being processed.

[0041] FIG. 4 is an example of at least one embodiment of the invention.

[0042] FIG. 4 shows a exemplary sample neighborhood G(x) of pixels x_Kabout a central pixel x, where K denotes the direction of the sample pixel adjacent to x or numerically designates a position about the pixel x. Each numerical value represents the intensity value of the pixel such as that derived from exposing those pixels in a sensor to capture an image of a scene. In the example, the pixel x has intensity value of 39 while its north neighbor x_Nhas a value of 42 and so on as shown.

[0043] One way of expressing the closing function in terms of set membership is:

[0044] closing(x)=erosion{G(y)} where y_i=dilation(x_i), x_iεG(x).

[0045] Referring to the pixel values given in FIG. 4, dilation (X_NW)=maximum {G(X_NW)}=maximum {33, 43, 20, 91, 33, 42, 3, 29, 39}=91 which is the pixel X₁₆. Likewise, dilation (X_N)=maximum {G(X_N)}=maximum {43, 20, 4, 33, 42, 5, 29, 39, 85}=85 which is pixel value X_E. Thus, by applying this procedure, the dilation (x) results in the pixel set X₁₆, X_E, X_E, X_E, X_E, X_E, X₁₁, and X₁₆.

[0046] The erosion (x) function is likewise computed by taking minimums around a neighborhood about each of pixels surrounding the pixel x. The closing (x) associates the two set operations as the erosion (dilation (x)). Thus, closing (x) would take an erosion of the set of pixels resulting from the dilation of the pixel x. Again, expressing the opening operation in set membership:

[0047] opening(x)=dilation{G(y)} where y_i=erosion(x_i), x_iεG(x).

[0048] In each dilation and erosion operation, the pixel x is included as part of the neighborhood G(x). In the example shown in FIG. 4, dilation(x) equals 85 (the value of X_E) while erosion(x) equals 5 (the value of X_NE). Likewise, the opening(x) and closing(x) are defined as closing(x)=erosion(dilation(x)) and opening(x)=dilation(erosion(x)).

[0049] If the above specified morphological filter f(x) is to applied upon x, then first x must be judged as belonging to a region of non-texture. Thus, according to the embodiment described in FIG. 3, the local variance V(x) for the sample neighborhood G(x) must be computed. To do so, the mean x′ of the sample set G(x) must be determined. The mean is: {x+x_N+x_NE+x_E+x_SE+x_S+x_SW+x_W+x_NW}/N, where N is the number of sample points. In this example, the mean x′ is 351/9 or 39. The variance V(x) then is 24.36, in accordance with the formula given above. The variance is then compared to a threshold value T. Experimental results have shown that for a 8-bit intensity value per pixel (i.e., 0 to 255), a threshold T between 60 and 70 yield good texture discrimination.

[0050] FIG. 5 is a system diagram of one embodiment of the invention.

[0051] Illustrated is a computer system 510, which may be any general or special purpose computing or data processing machine such as a PC (personal computer), coupled to a camera 530. Camera 530 may be a digital camera, digital video camera, or any image capture device or imaging system, or combination thereof that has a image sensor comprised of a plurality of sensor locations known as pixels and may be utilized to capture an image of a scene 540. Captured images are compressed by quantization and predictive encoding by an image processing circuit 532 so that they can be efficiently stored in an image memory unit 534, which may be a ROM, RAM or other storage device such as a fixed disk. The image contained within image memory unit 534 that is destined for computer system 510 is the result of compression within the camera 530 and thus may show ringing artifacts when decompressed. The ringing artifact reduction scheme of the invention in various embodiments is intended to provide an efficient way of reducing the ringing artifact while not disturbing the amount of compression achieved in camera 530. This allows camera 530 to compress images to a greater degree so that faster throughput or less memory utilization is realizable.

[0052] The stored compressed images in memory unit 534 are transferred to computer system 510 for local storage thereon and may be decompressed using a processor 512 such as the Pentium (a product of Intel Corporation) and a memory 511, such as RAM, which is used to store/load instruction addresses and result data. Once decompressed, the resulting pixel map of the image can be subjected to morphological filtering in order that ringing artifacts be reduced. Morphological filtering, as described in more detail above in various embodiments, consists of statistical analysis of a neighborhood of pixel locations (such as computing mean and variance), comparisons and then computation of filter operator including the finding of maxima and minima. The instructions for carrying out and executing such filtering may be stored to a disk 518 or memory 511. It would be readily apparent to one in the art to program a computing machine to perform ringing artifact reduction given that the methodology described in the various embodiments of the invention are adhered to.

[0053] Computer system 510 has a system bus 513 which facilitates information transfer to/from the processor 512 and memory 511 and a bridge 514 which couples to an I/O bus 515. I/O bus 515 connects various I/O devices such as a display adapter 516, disk 518 and an I/O port 517, such as a serial port. Many such combinations of I/O devices, buses and bridges can be utilized with the invention and the combination shown is merely illustrative of one such possible combination.

[0054] Image processing circuit 532 consists of ICs and other components which may execute, among other functions, the DWT compression of captured images. After such compression, and transfer to computer system 510, the compressed images are decompressed on computer system 510 using suitable application software (or hardware in combination thereof) executable with the aid of processor 512. After being decompressed, the morphological filtering procedure is executed and may result in the replacement of certain non-texture pixels by a filter operator result, as discussed above. The enhanced, ringing artifact-reduced image data may then be rendered visually using a display adapter 516 into a rendered image 550. In a videoconferencing application, the image data in its compressed form may be communicated over a network or communication system to another node or computer system in addition to or exclusive of computer system 510 so that a videoconferencing session may take place. Whatever the intended application or use, the end result will be a higher quality rendered image 550 that displayed onto monitor 520 or other nodes in a videoconferencing session as compared with conventional imaging. Advantageously, the level of compression need not consider as strongly the effect of a ringing artifact upon decompression, since such artifacts are filtered by the morphological operation described herein. Thus, higher compression ratios as desired by the application/end use and higher quality images after decompression and rendering are simultaneously achievable.

[0055] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will however be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are accordingly to be regarded as illustrative rather than restrictive.