Structure and Method for Reducing Spatial Noise

1. Field of the Invention

The present invention relates to digital video processing. In particular, the present invention relates digital filtering techniques for noise elimination.

2. Discussion of the Related Art

Video noise reduction filters are often provided for removing from a video image artifacts ("noise"), which are visible to a human viewer. The objective of noise removal is to create a visually pleasing image. Such noise-reduction filters include median filters, temporal recursive filters, and linear low-pass filters. Median filters often introduce additional artifacts which corrupt edges in the image. Temporal recursive filters often introduce into the image motion artifacts, such as motion blurs and ghosting. Linear low-pass filters often blur edges in the image. In general, these techniques are provided to remove visible defects from the image so that, for that purpose, introduction of such additional artifacts generally invisible to the eye is tolerated. However, in video signal processing, these "invisible" artifacts can be detrimental to other objectives, such as achieving a high compression ratio for storage and transmission of the video image. A lower compression ratio requires that the video processing system to operate at either a higher bit rate (in a variable bit rate encoding application) or a lower image quality (in a fixed bit rate encoding application).

In the prior art, the Wiener filter and its adaptive field/frame variants are noise-reduction digital filters which has been extensively studied. For example, some local Wiener filters are described in (i) "Digital Image Enhancement and Noise Filtering by Use of Local Statistics" by J. S. Lee, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-2, No 2, March 1985, pp. 165-168; (ii) "Refined Filtering of Image Noise Using Local Statistics", J. S. Lee, Computer Graphics and Image Processing 15, 380-389 (1981); and (iii) "Adaptive Noise Smoothing Filter for Images with Signal-Dependent Noise", Kuan et al, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-7, No. 2, March 1985, pp. 165-177. Specifically, local Wiener filters operate on each pixel of an image based on the mean value and the variance value of a finite number of pixels in the immediate vicinity of that pixel. Wiener filters are important in video compression processing for two reasons. First, Wiener filters remove noise that is not very visible by the eye, such as noise related to film grain. Second, as compared to the noise-reduction filters discussed above (e.g. the median filter), a Wiener filter is less prone to introduce new defects, especially those visible to the eye. Thus, Wiener filters are often used to improve compression efficiency.

The present invention provides both a circuit and a method for removing noise from a video image. Typically, the video image is formed by a sequence of successive frames, in which each frame includes a first field and a second field. The method of the present invention includes the steps of: (i) computing, for each pixel of a frame of the video image, a first variance ("frame-based variance") based on a predetermined number of pixels, in the pixel frame, adjacent that pixel; (ii) computing, for each pixel in the same frame of the video image, a second variance ("field-based variance") based on the predetermined number of pixels, selected from the field of that pixel, which are adjacent that pixel; (iii) selecting the lesser of the first variance and the second variance as a neighborhood variance; and (iv) filtering each pixel of the image using an adaptive filter having a filter coefficient which is a function of the neighborhood variance selected.

In one embodiment, an adaptive filter based on the principles of an adaptive Wiener filter is provided to filter the video image. The pixels used for computing the field-base and the frame-based variances are selected from 3.times.3 configurations, centered around the pixel to be filtered, in the respective field or frame. The adaptive filter is designed to have filter characteristics which both preserve sharpness of the video image at the edges and smooth regions of the image having a relatively low variance.

The present invention provides a filter that, while effective in removing visible artifacts, improves encoder efficiency. Encoder efficiency is achieved by elimination of invisible artifacts of the image in smooth regions, so that fewer bits are required to encode the image without degradation of the visible image. The filter of the present invention has high performance in reducing both frame-based noise (e.g. in still images, inverse telecined images, or deinterlaced images) and field-based noise (e.g. video images with high motion content). Because the filter of the present invention is adaptive on a pixel by pixel basis, the filter correctly preserves both frame based (e.g. in the portion of the image where there is little motion) and field based (in the portion of the image where there is motion) characteristics in the same image.

The present invention is better understood upon consideration of the detailed description below and the accompanying drawings.

FIG. 1 shows a digital filter 101 used to remove noise from a video signal prior to data compression.

FIG. 2 illustrates the operation of the filter 101 in accordance with the present invention.

FIG. 3a shows a pixel's 3.times.3 neighborhood formed by pixels in the same frame; the pixel to be filtered is shown in solid, and its eight neighbors in the neighborhood shown in shade.

FIG. 3b shows a pixel's 3.times.3 neighborhood formed by pixels of the same field; the pixel to be filtered is shown in solid, and its eight neighbors in the neighborhood shown in shade.

One embodiment of the present invention is used in a video data compression system, such as that shown in FIG. 1. FIG. 1 shows a stream 102 of digitized video data being provided to a digital filter 101 for noise removal. The filtered data stream 103 is provided to a conventional video compression system 104, such as the Minerva Compressionist, available from Minerva Systems, Santa Clara, Calif. The compressed data, indicated by reference numeral 106, can then be stored in a mass storage medium, indicated by reference numeral 105, to be retrieved later for further processing or reproduction.

Digital filter 101 can be provided by the filter sequence shown in FIG. 2. As shown in FIG. 2, as represented by step 201a, a neighborhood mean ("frame-based mean") and a neighborhood variance ("frame-based variance") are computed for each pixel, based on averaging and computing the variance of the nine pixels in the pixel's 3.times.3 pixel neighborhood, where these pixels are selected from the frame (both fields) of the pixel. FIG. 3a shows the nine pixels in the 3.times.3 neighborhood, showing the pixel to be filtered in solid and the pixel's eight adjacent pixels shaded. In addition, as represented by step 201b, a neighborhood mean ("field-based mean") and a neighborhood variance ("field-based variance") are computed for each pixel, based on averaging and computing the variance of the nine pixels of the same field in the pixel's 3.times.3 pixel neighborhood. FIG. 3b shows the pixels in that 3.times.3 neighborhood, showing the pixel to be filtered in solid and the eight adjacent pixels of the neighborhood shaded. Then, at the next step, designated by reference numeral 202, for each pixel g(i,j) in the frame, the smaller of the frame-based and field-based neighborhood variances, and its associated neighborhood mean, are chosen to be the neighborhood variance (designated .sigma..sub.g.sup.2 (i,j)) and neighborhood mean (designated g(i,j)), respectively, for that pixel. Independently, at step 205, the frame-based and field-based neighborhood variances obtained at steps 201a and 201b are summed and accumulated for the entire image. The resulting value is used, at step 203, to compute a noise variance (designated .sigma..sub.n.sup.2) for a global noise signal. .sigma..sub.n.sup.2 can be computed in one of several ways. In the present embodiment, for example, where the global noise signal is modelled as a white noise process, .sigma..sub.n.sup.2 is obtained by:

where a is a "preset" scaling constant. This approach of deriving a variance for the global noise signal has the advantage of simplicity, and the degree of noise filtering can be controlled by varying the scaling constant. However, this approach has the drawback of being sensitive to visual activities in an image. For example, this approach results in blurring ("over-filtering") of busy images.

Alternatively, noise variance .sigma..sub.n.sup.2 can also be obtained by estimating noise in one or more reference areas. The reference area or areas can be obtained by either (i) having the user selects a smooth area in a "typical" image of the source material, or (ii) automatically search for smooth areas in the images of the source material. The selected reference area, or areas, is then used to compute noise variance .sigma..sub.n.sup.2 using one of the known statistical techniques. The resulting noise variance .sigma..sub.n.sup.2 is then used estimate noise in all images of the source material. This alternative approach avoids the over-filtering mentioned above.

The filtered value p(i,j) of pixel g(i,j) is then obtained, at step 204, according to an adaptive Wiener filter model: ##EQU1##

The Wiener filter of the present embodiment has the characteristic that, if the neighborhood variance .sigma..sup.2.sub.g, whether obtained from the field-based or from the frame-based neighborhood variance, is relatively high, the output filtered value p(i,j) is close to the input pixel value g(i,j). Hence, the edges of an image will remain relatively sharp. However, if the neighborhood variance is relatively low, the output filtered value p(i,j) is close to the neighborhood mean value g(i,j). Thus, in a relatively smooth region of the image, the local area of the output pixels are made even smoother. Consequently, higher compression ratio can be achieved in these smooth regions without sacrificing image quality. By selecting the neighborhood variance and neighborhood mean from the smaller of the field-based and frame-based neighborhood variances, characteristics of various video formats can be exploited to achieve noise reduction. For example, if the video sequence is obtained by inverse telecine, adjacent lines (although representing pixels from different fields) are effective in providing noise reduction. However, if the video sequence is obtained from an interlaced source, where adjacent lines are separated temporally, the use of field-base neighborhood variances is more effective in providing noise reduction.

Performance of the digital filter described above has been compared with the performances of a number of median filters that are provided with edge maps to suppress edge filtering. Such median filters include a 3.times.3 median filter, a separable median filter, a median of medians filter, and a 1.times.5 point median filter. The digital filter described above results in consistently lower bit rates (5% to 20%), without creating any visible artifact. Additional tests also showed that video material filtered in accordance with the present invention can always be compressed at a lower bit rate than a corresponding adaptive Wiener filter which does not choose between frame-based and field-based variances.

The detailed description above is provided to illustrate the specific embodiments of the present invention and should not be taken as limiting the present invention. Numerous variation and modification within the present invention are possible. The present invention is defined by the following claims.