Adaptive Image Enhancement System

This invention relates to real-time image processing systems, and more particularly, to a video signal processing system for enhancing a video image by combined noise filtering and edge sharpening.

Altering the image signal to present the viewer additional information or insight into factors concerning the pre-enhanced image are known. For example, two useful image enhancement operations are noise reduction and edge sharpening. Various filtering techniques for reducing noise have been proposed. For example, in U.S. Pat. No. 3,009,016--Graham there is described a noise reduction system in which a running average of picture samples (pixels) is compared with each new sample of the video signal, and if the picture sample does not exceed a predetermined theshold, the average of the samples is substituted. In copending application Ser. No. 133,606 filed Mar. 24, 1980, entitled "Adaptive Enhancement of Signal-to-Noise Ratio in Television Imagery," assigned to the same assignee as the present application, there is described a refined noise reduction system which generates an average from samples over an area extending both vertically above, and horizontally back from the current sample of the video signal. While both of these arrangements improve picture quality by reducing noise in areas of the picture which have relatively low detail and uniform intensity, they provide no enhancement by sharpening the edge between regions of contrasting intensity. Noise filtering and image sharpening involve conflicting requirements since filtering to reduce noise can blur the image while circuits for enhancing image sharpness will also enhance the noise.

The present invention is directed to a video image enhancement system which adapts itself to noise filtering of the video signal in image regions of nearly uniform intensity while providing edge sharpening in a region of sharp contrast and high detail. The system adapts globally to the input signal-to-noise (SNR) ratio so that if if the noise level is very low, there is enhancement by edge sharpening of all detail, but if the noise level is high, noise filtering but no edge enhancement is provided. This global adaption to the signal-to-noise ratio is independent of picture detail. In addition, the system adapts locally to the gradient magnitude to provide either noise filtering or edge enhancement with a smooth transition between these two conditions.

The video enhancement system of the present invention is based on the concepts that human observers detect the presence of noise much easier in a nearly uniform region than near an edge, while responding more to an edge brightness difference than to the same brightness difference in regions not adjacent to each other. This suggests that by dividing the images into regions of nearly uniform brightness and regions of higher contrast, it is possible to limit noise filtering to the low contrast regions and edge sharpening to the higher contrast regions. Because such segmentation requires some kind of thresholding to define the two regions, resulting in loss in spatial resolution and introduction of artifacts, the present invention provides a system which is adaptive to image content as measured by the magnitude of the picture gradient. The degree of noise filtering is decreased gradually while the degree of edge sharpening is increased gradually as the magnitude of the gradient increases. At the same time, the rate of transition is made adaptive to the input signal-to-noise ratio of the image so that more adaptive noise filtering is performed if the image is noisier and more adaptive edge enhancement is performed if the image is cleaner.

This is accomplished by providing apparatus in which a succession of digitized samples of the video signal are generated and stored. With each new sample, a group of samples are averaged to generate an average sample while another group of samples are compared to generate the median sample of the group. Processing means generate a gradient sample corresponding to the absolute difference between the median and average samples. The gradient sample is scaled according to the noise level of the video signal independently of picture content. The scaled sample gradient is multiplied by the difference between the median and average samples and added to the average sample to generate an output sample. Successive output samples are converted to an output video signal for processing into an image.

For a better understanding of the invention reference should be made to the accompanying drawings, wherein:

FIG. 1 is a graphical representation of the idealized characteristics of the enhancement system;

FIG. 2 is a schematic block diagram of one embodiment of the invention;

FIG. 3 is a graphical representation of the characteristics of the system of FIG. 2;

FIG. 4 is a schematic block diagram of a preferred embodiment of the present invention;

FIG. 5 is a graphical representation of the characteristics of the embodiment of FIG. 4;

FIG. 6 is a schematic block diagram of a median filter;

FIG. 7 is a block diagram of the processor P.sub.2 of FIG. 2;

FIG. 8 is a block diagram of the processor P.sub.4 of FIG. 2;

FIG. 9 is a block diagram of the processor P.sub.6 of FIG. 4.

Referring to FIG. 1, there is shown the overall characteristics of a video adaptive image enhancement system having the required properties. As the picture gradient increases, the degree of average filtering to reduce noise is decreased and the degree of edge filtering increases. A family of curves for different levels of SNR, as indicated by .sigma., show the characteristic shifts to provide more or less average filtering while providing less or greater edge filtering as the noise level increases or decreases. It will be noted that the degree of noise filtering decreases gradually while the degree of edge sharpening increases gradually as the magnitude of the gradient increases. There is a smooth transition from the performance of the adaptive noise filter to the performance of the adaptive enhancer as the image content changes at different regions of the image. As shown by the family of curves, the rate of transition is made adaptive to the input signal-to-noise ratio of the image. Thus the curve .sigma..sub.1 represents an image generated from a video signal of relatively low SNR while the curve .sigma..sub.2 corresponds to a video signal of higher SNR. Thus a system having the characteristics of FIG. 1 adapts locally to the image content as measured by the gradient magnitude and globally to the input SNR of the image.

Referring to FIG. 2, there is shown a simplified block diagram of a video enhancement circuit according to the present invention which approximates the characteristics shown in FIG. 1. The performance characteristics of the circuit of FIG. 2 are shown in FIG. 3. A television video signal is applied to the input of the enhancement circuit. An analog-to-digital converter 10 quantizes the signal into a plurality of successive digitized samples or pixels. The signal amplitude represented by each pixel is defined as x(i,j), where i is the number of the row or line during one complete frame of the video image and j is the number of successive samples in one line. At least two complete rows are temporarily stored in a two-line store 12, which may be a shift register. As each new sample is transferred into the register 12, a sample from at least two rows above in the image is shifted out of the register.

For each sample x(i,j), a median sample m(i,j) is generated by a median filter 14. The median filter looks at a group of samples in the store 12 and selects the sample point of the group which has the middle magnitude. Preferably the median filter 14 looks at five image points or pixels arranged in a cross pattern, that is, three successive points in the same row or line, and three vertical points with the center point being common to both the horizontal and vertical lines. A suitable circuit for the median filter 14 is described below in detail in connection with FIG. 6.

At the same time and operating in parallel with the median filter is an average filter 16 which generates an output sample that is the average of a group or block of the samples stored in the store 12. Various averaging patterns can be used, such as a running line average, an area average, or a more complex averaging arrangement, such as described in the above-identified patent application. The samples stored may correspond to a 3.times.3 pattern of image pixels. To simplify division in obtaining the average value, only eight samples are used instead of nine. This allows a division by eight, which in a binary system only requires a three digit left shift operation. The resulting output sample from the average filter 16 is designated f(i,j).

The magnitude of the gradient at any point on the picture can be determined by the difference in the median sample value m(i,j) and the average f(i,j). The magnitude of the gradient at any point on the image is given by G(i,j)=m(i,j)-f(i,j). Thus the sample gradient is obtained by a subtracting circuit 18 which subtracts the digitized output from the average filter 16 from the digitized output of the median filter 14.

The gradient magnitude provided by the output of the subtracter is used to activate one of five processing units P.sub.1, P.sub.2, P.sub.3, P.sub.4 or P.sub.5 by comparing the gradient magnitude with a set of threshold values T.sub.1, T.sub.2, T.sub.3 and T.sub.4 derived from a table of threshold values stored in a threshold memory 20. Any one of a plurality of sets of four threshold values are selected, depending on the signal-to-noise ratio (SNR) of the video signal as determined by a noise monitor circuit 22. The noise monitor circuit 22, as hereinafter described in detail, generates a digitized value which is proportional to the SNR of the video input signal. In its preferred form, the noise monitor circuit utilizes the equalizing pulses of a received television signal as a reference, since the waveshape of the pulses are standardized. Any noise in the received signal can be detected as distortion of the equalizing pulse waveform. Such a noise monitoring system is described, for example, in U.S. Pat. No. 4,044,381. The output of the noise monitor circuit 22 is used to address the memory 20 to select a corresponding set of four threshold values from the table of thresholds stored in the memory 20.

The absolute value .vertline.G(i,j).vertline. of the gradient magnitude is compared with the first threshold value T.sub.1 by a comparator circuit 24. If the comparator indicates that the gradient value .vertline.G(i,j).vertline. is less than the first threshold value T.sub.1, it activates the processor P.sub.1. The processor P.sub.1 is a simple gate circuit which is activated by the output C.sub.1 from the comparator circuit 24 to gate the average sample from the average filter 16 to a common terminal 26 connected to the input of a digital-to-analog converter 28. Thus the average value sample f(i,j) is substituted for the input sample x(i,j) in generating an enhanced TV video signal.

The absolute value .vertline.G(i,j).vertline. of the gradient is also compared by a comparator 30 with a threshold value T.sub.2 which is larger than the value T.sub.1. If the gradient value is equal to or greater than the threshold value T.sub.2, the output of the comparator C.sub.2 activates the processor P.sub.2. The processor P.sub.2, in response to the output m(i,j) of the median filter 14, the output f(i,j) of the average filter 16, the absolute value .vertline.G(i,j).vertline. of the gradient and the noise level as indicated by the output of the noise monitor 22, generates an output sample P.sub.f (i,j) according to the following relation: ##EQU1## .alpha.(.sigma.) is a noise factor which is a function of the SNR value .sigma.. T(.sigma.) is the threshold value T.sub.2 which is also a function of the SNR value .sigma..

The processor P.sub.2 for carrying out the function of the equation 1 is shown in FIG. 7. The absolute value .vertline.G(i,j).vertline. of the gradient is scaled by the noise factor .alpha.(.sigma.) by a multiplier 32. The output is then multiplied by the median value m(i,j) by a multiplier 34. The output of the multiplier 34 is added to the average sample value f(i,j) by an adder 36 and applied to one input of a divider circuit 38. The output of the multiplier 32 is increased by one by an "add-one" circuit 40 and used as the divisor for the divider circuit 38. The output of the divider circuit 38 corresponds to the value P.sub.f (i,j) and is gated to the common terminal 26 by a gate 42 in response to the control signal C.sub.2 from the comparator 30 when the gradient magnitude is greater than T.sub.1 but less than T.sub.2.

If the absolute value .vertline.G(i,j).vertline. of the gradient is greater than the threshold T.sub.2 but less than the threshold T.sub.3, a comparator 44 turns on a control signal C.sub.3, activating the processor P.sub.3. The processor P.sub.3 is merely a gate for gating the median value m(i,j) to the common terminal 26.

If the absolute value .vertline.G(i,j).vertline. of the gradient is greater than the threshold value T.sub.3 but less than a threshold value T.sub.4, a comparator 46 turns on a control signal C.sub.4 which in turn activates the processor P.sub.4 having the following characteristic:

The processor P.sub.3 is shown in FIG. 8. The processor P.sub.4 again multiplies the absolute value .vertline.G(i,j).vertline. of the gradient with the noise factor .beta.(.sigma.) by a multiplier 47. The output of the multiplier 47 is added to the median value m(i,j) by an adder 48 to produce the desired edge enhanced value P.sub.e (i,j). This modified sample value is gated to the common terminal 26 through a gate 50 in response to the control signal C.sub.4.

Finally, if the absolute value .vertline.G(i,j).vertline. of the gradient is greater than T.sub.4, a maximum limit is set to prevent any excessive enhancement. The processor P.sub.5 modifies the median value m.sub.i,j by adding or subtracting a constant value C and applying the output to the common terminal 26 in response to a control signal C.sub.5 from the comparator 46.

The characteristics of the circuit of FIG. 2 are shown in FIG. 3. It will be seen that the characteristic of FIG. 3 is a close approximation to the desired characteristic of FIG. 1. From FIG. 3 it will be seen that if the magnitude of the gradient is below T.sub.1, the output sample corresponding to the average value f(i,j). Between T.sub.1 and T.sub.2, the adaptive noise filter sample value P.sub.f is used, which reduces the degree of noise filtering as the magnitude of the gradient increases. Once the magnitude of the gradient exceeds the threshold T.sub.2, the output sample of the median filter is used as the output sample, resulting in less noise filtering but improved edge detail. With the magnitude of the gradient between the range of T.sub.3 and T.sub.4, the edge enhanced sample P.sub.e, which varies with the change in the magnitude of the gradient, is used as the output sample. Finally, a maximum degree of edge enhancement is provided by using the m(i,j).+-.C as the sample output. FIG. 3 shows two different sets of thresholds, T and T' , corresponding to two different levels of noise, .sigma..sub.1 and .sigma..sub.2.

A preferred embodiment is shown in FIG. 4 in which a single processor P.sub.6 provides a modified sample value that provides both noise filtering and edge enhancement of the resulting image. The circuit is otherwise the same as the arrangement in FIG. 2 except that the threshold memory provides only two threshold values, T.sub.1 and T.sub.4. Processor P.sub.6 is activated between the full gradient range from T.sub.1 to T.sub.4. The processor P.sub.6 provides a combined noise filtering and edge enhancement according to the following relation: ##EQU2## For a fixed SNR, the absolute value .vertline.G(i,j).vertline. of the gradient acts as a non-linear adaptive gain factor which changes P(i,j) from P.sub.f (i,j) to P.sub.e (i,j) smoothly as the gradient increases. Since T(.sigma.) increases as .sigma. increases, this gain factor is de-emphasized as the input picture gets noisier. FIG. 9 shows a block diagram of processor P.sub.6.

The performance of the circuit depends largely on the size of the averaging and the median filters. For example, averaging a block of n.times.n pixels gives an improvement in SNR of 20 log.sub.10 (n). The larger that n is, the better is the noise filtering. With n=3, that is, averaging a 3.times.3 matrix of pixels, the noise filtering is improved by about 9.5 db. Performance of the median filter depends on both the size and the shape of the block of pixels used. Increasing the number increases the noise filtering but produces a greater loss of spatial resolution. Five pixels arranged in a cross has been found to produce satisfactory results.

One embodiment of a median filter is shown in the block diagram of FIG. 6. The five samples from the cross-shaped block, designated x.sub.0, x.sub.1, x.sub.2, x.sub.3 and x.sub.4, are applied to a group of comparator circuits 50 from the store 12. Ten comparison circuits are required to compare each of the five samples with each of the other samples. Each comparator provides a binary coded output in which 0 represents less than and 1 represents greater than or equal. The ten binary bits from the ten comparators are applied to a decision logic circuit 52 which determines which of the five input samples is in the middle of the five samples. The decision logic sets a switch to connect the median value of the input samples to an output register 56. The concept of a median filter has been described in the literature. See, for example, "Nonlinear (Nonsuperposable) Methods for Smoothing Data" by J. W. Tukey, Comp. Rec., 1974 EASCON, p. 673.

The averaging filter, in its simplest form, sums a group of the stored points, preferably arranged in a block, and divides by the number of points to generate an average value. An alternative averaging circuit is described in copending application Ser. No. 133,606 filed Mar. 24, 1980, entitled "Adaptive Enhancement of Signal-to-Noise Ratio in Television Imagery" by Curtis May and assigned to the same assignee as the present application.

From the above description it will be seen that a circuit is provided which provides both noise filtering and edge enhancement in the same image. The degree of noise filtering and edge enhancement adapts locally to image content and globally to the SNR of the image. As a result, degradation in spatial resolution or the introduction of artifacts such as "contouring" and "salt-and-pepper" effects are minimized. The averaging filter and the median filter can be tailored depending on application areas, giving the system considerable flexibility. Minimum storage is required and real-time operation for a TV system is no problem.