Signal Processing Method and Apparatus for Producing Interpolated Chrominance Values in a Sampled Color Image Signal

The invention relates to signal processing methods and apparatus for processing a sampled color image signal of the type having a highly sampled color component and a less highly sampled color component, and more particularly to such signal processing methods for providing interpolated values between sampled values of the less highly sampled color component.

Some color image sensing apparatus produce a sampled color image signal by sampling one color component of an image at a higher spatial sampling frequency than other color components. The more highly sampled colored component usually corresponds to the portion of the spectrum to which the human eye is most sensitive to image detail, and is often called the luminance component. The term "luminance component" as used herein refers to the most highly sampled color component, whether it be green, white, or some other color.

The less highly sampled color components, are often referred to as chrominance components and may comprise, for example, red and blue, or cyan and yellow. The term "chrominance component" as used herein refers to a less highly sampled color component of an image regardless of the particular color.

In some color image sensing apparatus of this type, one image sensor samples the luminance component of the image, and another image sensor samples two chrominance components. The image sensing elements on the image sensor that samples the chrominance components alternate between one color and the other.

In other color image sensing apparatus, a single image sensor is employed to sense the luminance and two chrominance components of the image. The image sensing elements of the image sensor alternate between elements for sampling luminance and elements for sampling one and then the other of the chrominance components.

When reconstructing an image from the sampled color image signals produced by these color image sensors, values of the luminance and chrominance components are provided for each sample location. This is generally accomplished by using some form of linear interpolation between sample values of the less highly sampled color components. Reconstruction of the color image by linear interpolation of the chrominance components results in the appearance of colored fringes in areas of image detail, due to a deviation of the interpolated values from the actual color values existing in the original image.

It is known to employ signal processing to reduce the appearance of these colored fringes in the reconstructed images. In U.S. Pat. No. 4,176,373, issued Nov. 27, 1979, Dillon and Bayer disclose analog signal processing apparatus for interpolating values between the sampled luminance and chrominance values in a sampled color image signal produced by a single image sensor having a checkerboard pattern of luminance (green) image sensing elements interspersed with chrominance component (red and blue) image sensing elements. The apparatus performs a linear interpolation between chrominance component samples, then adds a high spatial frequency portion of the luminance signal to the interpolated chrominance component signals. Adding the high spatial frequency portion of the luminance signal to the interpolated chrominance component signals reduces the appearance of color fringes in areas of image detail by driving the high spatial frequency portion of the image signal towards a neutral color. Although the signal processing disclosed by Dillon and Bayer is effective to reduce the appearance of color fringing in the detailed areas of the image, the method achieves this object at the expense of altering the sampled values of the chrominance component signals of the image signal. As a result, some hue shifts are introduced in areas of image detail even as the appearance of color fringing is reduced.

The object of the present invention is to provide a signal processing method for reducing the appearance of colored fringes in areas of fine image detail without introducing unwanted hue shifts.

The above-noted object is achieved by producing neighboring hue values representing a hue component of the image at neighboring chrominance component sample locations as a function of a luminance value and a chrominance value at the neighboring locations; producing an interpolated hue value representing the hue component of the image at an interpolation location as a function of the neighboring hue values; and producing an interpolated chrominance value as a function of the interpolated hue value and a luminance value at the interpolation location.

As used herein, the term "hue value" refers to a quantity relating the value of a less highly sampled color (chrominance component) to the value of the more highly sampled color (luminance component). By interpolating the hue value and deriving the interpolated chrominance value from the interpolated hue value, hues are allowed to change only gradually, thereby reducing the appearance of colored fringes in areas of image detail without introducing unwanted hue shifts.

In one embodiment of the invention, the sampled color image signal luminance and chrominance values are linear exposure values, the hue values are formed as the ratio of the sampled chrominance value to the luminance value at the respective chrominance component sample locations, and the interpolated chrominance value is formed as the product of the interpolated hue value and the luminance value at the interpolation location.

In another embodiment of the invention, the sampled color image signal luminance and chrominance values are logarithmic density values, the hue value is formed as the difference between the sampled chrominance values and the luminance values at the respective chrominance component sample locations, and the interpolated chrominance value is formed as the sum of the interpolated hue value and the luminance value at the interpolation location.

Modes of carrying out the invention are described below, with reference to the drawings wherein:

FIG. 1 schematically shows apparatus having two image sensors for producing, processing, and displaying a sampled color image signal;

FIG. 2 is a diagram useful in describing the signal processing method applied to the sampled color image signal produced by the image sensing apparatus shown in FIG. 1;

FIG. 3 is a block diagram illustrating the function performed by the signal processing method of the present invention;

FIG. 4 schematicaly shows apparatus having a single image sensor for producing a sampled color image signal;

FIG. 5 is a diagram useful in describing the signal processing method applied to the signal produced by the image sensing apparatus shown in FIG. 4;

FIG. 6 is a diagram showing an alternative color sampling pattern for the single image sensor shown in FIG. 4; and

FIG. 7 is a diagram showing an alternative color sampling pattern wherein 75% of the image sensing elements are luminance component sensing elements .

Apparatus for generating a sampled color image signal, processing the sampled color image signal, and displaying the processed sampled color image signal are shown in FIG. 1. The apparatus includes an optical system having a lens 10, a beam splitter 12 and a mirror 14 for forming an image on a pair of image sensors 16 and 18, such as CCD image sensing arrays. Image sensor 16 senses the luminance component of the image, and is provided with a uniform green color filter 20. Image sensor 18 senses the chrominance components of the image and is provided with a striped color filter array having alternate red and blue vertical stripes labeled R and B in the figure.

The sampled analog luminance signal produced by image sensor 16 is supplied to a digital computer 22 on line 24 through an analog-to-digital converter 26. The sampled analog chrominance component signals produced by image sensor 18 are supplied to the computer 22 on line 28 through an analog-to-digital converter 30.

The digital computer 22 performs the signal processing according to the invention to interpolate chrominance values between sampled chrominance values in the color image signal.

The digital computer 22 includes a central processing unit 32, a frame store 34 for storing the unprocessed and processed digital color image signals and random access and read-only memories RAM and ROM 36 and 38 for storing the control programs for the computer. The digital image signals are supplied to the computer on data bus 40. Processed image signals are supplied to a display device such as a CRT 42 or graphic arts scanner 44 via digital-to-analog converter 46.

Although the elements of the apparatus are shown physically connected in FIG. 1, the elements may be physically separated and the signals carried between the elements via storage media such as magnetic tape or disc. For example, the sampled analog signals from the image sensor may be recorded directly on a video magnetic tape recorder and later supplied to the computer for signal processing. Similarly, the processed sampled image signals may be recorded for later display. Although the invention was reduced to practice using a general purpose digital computer, it will be obvious to one skilled in the art that a programmed microprocessor or custom-designed integrated circuit can be employed to practice the invention.

The signal processing method performed by computer 22 will now be described with reference to FIG. 2. FIG. 2 shws a portion of a horizontal line of image sensing locations labeled G.sub.1 -G.sub.5 on luminance image sensor 16, and a portion of the corresponding horizontal line on the chrominance image sensor 18. The purpose of the signal processing method is to provide interpolated chrominance values (e.g. B.sub.1, R.sub.2, B.sub.3, etc.) in such a manner that color fringes are reduced in areas of image detail without introducing hue shifts in these areas. This requires that image intensity information be faithfully reproduced while only smooth hue changes are allowed from one sampling location to the next.

In an image of uniform hue, the values of the luminance (G) and one of the chrominance components (for example the red component R) at one sample location (Rx, Gx) are related to the values (Ry, Gy) at another location as follows:

in exposure space. If Ry represents an unknown chrominance value at an interpolation location, from equation (1),

where Rx/Gx represents the measured value of the hue component at another sample location.

In an image that does not have a uniform hue, as in a normal color image, smoothly changing hues are assured by interpolating the hue values between neighboring chrominance component sample locations. Thus, the interpolated chrominance value R' becomes:

where the primed hue value represents an interpolated hue value between neighboring chrominance component sample locations, and the luminance value G is the luminance value at the interpolation location.

This signal processing method is applied to the signal produced by the apparatus shown in FIGS. 1 and 2, by employing linear interpolation between the hue values at neighboring chrominance component sample locations. Interpolated chrominance values in exposure space are obtained as follows: ##EQU1## and similarly for the other chrominance value ##EQU2## Alternatively, the hue values can be interpolated in density space. In density space, equation (3) becomes:

Where the primed quantity in parentheses indicates an interpolated hue value between neighboring chrominance component sample locations. Applying linear interpolation to the signals represented in FIG. 2, the interpolated chrominance values in density space are obtained as follows:

and similarly for the other chrominance values,

It should be noted that equations (7) and (8) will provide slightly different results than will equations (4) and (5) since log [(A+B)/2] is not exactly the same as (log A+log B)/2, but is merely an approximation thereof.

FIG. 3 is a block diagram illustrating the essential functions that are performed by the digital computer 22 to accomplish the signal processing method of the present invention. Hue value signal producing means 50 receives the sampled luminance value G and the sampled chrominance values R, B and produces hue values H.sub.R, H.sub.B, by dividing the chrominance values by the luminance values as described above. Alternatively, in performing the operations in density space, the logs of the luminance values are subtracted from the logs of the chrominance values as described above.

Interpolation means 52 receives the hue values H.sub.R, H.sub.B and produces interpolated hue values H.sub.R ', H.sub.B ' representing the hue values at the interpolation locations. As disclosed above, the interpolation performed is preferably a linear interpolation. Alternatively, higher order interpolation may be performed. Means 54 for producing interpolated chrominance values receives the luminance value G and the interpolated hue values H.sub.R ', H.sub.B ' and produces the interpolated chrominance values R', B' by multiplying the luminance value by the interpolated hue value as described above.

Alternatively, in density space, the log of the luminance value G is added to the log of the interpolated hue values H.sub.R ', H.sub.B ' to produce the interpolated chrominance values R', B'.

In the previous example, the luminance values at every location were actual values measured at each sample location by image sensor 16. FIG. 4 shows an example of a single-chip, color image sensor where the luminance component sampling locations in the horizontal direction are separated by chrominance component sampling locations. The single-chip color image sensor 16' is provided with a vertically striped color filter array having the alternating pattern green, red, green, blue (G,R,G,B) of color filters. Other elements of the apparatus for processing and displaying the signal are the same as those shown in FIG. 1. Elements similar to those in FIG. 1 are numbered with primes (').

When the signal processing method is applied to the signals produced by the image sensor 16', luminance values (G) for the chrominance component sampling locations (R,B) are supplied by an appropriate form of interpolation. FIG. 5 shows a portion of one line of image sensing locations on the image sensor 16'. The interpolated luminance value at the chrominance component sampling location R.sub.2 could be generated, for example, by linear interpolation as follows:

where the prime indicates that the luminance value is an interpolated value. The other luminance values are obtained in a similar manner.

Using the interpolated luminance values, the signal processing method of the present invention is applied to the signal in exposure space to provide interpolated chrominance values as follows: ##EQU3## where the primed luminance values (G'.sub.n) represent interpolated luminance values. The blue chrominance values are generated in a similar manner. In the block diagram of FIG. 3, an interpolation means 56 is shown in phantom. This interpolation means provides the interpolated luminance values G'.sub.n.

In density space, the signal processing method is applied as follows to generate red chrominance values:

The blue chrominance values are generated in a similar fashion.

Another well known filter array pattern for a single-chip color image sensor is the checkerboard pattern. One checkerboard pattern, often called a Bayer array after the inventor, is described in U.S. Pat. No. 3,971,065, issued July 20, 1976, to B. E. Bayer. A portion of a Bayer array pattern is shown in FIG. 6 where some of the sample locations are numbered for purposes of the following description.

In the Bayer array, each chrominance component sampling location is surrounded by four immediately adjacent luminance sampling locations. One appropriate way of generating interpolated luminance values is to average these four nearest neighbors as follows: ##EQU4## where the prime indicates are interpolated value, and G.sub.n are the values of the nearest neighbors.

When applying the signal processing method of the present invention to the signals produced by an image sensor having a Bayer array sampling pattern, there are three different sampling configurations to consider. In the first configuration, indicated by the region labeled 58 in FIG. 6, the measured chrominance values lie on either side of the interpolation location. In the second configuration, labeled 60 in FIG. 6, the measured chrominance values lie above and below the interpolation location, and in the third configuration, indicated by the area labeled 62 in FIG. 6, the measured chrominance values lie at the four corners of the interpolation location.

In exposure space, the signal processing method is applied to the signal produced by the Bayer array as follows.

For the first configuration: ##EQU5## for the second configuration, ##EQU6## and for the third configuration, ##EQU7## where the primed luminance values indicate interpolated values. The interpolated values of the other chrominance component (e.g. the blue component) are produced in a similar manner.

In density space, the signal processing method is applied as follows:

and the other chrominance values (e.g. blue component) are determined in a similar manner.

In another two-dimensional array color filter pattern, 75% of the image sampling locations are luminance sampling locations, and each chrominance sampling location is completely surrounded by luminance sampling locations. A portion of such a 75% luminance component sampling array 64 is shown in FIG. 7 where some of the sample locations 66 are numbered for purposes of the following description. The signal processing method of the present invention is applied to the signals produced by the array shown in FIG. 7 as follows.

First, interpolated luminance values are determined at positions 6 and 16, for example by taking the average of the eight luminance values surrounding each chrominance component sample location.

In exposure space, a red hue value HR is determined for positions 6 and 16 as follows:

where G'.sub.6 is the interpolated luminance value, and similarly for HR.sub.16.

The hue values for the sample locations between the chrominance component sample locations are determined for example by linear interpolation as follows:

Finally, the interpolated chrominance values are found by multiplying the interpolated hue values by the luminance values as follows:

The values of the other chrominance component (blue) are reconstructed in a similar manner.

In density space, the signal processing method is applied as follows:

The values of the other chrominance component are reconstructed in a similar manner.

By employing this method of interpolation, the red and blue chrominance values are reconstructed without noticeable degradation.

A variety of photographic images were sampled to produce sampled color image signals, and the signal processing method according to the present invention was applied to the sampled colored image signals. The images reproduced from the processed sampled color image signals were found to have reduced color fringing in areas of image detail and were free from any noticeable hue shifts in these areas. A definite improvement was noted over images reproduced from signals processed according to prior art methods. Although the interpolated chrominance values produced when the image signals were processed in exposure space varied slightly from the values produced when the image signals were processed in density space, the differences in the appearance of the images reproduced from the processed signals were not significant enough so that one signal processing space could be determined to be preferable over the other.

Although the particular manner of interpolating the luminance values is not an essential feature of the general inventive concept, when processing a signal from a single chip color image sensor having a checkerboard-type sampling pattern like the Bayer array, best results have been achieved with a pattern recognition interpolation method of the type disclosed in copending patent application Ser. No. 649,001 entitled "Signal Processing Method and Apparatus for Sampled Image Signals" filed Sept. 10, 1984, by the present inventor. In the pattern recognition interpolation method, a plurality of interpolation routines appropriate for producing interpolated signal values that complete a respective plurality of known geometrical image features are employed. A neighborhood of sampled luminance values is examined to determine which geometrical image feature is represented by the local neighborhood of sample values, and the appropriate interpolation routine is used to produce the interpolated luminance value.

Although the invention has been described as being performed by signal processing apparatus having a full frame store, it will be apparent to one skilled in the art that the signal processing apparatus needs to store only a few lines of the image signal at one time in order to perform the signal processing method.

Furthermore, although the invention was reduced to practice using a general purpose digital computer, it will be apparent to one skilled in the art, that a programmed microprocessor, or custom-designed integrated circuits can be used to practice the invention.

The signal processing method according to the present invention has the advantage of reducing color fringing without introducing hue shifts in areas of image detail in an image reproduced from the processed sampled color image signal. The invention is useful in amateur or professional video and still imaging apparatus employing sample-type color image sensors, and in graphic arts apparatus for reproducing images from sampled color image signals.