Method of determining missing color values for pixels in a color filter array

[0002] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0003] 1. Field of the Invention

[0004] The present invention relates generally to computer graphics and more specifically to the manipulation of color filter arrays used to represent color images captured by a digital camera.

[0005] 2. Description of Related Art

[0006] Digital cameras that capture color images are rapidly becoming popular. These digital cameras are used for still photography and full motion video recording. The images captured by the camera may be transferred to a computer system and displayed on a computer monitor for viewing by the user. The increasing power of microprocessors permits many sophisticated and computationally intensive image processing operations to be performed by software executing on the computer system rather than by special purpose circuitry in the camera. Image processing methods implemented in software allow presentation of higher quality images to the user while lowering the cost of the digital camera, because the specialized circuitry for performing various image processing operations is no longer needed.

[0007] Most commercial digital cameras use a single-chip color filter array to capture images. The color filter array is located in the focal plane of the camera in the same place that film is located in a traditional film camera. A single-chip color filter array is a two-dimensional grid of picture elements, called pixels, covered by color filters. Each pixel is typically implemented through the use of a charge-coupled device (CCD) or CMOS circuit. To minimize the cost of the color filter array (and therefore the cost of the camera), only one of the red, green or blue colors is sensed at each pixel. Hence, information for two colors of a red, green, and blue (RGB) color combination is missing at each pixel of the sensed image. The colors that are sensed are arranged in a predetermined pattern in the color filter array so that the entire image may be represented to the user in an aesthetically pleasing manner, despite the missing color information. One pattern that is useful for representing such images is disclosed in U.S. Pat. No. 3,971,065, issued to B. E. Bayer. There are also many other patterns that can be used to represent color images.

[0008] An example of a color filter array with a Bayer pattern is shown in FIG. 1. Pixels are represented by squares in the grid of FIG. 1. Each pixel includes an electronic sensor which measures the light falling on it. Those pixels which have a red filter measure red light and are represented by an R in the pattern, those pixels which have a green filter measure green light and are represented by a G in the pattern, and those pixels which have a blue filter measure blue light and are represented by a B in the pattern. The Bayer pattern is replicated throughout the entire color filter array in both the horizontal and vertical directions.

[0009] A problem with this method of capturing color images is that only a single color is sensed and stored at each pixel in the color filter array, although three colors (red, green and blue) are needed to reconstruct the true color at each pixel. Typically, each pixel is represented by eight bits for a color. Hence, the amount of color information being collected at each pixel in the captured image is only eight bits, but 24 bits are needed for the complete set of red, green and blue color values for the pixel. Cameras which use a single color filter array must determine the missing color values at each pixel. For example, values of missing red and blue colors must be found for a pixel with a green filter.

[0010] Existing systems use color values of adjacent pixels to interpolate the missing color values. The simplest approach is to treat each color plane separately. A color plane is the representation of a single color for the whole color filter array. With the separate color plane method, missing red, green, and blue values are determined by reference to neighboring red, green, and blue pixels, respectively.

[0011] Some methods for computing missing color values use color values from other color planes. In most cases, values from the green plane are combined with values of adjacent red pixels to determine missing red values, and values from the green plane are combined with values of adjacent blue pixels to determine missing blue values. Generally, color array patterns (such as the Bayer pattern) use more green pixels than red or blue pixels. Using more green pixels than red and blue pixels improves luminance resolution or sharpness of the image without significantly degrading color quality. A method disclosed by P. D. Dillon, D. M. Lew, and F. G. Kaspar in “Color Imaging System Using A Single CCD Array”, IEEE Trans. Electron Devices, vol. ED-32, pp. 102-107, 1978, combines green with other colors, subtracts a purposely blurred green value from an original green value, and adds the difference to all three color planes. This method is an image sharpening technique also known as unsharp masking as discussed by R. C. Gonzalez and R. E. Woods in “Digital Image Processing”, published by Addison-Wesley of Reading, Mass., 1992. Another method, taught by J. A. Weldy in “Optimized Design For A Single-Sensor Color Electron Camera System”, Proceedings of SPIE, vol. 107, vol. 1071, pp. 300-308, 1989, linearly interpolates missing green values so that there are green values at all pixel locations, and then uses the difference between red and green at red pixel locations to interpolate missing red values, and the difference between blue and green at blue pixel locations to interpolate missing blue values.

[0012] These methods for interpolating missing color values suffer from two problems. Interpolation results in blurred images, and inaccurate interpolation of red and blue values tends to produce aliasing artifacts at object boundaries within the image. Object boundaries may appear as red or blue instead of their true colors. Most cameras reduce these aliasing artifacts by optically blurring the image before it reaches the sensor. This technique is discussed by J. E. Grievenkamp in “Color Dependent Optical Prefilter For The Suppression Of Aliasing”, Applied Optics, vol. 29, pp. 676-684, February 1990. A new method is needed which overcomes these problems by forming a compromise between blurring and aliasing reduction methods to provide excellent quality, true color images in a cost-efficient manner.

[0013] An embodiment of the present invention is a method for generating missing color values for pixels in an array, each pixel having one of three color values. The method includes the steps of determining values for all missing second color values for pixels in the array by interpolation of known second color values adjacent along diagonal lines from the missing second color values, and determining values for all missing first and third color values for the pixels from a sum of an interpolation term and a luminance correction term based on differences between second color values of adjacent pixels.

[0014] The features and advantages of the present invention will become apparent from the following detailed description of the present invention in which:

[0015] FIG. 1 is a diagram of a Bayer pattern (Prior Art).

[0016] FIG. 2 is a block diagram of a digital camera having a color filter image stored in a memory.

[0017] FIG. 3 is a high level flow diagram of the processing steps of the present invention.

[0018] FIGS. 4a-bare diagrams of sample Bayer patterns of pixels showing diagonal lines of processing according to the present invention.

[0019] FIG. 5 is a flow diagram showing the steps of determining missing green pixels.

[0020] FIGS. 6a-dare diagrams of possible color patterns to be processed by the present invention.

[0021] FIG. 7 is a diagram of a detailed example of a Bayer pattern as processed by the present invention.

[0022] In a single-sensor system, the three colors of information (typically red, green, and blue) cannot be sampled by each pixel of the sensor. However, for an aesthetically pleasing display of the current scene, three colors of information should be displayed at every pixel. Therefore, image reconstruction processing is used to create three color information at every pixel of the display. The present invention is a method of image reconstruction processing for determining missing color values for a color filter array which results in an excellent quality image without requiring purposely blurring of the image by the camera. It effectively allows for improving or increasing the pixel depth of a displayed image to provide a more accurate representation of a sensed image.

[0023] Embodiments of the present invention include methods for “upsampling” 8-bit color filter array values into 24-bit color values for a color image captured by a digital camera. However, the present invention can be used in any computer application where individual red, green, and blue color values are represented in an array and the depth of the pixels needs to be improved. Typically, the depth improvement will be from 8 bits to 24 bits. However, other depth improvement operations (for example, from 16 bits to 48 bits) are also within the scope and spirit of the present invention. The present invention also is applicable to color schemes using colors other than red, green, and blue. Furthermore, the present invention is discussed below in terms of an implementation in software, however, an implementation in hardware is also feasible.

[0024] FIG. 2 is a block diagram of a digital camera having a color filter image stored in a memory. Digital Camera 10 uses Lens 12 to accept light from the environment around the camera. The light passes through and is filtered by Color Filter Array 13 and is sensed by electronics Sensors 14. The resulting Color Filter Image 16 is stored in Memory 18. The Sensors may be CCDs, CMOS arrays, or other light sensing devices. The Color Filter Image is a representation of the image sensed by the camera. Memory 18 is coupled to Interfaces (I/F) 20, 22 to connect Digital Camera 10 to a Computer System 24 for displaying the captured images. Computer System 24 is a general purpose computer such as a personal computer (PC) or workstation having a Processor 26, Memory 28, and a Display 30. The Color Filter Image 16 generated by Digital Camera 10 is stored as Color Filter Image 16′ in Memory 28 in Computer System 24 and manipulated as a result of instructions being executed by Processor 26 according to the present invention.

[0025] FIG. 3 is a high level flow diagram of the processing steps of the present invention. The Color Filter Image 16′ has a plurality of individual pixels, each pixel being one of the red, green, and blue colors, with the other two colors being missing at each pixel. At Step 100, color values for all missing green pixels are determined by interpolation based on known green pixels adjacent along diagonal lines. At Step 102, the color values for all missing red and blue pixels are determined from the sum of an interpolation term and a luminance correction term based on the differences between the values of neighboring green pixels.

[0026] Missing green pixel values are computed at Step 100 using a cubic B-spline digital filter as disclosed by M. Unser, A. Aldroubi, and M. Eden in “Fast B-spline Transforms For Continuous Image Representation And Interpolation”, IEEE Trans. Pattern Anal. Machine Intell., vol. 13, pp. 277-285, 1991. In an illustrative embodiment of the present invention, green pixels are sampled along diagonals instead of the conventional approach along rows and columns in the Color Filter Array. FIGS. 4aand 4B are diagrams of sample Bayer patterns of pixels showing diagonal lines of processing according to the present invention. The diagonal lines of FIG. 4aare lines angled diagonally connecting green pixels. The diagonal lines of FIG. 4bare lines angled diagonally (perpendicular to the first type) connecting alternating red and blue pixels. FIG. 5 is a flow diagram showing the steps of determining missing green pixels corresponding to Step 100 of FIG. 3. First, values for green at the corners of green pixels which lie on the diagonal lines of FIG. 4aare determined at Step 110 using the cubic B-spline filter oriented along the diagonal lines. Next, at Step 112, values for green at the centers of pixels labeled red and blue are determined by orienting the cubic B-spline filter along the diagonal lines of FIG. 4band the interpolated green values at the corners of pixels determined in Step 110 are sampled.

[0027] The determination of the missing red and blue values of Step 102 of FIG. 3 can be illustrated by several examples using FIGS. 4aand 4bto indicate pixel positions in the Bayer pattern. The missing red value for the pixel at the first row of the second column, denoted R₁₂is computed according to the present invention as (R₁₁+R₁₃)/2+(G₁₂−(G₁₁+G₁₃)/2). The first term of the equation (referencing the adjacent red values) is called the interpolation term. This term is the linear interpolation of the neighboring red values. The second term (referencing the green values) is called the luminance correction term. This term represents the difference between green at the location of the interpolated value of red and green at the locations of measured red values used in the interpolation calculation. The luminance correction terms use results obtained from the green interpolation calculations of Step 100 of FIG. 3. The missing red value for the pixel at the second row of the first column, denoted R₂₁, is computed as (R₁₁+R₃₁)/2+(G₂₁−(G₁₁+G₃₁)/2). The missing red value for the pixel at the second row of the second column, denoted R₂₂, is computed as (R₁₁+R₁₃+R₃₁+R₃₃)/4+(G₂₂−(G₁₁+G₁₃+G₃₁+G₃₃)/4). Note that red values bordering edges of the image are determined by reference to only two adjacent red pixels; otherwise four diagonally adjacent red pixels are used for the interpolation term. In a similar manner, missing blue values are computed. The missing blue value for the pixel at the second row of the third column, denoted B₂₃, is computed according to the present invention as (B₂₂+B₂₄)/2+(G₂₃−(G₂₂+G₂₄)/2). The missing blue value for the pixel at the third row of the second column, denoted B₃₂, is computed as (B₂₂+B₄₂)/2+(G₃₂−(G₂₂+G₄₂)/2). The missing blue value for the pixel at the third row of the third column, denoted B₃₃is computed as (B₂₂+B₂₄+B₄₂+B₄₄)/4+(G₂₂−(G₁₁+G₁₃+G₃₁+G₃₃)/4). Note that blue values bordering edges of the image are determined by reference to only two adjacent blue pixels; otherwise four diagonally adjacent blue pixels are used for the interpolation term.

[0028] FIGS. 6a-dare diagrams of possible color patterns to be processed by the present invention. All are variations of Bayer patterns. FIG. 6ashows a pattern called “Bayer Pattern 1 Red Top.” FIG. 6bshows a pattern called “Bayer Pattern 2 Red Top.” FIG. 6cshows a pattern called “Bayer Pattern 1 Blue Top.” FIG. 6dshows a pattern called “Bayer Pattern 2 Blue Top.” Note the variations between the patterns. The present invention handles each of these patterns equally well.

[0029] FIG. 7 is a diagram of a detailed example of a Bayer pattern as processed by the present invention. One of the four possible Bayer patterns (the Bayer Pattern 1 Red Top pattern) is shown in FIG. 7. First, the green values for pixels marked as red (R) and blue (B) are computed by using diagonal splines whenever possible. Pixels along the boundaries of the sensed image must be handled differently than the pixels in the interior of the sensed image. It is not possible to use splines along the sensed image boundaries. Instead, linear interpolation is used along the boundaries. Linear interpolation is used to set an interpolated green value at a red or blue pixel equal to the average of the neighboring green pixels. For example, green values for two of the corners of the pattern of FIG. 7 are computed as G_0,0=(G_0,1+G_1,0)/2 and G_17,17=(G_16,17+G_17,16)/2. Those pixels with two green neighbors use the general form G_i,j=(G_i−1,j+G_i,j−1)/2. Other pixels on the top and bottom rows, and the first and last columns, have three green neighbors. Therefore, for the top row, green values are computed as G_i,j=(G_i,j−1+G_i,j+1+G_i+1,j)/3. For the bottom row, green values are computed as G_i,j=(G_i,j−1+G_i,j+1+G_i−1,j)/3. For the left column, green values are computed as G_i,j=(G_i−1,j+G_i+1,j+G_i,j+1)/3. For the right column, green values are computed as G_i,j=(G_i−1,j+G_i+1,j+G_i,j−1)/3. All other linearly interpolate green values are computed using the equation G_i,j=(G_i−1,j−1+G_i−1,j+1+G_i+1,j−1+G_i+1,j+1)/4.

[0030] The diagonal splines are computed by finding temporary values with the spline filter along one diagonal line, and then using those temporary values to find the final result with the spline filter along the other diagonal line. FIG. 7 shows an example of an eight tap filter with its top at G_0,7and bottom at G_7,14. The filter result is L_0,3, halfway between the fourth and fifth pixel along the diagonal. L_0,3represents an interpolated value in the green plane. It is computed as follows: L_0,3=C₀G_0,7+C₁G_1,8+C₂G_2,9+C₃G_3,10+C₄G_4,11+C₅G_5,12+C₆G_6,13+C₇G_7,14. The general form for a filter result is as follows:

[0031] L_i,k=the sum from m=0 to n−1 (C_mG_i,j+m) where n is the filter length, C_mis the filter coefficient and k=floor (j/2).

[0032] For example, in FIG. 7, i=0, j=7, n=8, and k=floor (7/2)=floor (3.5)=3. Once the L values are found, green values at red and blue pixel locations can be computed. In FIG. 7, the filter whose top is at L_0,3and whose bottom is at L_7,0computes G_7,7at the location of the circled B as G_7,7=C₀L_0,3+C₁L_1,3+C₂L_2,2+C₃L_3,2+C₄L_4,1+C₅L_5,1+C₆L_6,0+C₇L_7,0. The general form is as follows:

[0033] G_i,j=the sum from m=0 to n−1 (C_mL_i−n+1,k) where k=floor ((j−m)/2)

[0034] Note that in the example, all of the L_i,jvalues to the upper left of the diagonal line whose top is L_0,3are never used to compute green values because there are too few values (less than n) of L_i,jalong the diagonal.

[0035] The method for computing red and blue values according to the present invention can be generalized as follows. Interpolation of a red value at a location where the pixel in the original Bayer pattern (i.e., the sensed image) is green in a row of alternating reds is computed as R_i,j=(R_i,j+R_i,j+1)/2+(G_i,j−(R_i,j−1+R_i,j+1)/2). Interpolation of a red value at a location where the pixel in the original Bayer pattern is green in a row of alternating blues is computed as R_i,j=(R_i−1,j+R_i+1,j)/2+(G_i,j−(R_i−1,j+R_i+1,j)/2). Interpolation of a red value at a location where the pixel in the original Bayer pattern is blue is computed as R_i,j=(R_i−1,j−1+R_i−1,j+1+R_i+1,j−1+R_i+1,j+1)/4+(G_i,j−(G_i−1,j−1+G_i−1,j+1+G_i+1,j−1+G_i+1,j+1)/4). Interpolation of a blue value at a location where the pixel in the original Bayer pattern is green in a row of alternating blues is computed as B_i,j=(B_i,j−1+B_i,j+1)/2+(G_i,j−(B_i,j−1+B_i,j+1)/2). Interpolation of a blue value at a location where the pixel in the original Bayer pattern is green in a row of alternating reds is computed as B_i,j=(B_i−1,j+B_i+1,j)/2+(G_i,j−(G_i−1,j+G_i+1,j)/2). Interpolation of a blue value at a location where the pixel in the original Bayer pattern is red is computed as B_i,j=(B_i−1,j−1+B_i−1,j+1+B_i+1,j−1+B_i+1,j+1)/4+(G_i,j−(G_i−1,j−1+G_i−1,j+1+G_i+1,j−1+G_i+1,j+1)/4).

[0036] An embodiment of the present invention is represented below in the form of detailed pseudo-code in Appendix A.

[0037] Cubic spline interpolation produces better images than conventional methods by finding values with continuous and equal first derivatives and equal second derivatives at image points. Green pixels do not form a square grid along rows and columns of the image, but they do form a square grid along diagonals. Green pixels are evenly spaced and they are most closely spaced along diagonals. Cubic splines which interpolate halfway between two grid points, as in the present invention, can be computed with a digital filter which has a low computational complexity.

[0038] The luminance correction term corrects red and blue interpolation errors. Changes in green represent changes in luminance, which affects overall image sharpness. Interpolation of red and blue tends to produce a smooth transition between points in the image. The luminance correction term correlates red and blue color values with green so that if higher resolution green has a sharp transition, red and blue values are modified to take into account this transition. The luminance correction term increases image sharpness and reduces aliasing artifacts.

[0039] The combination of the effect of diagonal cubic spline interpolation and luminance correction according to the present invention provides sharp images with limited aliasing artifacts. The present invention also has low computational complexity. Results indicate that the optical blurring required to further suppress aliasing may be provided by focus errors and aberrations in low cost digital cameras alone without the need for additional blurring of the image by software.

[0040] Testing was accomplished by decimating good quality 24-bit color images so that the remaining pixel values are those of an 8-bit Bayer pattern, and then reconstructing the images according to the present invention. Differences between original and reconstructed images were difficult to detect when the original image was pre-filtered with a smoothing filter which removes only 15 percent of the high frequency content, rather than 50 percent as in prior art methods. Use of interpolated green values in the luminance correction term is supported by tests which show that interpolated green planes of images with high frequency content appear identical to the original green planes. The luminance correction increases image sharpness and reduces aliasing in images in which only the cubic spline filter is used. Results using the spline filter on red and blue planes do not differ noticeably from those results produced by using linear interpolation, so linear interpolation is used in the present method because it requires fewer operations. Testing results indicated that the present invention reconstructs images better than other conventional methods known in the art.

[0041] While this invention has been described with reference to illustrative embodiments, this description is not intended to be construed in a limiting sense. Various modifications of the illustrative embodiments, as well as other embodiments of the invention, which are apparent to persons skilled in the art to which the inventions pertains are deemed to lie within the spirit and scope of the invention.

[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]



[Inline image omitted. See PDF.]