Demosaicing for digital imaging device using perceptually uniform color space

[0002] The present application contains subject matter related to a concurrently filed U.S. Patent Application by Bo Tao entitled “DEMOSAICING USING WAVELET FILTERING FOR DIGITAL IMAGING DEVICE”. The related application is identified by application Ser. No. 09/412,351, is assigned to the same assignees, and is hereby incorporated by reference.

[0003] The present invention relates generally to digital video and still cameras, and more specifically, to such digital cameras using a single-channel, solid-state imaging sensor array.

[0004] Solid-state digital video and still cameras are becoming more and more popular. Most of the current digital cameras use a single channel, solid-state photosensor array. For example, most of the digital still cameras on the market now are so-called 1-CCD (single channel charge coupled device) cameras. The solid-state photosensors are CCDs are arranged in an array in a matrix fashion with each CCD defining an element in the matrix called a “pixel”. By placing a mosaiced color filter over the photosensor array, a single channel image is created where each pixel has a single color component. For example for a red-green-blue (RGB) system, a pixel has a single red (R), green (G), or blue (B) color component of a full color image. This forms a single channel mosaiced color image. The single channel mosaiced color image has to be demosaiced subsequently so that a full RGB (three-channel) image is created where each pixel contains the three RGB color components.

[0005] Each single-channel digital camera has its own unique demosaicing method. A number of patents have been filed on demosaicing methods such as U.S. Pat. No. 5,552,827 granted to Maenaka et al. titled “Color Video Camera with a Solid State image Sensing Device” and U.S. Pat. No. 4,716,455 granted to Ozawa et al. titled “Chrominance Signal Interpolation Device for a Color Camera”. However, these demosaicing methods did not take into account the filtering and sampling process performed by the optical lens and the CCD. Thus, they often generate images that are not sharp and/or have aliasing effects.

[0006] Many programs have been published and/or patented on demosaicing. However, currently, demosaicing is done in a perceptually nonuniform color space, such as RGB space. Thus, false color artifacts can appear and the picture sharpness is also limited. These have been long standing problems in the industry.

[0007] The present invention provides a demosaicing process which includes a first color transformation into a perceptually uniform color space for determining luminance and a luminance interpolation for interpolation of the missing luminance component values. The demosaicing process further includes a second color transformation for determining chrominance where the color components are known using the luminance values. A chrominance interpolation is performed to interpolate missing chrominance components and an inverse color transformation transforms the chrominance components into the RGB colors.

[0008] The present invention provides a digital camera having a lens for focusing light from an image through a RGB mosaiced filter on to a solid state photosensor array containing a matrix of CCDs. The CCDs provide a single channel mosaiced image output. A demosaicing process includes a first color transformation for determining luminance and a luminance interpolation for interpolation of the missing luminance component values. The demosaicing process further includes a second color transformation for transforming the other known color components using the luminance. A chrominance interpolation is performed to interpolate missing chrominance components and an inverse color transformation transforms the components into the RGB colors. The present invention further provides super-resolution imaging for any application involving color image data interpolation.

[0009] The present invention further provides for generating high-resolution image/video from a low-resolution one. The present invention further provides a method to interpolate an NTSC or CCIR-601 signal into a HDTV signal. The above and additional advantages of the present invention will become apparent to those skilled in the art from a reading of the following detailed description when taken in conjunction with the accompanying drawings.

[0010] FIG. 1 is a solid state digital camera incorporating the present invention;

[0011] FIG. 2 is a mosaiced image in a Bayer pattern;

[0012] FIG. 3 is a mosaiced image in a strip pattern;

[0013] FIG. 4 is a flow chart of the process used in the solid state digital camera of FIG. 1;

[0014] FIG. 5 is a flow chart of the process used in the demosaicing of the present invention;

[0015] FIG. 6 is a series of separated mosaiced image planes which are outputs of the separation process of the present invention;

[0016] FIG. 7 is a series of separated demosaiced image planes which are inputs to the combination process of the present invention;

[0017] FIG. 8 is the output of the second color transformation of the present invention;

[0018] FIG. 9 is the output of the chrominance interpolation of the present invention; and

[0019] FIG. 10 is a full RGB image output after the demosaicing of the present invention.

[0020] Referring now to FIG. 1, therein is shown a solid state digital camera 10 having a camera body 12 and an attached lens system 14 with a lens filter 15. A single lens reflex camera is shown as an example where the present invention is implemented, but it would be evident that the present invention would work for all still and motion cameras as well.

[0021] The camera 10 has a mirror 16 which pivots on a pivot 18 to initially direct light both from an image as well as ambient light through the lens filter 15 and the lens system 14 up to a prism 20 and out through an eyepiece 22.

[0022] When a picture is being taken, the mirror 16 is pivoted up so as to allow light to strike a recording medium 24. In a digital still camera (DSC), the digital recording medium 24 is a matrix of photosensors, such as analog output, single channel, charge-coupled devices (CCDs) 26. Each of the CCDs 26 represents one picture element, or pixel, of the picture and the full array is called the “mosaic” pattern. Although cyan, magenta, yellow, and green filters can be used, each of the CCDs 26 herein has a green (G), red (R), or blue (B) filter over it, generally designated as the mosaic filter 28. The image recorded initially by these CCDs 26 is then electronically or magnetically recorded digitally for later playback. A processing unit 30 incorporating the present invention performs the processing of the signals from the CCDs 26 to the output of the image.

[0023] Referring now to FIG. 2, therein is shown a portion of a mosaiced image 50 in a pattern having columns 52, labeled “i” with only columns 1 through 6 shown, and rows 54, labeled “j” with only rows 1 through 6 shown. One example of a mosaiced image 50 is a pattern disclosed in U.S. Pat. No. 3,971,065 granted to Bryce Bayer titled “Color Imaging Array”, which is known as a Bayer pattern. Row 1 starts with G and alternates with R. Row 2 starts with B and alternates with G. The subsequent rows alternate with GRG and then BGB.

[0024] Referring, now to FIG. 3, therein is shown a portion of another mosaiced image 55 in a pattern having columns 57, labeled “i” with only columns 1 through 6 shown, and rows 59, labeled “j” with only rows 1 through 6 shown. One example of a mosaiced image 55 is known as a “strip” pattern. Column 1 is all G; column 2 is all R; and column 3 is all B. The columns then repeatedly repeat G, R, and then B.

[0025] Referring now to FIG. 4, therein is shown a flow chart 70 of the solid state digital camera 10 with light from an image passing through the mosaic filter 28 on to the CCDs 26. The CCDs 26 photo-electrically convert the light falling on them into proportional analog electrical signals in photosensor conversion 71. The analog signals are converted to digital signals by an analog to digital (A/D) conversion 72. The digital signals are then output for demosaicing 74. After demosaicing 74, the output is subject to image processing 76. In the image processing 76, the output of the demosaicing 74 will go through the series of conventional image processing steps, including white balancing and compression, and finally be output to the image output 78 for user devices such as a recording unit or display device. Generally, as would be evident to those skilled in the art, the mechanisms or units for performing the steps of the method would be a matter of design based on the descriptions provided herein.

[0026] Referring now to FIG. 5, therein is shown the demosaicing 74 program using color transformations into perceptually uniform color spaces. The term “luminance (L)” is used herein in a broad sense to refer to the color vector which is the major contributor of luminance information. The term “chrominance” refers to those color vectors a* and b* other than the luminance color vectors which provide a basis for defining an image. The digital signals from the A/D conversion 72 are separated into image planes in a separation 200 process.

[0027] Referring now to FIG. 6, therein is shown a series of three mosaiced image planes 100, 102, and 104 which are outputs of the separation 200.

[0028] Referring back to FIG. 5, the outputs of the separation 200 undergo a first color transformation 202, which transforms the known color components into a perceptually uniform color space where L can be determined. L is then used to interpolate missing L components by luminance interpolation 204. The output of the luminance interpolation 204 then undergoes a second color transformation 206 which transforms the known color components into chrominance space. The output of the second color transformation 206 is inputted into chrominance interpolation 208. After interpolation of the missing chrominance components, an inverse color transformation 210 is applied which is combined in combination 212 process. The combination 212 outputs to the image output 78.

[0029] Referring now to FIG. 7, therein is shown a series of three demosaiced image planes 110, 112, and 114 which are the inputs to the combination 212.

[0030] Referring now to FIG. 8, therein is shown the output of the second color transformation is 206 which is an array 250 starting with a column with alternating L and Lb* alternating with columns with alternating La* and L.

[0031] Referring now to FIG. 9, therein is shown the output of the chrominance interpolation 208 which is an array 260 filled with La*b*.

[0032] Referring now to FIG. 10, therein is shown a full demosaiced RGB image 270 that is an array filled with RGB.

[0033] In use, light from the image would enter the camera 10 through the lens filter 15 and lens system 14 to be reflected by the mirror 16 which would be in the down position. The light would be reflected upwards into the prism 20 and be reflected through the prism 20 to exit out the eyepiece 22 where the user could see the exact image seen through the lens filter 15 and the lens system 14.

[0034] When a picture is being taken, the mirror 16 pivots up and out of the way so light passes through the mosaic filter 28 and strikes the single-channel solid-state CCDs 26. After a single-channel image is created from the mosaiced sensor array, each pixel has one color component, R, G, or B and a mosaiced image 50 in the Bayer pattern of FIG. 2 or a mosaiced image 55 in the strip pattern of FIG. 3 is formed. The mosaiced image 50 or 55 will be digitized by the A/D conversion 72.

[0035] Demosaicing is subsequently applied to generate the missing color components. For example, for the pixel at the upper-left corner (bold-faced and italicized) in FIG. 2 (PRIOR ART), only the G component is created from the CCDs 26. The missing R and B components have to be generated by demosaicing 74. As yet another example, for the pixel in the middle R (bold-faced and italicized), the missing G and B components will be generated through demosaicing. The output is subsequently subject to demosaicing 74 where the mosaiced image 50 is. demosaiced and the full RGB image 270 is generated. The full RGB image 270 after demosaicing is shown in FIG. 8.

[0036] The demosaicing 74 plays an important role in the digital camera 10. It has direct impact on the image quality. The filter used in demosaicing needs to have good frequency response, be interpolating, and be smooth. To meet these requirements interpolators, or filters, such as “wavelet filters” are used in the present invention. Wavelet filters have been used for the analysis and the synthesis of signals, especially when such signals correspond to sounds or images. Previously, they have been used for images predominantly in image compression. A “wavelet transform” allows the representation of any arbitrary signal as a superposition of wavelets. The wavelets are functions generated from a single function by dilations and translations and they allow decomposition of a signal into different levels (each of which is further decomposed with a resolution adapted to the particular level). A description of the wavelet filters is provided in concurrently filed U.S. Patent Application Ser. No. 09/412,351 entitled DEMOSAICING USING WAVELET FILTERING FOR DIGITAL IMAGING DEVICE, by Bo Tao, supra, which is incorporated by reference. The demosaicing 74 of the present invention is shown in detail in FIG. 5.

[0037] The mosaiced output from A/D conversion 72, as shown in FIG. 2, is provided for demosaicing 74, as shown in FIG. 4. The mosaiced output is first separated by the data separation 200 into the three separated mosaiced image planes 100, 102, and 104, as shown in FIG. 6. The mosaiced image planes 100, 102 and 104 are the inputs for first color transformation 202 such that:

[0038] L=f(G),

[0039] R′=f(R), and

[0040] B′=f(B),

[0041] where: f(·)=log(·) or f(·)=(·)^α where α can be chosen to be ⅓ or a reasonable number that approximates the retinal response of the human vision system.

[0042] In the alternative, f can be in other forms that approximates the retinal response of the human vision system.

[0043] The L component of the output of the first color transformation 202 is in a perceptually uniform color space by virtue of the above approximation and provides the luminance which can be subsequently used.

[0044] A number of methods can be used for estimating the missing L component values, including linear, bilinear, and bicubic, among others. The output image of the luminance interpolation has a known L component at each pixel position.

[0045] The luminance interpolation 204 uses a quincunx wavelet filter in the preferred embodiment. The filter is two-dimensional. The coefficients of one such filter are given in Table I below:

[0046] However, other wavelet filters can be used as well. Where F₁represents this filter and X_Lrepresents the L output, at pixel position (i,j), the interpolated data is given by the following equation: $Y_L\left(i,j\right)=\sum _{m=-6}^6\sum _{n=-6}^6{F}_1\left(m,n\right)\times X_L\left(i+m,j+n\right)/2^{15}\times \sqrt{2}$

[0047] where Y_Lis the interpolated L output as shown in FIG. 8.

[0048] An alternative embodiment of this invention can be realized by thresholding the filter F₁, i.e. setting those coefficients of F₁smaller than a threshold to zero. This can reduce the computational cost. One thresholded filter, F₁, is shown in Table II below with the threshold set at 15:

[0049] The luminance interpolation 204 performs the following summation: $Y_L\left(i,j\right)=\sum _{m=-4}^4\sum _{n=-4}^4{F}_1\left(m,n\right)\times X_L\left(i+m,j+n\right)/2^{15}\times \sqrt{2}/k$

[0050] with k given by: $k=\sum _{m=-4}^4\sum _{n=-4}^4{F}_1\left(m,n\right).$

[0051] Other thresholds can be used also. The output of the luminance interpolation 204 is provided for the second color transformation 206.

[0052] With L known at all pixel positions now, R′ and B′ are subject to a second color transformation 206 where the R and B components are known to determine the chrominance components:

[0053] a*=L−R′; and

[0054] b*=L−B′

[0055] After the second color transformation 206, the image output is in the form of the array 250 shown in FIG. 8.

[0056] The missing chrominance signals a* and b* are then subject to chrominance interpolation 208. A number of algorithms can be used in the chrominance interpolation 208, such as linear, bilinear, bicubic, and so forth. One such interpolation filter is given by:

[0057] F₂=[−0.045636 −0.028772 0.29563.6 0.557543 0.295636 −0.028772 −0.045636]

[0058] However, other filters can be employed too. Two filtering processes need to be performed, once in the horizontal direction and the other one in the vertical direction. The order in which to perform the two processes is arbitrary. In the following, the program does the vertical filtering first. When the filtering is applied in the vertical direction, at pixel position (i,j), the output is given by: $Z_{a^{*}}\left(i,j\right)=\sum _{m=-3}^3{F}_2\left(m\right)\times X_{a^{*}}\left(i+m,j\right)\times 2\times \sqrt{2}$

[0059] where: X_a*represents the a* output of the second color transformation 206.

[0060] After Z_a*has been computed, each row is flipped. Then:

[0061] Z′_a*(i,j)=Z_r(i,N−1−j)

[0062] where: Z′_a*represent this new signal with pixel index i=0, . . . , N−1, j=0, . . . , N−1.

[0063] Horizontal filtering is performed on Z′_a*and the a* output of the chrominance interpolation 208 is given by: $Y_{a^{*}}^{\prime }\left(i,j\right)=\sum _{n=-3}^3{F}_2\left(n\right)\times Z_{a^{*}}^{\prime }\left(i,j+n\right)\times \sqrt{2}.$

[0064] Each row of Y′_a*is flipped to get the output Y_a*. Then:

[0065] Y_a*(i,j)=Y′_a*(i,N−1−j).

[0066] The b* interpolation uses the same filter F₂as the a* interpolation. The only difference between these two interpolations is that in the b* interpolation, the two flipping processes are performed during vertical filtering, while no flipping is needed for horizontal filtering.

[0067] The output of the chrominance interpolation is in the form of the array 260 of FIG. 9 with La*b* values known at all pixel positions.

[0068] An inverse color transformation 210 brings the image 260 of FIG. 9 back into RGB space by doing the following computation:

[0069] G=f⁻¹(L)

[0070] R=f⁻¹(L−a*)

[0071] B=f⁻¹(L−b*)

[0072] After combination 212, the full RGB image 260 output is then provided to the image processing 76. In the image processing 76, the RGB image will go through the series of conventional image processing steps, including white balancing and compression, and finally be output as the image output 78.

[0073] As described above, the present invention provides super-resolution imaging for any application involving data interpolation and more particularly for generating high-resolution image/video from a low-resolution image/video. One application is in the interpolation of an NTSC or CCIR-601 signal into a HDTV signal.

[0074] While the invention has been described in conjunction with a specific best mode, it is to be understood that many alternatives, modifications, and variations will be apparent to those skilled in the art in light of the aforegoing description. Accordingly, it is intended to embrace all such alternatives, modifications, and variations which fall within the spirit and scope of the included claims. All matters set forth herein or shown in the accompanying drawings are to be interpreted in an illustrative and non-limiting sense.