Color interpolation apparatus and color interpolation method utilizing edge indicators adjusted by stochastic adjustment factors to reconstruct missing colors for image pixels

[0001] 1. Field of the Invention

[0002] The present invention relates to image processing, and more specifically, to a color interpolation apparatus and color interpolation method using edge indicators adjusted by stochastic adjustment factors to reconstruct missing colors for image pixels.

[0003] 2. Description of the Prior Art

[0004] In an imaging system, three components of color information must be captured simultaneously to precisely presenting an image. To create an analogous digital imaging system that simultaneously captures all three components of the color information would require three individual imaging detectors. This would be prohibitive due to the high cost and would cause the packaging to be very complex. To keep the size and cost of a digital video imaging system to a minimum, an image sensor array of the system (using silicon chips) must keep its size small as well. Therefore, the number of color samples must be kept low. An alternative approach is to have each detector of the imaging system gathering data for a single color to create a sparse color image. Therefore, the imaging systems typically use a mosaic filter generally called a color filter array (CFA), and acquire a scene image by sampling one of the three different color components to obtain an array that stores only one color component per pixel. The imaging system gets a raw sensory data having less color samples per pixel because it ignores the other two color components for each pixel. Since each filter of the color filter array covers a single pixel and only allows a color in a specific spectral band to pass, before the scene image is further processed or displayed, the missing colors of each image pixel must be reconstructed so that each image pixel contains all three color components.

[0005] Because of the tri-stimulus nature of human color perception, to reconstruct a full color image (typically red, green, and blue or cyan, magenta, and yellow) from the raw sensory data, a color interpolation scheme is required to estimate these two missing color components for each of the image pixels. Based on an assumption that an object in the physical world usually gives rise to image intensities that vary smoothly over the image of the object, in order to economize processing resources, color interpolation is typically based upon color information at image pixels in a small neighborhood around each image pixel to be reconstructed. Given a known color filter array (CFA), such as Bayer pattern described in U.S. Pat. No. 3,971,065, the remaining two colors for each image pixel location can be reconstructed using color information provided by neighboring image pixels. The conventional color interpolation method uses replication of the values of the nearest neighboring image pixels, or alternatively uses linear or logarithmic averaging techniques for obtaining an average value of the neighboring image pixels for reconstructing the missing color. The color interpolation process to convert raw sensory image data into a full color image by estimating the missing color components of each image pixel from its neighboring image pixels is known as Demosaicing. Due to the aliasing effects caused by averaging (low-pass filtering) pixel values across the edges, most demosaicing approaches often introduce image effect problems like: zipper effects, false colors, or blur the edges of the image where there are dense edges. There are some conventional color interpolation methods to reduce or solve the image effect problems mentioned, such as: bilinear method, color difference based method, gradient-based method, and C2D2 (Color Correlations and Directional Derivatives) method. Most of the conventional methods can be classified into two categories: non-adaptive color interpolation or edge-directed adaptive color interpolation.

[0006] The non-adaptive color interpolation method involves the color interpolation as mentioned above, uniformly applied across the entire image, while the adaptive color interpolation, on the other hand, adjusts the weights of the color samples in the color interpolation process by utilizing the edge information. Moreover, a real-time application commonly requires an interpolation algorithm with minimum computational complexity to provide high quality images that are sharp and do not contain false color. The false color usually occurs around the edges of an image due to high spatial frequency. Localized spatial features, such as image edges and areas where hues of adjacent pixels change abruptly, cause estimation of a color to often be inaccurate. Further, false color usually occurs when one of the color components is missing making imaging systems that utilizes a CFA to acquire images particularly susceptible to this problem. Moreover, to provide sharp images with no false color, a well-known sharpening filter is usually applied to improve the image quality before displaying the image. However, when some color components are missing for each pixel, the color artifacts are not corrected with a sharpening filter, but are typically further visually enhanced after a sharpening filter is applied.

[0007] Therefore, the effectiveness of an adaptive color interpolation method highly depends on the edge detecting capability and the computational complexity of the interpolation algorithm when realizing the imaging system. In other words, the edge detecting capability plays an important role in color interpolation. Taking the C2D2 method for example, this method suffers from blurred edges and false colors. Therefore, there is a need for a color interpolation algorithm for digital imaging systems that uses fewer samples per image pixel yet can accurately reconstruct the missing color components to provide sharp images without zipper effects or false colors.

[0008] It is therefore one of the objectives of the claimed invention to provide a color interpolation apparatus and color interpolation method using edge indicators adjusted by stochastic adjustment factors to accurately reconstruct missing colors for image pixels, to solve the above problems.

[0009] According to one embodiment of the claimed invention, a color interpolation apparatus for processing a plurality of image pixels corresponding to a color filter array to reconstruct missing color components for each image pixel having a single color component is disclosed. The color interpolation apparatus comprises: an edge detection circuit for detecting edges in a plurality of interpolating directions for a target image pixel and then generating a plurality of edge indicators respectively corresponding to the interpolating directions by summing absolute values of a first color difference and a second color difference and then normalizing the sum of the absolute values with a stochastic adjustment factor for normalizing standard deviations of the interpolating directions, where the first color difference is the difference between color components of a first image pixel in the interpolating direction and a second image pixel in another interpolating direction opposite to the interpolating direction and the second color difference is difference between color components of a third image pixel in the interpolation direction and the target image pixel, (the color components of the first and second image pixels being identical to the missing color component of the target image pixel, the color component of the third image pixel being identical to the color component of the target image pixel); and an interpolation circuit, coupled to the edge detection circuit, for reconstructing at least a missing color component of the target image pixel according to a plurality of neighboring image pixels respectively in the interpolating directions and the edge indicators of the interpolating directions, each of the neighboring image pixels having a color component identical to the missing color component.

[0010] According to another embodiment of the claimed invention, a color interpolation method for processing a plurality of image pixels corresponding to a color filter array to reconstruct missing color components for each image pixel having a single color component is disclosed. The color interpolation method comprises: (a) detecting edges in a plurality of interpolating directions for a target image pixel and then generating a plurality of edge indicators respectively corresponding to the interpolating directions, wherein an edge indicator for each interpolating direction is determined by summing absolute values of a first color difference and a second color difference and then normalizing a sum of the absolute values with a stochastic adjustment factor for normalizing standard deviations of the interpolating directions, where the first color difference is between color components of a first image pixel in the interpolating direction and a second image pixel in another interpolating direction opposite to the interpolating direction, the second color difference is between color components of a third image pixel in the interpolation direction and the target image pixel, (the color components of the first and second image pixels are identical to the missing color component of the target image pixel, the color component of the third image pixel are identical to the color component of the target image pixel); and (b) reconstructing at least a missing color component of the target image pixel according to a plurality of neighboring image pixels respectively in the interpolating directions and the edge indicators of the interpolating directions, wherein each of the neighboring image pixels has a color component identical to the missing color component.

[0011] The claimed invention provides better edge-sensing capabilities and finer images. It reconstructs the missing color components of each image pixel with high accuracy, because it is based on the detection of the spatial features present in the pixel neighborhood. Through the directionally edge-adaptive weighted color interpolation, it provides sharp images without false colors and with smooth transition in hue from pixel to pixel. The directionally edge-adaptive weighted color interpolation of the claimed invention can be applied to image capturing devices such as a digital camera or a digital video (DV) camcorder.

[0012] These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.

[0017] Please refer to FIG. 1. FIG. 1 is a block diagram illustrating an imaging system 10 for image capturing in a digital camera or a digital video (DV) camcorder according to an embodiment of the present invention. The imaging system 10 comprises a lens 11 for optically receiving an image; a color filter array (CFA) 12 having a specific color pattern to filter the image passing through the lens 11; an image sensor array 13 to sense output of the CFA 12; and a color interpolation processor 14 for performing the directional edge-adaptive weighted color interpolation of the present invention to reconstruct missing color components for each image pixel to get a full color image.

[0018] Please refer to FIG. 2. FIG. 2 is a diagram illustrating a color pattern 20 of the CFA 12 shown in FIG. 1. In this embodiment, the color pattern 20 is a Bayer pattern. However, please note that the present invention is not limited to using the Bayer pattern. As shown in FIG. 2, a missing color of a target image pixel B₄₄ is to be reconstructed. The color interpolation of the present invention makes use of twelve interpolating directions D1-D12 for reconstructing a missing green color. Based on the known Bayer pattern, the target pixel B₄₄ has a blue color component, the neighboring image pixels G₄₃, G₃₄, G₄₅, G₅₄, G₃₂, G₂₃, G₂₅, G₃₆, G₅₆, G₆₅, G₆₃, G₅₂ each having a green color component. In this embodiment, the color interpolation processor 14 shown in FIG. 1 will perform the color interpolation in all twelve interpolating directions D1-D12 because in each of the interpolating directions color values of neighboring image pixels are green, thus all twelve neighboring image pixels G₄₃, G₃₄, G₄₅, G₅₄, G₃₂, G₂₃, G₂₅, G₃₆, G₅₆, G₆₅, G₆₃, G₅₂ can provide needed color values for interpolating the wanted color component for the target image pixel B₄₄. On the other hand, if the missing color component of the target image pixel B₄₄ to be reconstructed, is not green, but lets say for example is red, then only the four nearest image pixels R₃₃, R₃₅, R₅₅, R₅₃ having the same red color component would be selected. The operation of the color interpolation processor 14 is detailed as follows.

[0019] Please refer to FIG. 3 in conjunction with FIG. 2. FIG. 3 is a block diagram of the color interpolation processor 14 shown in FIG. 1. The color interpolation processor 14 comprises an edge detection circuit 41 and a weighted interpolation circuit 42. In this embodiment, the imaging system 10 processes a plurality of image pixels corresponding to the color filter array CFA 12 in FIG. 1 to reconstruct missing color components for each image pixel. The edge detection circuit 41 is used to detect edges in the interpolating directions D1-D12 for the target image pixel B₄₄ and then generates a plurality of edge indicators I1-I12 respectively corresponding to the interpolating directions D1-D12. The weighted interpolation circuit 42, coupled to the edge detection circuit 41, is used to reconstruct the missing green color component of the target image pixel B₄₄ according to the neighboring image pixels G₄₃, G₃₄, G₄₅, G₅₄, G₃₂, G₂₃, G₂₅, G₃₆, G₅₆, G₆₅, G₆₃, G₅₂ respectively in the interpolating directions D1-D12. In addition, the weighted interpolation circuit 42 is further used to reconstruct the missing red color component of the target image pixel B₄₄ according to the neighboring image pixels R₃₃, R₃₅, R₅₅, R₅₃.

[0020] The edge detection circuit 41 determines an edge indicator I1, I2, . . . , I12 for an interpolating direction D1, D2, . . . , D12 by summing absolute values of a first color difference and a second color difference with an adjustment process to increase the resolution of the edge indicators. The first color difference is the difference between color components of a first image pixel in the interpolating direction and a second image pixel in another interpolating direction opposite to the interpolating direction, and the second color difference is the difference between color components of a third image pixel in the interpolation direction and the target image pixel, where the color components of the first and second image pixels are identical to the missing color component of the target image pixel, and the color component of the third image pixel is identical to the color component of the target image pixel.

[0021] For increasing the resolution of the edge indicators, according to the embodiment of the present invention a stochastic edge-detecting scheme is disclosed. The stochastic edge-detecting scheme has more precise edge-sensing ability than prior art edge-detecting schemes, such as the directional-derivative method (e.g. linear adjustment in C2D2), and it can be generalized to other sets of edge orientations. In the stochastic edge-detecting scheme, an image source having a locally stationary Gaussian distribution is utilized. Therefore, the image samples within a locally stationary region will have the same stochastic mean. Denote the current image sample as X(0), and a sample d pixels away from the current image sample X(0), as X(d). In this embodiment, image samples X(0) and X(d) are both within the stationary region to have the same stochastic mean. Then, the difference X(0)−X(d) will be a Gaussian random variable with zero mean, and the probability that the difference value is largely away from zero will be very small. Since the goal of the edge-adaptive weighted color interpolation is to assign small weight to an image sample during the interpolation if the likeliness of an edge between the image sample and the target image pixel is high, also, since the probability distribution will be different when the differences are calculated from image samples with different distance d, the edge indicator in this embodiment is calculated by normalizing the difference value of the image samples, thereby generating a fair measure of the edge strength.

[0022] To normalize the difference values, in this embodiment, the calculated difference values are multiplied by an adjustment factor κ such that the modified values become Gaussians with the same standard deviation (thus with the same distribution).

[0023] The derivation of the factor κ is as follows. Let Ω(d) denote the auto-correlation function:
Ω(d)=(E[(X(0)−μ_x)(X(d)−μ_x)])/(σ_x²)  Eq.(1)

[0024] In Eq. (1), μ_x is the stochastic mean in the stationary region and σ_x is the standard deviation. The standard deviation of the difference X(0)−X(d) will be:

[0025] $\begin{array}{cc}\begin{array}{c}\sqrt{E\left[{\left(X\left(0\right)-X\left(d\right)\right)}^2\right]}=\sqrt{E\left[{\left(\left(X\left(0\right)-{\mu }_X\right)-\left(X\left(d\right)-{\mu }_X\right)\right)}^2\right]}\\ =\sqrt{E\left[\begin{array}{c}{\left(X\left(0\right)-{\mu }_X\right)}^2-2\left(X\left(0\right)-{\mu }_X\right)\\ \left(X\left(d\right)-{\mu }_X\right)+{\left(X\left(d\right)-{\mu }_X\right)}^2\end{array}\right]}\\ =\sqrt{{\sigma }_X^2-2{\sigma }_X^2\Omega \left(d\right)+{\sigma }_X^2}\\ ={\sigma }_X\sqrt{\left(1-\Omega \left(d\right)\right)}\end{array}& \begin{array}{c}\mathrm{Eq}.\left(2.a\right)\\ \mathrm{Eq}.\left(2.b\right)\\ \mathrm{Eq}.\left(2.c\right)\\ \mathrm{Eq}.\left(2.d\right)\end{array}\end{array}$

[0026] Eq. (2.c) holds because of the stationary property. If a certain distance d₁ is set as the basic distance, for other distance d_n, the normalization factor κ_n will be:

[0027] $\begin{array}{cc}{\kappa }_n=\frac{{\sigma }_{\stackrel{\_}{X}}\sqrt{\left(1-\Omega \left(d_1\right)\right)}}{{\sigma }_{\stackrel{\_}{X}}\sqrt{\left(1-\Omega \left(d_n\right)\right)}}=\sqrt{\frac{\left(1-\Omega \left(d_1\right)\right)}{\left(1-\Omega \left(d_n\right)\right)}}& \mathrm{Eq}.\left(3\right)\end{array}$

[0028] κ_n can be derived when substituting Ω(d₁) and Ω(d₂) calculated by Eq. (1) into Eq. (3).

[0029] Taking the edge indicator I1 for example, it is calculated as follows:
I1=κ₁*[ABS (the color value of G₄₃−the color value of G₄₅)+ABS (the color value of B₄₂−the color value of B₄₄)]  Eq.(4)

[0030] In Eq. (4), G₄₃ is the above-mentioned first image pixel, G₄₅ is the above-mentioned second image pixel, and B₄₂ is the above-mentioned third image pixel. Therefore, the distance d₁ equals 2 pixels. As shown in FIG. 2, the distances d₁-d₁₂ can be divided into two categories: d_n=2 for 1≦n≦4, and d_n=2√{square root over (5)} for 5≦n≦12. So the value of κ_n will be of two kinds. Under the assumption that the autocorrelation function Ω(d) is of the form Ω(d)=exp(−d²/τ²), where τ is a image dependent factor, κ_n will be very close to 0.5, for 5≦n≦12, as long as the value Ω(d₁) is high. The setting of the factor κ_n can be summarized as follows:

[0031] $\begin{array}{cc}{\kappa }_n=\{\begin{array}{cc}1,& 1\le n\le 4\\ 0.5,& 5\le n\le 12\end{array}& \mathrm{Eq}.\left(5\right)\end{array}$

[0032] The calculation of the remaining edge indicators I2-I12 can be easily derived from the above description, Eq. (4) and Eq. (5).

[0033] In the present invention, the weighted interpolation circuit 42 shown in FIG. 3 comprises a weight calculation module 43 and an interpolation module 44. The weight calculation module 43 is used to calculate a plurality of weights W1-W12 for the neighboring image pixels G₄₃, G₃₄, G₄₅, G₅₄, G₃₂, G₂₃, G₂₅, G₃₆, G₅₆, G₆₅, G₆₃, G₅₂ according to the edge indicators I1-I12. The weights W₁-W₁₂ are calculated as follows:

[0034] $\begin{array}{cc}{W}_n=\left(\frac{1}{1+I_n}\right)/\sum _{n=1}^{12}\frac{1}{1+I_n}& \mathrm{Eq}.\left(6\right)\end{array}$

[0035] Please note that Eq. (6) is not the only way to calculate the weights. In another embodiment, other kinds of functions or a look up table can also calculate the weights, thus the present invention is not limited to Eq. (6).

[0036] The interpolation module 44, coupled to the weight calculation module 43, is used to reconstruct the missing green color component according to the weights W₁-W₁₂ calculated by Eq. (6) and color components of the neighboring image pixels G₄₃, G₃₄, G₄₅, G₅₄, G₃₂, G₂₃, G₂₅, G₃₆, G₅₆, G₆₅, G₆₃, G₅₂ by summing a plurality of products of the weights W1-W12 and a plurality of color values of the neighboring image pixels G₄₃, G₃₄, G₄₅, G₅₄, G₃₂, G₂₃, G₂₅, G₃₆, G₅₆, G₆₅, G₆₃, G₅₂ to reconstruct the missing green color component for the target image pixel B₄₄. That is, the reconstructed green color component is equal to: W₁*(color value of G₄₃)+W₂*(color value of G₃₄)+, . . . , +W₁₂ *(color value of G₅₂).

[0037] However, as described in “Effective color interpolation in CCD color filter arrays using signal correlation,” IEEE Trans. on Circuit and System for Video Technology, vol. 13, no. 6, Jun. 2003, because of the high correlation between R, G, and B color components of the image captured by the imaging system 10, the color difference values (G−R, G, G−B) are relatively smoother than the (R, G, B) values. As a result, this causes the aliasing effects to be less pronounced (i.e., less visible) in the color difference space. Thus, the interpolation module 44 can apply color difference values (G−R, G, G−B) instead of (R, G, B) to reconstruct the missing green color component for better image quality. That is, in another embodiment using color difference, the reconstructed green component is equal to: (color value of B₄₄)+W₁*(color difference value between green color component and blue color component of G₄₃)+W2*(color difference value between green color component and blue color component of G₃₄)+ . . . +W₁₂*(color difference value between green color component and blue color component of G₅₂).

[0038] As to reconstructing the missing red color component of the target image pixel B₄₄, the color interpolation operation is similar to that for reconstructing the missing green color component. Referring to Eq. (4) and the related description, the edge indicators I1′-I4′ are obtained by the edge detection circuit 41. Similarly, referring to Eq. (6) and the related description, the weights W₁′-W₄′ are obtained by the weight calculation module 43. In one embodiment, the reconstructed red color component is equal to: W₁′*(color value of R₃₃)+W₂′*(color value of R₃₅)+W₃′*(color value of R₅₅)+W₄′*(color value of R₅₃). In another embodiment using color difference, the reconstructed red color component is equal to: (reconstructed green color value of B₄₄)−W₁′*(color difference value between green color component and red color component of R₃₃)−W₂′*(color difference value between green color component and red color component of R₃₅)−W₄′*(color difference value between green color component and red color component of R₅₃).

[0039] In the above description, the color interpolation applied to reconstructing the missing color components for the target image pixel B₄₄ has been detailed. Based on the above description, those skilled in this art can easily realize that the same process can be applied to other pixels for reconstructing corresponding missing color components. Further description is omitted for brevity.

[0040] Please refer to FIG. 4. FIG. 4 is a flowchart illustrating the operation of reconstructing missing color components of a target image pixel according to an embodiment of the present invention. The operation of the directional edge-adaptive weighted color interpolation to reconstruct missing colors includes the following steps:

[0041] Step 300: Use a twelve-direction color Interpolation to reconstruct all missing green components of all target pixels.

[0042] Step 302: Use a four-direction color Interpolation to reconstruct the other missing components of the target pixels in Step 300.

[0043] Step 304: Use the twelve-direction color Interpolation to reconstruct missing color components of remaining pixels. Please note that the color interpolation is performed by utilizing the hardware circuitry as shown in FIG. 1. However, the color interpolation can be implemented by software computation as well. For instance, a processor and a storage device are implemented to build the desired color interpolation apparatus, where the storage device stores a color interpolation program. Therefore, when the color interpolation program is loaded and executed by the processor, the processor is capable of performing the above-mentioned weighted color interpolation. In other words, the processor running the color interpolation program is functionality identical to that of a hardware-based color interpolation apparatus mentioned above.

[0044] In contrast to the related art, the present invention provides better edge-sensing capability and finer images. It reconstructs the missing color components of each image pixel with high accuracy, because it is based on the detection of the spatial features present in the pixel neighborhood. Through the directionally edge-adaptive weighted color interpolation, it provides sharp images without false colors and with smooth transition in hue from pixel to pixel. The directionally edge-adaptive weighted color interpolation of the present invention can be applied to image capturing devices such as a digital camera or a digital video (DV) camcorder.

[0045] Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.