Sparsely sampled image sensing device with color and luminance photosites

[0002] The invention relates generally to the field of image capture, and more specifically to an image sensing device with different color types of photosites.

[0003] Electronic image sensing devices have employed a technique of sparsely sampling photographic images with an arrangement of photosites having different color responses to light. The term sparsely sampling as used herein refers to the fact that all colors are not sensed at all photosites in the image sensing device. In 1975 Bayer disclosed a sparsely sampled image sensing device in the form of a color filter array of photosites in U.S. Pat. No. 3,971,065 issued Jul. 20, 1976 to Bayer. The image sensing device described by Bayer was configured in a rectangular array of photosites with luminance sensitive photosites occurring every other photosite of every row of photosite with the pattern of luminance sensitive photosites staggered by one pixel in adjacent rows of photosite. Thus the luminance sensitive photosites formed a checkerboard pattern. The other photosites of Bayer's image sensing device used two types of chrominance, or color, sensitive photosites with a first chrominance photosite type used in the odd rows of photosites, and a second chrominance photosite type used in the even rows of photosites.

[0004] Bayer also disclosed an alternative configuration for a sparsely sampling image sensing device in U.S. Pat. No. 3,971,065 that over time has proven to be more useful. In this alternative configuration, green sensitive photosites are used to sense the luminance signal and red and blue sensitive photosites are used for the two chrominance types of photosites as shown in FIG. 2a. The advantage of Bayer's alternative configuration is a sensor with nearly equalized photo response between the three types of photo elements, i.e. the three types of photosites saturate at nearly the same incident light level when exposed to white light. A disadvantage of both the Bayer configurations is evident when over exposure conditions arise. Since the green sensitive photosites have a closer relationship to the image luminance signal, many real world objects have more green image content than red or blue image content. Consequently, if a given scene is overexposed, the green sensitive photosites generally saturate (reach the maximum possible signal strength) faster than the red or blue sensitive photosites. The resulting pixel values corresponding to the saturated photosites are clipped (reach a maximum pixel value) resulting in a loss of spatial detail. FIG. 2bshows the Bayer pattern with X's at the locations of the saturated green photosites. These saturated green photosites may obscure image detail. Not only is there a loss of spatial detail but there is also a loss of green color information at these photosites.

[0005] U.S. Pat. No. 5,323,233 issued Jun. 21, 1994 to Yamagami et al. discloses an image sensing device color filter array of photosites with a different configuration as a variation on Bayer's pattern. The Yamagami pattern of photosites is shown in FIG. 3awith the Y labeled photosites responding to luminance light, and R, G, and B labeled photosites responding to red, green, and blue light respectively. The Yamagami pattern contains 50% luminance photosites, 25% green photosites, 12½% red photosites, and 12½% blue photosites. The Yamagami pattern for overexposed conditions is shown in FIG. 3bwith saturated luminance photosites indicated by X's. The red, green, and blue photosites may still produce modulated signals. However, for overexposed conditions the Yamagami pattern does suffer a loss of spatial resolution due to the fact that 50% of the array are luminance response photosites.

[0006] There is a need therefore for an improved pattern of photosites for a image sensing device that employs a sparsely sampling technique.

[0007] The need is met according to the present invention by providing an image sensing device having an array of light-sensitive elements, including a first type of element sensitive to a spectral region corresponding to luminance; a second type of element sensitive primarily to red light; a third type of element sensitive primarily to green light; and a fourth type of element sensitive primarily to blue light, the four types of elements occurring in repeating patterns. In a preferred embodiment, the four types of elements of the image sensing device which occur in repeating patterns can be configured such that over at least a major portion of the array all four types of elements occur at every other element position along both of two orthogonal directions of the array of light-sensitive elements.

[0008] An advantage of the present invention that the image sensing device minimizes the loss of spatial detail for overexposed conditions.

[0009] FIG. 1 is a diagram showing the photosite pattern according to the invention;

[0010] FIG. 2ais a diagram showing the prior art Bayer photosite pattern;

[0011] FIG. 2bis a diagram showing the prior art Bayer photosite pattern for overexposure conditions;

[0012] FIG. 3ais a diagram showing the prior art Yamagami photosite pattern;

[0013] FIG. 3bis a diagram showing the prior art Yamagami photosite pattern for overexposure conditions;

[0014] FIG. 4 is a diagram showing the photosite pattern according to the invention for overexposed conditions;

[0015] FIG. 5 is a block diagram of a digital imaging system utilizing an image sensing device with the photosite pattern and a CFA interpolator 26 according to the invention;

[0016] FIG. 6 is a diagram of pixels used by the CFA interpolator 26 to interpolate pixel values corresponding to the luminance photosites;

[0017] FIG. 7 is a diagram of pixels used by the CFA interpolator 26 to interpolate pixel values corresponding to the red photosites;

[0018] FIG. 8 is a diagram of pixels used by the CFA interpolator 26 to interpolate pixel values corresponding to the blue photosites;

[0019] FIG. 9 is a diagram of pixels used by the CFA interpolator 26 to interpolate pixel values corresponding to the green photosites; and

[0020] FIG. 10 is a diagram of pixels used by the CFA interpolator 26 to interpolate pixel values corresponding to the luminance photosites when the luminance pixels are clipped.

[0021] Image sensing devices, such as a charge-coupled device (CCD) and CMOS image sensors, are commonly found in such products as digital cameras, scanners, and video cameras. Many image sensing devices employ an array of color filters (color filter array (CFA)), to alter the color response to light of different light sensitive elements or photosites. Thus for the geometrical locations of the photosites in a sensed image, only one characteristic response to light or color is sampled. Hence, the image sensing devices which employ a color filter array are termed sparsely sampled image sensing devices. Digital pixel values (pixels) are derived from the electronic signals produced by the photosites with electronic analog-to-digital converting circuitry. Pixels derived from the signal produced by a photosite type that has a primarily red color response to light are termed red pixels. Similarly, green and blue pixels have corresponding photosite types responsive primarily to green and blue light respectively. Luminance pixels have corresponding photosite types responsive to red, green, and blue (white) light.

[0022] The present invention employs a color filter array CFA pattern of light sensitive elements as shown in FIG. 1. The photosite locations labeled with an L are sensitive to white light and generate luminance pixel data. Similarly, the photosite locations labeled with R, G, and B are sensitive primarily to red, green, and blue light and correspondingly generate red, green, and blue pixel data. Each of the four types of photosites are arranged in a rectangular pattern of every other photosite of every other row of photosites such that over at least a major portion of the array, all four types of elements occur at every other element position along both of two orthogonal directions of the array of light-sensitive elements. Note that each of the four photosite types are represented with equal frequency, i.e. each photosite type constitutes 25% of the total of photosites.

[0023] An important aspect of the present invention is evident for overexposure conditions. If too much light is received by the image sensing device, the luminance photosites will saturate, i.e. the luminance photosites reach the maximum signal condition before the red, green, and blue photosites. When an overexposure condition occurs, the corresponding luminance pixel values become clipped, i.e. they assume the maximum numerical value possible. In a typical sensed image, some of the luminance photosites will saturate while other receiving less light will not. Thus the saturation condition is experienced locally. FIG. 4 shows the CFA pattern of photosites for a local region of the image sensing device that has driven the luminance photosites into saturation by overexposure. The overexposed photosites are labeled with X's. Note that the red, green, and blue photosites may still record signal variations as they may not have been saturated by overexposure. Thus the present invention minimizes the loss of spatial detail for overexposed conditions by virtue of the fact that only 25% of the photosites are luminance photosites.

[0024] Imaging devices employing electronic sensors are well known, therefore the present description will be directed in particular to elements forming part of, or cooperating more directly with, apparatus in accordance with the present invention. Elements not specifically shown or described herein may be selected from those known in the art. Note that as used herein, the term image is a two dimensional array of pixel values. An image may be a two dimensional subset of another image. The present invention can be implemented in whole or in part within a digital camera.

[0025] The essential elements of the present invention are shown as a functional block diagram in FIG. 5. Light from an object or scene is incident upon a lens 2, forming a photographic image on the image sensing device 10 such as a charged-coupled device (CCD). Note that other devices, such as CMOS devices, may be used as the image sensing device 10. An optical low pass filter 6, placed between the lens 2 and the image sensing device 10 performs a slight blurring of the imaged light in order to reduce the occurrence of aliasing. An A/D converter 14 receives the voltage signal corresponding to the imaged light from the image sensing device 10 and produces an image signal corresponding to the voltage signal. The output of the A/D converter 14 is a sparsely sampled digital image 201. A sparsely sampled digital image is defined as a digital image that was captured with an image sensing device having a single image sensor with multiple types of photosites. The CFA interpolator 26 receives the sparsely sampled digital image 201 from the A/D converter 14 and generates an interpolated color digital image 202 by applying a color filter array CFA interpolation filter to the sparsely sampled digital image 201. The CFA interpolator 26 generates color pixel values for pixel locations for which the corresponding color was not sensed by the image sensing device 10. For example, the CFA interpolator 26 generates red pixel values for pixel locations corresponding to green photosites. Thus the interpolated color digital image 202 has defined red, green, and blue pixel values for all pixel locations.

[0026] The A/D converter 14 shown in FIG. 5 converts the voltage signal produced by the image sensing device 10 into an image signal, i.e. a stream of digital pixel values corresponding to the voltage signal produced by the photosites of the image sensing device 10. More specifically, the A/D converter 14 converts the voltage signal, nearly linear with respect to the intensity of the incident light, from the image sensing device 10 to a discrete digital image signal, e.g. a 10 bit signal where the linear encoded values range from 0 to 1023. The A/D converter 14 may also perform processing to convert the linear code value domain image signal to a nonlinear code value domain image signal, such as an 8 bit logarithmic signal as is commonly performed in the art.

[0027] Generally, the CFA interpolator 26 operates by considering the pixel values of the corresponding photosite and the pixel values of associated surrounding photosites. While a variety of known CFA interpolation filters may be used, the present invention uses a modified version of the CFA interpolation filter disclosed by Adams et al. in U.S. Pat. No. 5,652,621 issued Jul. 29, 1997 to Adams, Jr. et al. The CFA interpolator 26 produces the interpolated color digital image 202 from the sparsely sampled digital image 201 in a three step procedure. In the first interpolation step, a first interpolated pixel value for all four types of pixels is generated. Although a color digital image can be constructed directly from these first interpolated pixel values, the present invention uses the first interpolated pixel values in a second interpolation step to generate color difference pixel values. In a third interpolation step, the color difference pixel values and the first interpolated luminance pixel values are used to generate second, or final, interpolation pixel values for improved results.

[0028] In the first interpolation step, a first interpolation pixel value or missing color pixel value, i.e. a color pixel value not generated directly by a photosite sensing the corresponding color of light, is generated by considering neighboring pixel values. This interpolation step is performed for each missing pixel value at each pixel location. Therefore, first interpolated red R′, green G′, and blue B′ pixel values are generated at pixel locations corresponding to the luminance photosites. Similarly, first interpolated luminance L′, green G′, and blue B′ pixel values are generated at pixel locations corresponding to the red photosites. First interpolated luminance L′, red R′, and blue B′ pixel values are generated at pixel locations corresponding to the green photosites. First interpolated luminance L′, red R′, and green G′ pixel values are generated at pixel locations corresponding to the blue photosites.

[0029] FIG. 6 shows the pattern of pixels for generating first interpolating pixel values for red R′, green G′ and blue B′ pixels at a pixel location corresponding to a luminance photosite. The first interpolated red pixel value R′₁₃corresponding to the pixel location at L₁₃is calculated in two terms. The first term is an average of two nearest neighboring red pixel values R₁₂and R₁₄. The second term is a Laplacian pixel value given by subtracting the average of the two nearest neighboring luminance pixel values L₁₁and L₁₅from the center luminance pixel value L₁₃that are in a line, in this case a horizontal line, as the nearest neighboring red pixels. The first interpolated blue pixel value B′₁₃corresponding to the pixel location L₁₃is calculated in similar fashion with the luminance and blue pixel values sampled in the vertical direction about the center luminance pixel L₁₃.

[0030] The green photosites do not line up in either a horizontal or vertical line about the luminance photosites. However, the green photosites do lineup with the luminance photosites along the two diagonal directions. One of the diagonal sets of pixels is used to calculate the first interpolated green pixel value based on the magnitude of the gradients along the two diagonal directions. Specifically, the absolute value of the difference between pixels G₇an G₁₉(gradient pixel value) is compared with the absolute value of the difference between pixels G₁₉an G₁₇(gradient pixel value) shown in FIG. 6. If the magnitude of the gradient pixel value formed from pixels G₇and G₁₉is less than the magnitude of the gradient pixel value formed from pixels G₉and G₁₇, then pixels L₁, L₁₃, L₂₅, G₇, and G₁₉are used in to calculate the green first interpolated pixel value G₁₃corresponding to the pixel location L₁₃. Otherwise, pixels L₅, L₁₃, L₂₁, G₉, and G₁₇are used in to calculate the first interpolated green pixel value G′₁₃corresponding to the pixel location L₁₃. Equation (1) describes the mathematical formula for the first interpolated red, green, and blue pixel values R′₁₃, G′₁₃, and B′₁₃respectively corresponding to the pixel location L₁₃.

[0031] R′₁₃=(R₁₂+R₁₄)/2+(−L₁₁−2L₁₃−L₁₅)/4

[0032] B′₁₃=(B₈+B₁₈)/2+(−L₃−2L₁₃−L₂₃)/4

[0033] G′₁₃=(G₇+G₁₉)/2+(−L₁−2L₁₃−L₂₅)/4 for |G₇−G₁₉|<|G₉−G₁₇|

[0034] G′₁₃=(G₉+G₁₇)/2+(−L₅−2L₁₃−L₂₁)/4 for |G₉−G₁₇|<=|G₇−G₁₉|  (1)

[0035] FIG. 7 shows the pattern of pixels used for interpolating the first interpolated luminance L′₁₃, green G′₁₃, and blue B′₁₃pixel values corresponding to the pixel location R₁₃. For this case, the red and green pixels line up along the vertical direction while the red and luminance pixels line up along the horizontal direction. The blue and red pixels line up along the two diagonal directions. Equation (2) describes the mathematical formula for the first interpolated luminance, green, and blue pixel values L′₁₃, G′₁₃, and B′₁₃respectively, corresponding to the pixel location R₁₃.

[0036] L′₁₃=(L₁₂+L₁₄)/230 (−R₁₁2R₁₃−R₁₅)/4

[0037] G′₁₃=(G₈+G₁₈)/230 (−R₃−2R₁₃−R₂₃)/4

[0038] B′₁₃=(B₇+B₁₉)/230 (−R₁−2R₁₃−R₂₅)/4 for |B₇−B₁₉|<|B₉−B_17|

[0039] B′₁₃=(B₉+B₁₇)/230 (−R₅−2R₁₃−R₂₁)/4 for |B₉−B₁₇|<=|B₇−B₁₉|  (2)

[0040] FIG. 8 shows the pattern of pixels used for interpolating the first interpolated luminance L′₁₃, green G′₁₃, and red R′₁₃pixel values corresponding to the pixel location B₁₃. For this case, the luminance and blue pixels line up along the vertical direction while the green and blue pixels line up along the horizontal direction. The blue and red pixels line up along the two diagonal directions. Equation (3) describes the mathematical formula for the first interpolated red, green, and luminance pixel values R′₁₃, G′₁₃, and L′₁₃respectively corresponding to the pixel location B₁₃.

[0041] G′₁₃=(G₁₂+G₁₄)/2+(−B₁₁−2B₁₃−B₁₅)/4

[0042] L′₁₃=(L₈+L₁₈)/2+(−B₃−2B₁₃−B₂₃)/4

[0043] R′₁₃=(R₇+R₁₉)/2+(−B₁−2B₁₃−B₂₅)/4 for |R₇−R₁₉|<|R₉−R₁₇|

[0044] R′₁₃(R₉+R₁₇)/2+(−B₅−2B₁₃−B₂₁)/4 for |R₉−R₁₇|<=|R₇−R₁₉|  (3)

[0045] FIG. 9 shows the pattern of pixels used for interpolating the first interpolated luminance L′₁₃, red R′₁₃, and blue B′₁₃pixel values corresponding to the pixel location G₁₃. For this case, the red and green pixels line up along the vertical direction while the green and blue pixels line up along the horizontal direction. The green and luminance pixels line up along the two diagonal directions. Equation (4) describes the mathematical formula for the first interpolated red, blue, and luminance pixel values R′₁₃, B′₁₃, and L′₁₃respectively corresponding to the pixel location G₁₃.

[0046] B′₁₃=(B₁₂+B₁₄)/2+(−G₁₁−2G₁₃−G₁₅)/4

[0047] R′₁₃=(R₈+R₁₈)/2+(−G₃−2G₁₃−G₂₃)/4

[0048] L′₁₃=(L₇+L₁₉)/2+(−G−2G₁₃−G₂₅)/4 for |L₇−L₉|<|L₉−L₁₇|

[0049] L′₁₃=(L₉+L₁₇)/2+(−G₅−2G₁₃−G₂₁)/4 for |L₉−L₁₇|<=|L₇−L₁₉|  (4)

[0050] In the second interpolation step, two color difference pixel values are generated for each pixel location, i.e. a green-magenta pixel value (GM′) and an illuminant pixel value (ILL′). The mathematical formulas for calculating these two color difference pixel values are given in Equation (5)

[0051] GM′=(2G′−R′−B′)/4

[0052] ILL′=(B′−R′)/2  (5)

[0053] where the R′, G′, and B′ variables represent the interpolated red, green, and blue pixel values respectively corresponding to a luminance photosite. For the red, green, and blue photosite locations, Equation (5) is used to calculate a GM′ and ILL′ pixel value, however, when possible, the original pixel values rather than the first interpolated pixel values are used. For example, at the red photosites, the original red pixel values R and the interpolated green G′ and blue B′ pixel values are used in Equation (5).

[0054] After the second step, an L′, GM′, and ILL′ pixel is defined for all pixel locations. In the third interpolation step, the L′, GM′, and ILL′ pixel values are used to generate the set of second interpolated red, green, and blue pixel values R″, G″, and B″ respectively. The inverse transform of Equation (5) is used to generate the R″, G″″, and B″″ pixel values given by Equation (6).

[0055] R″=L′−(2GM′)/3−ILL′

[0056] B″=L′−(2GM′)/3+ILL′

[0057] G″=L′+(4GM′)/3  (6)

[0058] At the luminance photosite locations, the original L pixel values re used in Equation (6) in place of the L′ variable. The original pixel values R, and B are uses as the final interpolated pixel values at the locations of the red, green, and blue photosites respectively. For example, the second interpolated red pixel values R″ form the red pixel values of the interpolated color digital image 202 for the luminance, green, and blue photosite pixel locations.

[0059] As shown in FIG. 4, when the imaging sensing device 10 experiences an overexposure condition resulting from too much light received, the luminance pixel values clip to the maximum allowable pixel value as produced by the A/D converter 26. The present invention remedies the overexposure condition by using a nearest neighbor interpolation method. For the clipped luminance pixel location indicated by the X's in FIG. 10, the second interpolated blue pixel value B″ is calculated by averaging the two adjacent vertical original blue pixel values. Similarly, the second interpolated red pixel value R″ is calculated by averaging the two adjacent horizontal original red pixel values. The second interpolated green pixel value G″ is calculated by first comparing the magnitude of the different between the two sets of diagonal green pixel values. The two diagonal pairs of green pixel values with the lesser magnitude gradient pixel values are averaged to calculate the second interpolated green pixel value G″. Equation (7) describes the mathematical calculation of second interpolated red R″, green G″, and blue B″ pixel values for the clipped pixel corresponding to the pixel location X₅shown in FIG. 10.

[0060] R₅″=(R₄+R₆)/2

[0061] B₅″=(B₂+B₈)/2

[0062] G₅″=(G₁+G₉)/2 for |G₁−G₉|<|G₃−G₇|

[0063] G₅″=(G₃+G₇)/2 for |G₃−G₇|<=|G₁−G₉|  (7)

[0064] For the color pixel locations surrounding the clipped luminance pixel the first interpolated pixel values are used. For example, at pixel location G₁, R′ and B′ are used as the final interpolated pixel values R″ and B″ respectively. Similarly, at pixel location R₁, G′ and B′ are used as the second interpolated pixel values G″ and B″ respectively. At pixel location B₁, G′ and R′ are used as the second interpolated pixel values G″ and R″ respectively.

[0065] The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.

[0066] 2 lens

[0067] 6 optical lowpass filter

[0068] 10 image sensing device

[0069] 14 A/D converter

[0070] 26 CFA interpolator

[0071] 201 sparsely sampled digital image

[0072] 202 interpolated color digital image