Automatic color calibration of an image sensor

[0001] Embodiments of the invention relates to an image capturing device. In particular, embodiments of the invention relate to determining color correction parameters for use by a color correction system for an image capturing device.

[0002] In digital color image capturing devices, or systems, such as, for example, digital cameras, digital endoscopes, and the like, a scene is typically captured by an image sensor, such as, for example, a Charge Coupled Device (CCD), a CMOS sensor, or the like. The scene captured by the image sensor is converted into image data by the image sensor. The image data is in the form of pixel values corresponding to the image. The image data is then processed through an imaging pipeline by making use of, for example, appropriate algorithms, or the like, so as to produce an output color image. A reason for processing the image data through the imaging pipeline is to produce an accurate color output image from the image capturing device, or system, at a specified output resolution.

[0003] Typically, the output color image has three primary color values per pixel. A typical color image is defined by three primary color planes of pixels, where each pixel is typically defined by a 3-tuple in a color space, such as, for example, the Red, Green and Blue (RGB) color space. Although RGB is one of the most common color spaces currently in use, there are many other color spaces. Some examples of such other color spaces include the CMY (Cyan, Magenta and Yellow), the YUV (where Y is the luminance and U and V represent chrominance), and the HSL (Hue, Saturation, and Lightness) color spaces.

[0004] Color accuracy, namely whether the color in the scene is accurately described by the output color image, is an important consideration in providing color image output quality. Color accuracy of an image capturing device, or system, may be influenced by, for example, the physical characteristics of the capturing device, the lighting condition in the scene during image capture, etc.

[0005] To produce an accurate color output image, it is important to compensate for distortion caused by the physical characteristics of the image capturing device. It is also important to compensate for distortion which can be caused when different lighting conditions, such as daylight, artificial lighting, are present in the scene. Such compensation may be provided by means of a color correction system of the image capturing device. For such a color correction system to perform properly, it is necessary to calibrate the color correction system. Calibration of the color correction system involves determining appropriate color compensation parameters. Once appropriate compensation parameters have been determined, the compensation parameters can be employed by the color compensation system to enable the image capturing device to produce relatively accurate color output images.

[0006] Appropriate compensation parameters can include gamma curves, color channel gains and offsets, color correction parameters, and the like.

[0007] According to a first aspect of the invention, there is provided a method comprising (a) determining a white balance correction parameter for a white balance correction processing block for an image sensor, and including the white balance correction parameter in the white balance correction processing block; (b) determining a gamma correction parameter for a gamma correction processing block for the image sensor, and including the gamma correction parameter in the gamma correction processing block; and (c) determining a color correction parameter for a color correction processing block for the image sensor, and including the color correction parameter in the color correction processing block.

[0008] According to a second aspect of the invention, there is provided a method, comprising (a) disabling a gamma correction function, a gain correction function, an offset correction function, and a color correction function for a sensor; (b) capturing a first image of a color test chart with the sensor, wherein a pixel value corresponding to each color patch of the color test chart is known; (c) determining a first measured pixel value for each color patch in the first image; (d) determining gamma correction parameters for the sensor based on differences between each first measured pixel value for a color patch and the known pixel value for that color patch; (e) enabling the gamma correction function for the sensor using the gamma correction parameters determined in step (d); (f) with the gamma correction function enabled, capturing a second image of the color test chart with the sensor, and determining a second measured pixel value for each color patch in the second image; (g) determining gain and offset parameters for the gain and offset corrections, respectively based on differences between each second measured pixel value for a color patch and the known pixel value for that color patch; (h) enabling the gain correction function and the offset correction function for the sensor, wherein for the gain correction function the channel gain parameters determined in step (g) are used, and for the offset correction function the offset parameters determined in step (g) are used; (i) capturing a third image of the color test chart with the sensor and with the gamma correction function, the gain correction function, and the offset correction function enabled; (j) determining a third measured pixel value for each color patch in the third image; and (k) determining color correction parameters for the color correction function based on differences between each third measured pixel value for a color patch and the known pixel value for that color patch.

[0009] According to a third aspect of the invention, there is provided a computer-readable medium having stored thereon, a sequence of instructions which when executed by a processor, cause the processor to perform a method comprising (a) disabling a gamma correction function, a gain correction function, an offset correction function, and a color correction function for a sensor; (b) capturing a first image of a color test chart with the sensor, wherein a pixel value corresponding to each color patch of the color test chart is known; (c) determining a first measured pixel value for each color patch in the first image; (d) determining gamma correction parameters for the sensor based on differences between each first measured pixel value for a color patch and the known pixel value for that color patch; (e) enabling the gamma correction function for the sensor using the gamma correction parameters determined in step (d); (f) with the gamma correction function enabled, capturing a second image of the color test chart with the sensor, and determining a second measured pixel value for each color patch in the second image; (g) determining gain and offset parameters for the gain and offset corrections, respectively based on differences between each second measured pixel value for a color patch and the known pixel value for that color patch; (h) enabling the gain correction function and the offset correction function for the sensor, wherein for the gain correction function the channel gain parameters determined in step (g) are used, and for the offset correction function the offset parameters determined in step (g) are used; (i) capturing a third image of the color test chart with the sensor and with the gamma correction function, the gain correction function, and the offset correction function enabled; (j) determining a third measured pixel value for each color patch in the third image; and (k) determining color correction parameters for the color correction function based on differences between each third measured pixel value for a color patch and the known pixel value for that color patch.

[0010] According to a fourth aspect of the invention, there is provided a system, comprising a processor; and a memory coupled to the processor, the memory storing instructions which when executed by the processor, cause the processor to perform a method comprising (a) disabling a gamma correction function, a gain correction function, an offset correction function, and a color correction function for a sensor; (b) capturing a first image of a color test chart with the sensor, wherein a pixel value corresponding to each color patch of the color test chart is known; (c) determining a first measured pixel value for each color patch in the first image; (d) determining gamma correction parameters for the sensor based on differences between each first measured pixel value for a color patch and the known pixel value for that color patch; (e) enabling the gamma correction function for the sensor using the gamma correction parameters determined in step (d); (f) with the gamma correction function enabled, capturing a second image of the color test chart with the sensor, and determining a second measured pixel value for each color patch in the second image; (g) determining gain and offset parameters for the gain and offset corrections, respectively based on differences between each second measured pixel value for a color patch and the known pixel value for that color patch; (h) enabling the gain correction function and the offset correction function for the sensor, wherein for the gain correction function the channel gain parameters determined in step (g) are used, and for the offset correction function the offset parameters determined in step (g) are used; (i) capturing a third image of the color test chart with the sensor and with the gamma correction function, the gain correction function, and the offset correction function enabled; (j) determining a third measured pixel value for each color patch in the third image; and (k) determining color correction parameters for the color correction function based on differences between each third measured pixel value for a color patch and the known pixel value for that color patch.

[0011] Other aspects of the invention will be apparent from the detailed description below.

[0012] Embodiments of the invention will now be described, by way of example, with reference to the accompanying diagrammatic drawings, in which:

[0013] FIG. 1 shows a schematic block diagram indicating a color correction system of an image capturing device;

[0014] FIG. 2 shows an example of a gamma response curve used in a color correction system of an image capturing device;

[0015] FIG. 3 shows a schematic block diagram indicating a method of calibrating a color correction system of an image capturing device, in accordance with an embodiment of the invention;

[0016] FIG. 4 shows a color test chart containing an array of color patches, the color values of which patches are known; and

[0017] FIG. 5 shows an example of hardware that may be used to implement the embodiments of the invention.

[0018] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention. It will be apparent, however, to one skilled in the art that the invention can be practiced without these specific details. In other instances, structures and devices are shown at block diagram form only in order to avoid obscuring the invention.

[0019] Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments but not other embodiments.

[0020] Referring to FIG. 1 of the drawings, an example of a color correction system of an image capturing device is schematically and generally indicated by reference numeral 10. The color correction system 10 includes a front-end image processing block 12, a color correction block 14, and a back-end image processing block 16. In this example, the front-end processing block includes a demosaicing step 12.1, a channel gain and offset step 12.2, and a gamma correction step 12.3. Generally the front-end processing block can include other additional steps.

[0021] The front-end image processing block 12 receives captured color image data, or raw sensor data, from a sensor, such as a Charge Coupled Device (CCD), or CMOS sensor, or the like, as indicated by arrow A. The front-end image processing block 12 typically performs a demosaicing procedure 12.1 on the received data so that each pixel is defined by three primary colors in a color space, such as, for example, RGB (Red, Green, Blue), or the like.

[0022] The front-end image processing block 12 performs a color channel gain and offset step 12.2 with color gain and offset parameters B1 and a gamma correction step 12.3 with gamma correction parameters B2, on the data. The color gain and offset step 12.2 modifies the gain and offset in each individual channel so as to enhance white balance, namely, to cause white color in the scene to be rendered as white color in a color output image. The gamma correction step 12.3 compensates for non-linearity in light to electron transfer by the sensor. After the color gain and offset step 12.2 and the gamma correction step 12.3, pixel data are typically output as indicated by arrow C. The pixel data output at C can be defined as (r′_m,n, g′_m,n, b′_m,n), where r′_m,n, g′_m,n and b′_m,n are the respective red, green and blue pixel values at a coordinate (m, n).

[0023] Since the physical characteristics of the image capturing device, as well as the fact that the lighting condition in the scene can typically vary, for example, the lighting may be natural, or artificial, the values (r′_m,n, g′_m,n, b′_m,n) output at C may not represent true reflection of the actual color in the scene. It can therefore be necessary for color correction, as indicated by the block 14, so as to transform the pixel values (r′_m,n, g′_m,n, b′_m,n) into output pixel values (r_m,n, g_m,n, b_m,n), as indicated by the arrow D, that more accurately reflect the actual color in the scene. Color correction at 14 is typically performed using color compensation parameters as indicated by arrow E.

[0024] An example of equations that may be used by the front-end image processing block 12 to modify the input data A so as to compensate for gain, offset and gamma correction γ( ), are as follows:
r′_m,n=γ(λ_r{tilde over (r)}_m,n+ρ_r)
g′_m,n=γ(λ_g{tilde over (g)}_m,n+ρ_g)  (1)
b′_m,n=γ(λ_b{tilde over (b)}_m,n+ρ_b)

[0025] where the parameters λ_r, ρ_r, λ_g, ρ_g, λ_b and ρ_b represent the color gains and offsets in each channel (RGB, for example), γ( ) represents gamma correction, and {tilde over (r)}_m,n,{tilde over (g)}_m,n,{tilde over (b)}_m,n represent the captured color image data at A.

[0026] The gamma correction γ( ) can typically be represented by a non-linear curve as shown in FIG. 2 of the drawings, where the input is shown at 20 and the output is shown at 22.

[0027] The color correction block 14 may include a mapping between input and output color vectors. Such a mapping may be performed by, for example, matrix multiplication, or non-linear mathematical mapping, or using table look up approaches, or the like. The mapping can be performed in RGB or any other appropriate color spaces, such as L*a*b*, for example.

[0028] For example, color correction can be performed by using a 3×3 matrix, as follows: $\begin{array}{cc}\left(\begin{array}{c}{r}_{m,n}\\ g_{m,n}\\ b_{m,n}\end{array}\right)=\left(\begin{array}{ccc}{a}_{0,0}& a_{0,1}& a_{0,2}\\ a_{1,0}& a_{1,1}& a_{1,2}\\ a_{2,0}& a_{2,1}& a_{2,2}\end{array}\right)\left(\begin{array}{c}{r}_{m,n}^{\prime }\\ g_{m,n}^{\prime }\\ b_{m,n}^{\prime }\end{array}\right).& \left(2\right)\end{array}$

[0029] Instead, for example, color correction may be performed by using difference values of the color channels, as follows: $\begin{array}{cc}\left(\begin{array}{c}{r}_{m,n}\\ g_{m,n}\\ b_{m,n}\end{array}\right)=\left(\begin{array}{c}{r}_{m,n}^{\prime }\\ g_{m,n}^{\prime }\\ b_{m,n}^{\prime }\end{array}\right)+\left(\begin{array}{cccc}{d}_{0,0}& d_{0,1}& \cdots & d_{0,K-1}\\ ⋮& & & ⋮\\ d_{2,0}& d_{2,1}& \cdots & d_{2,K-1}\end{array}\right)\left(\begin{array}{c}{f}_0\left(r_{m,n}^{\prime },g_{m,n}^{\prime },b_{m,n}^{\prime }\right)\\ f_1\left(r_{m,n}^{\prime },g_{m,n}^{\prime },b_{m,n}^{\prime }\right)\\ ⋮\\ f_{K-1}\left(r_{m,n}^{\prime },g_{m,n}^{\prime },b_{m,n}^{\prime }\right)\end{array}\right)& \left(3\right)\end{array}$

[0030] where f_i( )'s are typically functions of the RGB values at pixel location (m, n).

[0031] The color compensation parameters a_i,j's or d_i,j's in the equations (2) and (3) typically vary with the physical characteristics of the image capturing device and the lighting condition in the scene. However, they typically do not change from one pixel to the next. For an image capturing device with a given sensor, a plurality of sets of color correction parameters can be stored for use in different types of lighting condition. For example, one set of parameters can be stored for a daylight lighting condition, one set of parameters can be stored for a tungsten light lighting condition, one set of parameters can be stored for a fluorescent light lighting condition, and the like. During actual image capture, an appropriate set of color correction parameters corresponding to the type of lighting condition in the scene is typically selected, as indicated by arrow E in FIG. 1 of the drawings.

[0032] Determining the gamma, the gain, the offset, and the color correction parameters can be problematic. It is possible to set up the image capturing device by manually adjusting the various parameters through trial and error in various lighting conditions, until the output colors are satisfactory. However, such a procedure is impractical for a number of reasons. Firstly, accurate color correction is typically determined by a combination of all the parameters. Adjustment of a parameter on its own typically cannot compensate for the color characteristics of the image capturing device properly. Accordingly, adjustment of each parameter individually typically can only bring the color response of the system to an intermediate state. Therefore, if the parameters are adjusted one by one, it is typically not possible to determine if the value for each particular parameter is at an optimum. Secondly, the parameters are not independent of one another. Accordingly, changing one parameter can alter the effect of the others. It is therefore not possible to anticipate what effect other parameters will have on the system when changing any particular parameter. Thirdly, each parameter can be quantized so that each parameter can take on one of L possible discrete values and then a search can be performed over the discrete space of the parameters to find optimum values. However, the size of such a discrete space grows exponentially with the number of parameters. Accordingly, an exhaustive search is normally unfeasible even for a relative small value of L, such as 32, for example. L=32 is typically not a fine enough amount of quantization for the parameters.

[0033] An embodiment of a method of determining color compensation parameters for a color image capturing device, in accordance with the invention, will now be described with reference to FIGS. 3 and 4 of the drawings.

[0034] In accordance with such a method, color test charts 50, as indicated in FIG. 4 of the drawings, are used as test data. A standard color test chart 50 typically comprises a number of color patches 50.1.1, 50.1.2, 50.1.y, . . . , 50.x.1, . . . , 50.x.y. The correct color values for the patches on the test chart 50 are known. The color image capturing device can then be calibrated as follows:

[0035] Firstly, as can best be seen in FIG. 3 of the drawings, the test chart 50 is imaged through the image capturing device 52 as indicated by arrow F, while the gamma, the gain and the offset, and the color correction steps in the image capturing device 52 are disabled. An automatic color correction parameter determination system 54 compares a set of measured output values, indicated by arrow G, with the known values of the color chart 50, as indicated by arrow H, to determine a gamma response curve γ( ) as indicated by arrow I.

[0036] The gamma response thus determined is then enabled at 52. The test chart 50 is then imaged again. A new set of measured output values G is then used to determine the channel gain and the offset parameters λ_r, ρ_r, λ_g, ρ_g, λ_b and ρ_b at 54 as indicated by arrow I.

[0037] The gamma response, the channel gain and the offset parameters are then enabled at 52. A new set of measured output values G is then used to determine the color correction parameters, a_i,j's or d_i,j's at 54 as indicated by arrow I. In this way all the color compensation parameters can be determined.

[0038] To improve accuracy, the measured value of each color patch at each stage of the procedure is typically obtained by sampling pixels within the color patch in the captured image. Accordingly, an average measured sample value of a selected region within each color patch can be computed by: $\begin{array}{cc}{r}_i^e=\underset{P_i}{\mathrm{avg}}\left(r_{m,n}^{\prime }\right)g_i^e=\underset{P_i}{\mathrm{avg}}\left(g_{m,n}^{\prime }\right)b_i^e=\underset{P_i}{\mathrm{avg}}\left(b_{m,n}^{\prime }\right)& \left(4\right)\end{array}$
where P_i represents the selected region within the i^th color patch.

[0039] After such a calibration procedure has been performed, the set of measured color patch values and its corresponding set of known color patch values form a pair that define an input-output characteristic of the color correction system. Accordingly, if the color correction system receives a pixel value during actual image capture that is equal to one of the known color patch values, then the color correction system should produce the corresponding true output color value as output.

[0040] The determination of the gamma response curve will now be described in greater detail. To determine the gamma response curve, a sequence of gray color patches at various gray level values can be used. Initially, the gain parameters λ_r, λ_g and λ_b are typically set to one and the offset parameters ρ_r, ρ_g and ρ_b are typically set to zero. The input intensity of the device may be set up so that the response of the image sensor to an input white patch is just below saturation. The relationship between the known RGB values of the input gray patches and the measured gray level values (r′_m,n, g′_m,n, b′_m,n) then gives the transfer characteristics of the sensor. The average measured sample value over a selected region within each color patch is typically determined by equation (4) above, for example. The gamma response curve γ( ) may be the inverse characteristics of the mapping in the sensor, namely, the gamma curve maps the measured values (r^e_i, g^e_i, b^e_i) to the known RGB values of the gray patches.

[0041] The next step in the color calibration process is to determine the color channel gains and offsets. The purpose of this step is to compensate for the color distortion caused by the type of lighting condition in the scene. For example, if the imaging system is used under incandescent lighting, the color of the light will cause a white patch to appear yellowish. To produce accurate output color in the imaging system, this should be corrected. Accordingly, the color calibration process should be performed under different types of lighting conditions so as to provide color compensation parameter sets for different lighting conditions.

[0042] The color channel gains and offsets are then determined by reducing the differences between the three color channels when measured over gray colored patches. Two relative gain parameters are defined as follows: ${\mu }_g=\frac{{\lambda }_g}{{\lambda }_r}\mathrm{and}{\mu }_b=\frac{{\lambda }_b}{{\lambda }_r}.$

[0043] A cost function is then defined as follows: $D=\sum _i\lfloor {\left(r_i^e-{\mu }_g{g}_i^e\right)}^2+{\left(r_i^e-{\mu }_b{b}_i^e\right)}^2+{\left({\mu }_g{b}_i^e-{\mu }_b{b}_i^e\right)}^2\rfloor$

[0044] where r_i^e, g_i^e and b_i^e are measured color values over the i^th gray patch. The measured color values are typically averaged as shown in equation (4) over a selected area of each gray patch to reduce measurement noise. The optimum relative gain parameters μ_g and μ_b can then be found by minimizing D. As an example, one method to minimize D is by setting the partial derivatives of D with respect to μ_g and μ_b to zero, which gives the optimum solution as follows: ${\mu }_g=\frac{2\sum _i{\left(b_i^e\right)}^2\sum _i{r}_i^e{g}_i^e+\sum _i{g}_i^e{b}_i^e\sum _i{r}_i^e{b}_i^e}{4\sum _i{\left(g_i^e\right)}^2\sum _i{\left(b_i^e\right)}^2-{\left(\sum _i{g}_i^e{b}_i^e\right)}^2}$ ${\mu }_b=\frac{2\sum _i{\left(g_i^e\right)}^2\sum _i{r}_i^e{b}_i^e+\sum _i{g}_i^e{b}_i^e\sum _i{r}_i^e{g}_i^e}{4\sum _i{\left(g_i^e\right)}^2\sum _i{\left(b_i^e\right)}^2-{\left(\sum _i{g}_i^e{b}_i^e\right)}^2}$

[0045] From the two relative gain parameters, the channel which has the largest absolute gain value of the three is determined. The absolute gain of the channel which has the largest gain is then set to 1. The absolute gain of the remaining two channels is then determined. Finally, the absolute gains are normalized using the measured and target data. The procedure can be summarized by the following pseudo code, as follows:

	if (μg > 1){
	if (μg > μb){
	λr = 1/μg; λg = 1; λb = μb/μg;
	}else{
	λr = 1/μb; λg = μg/μb; λb = 1;
	}
	}else{
	if (μb = 1){
	λr = 1/μb; λg = μg/μb; λb = 1;
	}else{
	λr = 1; λg = μg; λb = μb;
	}
	}
	κ = ⁢ min ( γ - 1 ⁡ ( r white ′ ) - γ - 1 ⁡ ( r black ′ ) λ r ⁡ ( r ~ white - r ~ black ) , γ - 1 ⁡ ( g white ′ ) - γ - 1 ⁡ ( g black ′ ) λ g ⁡ ( g ~ white - g ~ black ) , ⁢ γ - 1 ⁡ ( b white ′ ) - γ - 1 ⁡ ( b black ′ ) λ g ⁡ ( b ~ white - b ~ black ) ) ;
	λr = κλr; λg = κλg; λb = κλb;
	ɛ = avg ⁢   ⁢ ( γ - 1 ⁡ ( r black ′ ) , γ - 1 ⁡ ( g black ′ ) , γ - 1 ⁡ ( b black ′ ) ) ;
	ρ r = ɛ - λ r ⁢ r ~ black ; ρ g = ɛ - λ g ⁢ g ~ black ; ρ b = ɛ - λ b ⁢ b ~ black ;

[0046] The determination of the color compensation parameters will now be described in greater detail.

[0047] Equation (2) or (3), depending on which one is used, can be solved for the coefficients so as to determine the optimum color correction parameters. Equations (2) and (3) have very similar structures and they can typically be solved in a similar manner. As an example, consider a solution for the equation (2). Suppose there are N color patches in the color test chart of FIG. 4, so that equation (2) is applicable to the color values corresponding to all N patches. The mapping can then be as follows: $\begin{array}{cc}\left(\begin{array}{ccc}{r}_0^e& g_0^e& b_0^e\\ r_1^e& g_1^e& b_1^e\\ ⋮& & ⋮\\ r_{N-1}^e& g_{N-1}^e& b_{N-1}^e\end{array}\right)\left(\begin{array}{c}{a}_{0,0}\\ a_{0,1}\\ a_{0,2}\end{array}\right)=\left(\begin{array}{c}{r}_0^t\\ r_1^t\\ ⋮\\ r_{N-1}^t\end{array}\right)& \left(5\right)\\ \left(\begin{array}{ccc}{r}_0^e& g_0^e& b_0^e\\ r_1^e& g_1^e& b_1^e\\ ⋮& & ⋮\\ r_{N-1}^e& g_{N-1}^e& b_{N-1}^e\end{array}\right)\left(\begin{array}{c}{a}_{1,0}\\ a_{1,1}\\ a_{1,2}\end{array}\right)=\left(\begin{array}{c}{g}_0^t\\ g_1^t\\ ⋮\\ g_{N-1}^t\end{array}\right)& \left(6\right)\\ \left(\begin{array}{ccc}{r}_0^e& g_0^e& b_0^e\\ r_1^e& g_1^e& b_1^e\\ ⋮& & ⋮\\ r_{N-1}^e& g_{N-1}^e& b_{N-1}^e\end{array}\right)\left(\begin{array}{c}{a}_{2,0}\\ a_{2,1}\\ a_{2,2}\end{array}\right)=\left(\begin{array}{c}{b}_0^t\\ b_1^t\\ ⋮\\ b_{N-1}^t\end{array}\right)& \left(7\right)\end{array}$
where (r_i^t, g_i^t, b_i^t) represent the true color values for the i^th color patch and (r_i^e, g_i^e, b_i^e) represent the average measured color values for the i^th color patch as in equation (4). An aim in the color correction system is to find the color correction parameters a_0,0, a_0,1, . . . a_2,2 such that (5)-(7) can be satisfied.

[0048] Let A be the N by 3 matrix, given by: $\begin{array}{cc}A=\left(\begin{array}{ccc}{r}_0^e& g_0^e& b_0^e\\ r_1^e& g_1^e& b_1^e\\ ⋮& & ⋮\\ r_{N-1}^e& g_{N-1}^e& b_{N-1}^e\end{array}\right).& \left(8\right)\end{array}$

[0049] Preferably, in order to solve A for the values of a_0,0, a_0,1, . . . a_2,2, the total number of color patches N should not be less than 3. Equations (5)-(7) can be solved for any value of N larger than, or equal to, 3 using, for example, a least square fit. However, it is desirable that N is larger than 3. In such a case, each matrix equation is an over-determined set of linear equations, from which a least square solution can be found. The cost of finding the solution increases with N. Since the solution is computed off-line (i.e. not during real image capture), hence the size of N does not cause a burden regarding the functioning of the color correction system.

[0050] All three equations (5)-(7) can be solved by finding a pseudo inverse of A in equation (8). Since all three equations (5)-(7) have the same A matrix on the left hand side, the typically most computationally intensive portion of the solution, that of finding a pseudo inverse, will only have to be performed once.

[0051] To find a least square solution to a system of over-determined linear equations, a method based on the Singular Value Decomposition (SVD) of matrices can be used. Specifically, the SVD of the N by 3 matrix A (with N≧3) is given by:
A=UΣV^T

[0052] where U is an N by 3 orthogonal matrix, V is a 3 by 3 orthogonal matrix, T represents matrix transpose, and Σ is a 3×3 diagonal matrix given by: $\Sigma =\left[\begin{array}{ccc}{s}_0& 0& 0\\ 0& s_1& 0\\ 0& 0& s_2\end{array}\right]$

[0053] where S_i's are the singular values of A.

The pseudo inverse of A is:
A⁻¹=VΣ⁻¹U^T

[0054] where Σ⁻¹ is a diagonal matrix given by: ${\Sigma }^{-1}=\left[\begin{array}{ccc}\mathrm{inv}\left(s_0\right)& 0& 0\\ 0& \mathrm{inv}\left(s_1\right)& 0\\ 0& 0& \mathrm{inv}\left(s_2\right)\end{array}\right]$ $\mathrm{where}$ $\mathrm{inv}\left(\alpha \right)=\{\begin{array}{cc}1/\alpha & \mathrm{if}\alpha >\varepsilon \\ 0& \mathrm{otherwise}\end{array}$
for some pre-determined small value . As an example, the value ε=10⁻⁶ can be used.

[0055] Using the SVD, the optimum color correction coefficients can be found by: $\begin{array}{cc}\left[\begin{array}{c}{a}_{0,0}\\ a_{0,1}\\ a_{0,2}\end{array}\right]=V{\Sigma }^{-1}{U}^T\left(\begin{array}{c}{r}_0^t\\ r_1^t\\ ⋮\\ r_{N-1}^t\end{array}\right)& \left(9\right)\\ \left[\begin{array}{c}{a}_{1,0}\\ a_{1,1}\\ a_{1,2}\end{array}\right]=V{\Sigma }^{-1}{U}^T\left(\begin{array}{c}{g}_0^t\\ g_1^t\\ ⋮\\ g_{N-1}^t\end{array}\right)& \left(10\right)\\ \left[\begin{array}{c}{a}_{2,0}\\ a_{2,1}\\ a_{2,2}\end{array}\right]=V{\Sigma }^{-1}{U}^T\left(\begin{array}{c}{b}_0^t\\ b_1^t\\ ⋮\\ b_{N-1}^t\end{array}\right)& \left(11\right)\end{array}$

[0056] There are a number of reasons why Singular Value Decomposition can be a preferred way of solving (5)-(7) in the color correction system. Firstly, SVD is relatively effective in compacting the “energy” of matrices into a set of orthogonal components. In other words, SVD provides a relatively efficient decomposition of any given matrix into its main components. When the color correction system finds the pseudo inverse using SVD, the major modes in the A matrix are captured as given by the data points. This implies robustness in the solution.

[0057] Secondly, SVD always exists for any matrix of any size. As a result, it is normally not necessary to constrain the nature of the N data points. This does not necessarily imply that the “richness” of the color patches is unimportant. It is typically still necessary to use a color test chart so that the color patches span a large portion of the entire color gamut. Even if the color correction system results in a set of measured data forming a sub-space of less than 9 dimensions, the SVD method could still find a solution that adequately captures the characteristics of the data.

[0058] Thirdly, SVD can typically be computed relatively efficiently and the solution is robust. Once the SVD is computed, the optimum color correction coefficients for the given sensor and lighting condition can be determined using equations (9)-(11). These values can then be used during capture in (2) to obtain accurate color image data.

[0059] If equation (3) is used instead, the optimum solution for the color coefficients d_i,j can be solved in a similar fashion using SVD. In such a case, equations (5)-(7) should be modified to $\left(\begin{array}{cccc}{f}_0\left(r_0^e,g_0^e,b_0^e\right)& f_1\left(r_0^e,g_0^e,b_0^e\right)& \cdots & f_{K-1}\left(r_0^e,g_0^e,b_0^e\right)\\ f_0\left(r_1^e,g_1^e,b_1^e\right)& & & ⋮\\ ⋮& & & ⋮\\ f_0\left(r_{N-1}^e,g_{N-1}^e,b_{N-1}^e\right)& \cdots & \cdots & f_{K-1}\left(r_{N-1}^e,g_{N-1}^e,b_{N-1}^e\right)\end{array}\right)\left(\begin{array}{c}{d}_{0,0}\\ d_{0,1}\\ ⋮\\ d_{0,K-1}\end{array}\right)=\left(\begin{array}{c}{r}_0^t-r_0^e\\ r_1^t-r_1^e\\ ⋮\\ r_{N-1}^t-r_{N-1}^e\end{array}\right)$ $\left(\begin{array}{cccc}{f}_0\left(r_0^e,g_0^e,b_0^e\right)& f_1\left(r_0^e,g_0^e,b_0^e\right)& \cdots & f_{K-1}\left(r_0^e,g_0^e,b_0^e\right)\\ f_0\left(r_1^e,g_1^e,b_1^e\right)& & & ⋮\\ ⋮& & & ⋮\\ f_0\left(r_{N-1}^e,g_{N-1}^e,b_{N-1}^e\right)& \cdots & \cdots & f_{K-1}\left(r_{N-1}^e,g_{N-1}^e,b_{N-1}^e\right)\end{array}\right)\left(\begin{array}{c}{d}_{1,0}\\ d_{1,1}\\ ⋮\\ d_{1,K-1}\end{array}\right)=\left(\begin{array}{c}{g}_0^t-g_0^e\\ g_1^t-g_1^e\\ ⋮\\ g_{N-1}^t-g_{N-1}^e\end{array}\right)$ $\left(\begin{array}{cccc}{f}_0\left(r_0^e,g_0^e,b_0^e\right)& f_1\left(r_0^e,g_0^e,b_0^e\right)& \cdots & f_{K-1}\left(r_0^e,g_0^e,b_0^e\right)\\ f_0\left(r_1^e,g_1^e,b_1^e\right)& & & ⋮\\ ⋮& & & ⋮\\ f_0\left(r_{N-1}^e,g_{N-1}^e,b_{N-1}^e\right)& \cdots & \cdots & f_{K-1}\left(r_{N-1}^e,g_{N-1}^e,b_{N-1}^e\right)\end{array}\right)\left(\begin{array}{c}{d}_{2,0}\\ d_{2,1}\\ ⋮\\ d_{2,K-1}\end{array}\right)=\left(\begin{array}{c}{b}_0^t-b_0^e\\ b_1^t-b_1^e\\ ⋮\\ b_{N-1}^t-b_{N-1}^e\end{array}\right)$

[0060] In such a case, the matrix A for which the SVD needs to be found would be of dimension N×K. Other than that, the solution can be found in a fashion similar to that described above.

[0061] Referring to FIG. 5 of the drawings, an example of hardware 60 that may be used to implement the color calibration methods, in accordance with one embodiment of the invention. The hardware 60 typically includes at least one processor 62 coupled to a memory 64. The processor 62 may represent one or more processors (e.g., microprocessors), and the memory 64 may represent random access memory (RAM) devices comprising a main storage of the hardware 60, as well as any supplemental levels of memory e.g., cache memories, non-volatile or back-up memories (e.g. programmable or flash memories), read-only memories, etc. In addition, the memory 64 may be considered to include memory storage physically located elsewhere in the hardware 60, e.g. any cache memory in the processor 62, as well as any storage capacity used as a virtual memory, e.g., as stored on a mass storage device 70.

[0062] The hardware 60 also typically receives a number of inputs and outputs for communicating information externally. For interface with a user or operator, the hardware 60 may include one or more user input devices 66 (e.g., a keyboard, a mouse, etc.) and a display 68 (e.g., a Cathode Ray Tube (CRT) monitor, a Liquid Crystal Display (LCD) panel).

[0063] For additional storage, the hardware 60 may also include one or more mass storage devices 70, e.g., a floppy or other removable disk drive, a hard disk drive, a Direct Access Storage Device (DASD), an optical drive (e.g. a Compact Disk (CD) drive, a Digital Versatile Disk (DVD) drive, etc.) and/or a tape drive, among others. Furthermore, the hardware 60 may include an interface with one or more networks 62 (e.g., a local area network (LAN), a wide area network (WAN), a wireless network, and/or the Internet among others) to permit the communication of information with other computers coupled to the networks. It should be appreciated that the hardware 60 typically includes suitable analog and/or digital interfaces between the processor 62 and each of the components 64, 66, 68 and 72 as is well known in the art.

[0064] The hardware 60 operates under the control of an operating system 74, and executes various computer software applications, components, programs, objects, modules, etc. e.g. a program or module 76 which performs operations described above. Moreover, various applications, components, programs, objects, etc. may also execute on one or more processors in another computer coupled to the hardware 60 via a network 72, e.g. in a distributed computing environment, whereby the processing required to implement the functions of a computer program may be allocated to multiple computers over a network.

[0065] In general, the routines executed to implement the embodiments of the invention, may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as “computer programs.” The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processors in a computer, cause the computer to perform operations necessary to execute elements involving the various aspects of the invention. Moreover, while the invention has been described in the context of fully functioning computers and computer systems, those skilled in the art will appreciate that the various embodiments of the invention are capable of being distributed as a program product in a variety of forms, and that the invention applies equally regardless of the particular type of machine or computer-readable media used to actually effect the distribution. Examples of computer-readable media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others, and transmission type media such as digital and analog communication links.