Method and apparatus for automated optimization of white and color balance on video camera

[0002] (1) Field of the Invention

[0003] The field of the invention is calibration of a video capture device, more specifically the present invention is a method and an apparatus for automated optimization of white and color balance on a video camera.

[0004] (2) Related Art

[0005] Presently available video cameras are often equipped with various manual calibration features including white balance which allows the manual calibration of white images. Lighting is an important factor in the process of accurate video capture of clear white and color balanced images. Typically, when a video camera is turned on in an arbitrary lighting environment, the target captured by the video camera may not be perceived in the correct color. For example, whites may not be perceived as white and human faces may be captured with a slight green tint.

[0006] There are various disadvantages inherent in the video capture method of prior art technology. For example, presently available video cameras are typically equipped with manual white balancing function which adjusts “white” targets captured by a video camera and fine tunes the “white” into a more realistic shade of white. Unfortunately, it is difficult to correctly perform manual white balance due to the operator judgment and intervention required to set color controls. Furthermore, the manual white balancing function of prior art is catered to adjusting white and does not necessarily work for adjusting colors. Additionally, an auto white balancing function available in some video cameras may work poorly since no white reference is provided for the video camera to make the white color adjustment.

[0007] It is therefore desirable to provide an automatic method and apparatus to calibrate white as well as color for targets captured by a video camera.

[0008] A method for optimizing white and color balance on a video capture device is disclosed. The method comprises the steps of capturing an image of an optical target through the video capture device and calibrating the captured image using a predetermined reference image having desired white and color parameters. The calibration results in the captured image matching the predetermined reference image.

[0009] FIG. 1 is an exemplary system block diagram of the present invention.

[0010] FIG. 2 is an exemplary optical target used by the present invention in adjusting white and color balance.

[0011] FIG. 3 is a table illustrating derivation of YUV color and white representation as derived from standard RGB color.

[0012] FIG. 4 is a flow diagram illustrating the general steps followed by the present invention in performing image recognition and data extraction from an optical target prior to performing white and color balance.

[0013] FIGS. 5a-5care flow diagrams illustrating the general steps followed by the present invention in performing white and color balance of a target image.

[0014] FIG. 6 is a block diagram of an exemplary video camera receiving gain and offset values from a computer.

[0015] The present invention is a method and an apparatus for optical target and calibration software for automated optimization of color balance on video cameras. The present invention uses an optical target in combination with calibration software capable of recognizing the optical target. The calibration software adjusts the camera until the perceived image matches the correct image available to the calibration software. More specifically, the optical target includes both white and non-white colors of known intensity. Since the calibration software has knowledge of the correct appearance of the optical target, the calibration software is capable of adjusting the video camera's response such that the image observed matches the image's known characteristics. Once having been adjusted for a given light condition, any other objects then observed under the same lighting conditions will appear in their true and accurate colors as captured by the video camera.

[0016] The present invention may reduce operator intervention. Further, the present invention may automatically align the observed image of the optical target with a stored reference image and adjusts for a minimum difference between the observed image and the reference image.

[0017] FIG. 1 is an exemplary illustration of the system of the present invention. A personal computer (PC) 103 is coupled to a video camera 102. The video camera 102 is directed to an optical target 100 with a Y, U, V color chart optimized for image recognition and a gray background denoted by hatching.

[0018] More specifically, Y represents luminance, separate from color, and denotes a brightness scale somewhere between black and white. U and V are color difference values describing colors. A more detailed specification of the Y, U, V color representation may be found in Recommendation ITU-R BT.601-4, Encoding Parameters of Digital Television for Studios, International Telecommunications Union, published in 1994.

[0019] The computer 103 is further coupled to a display device 104 such as a computer monitor to display the optical target as captured by the video camera and an input device (not shown) such as a keyboard and/or mouse allowing a user to initiate the automatic color and white balance using the present invention's image recognition 112 and calibration 114 mechanisms. The image recognition mechanism 112 and calibration mechanism 114 reside in a storage element 110 which is coupled to a processor 106 by a bus 108. The calibration mechanism 114 has a white balancer 116, a color balancer 118 and a YUV-RGB converter 120 all used in automatically calibrating the video capture device 102.

[0020] The processor 106 runs the image recognition mechanism 112 and the calibration mechanism 114 upon initiation of the mechanisms by a user input through a graphic user interface (GUI) (not shown). Given the detailed description of the present invention provided herein, a person skilled in the art would be able to generate a GUI suitable to initiate the image recognition and calibration mechanisms 112 and 114, respectively, of the present invention.

[0021] The image recognition mechanism 112 and the calibration mechanism 114 of the present invention may also reside in a remote storage element coupled to the computer 103 for example through a network such as a wide area network (WAN) or a local area network (LAN).

[0022] FIG. 2 is an exemplary reference image which is compared to the perceived image of the optical target. The optical target perceived by the video camera is identical in format and actual white and color configuration as its corresponding reference image. Any deviation is created by the video camera's inaccurate perception of the white and color values of the optical target. Additionally, as was shown in FIG. 1, the optical target has a background such as gray which separates each patch in the target. As will be described later, the background is only necessary for the image recognition mechanism of the present invention in distinguishing and ultimately identifying each patch in the optical target and is not necessary to perform the comparison between the perceived optical target and the reference image. Thus, the background is irrelevant in the case of the reference image illustrated in FIG. 2.

[0023] With the present invention, a video capture device first captures an image of the optical target. The video capture device then aligns its frame against the optical target by utilizing an image recognition mechanism to recognize that there are 5 columns of patches (or 5 rows of patches) at an equal distance from each other. Once the video camera frame is properly aligned with the optical target, the video capture device and the present invention's calibration mechanism is ready to calibrate the white and color balance of any image under the same lighting.

[0024] The present invention's calibration mechanism has access to a “perfect” copy of the optical target as represented in numerics of Y, U, V values also referred herein as a reference image. The optical target as captured by the video camera is then compared against the reference image by the calibration mechanism to calibrate the video capture device.

[0025] More specifically, a reference image 200 is an array of 25 rectangles having Y, U, V values depicting various specially selected colors. The reference image 200 (as is the optical target) represents a bit-map where each color unit (pixel) may be located in an X-Y axis configuration.

[0026] The exemplary reference image specifies varying YUV values for each patch. From left to right, row 1 patch 1 contains the CCIR-601 specification defined legal minimum for Y (luminance). Row 1 patch 2 contains a Y value which is half-way between CCIR-601 specification defined minimum and median. Row 1 patch 3 contains a Y value which is a CCIR-601 specification defined median. Row 1 patch 4 contains a Y value which is half-way between CCIR-601 specification defined median and maximum values. Finally, Row 1 patch 5 contains a Y value which is a CCIR-601 specification defined maximum. The Y values are defined as such all while holding U and V values at 128 which is neutral and represents no color. The U and V values are held at 128 for setting the gain and offset of Y, without interaction from the U and V color difference values.

[0027] Rows 2 and 3 are tailored to exercise the U color channel only, except for column 3, which sets the color vector to 315 degrees and 135 degrees, respectively. The vectors in column 3 are chosen to bisect the +U, −V and the −U, +V axis, providing a good linearity check for color. In row 3, the U channel sequentially assumes the following values: patch 1 has a U value which is near the CCIR-601 specification-defined legal minimum, patch 2 has a U value which is half-way between the CCIR-601 specification defined minimum and median values, patch 3 has a U value which sets the color vector to 135 degrees, patch 4 has a U value which is half-way between the CCIR-601 specification defined median and maximum values, and patch 5 has a U value near the CCIR-601 specification defined maximum legal value. The U values are defined all while holding the V value at 128 (except for column 3), and ensuring that the corresponding R, G, or B values do not saturate beyond legal bounds in their equivalent color space. The table in FIG. 3 describes the YUV to RGB relationships as defined in CCIR-601 in more detail.

[0028] The resulting values of the Y values from row 3 are used in row 2 which has monochromatic patches (except for column 3). The row 2 monochromatic patches provide luminance linearity references. More specifically, luminance is a linear function and the degree of luminance may be verified by checking multiple points along the line as defined by the linear function. Here, the monochromatic patches are the reference points which may be used to verify luminance.

[0029] Rows 4 and 5 are tailored to exercise the V color channel only, except for column 3, which sets the color vector to 45 degrees and 225 degrees, respectively. The vectors in column 3 are chosen to bisect the +U, +V and the −U, −V axis, providing another good linearity check for color. In row 5, the V color channel sequentially assumes the following values: patch 1 has a V value near the CCIR-601 specification-defined legal minimum, patch 2 has a V value half-way between the CCIR-601 minimum and median, patch 3 has a V value which sets the color vector to 225 degrees, patch 4 has a V value which is half-way between the CCIR-601 median and maximum values, and patch 5 has a V value near the CCIR-601 maximum legal value.

[0030] The V values are set while holding U at 128 (except for column 3), and ensuring that the corresponding R, G or B values do not saturate beyond legal bounds in their equivalent color space. The resulting values of Y from row 5 are used in row 4, which has monochromatic patches (except for column 3). The monochromatic patches provide more luminance linearity references.

[0031] Other suitable optical targets with varying configurations of YUV and/or RGB values may be realized by a person skilled in the art given the detailed description provided herein.

[0032] FIG. 3 is a table illustrating the generation of a YUV color and white representation derived from the CCIR-601 RGB color and white standard. More specifically, the table illustrates the algorithm utilized by the calibration mechanism 114 in performing conversions 300 between YUV and RGB values. The conversion if performed when determining offset and gain values to be provided to the video camera for calibration of the perceived white and color values to match the “perfect” copy of the optical target (i.e. the reference image).

[0033] In the illustrated exemplary conversion method, it is assumed that YUV is CCIR-601 compliant. Further, the given method is exemplary for RGB values having a range of 0 to 255, Y value having a range of 16 to 235 and U and V values having a range of 16 to 240.

[0034] FIG. 4 is a flow diagram illustrating the general steps followed by the present invention in performing image recognition and data extraction from the optical target using the image recognition mechanism of the present invention. As was illustrated in FIG. 1, an optical target is placed such that the optical target fills the video camera's field of view with all given color patches and the surrounding gray background visible on a monitor coupled to the video camera. The black patch may be oriented in any of the corners of the optical target. The calibration software is then invoked on a computer coupled to the video camera and the monitor. Precise alignment of the optical target with the video camera is unnecessary since the present invention will find a cluster of pixels from the center of each patch as it identifies the patches by image recognition.

[0035] More specifically, in step 401, the image recognition mechanism of the present invention scans across successive horizontal lines of the optical target captured by the video camera beginning from the top of the optical target. In step 402, contrast transitions indicative of a transition between the background to the patches are noted from line to line and the top edge of the first row of patches is identified. The background surrounds all patches on the optical target. The contrast transitions are sudden changes in luminance between patches and the background.

[0036] The image recognition mechanism then scans successive lines to discern the horizontal structure on the optical target by counting the top edges of each patch, or contrast transitions across horizontal lines and noting that there are five differing wide regions (patches) which are separated by the background. In step 403, successive lines on the captured optical target is scanned and left and right edges (bounds) of each patch are identified by noting the contrast transitions. In step 404, once the left and right bounds of the patches are identified, the center of each patch on an X-axis is calculated For example, the center of a patch on the X-axis may be calculated by subtracting the X-axis value of the left edge of a patch from the X-axis value of the right edge of the patch and dividing the result by two. In step 405, the process is repeated for consecutive horizontal lines until the bottom edge of the top row of patches is reached where the horizontal transitions are no longer seen while the interposing background is being scanned.

[0037] In step 406, once the top and bottom bounds of the patches are identified, the location of the center of each patch on the Y-axis is calculated. For example, the center of a patch on the Y-axis may be calculated by subtracting the Y-axis value of the bottom edge of a patch from the Y-axis value of the top edge of the patch and dividing the result by two.

[0038] In step 407, after continuing the scan process for several more successive horizontal lines (the height of the background strip between rows of patches), additional contrast transitions are again noted from line to line when the top edge of the second row of patches is scanned. The process is repeated until the left, right, top and bottom boundaries of each row of patches are identified and the center of each patch on the X-axis and the Y-axis are determined.

[0039] Since vertical, horizontal and rotational alignment of the optical target and the video camera will likely be imperfect, the exact edge placement of the patches as discerned by the image recognition mechanism will not constitute precise horizontal rows of patches and vertical columns of patches. Consequently, the transitions within a given scan line from within a background boundary to within, for example, five patches, may occur over several scan lines in numbers of increments proportional to the rotational skew of the optical target versus the video camera. The present invention's image recognition approach is intentionally tolerant of these issues since it uses only the greatest observed X and Y extents for each patch and then calculates to find the center of each patch.

[0040] In step 408, an array of pixels from the center of each patch is sampled. For example, a 4×4 array of 16 pixels are sampled. In step 409, the bit values of the pixels representative of YUV values for each patch are then averaged. The resulting averaged YUV values one for each patch represent the pre-calibrated response of the video camera to the stimulus of the optical target.

[0041] FIGS. 5a-5care flow diagrams illustrating the general steps followed by the present invention in performing the white and color balance. YUV as referred in FIG. 5 and 6 and the accompanying detailed description is the raw, uncorrected data as perceived by the video camera from the optical target, and Y′U′V′ is the gain and offset compensation to the YUV necessary to make them correct.

[0042] In step 501, execution of the white balancer of the calibration mechanism is initiated. By convention, white balance must be performed in the RGB (red, green and blue) color space. More specifically, white balance requires calibration of the G value for white. Additionally, the R and B values must be comparatively calibrated against the G value. The calibration mechanism of the present invention must therefore either calculate R′G′B′ from the Y′U′V′ it obtains from the video camera or a decoder, or use a given R′G′B′ directly if the R′G′B′ is supplied directly. This YUV to RGB conversion is illustrated in FIG. 6's calibration adjustment and color space conversion matrices.

[0043] In step 502, the Y, U and V gain settings are initialized by being set to a nominal value. More specifically, gain is a multiplicative scale factor ranging, for example, from 0.5 to 1.5 used in adjusting raw uncorrected YUV values. A nominal value for a gain setting ranging from 0.5 to 1.5 is 1.0. In step 503, the corresponding offset controls are set to zero. More specifically, offset is an additive correction value added to the raw YUV values and ranges from −20 to +20. The nominal or default value for an offset is 0.

[0044] In step 504, the lightest monochromatic patch is examined and its green value is noted by the calibration mechanism. In step 505, the darkest monochromatic patch is examined and its green value is noted. In step 506, the difference in the green values between the lightest monochromatic patch and the darkest monochromatic patch is obtained resulting in a dynamic range of the color green.

[0045] In step 507, the red (R) and blue (B) values of the lightest and darkest monochromatic patches are measured as was done for the green (G) value and their differences are computed to produce a dynamic range of red and blue values. In step 508, the red gain is then set such that the resulting dynamic range of the red value from the lightest to the darkest monochromatic patches matches the dynamic range of the green value. In this way, the red, green and blue values are balanced such that white appears white and colors appear in their correct image. In step 509, similarly, the blue gain is set such that its resulting dynamic range from the lightest to the darkest monochromatic patches matches the dynamic range of the green value.

[0046] As was described earlier, a gain is a multiplicative scale factor ranging, for example, from 0.5 to 1.5 used in adjusting raw uncorrected YUV values. Once the dynamic ranges of red and blue values are determined, the gain may be determined by generating a multiplicative scale factor which when multiplied with the dynamic ranges of the red and blue values results in the dynamic range of the green value. For example, the following depicts an exemplary generation of the gain for the red value of a given patch:

[0047] gain for R=dynamic range of G/dynamic range of R.

[0048] In step 510, if the red and blue gains do not have enough range to match the green value in dynamic range, for example, due to saturation from Y, U or V gain or offset errors, the second darkest and the second lightest monochromatic patches are used as bounds for measuring a dynamic range for the red and blue values. For example, the second darkest patch in the optical target illustrated in FIG. 2 is in the second row, fifth column, and the second lightest patch is in the second row, first column. The process of determining the gain red and blue values are repeated for successively lighter and darker patches until the dynamic range of the red and blue values match those of the green value. Once the white balance has been calibrated in this way, the execution of the color balancer of the calibration mechanism performs the color balance beginning with determining the Y′ gain.

[0049] In step 511, the lightest and the darkest monochromatic patches are examined with the benefit of the correct white balance. In step 512 the observed dynamic range of Y′ from the lightest to the darkest monochromatic patches is compared with the pre-determined dynamic range of the corresponding patches on the reference image. In step 513, the Y′ gain is adjusted such that the dynamic ranges between the observed optical target and reference image match. In step 514, the Y′ offset is adjusted such that the absolute values of Y′ match the reference values shown in the chart for both light and dark patches.

[0050] Next, the U′ gain is calibrated. In step 515, the patches having the highest and lowest levels of U′ are examined with the benefit of the correct white balance. In step 516, the observed dynamic range of U′ from the highest and lowest levels of U′ patches is compared with the pre-determined dynamic range of the corresponding patches on the reference image. In step 517, the U′ gain is adjusted such that the dynamic ranges between the observed target and the reference image match. In step 518, the U′ offset is adjusted such that U′ is set to 128 (neutral=no color) when all of the top row of monochromatic patches is examined.

[0051] Next, the V′ gain is calibrated. In step 519, the patches having the highest and lowest levels of V′ are examined with the benefit of the correct white balance. In step 520, the observed dynamic range of V′ from the highest and lowest levels of V′ patches is compared with the pre-determined dynamic range of the corresponding patches on the reference image. In step 521, the V′ gain is adjusted by increasing or decreasing the gain such that the dynamic ranges between the observed target and the reference optical target match. In step 522, the V′ offset is adjusted such that V′ is set to 128 (neutral=no color) when all of the top row of monochromatic patches is examined. In this way, both the white balance and the color balance are calibrated.

[0052] In the illustrated embodiment, to ensure color calibration accuracy, a “best fit” curve fitting approach may be used to minimize linearity errors at the cost of more computer host-based computing effort.

[0053] FIG. 6 illustrates exemplary calibration adjustment and color space conversion matrices. The video camera 102 captures an image of the optical target 100 through an optical sensor 600 and forwards the information to the computer 103 running the present invention's calibration mechanism. The information is converted from RGB to YUV values by conversion block 602 and is converted from analog to digital through for example an analog-to-digital converter (ADC) (not shown). The Y′U′V′ values are converted to R′G′B′ by a conversion block 604 in the computer 103. Once the gain and offset values for the YUV and the corresponding RGB values are automatically calculated by the computer through the calibration mechanism 114, the gain and offset values are forwarded to the video camera 102 after they are converted from digital to analog form through for example a digital-to-analog converter (DAC) (not shown). The video camera 102 then utilizes the gain and offset values to calibrate its white and color perception under the given lighting. The method by which a video camera calibrates white and color perception once gain and offset values are provided is well known in the art and needs no further explanation.

[0054] What has been described is a method and apparatus for automatically calibrating a video capture device given an optical target with a plurality of white and color patches. Once the video capture device perceives an image of the optical target, an image recognition mechanism identifies the perceived image. The identified image is then compared to a “perfect” copy of the optical target (also referred herein as a reference image). The calibration mechanism then determines gain and offset values for each patch. The gain and offset values generated will calibrate the video capture device to have a perception of the optical target which matches the reference image. Once generated, the gain and offset values are forwarded to the video capture device to complete the calibration process.

[0055] While certain exemplary embodiments have been described in detail and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention is not to be limited to the specific arrangements and constructions shown and described, since various other modifications may occur to those with ordinary skill in the art.