RGB Image Correction Using Compressed Flat Illuminated Files and a Simple One or Two Point Correction Algorithm

Field of the Invention

The present invention relates to electronic color imaging and more particularly to a memory efficient method and means for storing color or infrared correction information for liquid crystal filters that are used with electronic color imaging CCD or CMOS cameras.

Prior Art

Typically in the electronic imaging art a 4K.times.4K CCD image sensor may be operated behind a liquid crystal tuned (controlled) filter (LCTF), to sense an image in red, green, and blue colors and thus produce color images. In such instances, it is not uncommon for the liquid crystal tuned filter to cause some color non-uniformities across the resulting image field of view. For example, when the CCD imaging array is subjected to flat field white illumination, the resulting image usually contains an imbalance in the red, green, and blue colors across its field of view. This liquid crystal filter color distortion appears superimposed on other offset and multiplicative noise from the CCD, optics, and camera with either a coded or uncoded image sensor.

Problem to be Solved:

It is accordingly a problem to eliminate the color non-uniformities that occur across the image field of view that are produced when using an electronic color image sensor, e.g., a CCD sensor, in combination with a liquid crystal tuned filter for snap-shot, slow framing, or video rate applications.

Objects:

It is therefore an object of the present invention to provide a method and means for controlling color non-uniformities in color images produced by CCD image sensors, particularly uncoded as well as coded sensors, due to use with liquid crystal tuned filters.

It is also an object of the invention to provide a memory efficient method and means for storing color correction information for liquid crystal tuned filters that are used with electronic color imaging cameras.

It is a further object of the present invention to make it possible for a camera operator to invoke the stored color correction information to automatically take control of image enhancement when using a CCD image sensor in combination with a liquid crystal tuned filter.

The present invention involves a system including a memory efficient method and means of storing color correction information for liquid crystal tuning filters when used with color electronic imaging cameras. The stored correction information may be used in minimizing color imbalance or distortion in an image produced by a camera, which distortion is due to color non-uniformities, caused by the filters, across the image field of view. The actual correction may be done by software or high speed digital signal processing in the case of video or framing cameras. For example, when utilizing a 4K.times.4K CCD image sensor, the sensor used to capture the RGB non-uniformities is binned internally and corrected for its own non-uniformities and blemishes to decouple the CCD non-uniformities from filter color non-uniformities. The color correction image from the 4K.times.4K array is also binned externally to a 500.times.500 or 1K.times.1K file, and may be compressed by applying the JPEG and/or wavelet compression algorithm with either a default configuration or a custom quantization table that emphasizes low frequency changes with more bits than are used for high frequency changes. At the completion of compression, the compressed R, G, B files and a one or two or more point correction executable algorithm are stored, e.g., on a floppy disk or CD ROM, so that the stored files can be automatically decompressed and executed to take control of image enhancement when invoked by the camera operator or photographer during subsequent use.

FIG. 1 is a system for illustrating the manner in which a color image is sensed in accordance with the present invention.

FIG. 2 illustrates one form of CCD sensor that may be used in the system of FIG. 1, involving a three-shot CCD color camera with a regular uncoded CCD sensor for three-shot operation.

FIG. 3 illustrates another form of CCD sensor that may be used in the system of FIG. 1, involving a single-shot coded image sensor that stores R, G, and B samples on the same sensor surface under opaque memory areas.

FIG. 4 is a diagram illustrating the clock signals and timing for storing calibration files with the coded sensor shown in FIG. 3.

FIG. 5 is an exemplary illustration of color distortion (flat files) as seen on a display.

FIG. 6 shows a color correction flat (white) file composed of RGB flat files including histograms for flat field RGB files used in correction, before and after processing.

FIG. 7 is a flow chart illustrating a representative set, of steps involved in carrying out the present invention.

FIG. 8 is an illustration of the selection of an interpolated F/# value in response to color correction information stored in accordance with the invention.

FIG. 9 is a schematic diagram illustrating a hardware implementation of a snap-shot camera or video camera containing the LCTF filter correction feature.

FIGS. 10 (a) to 10 (d) illustrate the operation of a 2-point correction algorithm as applied to the red (R) image portion of a coded sensor image produced through an LCTF filter, using 2.times.2 resolution.

FIG. 11 illustrates a system for applying a strong gamma (non-linear) transform or projection to the image data for an image containing color shading before or after color correction files are used to correct the image.

The present invention is embodied in a system including an improved method and means for enhancing electronic color images that are produced by electronic image sensors, such as CCD sensors, in combination with liquid crystal tuned filters (LCTF). The system efficiently stores color correction information for use in minimizing color imbalance or distortion in the subsequently produced electronic images that is due to filter-caused color non-uniformities across the image field of view. Such color imbalance may be corrected for both sequential three-shot color cameras and snap-shot color cameras using software and hardware in accordance with the invention.

More specifically, a basic set up or system for illustrating the manner in which a color image is sensed is shown in FIG. 1 wherein light from a light source 10 passes through a flat field diffuser 11 and is directed by suitable optics 12, e.g., a spherical illuminator, to uniformly illuminate an LCTF filter 13, controlled for R, G, and B image passage. The colored light fields or frames from the filter 13 are directed by further suitable optics 14 to be sensed by an electronic image sensor 15. The selection and positioning of the optics may be varied to suit a particular application, and the sensor 15 may be embodied, for example, in a three-shot CCD color camera as a regular uncoded CCD sensor 15' for three-shot operation as shown in FIG. 2, or a single-shot coded image sensor 15" that stores R, G, and B samples on the same sensor surface 16 under opaque memory areas 16' as shown in FIG. 3. Accordingly, the flat field RGB files are collected with the filter 13 in the R, G, and B states in either separate three-shot image files in a three-shot CCD imager, from which separate R, G, and B frames are readout, or on the same surface in a single-shot coded image sensor. In the latter instance, two out of each triad of pixels is covered to store R, G, and B subframes during flat field calibration and the subframes are then interpolated to the complete R, G, and B frames for correction purposes. Specifically, the image charge for each color is developed through a clear aperture sensor site and the first two charge packets are successively transferred to two opaque memory sites so as to store the R, G, and B information separately. A diagram illustrating the clock signals and timing for storing the calibration files on the coded sensor 15" in this manner is shown in FIG. 4. The calibration fields for color correction are the three separate R, G, and B subfields (frames) developed by the coded sensor 15" when it images a flat field while being clocked using three-phase clocks (.phi..sub.1, .phi..sub.2, .phi..sub.3) that are synchronized with the LCTF color filter switching signals.

Thus, individual dark (shutter closed) and light flat fields are captured through the LCTF filter, and the non-uniformity components are amplified across the intensity scale preparatory to being stored. Preferably, the fields are stored in compressed form.

FIG. 5 is an exemplary illustration of color distortion (flat files) as seen on a display. The offset is subtracted, but recorded, and the RGB fields are balanced before compression for generating the maximum precision correction files. Also, 32-bit to 40-bit (wide word) floating point arithmetic can be used for calculations to avoid residual non-uniformity and enhance correction accuracy. The offset is shown in FIG. 6 which illustrates, on the left, a color correction flat (white) file composed of raw histograms for flat field RGB files used in correction, before processing, and on the right, a processed histogram in the form after processing. Any offset component subtracted from RGB histograms is recorded for later use during image correction of the snap-shot or three-shot color image. The stored offset numbers are used in restoring filter transmission data to optimize color crosstalk correction of the coded image sensor required as a result of light leakage under the light shielded areas of the coded sensor's FPA. The offset may correspond to CCD dark current or filter transmission.

The amplified red (R), green (G), and blue (B) image files are then preferably compressed using the JPEG and/or wavelet compression algorithm. The quantization tables of the JPEG algorithm are edited to emphasize low frequency variation by using more bits than for the high frequency changes. All high frequency terms are sampled with fewer bits. Compression ratios of x30 to x300 are possible on each of the R, G, and B image information files while maintaining the most significant color distortion details needed to perform flat field correction of the image.

The CCD sensor used to capture the RGB non-uniformities is binned and corrected for its own non-uniformities to decouple the CCD non-uniformities from filter color non-uniformities. For instance, as shown in FIG. 7, in the first step of an exemplary procedure, an n.times.n, e.g., 4K.times.4K, photo response non-uniformity (PRNU) image array 70 is produced by the LCTF, and in the second step, the array is binned internally to CCD and externally to m.times.m, where m<n, e.g., m=500.times.500 or 1000.times.1000 elements 71, either on-chip or off-chip. Binning reduces chip non-uniformities and blemishes. The calibration frames can be collected by the same CCD chip or a different chip. In all cases, blemished pixels are corrected or substituted for by neighboring pixels.

In the third step, the external binned image is then compressed, e.g., with the default JPEG compression algorithm 72, using a JPEG luminance quantization table if high quality correction is desired. The resulting compressed file has a size 1/30 to 1/300 times that of the uncompressed RGB (flat field) file. The quantization table suggested for the JPEG algorithm, which is based on psychovisual thresholding and derived empirically using luminance and 2:1 horizontal subsampling, is in the form of TABLE 1 below. The user may edit the quantization table or JPEG compression quality factor to improve correction accuracy and reduce residual color shading or blocking artifacts.

TABLE 1 provides the "high quality" correction compression produced by the JPEG algorithm (72) as shown in the system in FIG. 7. TABLE 2, following TABLE 1 below, additionally or alternatively, provides the low-frequency "low quality" correction compression path (73) of the system using compression with low frequency parameters only.

The resulting compressed correction files (from 72 and/or 73) may be stored, in the next step, e.g, on a floppy disk 74, in inverted or direct mode, along with an executable algorithm to be subsequently used for color correction 75. The floppy disk 74 may contain the flat field RGB correction files obtained using several lenses, e.g., 80 mm, 200 mm, etc., to correct for illumination non-uniformity produced by each lens surface and chromatic aberration produced by different lens systems. Also, several flat fields (of n-points, e.g., 2 or 3 or 4 points) of the LCTF calibration may be stored at different intensities to correct for any non-linearity in the LCTF transmission per color channel.

The compressed R, G, B files are either expanded back to 4K.times.4K resolution or supplied in 500.times.500 resolution with the executable algorithm to repeat some corrections parameters on each four columns and four rows. The use of 500.times.500 binning avoids the need for editing the JPEG source code to the 4K.times.4K level.

The information used in image correction involves:

The 1/R, 1/G, and 1/B represent a look up table (LUT) or inverse (division or multiply by quotient) operation, such as typically done on an IR-imaging FPA for uniformity correction, and is repeated for the 4K.times.4K array in the appropriate steps determined by the binary procedure.

The technique described separates the CCD effects from the LCTF color effects. The same 1/R, 1/B, 1/G tables are stored for 3 or 4 different F/# on the lens, as the floppy disk size may make it hard to store each F/# color correction matrix. These tables have been found to be enough to correct the image under all F/# or exposure conditions. With an 80 mm F1 lens, the three F/# correction files that represent the filter non-uniformities are the RGB JPG files for F 2.8, F 5.6, and F 8 or F 11. The last file is good all the way to the F 22 setting. The camera operator or photographer will be prompted to enter the F/# used in building the image and the executable algorithm will interpolate the pixel correction values 1/R, 1/G, 1/B between the two nearest stored F/# settings, as illustrated in FIG. 8. The executable algorithm may alternatively use the file with the nearest F/# as will be seen from TABLE 3. The R, G, B or 1/R, 1/G, 1/B values are expanded from storage before interpolation using the JPEG and/or wavelet decompression algorithm.

Thus, flat field correction files of sampled F/#s and sampled lenses with different focal lengths may be stored on a floppy diskette for later use in image correction and enhancement when an image, created with these components, is to be displayed. Either compressed composite color files or separate color files may be stored in direct (R,G,B+offset) or inverted (1/R, 1/G, 1/B) form. For the case of slow framing or video cameras used with LCTF filters, this color correction information is kept in solid state memory for rapid real time correction of image data using subtraction in an arithmetic logic unit (ALU) and solid state multiplier. A system for implementing these functions and incorporating suitable hardware for these purposes is shown in FIG. 9 which illustrates a hardware implementation of a snap-shot camera or video camera containing the LCTF filter correction. As seen in FIG. 9, a subject 100 is imaged through suitable optics 101 and an LCTF filter 130 onto the sensor surface 102 of a camera 103 having appropriate drive electronics 104. The analog video output of the camera 103 is fed through an analog-to-digital (A/D) converter 105 to one input of an arithmetic logic unit (ALU) 106. The other input of the ALU 106 receives dark field signals from an average dark field image file memory 107 under the control of a timing generator 108. Timing generator 108 also controls a color correction memory and look-up-table (LUT) component 109 with a correction executable algorithm that outputs stored RGB color correction information signals to a digital multiplier 110. The digital multiplier 110, along with the color correction signals, receives signals from the ALU 106 for the three color images sensed by the sensor 102, which signals have been dark field corrected (.nu..sub.iR -.nu..sub.d, .nu..sub.iG -.nu..sub.d, .nu..sub.iB -.nu..sub.d) but not color corrected. The output of the digital multiplier 110 is in turn a dark field and color corrected digital image of the R, G, and B frames, that is then converted by a digital-to-analog (D/A) converter 111 for presentation on a suitable analog display 112, or a printer or other output device.

An illustration of the operation of a 2-point correction algorithm on a coded sensor image produced through an LCTF filter and using 2.times.2 resolution is shown in FIGS. 10 (a) to 10 (d). To begin with, FIG. 10 (a) shows a portion of an exemplary captured real (RGB) image, with the .nu..sub.i counts in ADU units, i.e., analog-to-digital converter units, and FIG. 10 (b) shows the related dark field image with the .nu..sub.d counts also in ADU units. FIG. 10 (d) shows the dark field corrected image, i.e., .nu..sub.i -.nu..sub.d. FIGS. 10 (c) to 10 (e) illustrate red (R) image correction, which is applied similarly to blue (B) and green (G) images for complete RGB image correction. FIG. 10 (c) shows the red (R.sub.i) correction flat field that is compared with the dark field corrected image of FIG. 10 (d) to produce the corrected red image of FIG. 10 (e) corrected by the red flat field.

Additionally, as shown in FIG. 11, strong gamma (non-linear) transform or projection of image data may be applied to the image produced by the photographer or user before or after the color correction files are used to correct the image. The purpose of the strong gamma (.gamma.) correction is to avoid the need to apply large gains to the image information to see the color details of the image. The high gain may cause residual errors (non-uniformities of the shading) in the color correction scheme to appear on the image. More particularly, it will be seen from FIG. 11 that the sensed image 80 produced by the photographer and containing color shading is electronically input to a gamma control and color processing device 81, the output of which is provided to a component 82 for calculating the LCTF filter correction equations. Component 82 also receives color correction information from the color correction file storage device 83 and applies it before or after the gamma correction information, to produce a corrected image 84 with no shading. If color correction files are applied after gamma correction, then they can be treated by the same or similar .gamma. transform.

It will therefore be seen that a memory efficient system for storing color correction information for liquid crystal tuning filters, when used with color electronic imaging cameras, has been described wherein the stored correction information is available to minimize color imbalance or distortion in an image produced by a camera, which distortion is due to color non-uniformities, caused by the filters, across the image field of view. The actual correction may be done by software or high speed digital signal processing in the case of video or framing cameras. The sensor used to capture the RGB non-uniformities is binned internally and corrected for its own non-uniformities to decouple the CCD non-uniformities from filter color non-uniformities, and the color correction image is binned externally to a smaller file which may then have the JPEG and/or wavelet compression algorithm applied with either a default configuration or a custom quantization table that emphasizes low frequency changes with more bits than are used for high frequency changes. At the completion of compression, the compressed R, G, B files and a one or more point correction executable algorithm are stored, e.g., on a floppy disk or CD ROM, and are automatically decompressed and take control of image enhancement when invoked by the camera operator or photographer during subsequent use.