Mathematical model for gray scale and contrast enhancement of a digital image

[0002] 1. Field of the Invention

[0003] The invention relates generally to image processing. More specifically, the invention relates to enhancement of image characteristics such as contrast, gray level, etc.

[0004] 2. Description of the Related Art

[0005] A digital image such as that captured by a digital camera or scanner can be viewed as consisting of an array of pixel locations each having a particular intensity value. In a Bayer pattern image sensor, each pixel consists of an intensity value associated with one of three color planes, Red, Green or Blue. The image when captured in this form is essentially a gray scale image. Through a process known as color interpolation, the missing two color plane component values for each pixel may be estimated. This process yields a full color image, but may distort or degrade certain image features.

[0006] Often in such captured images, it is necessary to highlight or suppress certain image pixels since they were obtained due to poor (low light) or abnormally bright lighting conditions. Such pixels may be “adjusted” for their gray scale intensity values. However, such gray scale adjustment cannot be adequately or efficiently performed after color interpolation. In some cases, even after an image is so adjusted and color interpolated, the contrast of the image may be pleasing to human vision. Enhancing the contrast of an image, while perhaps possible after color interpolation, may not properly work after gray scale adjustment. Additionally, if contrast enhancement is first performed before gray scale adjustment, the adjustment may yield poor results. Color shift and loss of color information may result if such techniques are not properly employed.

[0007] Thus, there is a need for an integrated gray scale adjustment and contrast enhancement technique that can be applied directly to a captured image.

[0008] What is disclosed is a method that specifies identifying a category to which a pixel belongs and performing integrated contrast enhancement and gray scale adjustment on the pixel in accordance with the identified category of the pixel.

[0009] The objects, features and advantages of the method and apparatus for the present invention will be apparent from the following description in which:

[0010] FIG. 1 is a flow diagram illustrating an embodiment of the invention.

[0011] FIG. 2 illustrates the application of one or more embodiments of the invention.

[0012] FIG. 3 illustrates a flow diagram for obtaining enhanced/adjusted pixel values according to at least one embodiment of the invention.

[0013] FIG. 4 illustrates a flow diagram for converting captured image pixel values to enhanced/adjusted pixel values according to at least one embodiment of the invention.

[0014] FIG. 5 is a block diagram of an image processing apparatus according to an embodiment of the invention.

[0015] FIG. 6 is a system diagram of one embodiment of the invention.

[0016] Referring to the figures, exemplary embodiments of the invention will now be described. The exemplary embodiments are provided to illustrate aspects of the invention and should not be construed as limiting the scope of the invention. The exemplary embodiments are primarily described with reference to block diagrams or flowcharts. As to the flowcharts, each block within the flowcharts represents both a method step and an apparatus element for performing the method step. Depending upon the implementation, the corresponding apparatus element may be configured in hardware, software, firmware or combinations thereof.

[0017] FIG. 1 is a flow diagram illustrating an embodiment of the invention.

[0018] In a digital image, pixels can be categorized according to their intensity (relative response of a sensor to an illuminant) values. For instance, according to step 110, pixels may be classified into categories using the range of the pixel's intensity response. Due to different lighting and exposure conditions during image capture, it may be desirable to enhance the overall contrast in an image. Thus, a first set of parameters, referred to as “contrast enhancement” parameters are initialized into the system (step 120). Contrast enhancement parameters may vary from captured image to captured image or depending on a mode or desired application may be fixed for all images, or fixed for all images regardless of the intended use or device mode. According to one embodiment of the invention, contrast enhancement is achieved while also being able to highlight/suppress certain image features making them more pleasing to the human visual system. This highlighting/suppression can take the form of gray scale intensity adjustments. Thus, according to step 130, a second set of parameters known as gray scale adjustment parameters are also initialized. Again, these parameters may be fixed or image specific depending upon the image content.

[0019] Once these parameters are identified, the integrated process of gray scale adjustment and contrast enhancement can commence. First, each pixel is fetched and its category association is identified, according to its intensity level (step 140). Depending on desired system response or use, the process of enhancing and adjusting a pixel may be performed after all pixels have been captured, or while pixels are being captured. Using the identity of the category of the pixel, the enhancement technique is applied using parameters that are appropriate (step 150). An exemplary operation of step 150 is shown in FIG. 3 and described below. As each pixel is fetched and enhanced it is stored separate from the original image data or may replace (overwrite) it (step 160). The process described in steps 140 through 160 repeat until there are no more pixels to enhance (step 165). This may involve only enhancing pixels in a particular geographical area or all pixels in the image.

[0020] FIG. 2 illustrates the application of one or more embodiments of the invention.

[0021] FIG. 2 shows an unaltered image 210 and an enhanced and adjusted image 220. Image 210 is shown having pixels designated generally by CXY, where C refers to the color plane association (Red-R, Green-G or Blue-B) of the pixel, while X and Y refer to the pixel's location, specifically the pixel's row and column index, respectively. Image 210 exemplifies a Bayer pattern captured image which has G associated pixels on every row and alternates B and R pixels on every other row. While image 210 is shown as a Bayer pattern image, any type of image, color or monochrome, in any pattern, may utilize the techniques described in the various embodiments of the invention. The unaltered image 210 may be composed of many more than the seven columns and six rows of pixels shown in FIG. 2, that configuration being merely exemplary.

[0022] Once the integrated image enhancement and gray scale adjustment procedure is complete, the enhanced and adjusted image 220 will result. Image 220 is shown to also consist of the same number of pixel locations as the image 210, but with each pixel given a new enhanced/adjusted value E(P) where P is the original pixel in image 210. Each E(P) pixel is both contrast enhanced and gray scale adjusted and may or may not retain the same intensity value as its corresponding counterpart in unaltered image 210. Also, the color plane association of the pixels is maintained so that the pixel E(G11) is also Green associated as was G11 in the unaltered image. In an alternate embodiment, the pixels E(P) may instead represented a color interpolated pixel set such that each pixel location has estimated values for all three color planes, R, G and B, rather than only one value. Each E(P) may have the same precision/resolution as the unaltered counterpart or may differ depending on the desired system design.

[0023] FIG. 3 illustrates a flow diagram for obtaining enhanced/adjusted pixel values according to at least one embodiment of the invention.

[0024] In one embodiment of the invention three categories of the pixels are classified These three categories may be determined with reference to two cutoff values C1 an C2. If P is the intensity value of a pixel, then it belongs to one of these three categories:

[0025] I: P<C1

[0026] II: C1 P C2

[0027] III: P>C2

[0028] Pixels whose intensity values are associated with category I are of low intensity due to insufficient lighting condition while category III pixels are of relatively high intensity. Both the pixels in categories I and III have non-uniform behavior. The pixels in category II are uniformly generated due to uniform lighting condition.

[0029] The embodiment of FIG. 3 illustrates a real-time enhancement and adjustment procedure. Accordingly, each pixel's intensity value P is fetched in turn (step 320). If P<C1 i.e., if it belongs to category I (checked at step 330), then a first formula is applied (step 335) to bring up the contrast and gray scale values to a higher level. If not, then P may belong to category II or III. If P lies between C1 (inclusive) and C2 (checked at step 330), then a second formula is applied (step 345) which is just result of scaling the intensity by a constant factor. If not, then P must by default belong to category III (having a value greater than C2) and thus a third formula is applied (step 350). Each formula models the desired response in the particular category (i.e., intensity classification) and performs both contrast enhancement and gray scale adjustment in a single computation. Given below are three such formulas.

[0030] E(P)=k₁*C₁₋₍k₁*C₁−P)^α1  Category I

[0031] E(P)=k₃*P  Category II

[0032] E(P)=k₂*C₂+|k₂*C₂−P|^α2  Category III

[0033] where k₁, k₂and k₃are contrast enhancement parameters and α1, α2 are gray scale adjustment parameters. The gray scale adjustment parameters α1 and α2 are not utilized in the Category II formula and thus, no gray scale adjustment is needed for pixels in this category but the contrast is enhanced by multiplying with the scale factor k₃. However, category I and category II pixels are gray scale adjusted in accordance with α1 and α2 , respectively. The scale factors k₁and k₂are used to enhance the contrast values up to a fixed level depending on C₁and C₂whereas the exponents α1 and α2 produces a form of gamma correction of the pixels to correct the non-uniformity in order to make it pleasing to human vision. The logic illustrated in FIG. 3 represents a real time approach to the integrated contrast enhancement and gray scale adjustment. Since the intensity value range is dependent upon a fixed resolution per pixel in most image sensors, it may be possible to pre-compile a table that maps each possible intensity value to an enhanced/adjusted values.

[0034] FIG. 4 illustrates a flow diagram for converting captured image pixel values to enhanced/adjusted pixel values according to at least one embodiment of the invention.

[0035] If each pixel location has a known, fixed and uniform resolution, which is often the case in image sensing devices, then a look-up table (LUT) may be constructed which maps each possible intensity value to its corresponding enhanced/adjusted value. Such a LUT may be constructed as shown in FIG. 4. First, the pixel categories should be classified (step 410). Next, parameters such as contrast enhancement and gray scale adjustment parameters should be set or initialized (step 420). If each pixel in the image sensing array has a N-bit depth or resolution, then any one of a total of 2^Nintensity values are possible for a given pixel's intensity reading. Using a counting value I, the enhanced/adjusted equivalent for each possible intensity value may be computed. If the intensity readout ranges from 0 to 2^N−1in a sensor, the count I could first be initialized to 0 (step 430). Next, the technique (formula(s) described above) would be applied based upon the category that I is identified with (step 440). The value of I and the computed value of E(I) is then stored in the table both on the same row such that a pixel value may be used as an address (or as an offset to an address such that only the computed E(I) values need to be stored) (step 450). I is then incremented such that the next enhanced/adjusted E(I) may be computed. If I is less than 2^Nthen steps 440 through 460 are repeated using the new, incremented I (checked at step 470). If I exceeds 2^N−1then the LUT is considered completed and may be utilized to look up values for converting a captured image into an enhanced image. By utilizing a LUT, repetitive calculations are avoided and thus allows image processing to occur more efficiently. Each intensity value in the pixel can be directly replaced by a mere look up in the LUT, which could be reconstructed if changing parameters, pixel categories or formulations of the enhancement/adjustment are desired.

[0036] FIG. 5 is a block diagram of an image processing apparatus according to an embodiment of the invention.

[0037] FIG. 5 is a block diagram of internal image processing components of an imaging device incorporating at least one embodiment of the invention including an adaptive encoder. In the exemplary circuit of FIG. 5, a sensor 500 generates pixel components which are color/intensity values from some scene/environment. The M-bit pixel values generated by sensor 500 are sent to a capture interface 510. Sensor 500 in the context relating to the invention will typically sense one of either R, G, or B components from one “sense” of an area or location. Thus, the intensity value of each pixel is associated with only one of three color planes and may form together a Bayer pattern raw image. Capture interface 610 resolves the image generated by the sensor and assigns intensity values to the individual pixels. The set of all such pixels for the entire image is in a Bayer pattern in accordance with typical industry implementation of digital camera sensors.

[0038] It is typical in any sensor device that some of the pixel cells in the sensor plane may not respond to the lighting condition in the scene/environment properly. As a result, the pixel values generated from these cell may be defective. These pixels are called “defective pixels.” The “pixel substitution” unit 515 replaces each dead pixel by the immediate previously valid pixel in the row. A RAM (Random Access Memory) 516 consists of the row and column indices of the dead pixels, which are supplied by the sensor. This RAM 516 helps to identify the location of dead pixels in relation to the captured image.

[0039] This sensor data may then be passed to a Integrated Gray Scale Adjustment and Contrast Enhancement module 528 which executes gray scale adjustment and contrast enhancement upon the captured image. Module 528 may either act in real-time by implementing directly the computation for the adjustment/enhancement process illustrated in FIG. 3 or utilize a look-up table constructed in accordance with FIG. 4 that is loaded via bus 560 from an external device. Such a look-up table which already maps all possible intensity values to corresponding enhanced/adjusted values may be stored in a RAM 531 that can be accessed with reference to the data values used as addresses or offsets to addresses for RAM 531. By applying gray scale adjustment and contrast enhancement directly to the captured image and prior to any color interpolation color shift is avoided.

[0040] A compressor/encoder 530 receives sensor image data and performs image compression and binary encoding. A RAM 531 can be used to store coefficients and/or quantization thresholds as desired in executing these compression techniques. Compressor/encoder 530 reduces the total size of the image data by exploiting factors such as data correlation. Compressed/encoded data can then be stored in storage arrays 540.

[0041] Each of the RAM tables 516, 526, 531 and 529 can directly communicate with a bus 560 so that their data can be loaded and then later, if desired, modified. Further, those RAM tables and other RAM tables may be used to store intermediate result data as needed and may be part of the same physical memory space even though depicted in FIG. 5 as being distinct.

[0042] FIG. 6 is a system diagram of one embodiment of the invention.

[0043] Illustrated is a computer system 610, which may be any general or special purpose computing or data processing machine such as a PC (personal computer), coupled to a camera 630. Camera 630 may be a digital camera, digital video camera, or any image capture device or imaging system, or combination thereof and is utilized to capture an image of a scene 640. Essentially, captured images are processed by an image processing circuit 632 so that they can be efficiently stored in an image memory unit 634, which may be a ROM, RAM or other storage device such as a fixed disk. The image contained within image memory unit 634 that is destined for computer system 610 can be according to one embodiment of the invention, stored directly as gray scale adjusted and contrast enhanced data. In most digital cameras that can perform still imaging, images are stored first and downloaded later. This allows the camera 630 to capture the next object/scene quickly without additional delay. The invention in its various embodiments, particularly in providing a compressed image that is directly enhanced/adjusted from the captured raw image, enhanced the quality of images generated by camera 630.

[0044] The image processing circuit 632 may carry out the integrated contrast enhancement and gray scale adjustment prior to the images being downloaded from camera 630. This prevents any loss from the compression process from being introduced before enhancement/adjustment which may adversely impact image quality. When an enhanced/adjusted image that has been compressed is downloaded to computer system 610, it may be decoded and then rendered to some output device such as a printer (not shown) or to a monitor device 620. Image decoding and rendering may be achieved using a processor 612 such as the Pentium™ (a product of Intel Corporation) Processor and a memory 611, such as RAM, which is used to store/load instruction addresses and result data.

[0045] Where image processing circuit 632 utilizes a look-up table to obtain enhanced/adjusted values, the table construction techniques described above may be achieved in a software application running on computer system 610 rather than directly in camera 630. The constructed table may then be loaded into camera 630. In such an embodiment, the image processing circuit may advantageously store only the mapping table and thus be relieved of the computational duty involved in implementing enhancement/adjustment formulas. The application(s) used to perform the table construction and/or decoding after download from camera 630 may be from an executable compiled from source code written in a language such as C++. The instructions of such executable/source files may be stored to a disk 618 or memory 611. Further, such application software may be distributed on a network or a computer-readable medium for use with other systems. In yet another embodiment, the entire enhancement/adjustment technique may be performed on the computer system 610, with a look-up table stored in disk 618, memory 611 or by implementing the process of FIG. 3, which is a real-time method on computer system 610 by use of application software executed thereon.

[0046] When an image, such as an image of a scene 640, is captured by camera 630, it is sent to the image processing circuit 632. Image processing circuit 632 consists of ICs and other components which execute, among other functions, the integrated enhancement/adjustment o a captured image. The image memory unit 634 will store the compressed and encoded data. Once all pixels are processed and stored or transferred to the computer system 610 for rendering the camera 630 is free to capture the next image. When the user or application desires/requests a download of images, the encoded image data in the image memory unit, are transferred from image memory unit 634 to the I/O port 616. I/O port 617 uses the bus-bridge hierarchy shown (I/O bus 615 to bridge 614 to system bus 613) to temporarily store the data into memory 611 or, optionally, disk 618. Computer system 610 has a system bus 613 which facilitates information transfer to/from the processor 612 and memory 611 and a bridge 614 which couples to an I/O bus 615. I/O bus 615 connects various I/O devices such as a display adapter 616, disk 618 and an I/O port 617, such as a serial port. Many such combinations of I/O devices, buses and bridges can be utilized with the invention and the combination shown is merely illustrative of one such possible combination.

[0047] The exemplary embodiments described herein are provided merely to illustrate the principles of the invention and should not be construed as limiting the scope of the invention. Rather, the principles of the invention may be applied to a wide range of systems to achieve the advantages described herein and to achieve other advantages or to satisfy other objectives as well.