Method for improved contrast mapping of digital images

[0001] The present invention relates generally to digital image enhancement and more specifically to a method of local contrast mapping that changes the dynamic range of an original image to more closely match the dynamic range of the medium used for the reproduction.

[0002] When taking photographs of an outdoor scene, the brightness ratio, or dynamic range, of an area in full sunlight to another area in deep shade can exceed 1000:1 (see FIG. 1). When a reflective print is created of this scene the print is typically limited to a dynamic range of about 100:1. Because of the large mismatch between the dynamic range of the reflective print and the original scene, there needs to be some mapping or transformation of the dynamic range of the scene onto the available dynamic range of the reflective print to create a pleasing reproduction.

[0003] A common technique used to map one dynamic range onto another dynamic range is the use of a nonlinear curve, often implemented digitally with a tone map. Tone maps are lookup tables that transform numbers from an input range (1000:1 in an outdoor scene) to a new range (100:1 for a reflective print). Tone maps are typically applied globally; all points in an image that have a certain numerical value are mapped to precisely the same output value. Unfortunately when using a tone map to map a large dynamic range onto a much smaller dynamic range the resulting image often produces a sensation of reduced contrast. This reduction in contrast makes the image appear flat and unappealing (see FIG. 2).

[0004] Photographers have addressed this problem by darkroom manipulations called burning and dodging. Dodging is a technique, performed while printing a negative onto photographic paper, in which dark areas of the image are lightened by casting a shadow over the area for some portion of the total exposure time. The shadow reduces the amount of light exposure to the print thereby lightening that part of the image. The shadow is commonly produced by blocking light projected by the enlarger lens with a piece of cardboard on a piece of wire, or even with the photographer's hand. Burning is the opposite process in which additional exposure time is given to the photographic paper by increasing the amount of light exposure to some area of the print. Typically a card with a small hole is held over the print with the light falling through the hole exposing only the area of the image to be darkened.

[0005] Dodging and burning work well for large areas of slowly changing brightness, but produce halos in areas that have complex shapes. For example the tree branches in the lower left corner of FIG. 1 would be very difficult to lighten, or dodge, without leaving a halo or bright area around and between the tree branches.

[0006] Another technique used to reduce the dynamic range in a scene is the Retinex algorithm developed by John McCann of Polaroid. Retinex is based on the Retina and Cortex theory of human vision developed by Edward Land. The basic Retinex algorithm is described in U.S. Pat. No. 4,384,336 Method and Apparatus for lightness imaging which is hereby incorporated by reference. The Retinex algorithm reduces the dynamic range of an image by removing much of the slowly varying changes in illumination in the scene. For images that have slowly changing illumination the original Retinex algorithm can make significant improvements in the appearance of the mapped reflective print. For images that have areas with rapidly changing levels of illumination, or even abrupt boundaries between different sources of Illumination, the original Retinex algorithm may only produce subtle improvements. For example, FIG. 3 is a reproduction of the same image as FIG. 1 except that the image in FIG. 3 received processing by the original Retinex algorithm. The bushes in the foreground and the tree in the lower left of the image have been lightened somewhat, but the effect is subtle. Overall the reproduction in FIG. 3 fails to convey the visual impact that was present in the original scene.

[0007] What is needed is a method that dramatically lightens the dark foreground in FIG. 1 without significantly changing the brightness of the cloud reflections in the water. This would imply that the large brightness ratio, or large contrast, between the reflected clouds and the tree branches in the lower left of FIG. 1 must be greatly compressed. At the same time, the small, local brightness ratios in the reflected clouds and in the foreground areas must be preserved, without compression. This is important to preserve the sensation of contrast when viewing the image. Therefore there is a need for a system that can compresses large contrast differences between different areas of an image while preserving small contrast differences between different areas of an image.

[0008] A method of local contrast mapping that changes the dynamic range of an original image to more closely match the dynamic range of the medium used for the reproduction. The method compresses large contrast differences between different areas of an image while preserving small contrast differences between different areas of an image.

[0009] Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.

[0010] The file of this patent contains at least one drawing executed in color. Copies of this patent with color drawings will be provided by the Patent and Trademark Office upon request and payment of the necessary fee.

[0011] FIG. 1 is a print of an image where the image has a large dynamic range of illumination.

[0012] FIG. 2 is a print of the image of FIG. 1 after modification with a global tone map.

[0013] FIG. 3 is a print of the image of FIG. 1 after modification by the original Retinex algorithm.

[0014] FIG. 4 is a print of the image of FIG. 1 after modification with a 1-to-1 slope using the method according to the current invention.

[0015] FIG. 5 is a print of the image of FIG. 1 after modification with a 1.8 slope using the method according to the current invention.

[0016] FIG. 6 is a tone map with a one-to-one slope in accordance with the present invention.

[0017] FIG. 7 is a tone map with an increased slope in accordance with the present invention.

[0018] FIG. 8 is a tone map with a smooth shape in accordance with the present invention.

[0019] A method of compressing large contrast differences between different areas of an image while preserving small contrast differences between different areas of an image can vastly improve the appearance of the reproduction of an image that has a large range of illumination intensity. Algorithms that locally darken or lighten an image by computing contrast differences between various portions of a scene can be significantly improved by maintaining the small contrast differences and compressing the large contrast differences. One such algorithm is the original Retinex algorithm.

[0020] In the original Retinex algorithm an output image O is iteratively formed from an input image according to the following relationship: $\begin{array}{cc}{O}_{x,y}=\frac{\mathrm{ClipToWhite}\left(I_{x,y}-I_{\mathrm{xs},\mathrm{ys}}+O_{\mathrm{xx},\mathrm{ys}}\right)+O_{x,y}}{2}.& \mathrm{Equation}1\end{array}$

[0021] Where I_x,y refers to a segmental area of the input image at location x, y. I_xs,ys refers to a segmental area in a shifted version of the input image where the image has been displaced by some distance dx and dy relative to the original image. Both images I and O are logarithmically encoded according to a measure of brightness (or are in log space) so taking a difference is the same as computing a ratio. For example the term I_x,yI_xs,ys in equation 1 is the ratio of the intensity (or radiance) at one location in the input image with respect to the intensity (or radiance) of the image at a different location in the input image. In log space adding a term is the same as multiplying. For example adding the term O_xs,ys in equation 1 is equivalent to taking the product of the ratio of the input intensities (I_x,yI_xs,ys) with the intensity of the displaced output image (O_xs,ys). The ClipToWhite function constrains the result of the product of the ratio to remain within the maximum or minimum intensity values. For example when the range of input intensity (or radiance) values are between 0 and 255, the ClipToWhite function would reduce any value over 255 to 255 and increase any value less than zero to zero. When the result of the product of the ratio (I_x,yI_xs,ysO_xs,ys) is 270 the ClipToWhite function would reset the value to 255. In the original Retinex algorithm, the input images and output image are shifted relative to each other starting with large displacements. A typical initial shift would be a power of two that is about one half the width or height of the input image. The image displacements proceed in a circular pattern for several revolutions. Then, the displacements are halved and the processes is repeated, until the displacement reaches zero (See U.S. Pat. No. 4,384,336).

[0022] The ratio of the intensity (or radiance) of the image at two different locations (I_x,yI_xs,ys) is a measure of the relative contrast difference (or contrast ratio) of one part of the image compared to another part of an image. For example, the illumination intensity of a pixel in of one of the branches in the tree in the lower left-hand corner of image 1 is 8. The illumination intensity of a pixel in the water behind the branch is 220. Therefore the ratio of the intensities between the branch and the water is 220/827.5 (or 1.44 in log space). The illumination intensity of one part of a cloud in the sky is 247. The illumination intensity of a different part of the sky is 215. The ratio of the intensities between these two different parts of the sky is 247/2151.15 (or 0.06 in log space).

[0023] The current invention compresses the large intensity ratios and preserves the small intensity ratios. To compress the large ratios and preserve the small ratios a tone map or look up table (LUT) can be used. The LUT is used to modify the ratio of intensity differences between different areas of the input image (see equation 2). $\begin{array}{cc}{O}_{x,y}=\frac{\mathrm{ClipToWhite}\left(\mathrm{LUT}\left(I_{x,y}-I_{\mathrm{xs},\mathrm{ys}}\right)+O_{\mathrm{xx},\mathrm{ys}}\right)+O_{x,y}}{2}.& \mathrm{Equation}2\end{array}$

[0024] The LUT or tone map consists of two parts. A line slope of 1:1 passing through the origin (602) is used to preserve the small differences (see FIG. 6). A hard limit (604) is used to compress the large contrast differences. The hard limit is set to limit the maximum ratio of illumination, for example a hard limit of 4-to-1 (or 0.6:1 in log space) may be used. Using Equation 2 with a line slope of 1-to-1 and a hard limit of 4-to-1 (0.6 in log space), image 1 was modified to create FIG. 4. By compressing the large contrast differences while preserving the small contrast differences the overall sensation of contrast has been maintained while the overall dynamic range of the scene has been compressed. These results are impossible to recreate in a darkroom without leaving some halo around the branches in the tree in the lower left corner of the image.

[0025] The slope of the line passing through the origin in the tone map can be increased (see FIG. 7) or decreased (not shown) to adjust the sensation of local contrast without strongly affecting the compression of the large contrast differences of the image. For example, a slope through the origin of about 0.85 can improve harsh lighting from an electronic flash in an image of a person. FIG. 5 was created, using equation 2, with a slope of 1.8-to-1 and a hard limit of 4-to-1. FIG. 5 has a higher level of local contrast and has maintained the compression of the large contrast differences.

[0026] Adjusting the hard limit of the ratio of intensities primarily controls the amount by which the darkest areas in an image are brightened. A higher limit (such as a density ratio of 1.5) reduces the amount of brightening and compression. A lower limit (such as a density ratio of 0.4) increases the amount of brightening and compression. The shape of the curve in the tone map can also be changed. For example the sharp change from the line passing through the origin to the hard clip limit can be modified to create a smoothly changing function (see FIG. 8).

[0027] Forming the output image 0 iteratively from the shifted input image at the full resolution of the input and output images, is a computationally intensive process. U.S. Pat. No. 4,750,211 Method and Apparatus for Image Processing With Field Portions, hereby included by reference, discloses a method of processing images at reduced resolution. Combining the method disclosed in U.S. Pat. No. 4,750,211 with the current invention can reduce the number of calculations required to produce an improved output image.

[0028] The foregoing description of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and other modifications and variations may be possible in light of the above teachings. The embodiment was chosen and described in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and various modifications as are suited to the particular use contemplated. It is intended that the appended claims be construed to include other alternative embodiments of the invention except insofar as limited by the prior art.