Gamut Restriction of Color Image

The present invention relates to methods for restricting the colors specified in pixels of an image to fall within the limitations of an image printing or display device.

The human eye can perceive a wider range or gamut of colors than can be presented by devices for generating images such as computer monitors, photographs, and printers. An image represented in digital form which contains colors falling beyond the gamut of the device for presenting the image must be restricted by clipping or by compression or by a combination of the two. For a thorough explanation of the problem, see U.S. Pat. No. 5,416,890 to Beretta, assigned to Xerox Corporation. Also, even if the image is to be presented in a device with as broad a gamut of colors as contained in the original image, such as when the original image is captured on film and is to be presented on film, it is often desirable to adjust the brightness or contrast or saturation or hue. Any of these adjustments can cause some of the colors to be generated to exceed the color gamut of the output device.

For digital storage and manipulation by a computer, images are typically stored in a three color cubic space, such as RGB (red, green, blue) as shown in FIG. 3. A simple compression method for restricting the colors of an image to the gamut of a device is to reduce each of the R, G, and B values by a ratio until all values fall within the gamut of the device. This is unsatisfactory because it does not take advantage of a large portion of the colors available and produces an image of poor color quality. A solution which takes advantage of all colors available is to simply clip any values of R, G, and B which exceed the permissible range. This will cause any pixel which is clipped to change its color.

Beretta explains how the color representation can be transformed from a three color representation to a "hue", "chroma" (or saturation), and "lightness" (or value or brightness) representation. There are many mathematical models which implement such a representation system, including CIELAB, HSV, HLS, and TekHVC. See Beretta and Foley, J., COMPUTER GRAPHICS SECOND EDITION, Addison Wesley, 1993, chapter 13.3. Using any one of these mathematical models, the hue can be held constant while the lightness or chroma are clipped or compressed to remap the colors of each pixel so that they fall within the gamut of the output device. In the prior art, it is also known to clip or compress both the lightness and the chroma to reach a color which is the shortest perceptual displacement from the original color (CIELAB.DELTA.E). See U.S. Pat. No. 5,611,030 to Stokes.

Unfortunately, there is no method of clipping or compressing based on a single model which produces the best result in each case. Depending upon the characteristics of the particular image, one or another model using one or another of the methods of clipping or linear compression or non-linear compression may produce the best result.

The invention is a method for allowing a user to subjectively select the most effective balance between two or more alternative methods for clipping or compressing the colors of an image into the gamut of an output device. The preferred embodiment is a software system which runs on a computer system with a color monitor whose color gamut is, in all dimensions, greater than or equal to the color gamut of the selected output device. The color gamut of the output device can then be entirely expressed on the computer display. At least two algorithms for limiting the colors of the image to the gamut of the output device are programmed. The image in question is then processed with both of the limiting algorithms. A weighted average of the two outputs is then computed with the weighting selected by the user. The resulting image is then displayed. The user adjusts the weighting to achieve the most desired image as selectively judged by the user.

Alternatively, if the color monitor cannot display all of the colors of the output device, many copies of the image, each with a different weighting, are transmitted to the output device for review by the user and selection of the preferred weighting.

The principal application for the invention is to allow users to adjust the colors of photographs taken with digital cameras prior to generating the final output in the form of a computer displayed image, a computer printed image, or computer generated film output. In addition to single or still images, the invention also applies to sequences of images such as video. In this case, the user will select a typical image for each scene and adjust the color limiting methods to produce the best results for that scene. The system then applies the same balance of limiting methods to all of the images from that scene. When applied to a television, the viewer can adjust the balance between two or more limiting algorithms at any time as the show continues.

In addition to choosing between two alternative algorithms for limiting the colors to fit the gamut of the output device, the user can be given three or more competing algorithms from which to choose weightings for an average. If the user is presented with three alternative methods for limiting the colors, the user interface can be a cursor placed within a triangle. Moving the cursor to a corner of the triangle uses one of the methods; placing the cursor on an edge of the triangle achieves a weighted blending between two methods; and placing the cursor anywhere within the interior of the triangle achieves a weighted blend of three methods.

In addition to a control for choosing weighting between alternative methods of limiting the color gamut, the user is typically also given controls for adjusting the contrast, brightness, saturation, and hue. Even if the gamut of the output device is equal to that of the input device, adjustments to any of these controls can cause the output colors to exceed the gamut of the output device. If the lightness (or brightness or value) is increased, some pixels of the image will likely exceed the gamut by being too light. Conversely, if the lightness is decreased, some pixels will be too dark. If the contrast is increased, the lightness (or brightness or value) of pixels may exceed the gamut in both being too light and in being too dark.

This invention allows the user to create more detailed images by allowing the user to use more contrast and brightness when editing digital images. It allows the user a choice of using more extreme settings while still retaining an attractive image. Greater accuracy can be retained in images which are undisplayable because the color hue can be retained at the expense of saturation (or chroma) and lightness (or brightness or value).

FIG. 1a shows the user interface with the color limit balance control for balancing between two algorithms.

FIG. 1b show the color limit balance control for balancing between three algorithms.

FIG. 2 shows a flow chart of the method.

FIG. 3 shows the RGB color model.

FIG. 4 shows the HSV color model.

FIG. 5 shows the HLS color model.

FIG. 6 shows the TekHVC color model.

FIG. 1a shows the user interface for controlling the software which implements the method. The color limit balance control 1 allows the user to choose the weighting between two algorithms for balancing limits on the colors to fall within the gamut of the output device. Where three algorithms are considered rather than two, the color limit balance control 16 is a triangle with a cursor marking the balance between the three algorithms as shown in FIG. 1b.

The hue, brightness, contrast, and saturation controls 2, 3, 4, and 5 are each presented to the user as a thumb wheel, the surface of which is dragged with the cursor to the right or the left. This type of control allows high precision without requiring a large screen display. In the preferred embodiment, each square on the presented edge of the thumb wheel is displayed with a texture or color which shows the effect it will produce. The hue control shows colors ranging from balanced flesh tones to extreme colors, like on a television. The brightness control shows varying shades of gray from white to black. The saturation control shows three fully saturated colored bars in each square on one end and the same three bars in each square progressively less saturated to gray at the other end. The contrast control shows two bars in each square, white and black at one end progressing to gray for both bars at the other end.

FIG. 2 shows the method which is employed when the user is adjusting the color limit balance of an image. The color gamut of the output device is input to the system, step 6. In the preferred embodiment, the software comes with standard devices preloaded for their color gamuts and the user merely selects the output device. The user then inputs an image, step 7, such as by retrieving via a PCMCIA port an image file created with a digital camera. Alternatively, the image might be a typical image from a scene in a video program or a computer generated image. For the first algorithm, the image is automatically transformed into a first color model (Color Model A in FIG. 2) and clipped or compressed to the selected gamut, step 8. For the second algorithm, it is also transformed to a second color model (Color Model B in FIG. 2) and clipped or compressed to the gamut, step 9. Steps 8 and 9 can use the same color model but different clipping or compression algorithms. Although it may make the mathematics more complicated, steps 10 and 11 can be omitted and the clipping or compression can be performed on the original representation of the image, whatever it happens to be.

In the preferred embodiment, the first algorithm is a transformation to the HSV (hue/saturation/value) color model as shown in FIG. 4. In this model, a fully saturated (saturation=1.0) and fully bright color is represented with the maximum "value" of 1.0. White is shown with the same value (1.0) and a saturation of 0. For the first algorithm, "values" greater than the output gamut, typically 1, are clipped to the limit of the output gamut (1.0), which tends to retain the hue and saturation of each pixel at the expense of variation in brightness. Consequently, detail which is carried in portions of the image which are at the limit of brightness is lost and those areas of the image that are over-saturated, but not over-bright, tend to become darkened. Values less than permitted by the output gamut are also clipped to the gamut and saturation numbers are also clipped to the gamut.

The second algorithm transforms the image into the HLS (hue/lightness/saturation) color model as shown in FIG. 5. In this model, fully saturated colors have a "lightness" of 0.5. White has a lightness of 1.0 and black has a lightness of 0. Saturation is constrained to 0 at both white and black, and its limit varies in proportion to its distance from either white or black. After the image is transformed to this model, both saturation and lightness are clipped to the color gamut of the output device within this model. Because this model represents as white any pixel with a lightness of 1, regardless of its hue or saturation, pixels with a lightness clipped to 1 will be represented as white. Unfortunately, this method loses all of the color information in the over bright areas of the image.

The transformation from the RGB color model to the HLS and HSV color models is a one to one mapping with a well known, reversible mathematical expression. Once the clipping or compression is completed, each of the two representations is transformed back to RGB by the reverse process, steps 10 and 11.

The software then causes the system to receive user input on the desired weighting between the two algorithms, step 12. The default is even weighting, so the color balance limit control 1 is initially shown as centered. The user drags the slider to the left or to the right to adjust the weighting given to the two algorithms. The representations of the image in the RGB model are combined with the user selected weighting and the combined image is displayed, step 13. When the user is satisfied, step 14, the image is sent to output, step 15, which might be a saving of the image in a memory, transmission of the image across the network, or sending the image to a printer.

The essence of the invention is that out of gamut colors are clipped or compressed with two different methods and the user is allowed to choose a balance between the methods for the output. Instead of the HLS and HSV color models, any other color model can be used, including RGB as shown in FIG. 3 or a model intended to be perceptually uniform such as TekHVC as developed by Tekronix, Inc. and shown in FIG. 6 or CIELAB adopted by the CIE (Commission Internationale de l'Eclairage) in 1976. The model for both of the alternative algorithms can be the same. The clipping or compression methods used must yield different results. For example, while retaining hue constant, one of the methods can clip lightness (or brightness or value) while the other method clips chroma (or saturation). Instead of clipping, colors which lie outside the gamut can be transformed along a line toward the center of the gamut until they pass within the gamut boundary, while also transforming all of the more interior colors on the same line to be closer to the center of the gamut. If done uniformly, this is linear compression. The linear compression can keep the lightness constant, in which case each line of compression will be horizontal in a color model with constant lightness in each horizontal plane, or the compression can be along a line which passes through a single center point, such as through the point of saturation 0, lightness 0.5 in the HVC model of FIG. 5. The compression can be non-linear by having a proportionately larger effect on colors near the gamut boundary than on colors near the center of the gamut.

From the foregoing teachings, it will be appreciated by persons having skill in the art that many alternative algorithms can be constructed for implementation of the invention. The user can then be presented with two or more alternative clipping or compression methods which produce different trade-offs, some of which are better for some images and others of which are better for other images. The user can then use the immense visual processing power of the human brain to choose the best balance for the image. Consequently, the invention should not be understood to be limited by any of the above description but rather by the following claims.