Method and apparatus of enhancing a digital image using multiple selected digital images

[0001] The present invention relates to the field of digital imaging, and more particularly to processing an digital image making use user selected digital images.

[0002] Some interactive digital imaging software applications have been designed and implemented that allow the user to view multiple renditions of a source digital image on a monitor and select a most pleasing rendition. Specifically, applications have been employed where the multiple renditions represent different balances of the source digital image, i.e. different brightness renditions or different color renditions. The user when presented with the multiple renditions selects one of the renditions. This type of user interactive display is sometimes referred to as a ring-a-round when a default rendition of the source digital image is displayed in a center position with rendition variations presented about the center position. The digital imaging application then records the user's rendition selection and uses the corresponding image processing parameters associated with the rendition selection to enhance the source digital image.

[0003] One of the problems with current multiple rendition digital imaging applications is experienced when the ideal rendition of the digital image is not represented by one of the displayed renditions. For this case the user must select the rendition closest to the hypothetical ideal rendition. Other types of user interactive applications use slider bars or numerical input parameters that allow a user to fine tune the image characteristics of the final image rendition. These types of user interface mechanisms are sometimes difficult to use or may be slow due to the image processing required to generate the next rendition of the digital image for viewing.

[0004] What is needed is a user interface that combines the ease of use characteristics of the multiple rendition design interface with the fine tuning characteristics of the slider bar design interface.

[0005] According to the present invention there is provided a solution to the problems discussed above.

[0006] According to a feature of the present invention there is provided a method for deriving enhanced image processing parameters for a source digital image, comprising the steps of:

• • [0007] a) providing a source digital image;
  • [0008] b) generating at least three rendered digital images from the source digital image with at least one image processing parameter associated with each rendered digital image;
  • [0009] c) displaying the rendered digital images on a display;
  • [0010] d) selecting two or more of the rendered digital images; and
  • [0011] e) using the image processing parameters associated with the rendered digital images to generate enhanced image processing parameters.

[0012] The invention has the following advantages.

[0013] 1. A user interface for a display that displays a source digital image that combines the ease of use characteristics of a multiple rendition design interface with the fine tuning characteristics of a slider bar design interface.

[0021] In the following description, a preferred embodiment of the present invention will be described as a software program. Those skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. Because image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance with the present invention. Other aspects of such algorithms and systems, and hardware and/or software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein may be selected form such systems, algorithms, components, and elements known in the art. Given the description as set forth in the following specification, all software implementation thereof is conventional and within the ordinary skill in such arts.

Description of an Image and a Digital Image

[0022] An image can refer to any form of visual information in recorded or displayed form. Examples of recorded images may include, but are not limited to, a photographic film negative, a photographic film transparency, a motion picture film, and a photographic print. Displayed forms of images may include, but are not limited to, visual presentations made on electronic displays such as CRT monitors, LCD panels, electroluminescent devices, and LASER projection systems.

[0023] A digital image is comprised of one or more digital image channels. Each digital image channel is comprised of a two-dimensional array of pixels. Each pixel value relates to the amount of light received by the image capture device 10 (FIG. 2) corresponding to the geometrical domain of the pixel. For color imaging applications a digital image will typically consist of red, green, and blue digital image channels. Although the preferred embodiment of the present invention is practiced with digital images produced with a capture device 10, the present invention is not limited to pixel data relating to photographs. For example, graphic or other synthetic data may be merged with photographically captured pixel data and still be considered a digital image. Other configurations are also practiced, e.g. cyan, magenta, and yellow digital image channels. For monochrome applications, the digital image consists of one digital image channel. Motion imaging applications can be thought of as a time sequence of digital images. Those skilled in the art will recognize that the present invention may be applied to, but is not limited to, a digital image for any of the above mentioned applications. Although the present invention describes a digital image channel as a two dimensional array of pixel values arranged by rows and columns, those skilled in the art will recognize that the present invention can be applied to mosaic (non rectilinear) arrays with equal effect.

Description of a Image Capable Computer

[0024] The present invention may be implemented in a combination of computer hardware software as shown in FIG. 3 within an image capable computer 100 as shown in FIG. 2. Although the preferred embodiment of the present invention is practiced with a personal computer, any image capable computer 100 can perform the tasks required of the present invention such as but not limited to a hand held computer, kiosk computer, laptop computer, or a computer remotely connected via a communications network 110. The following description relates to a digital imaging system as a subset of the components of a personal computer 100 which includes an image capture device 10, a digital image processor 20, an image output device 30, and a general control processor 40. The system includes a monitor device 42 for viewing digital images. The system also includes an input control device 44 such as a keyboard and or mouse pointer. The input control device 44 is used to make selections of digital image displayed on the monitor device 42.

[0025] The digital image processor 20 may be used with digital images derived from a variety of imaging devices. For example, the image capture device 10 can be a conventional photographic film camera combined with a film scanner device. The photographic film camera captures a scene on photographic film negative or photographic film transparency and a film scanner device for scanning the developed image on the film and producing a digital image. The present invention can also be used with other image capture devices 10; such as but not limited to, digital cameras and print scanners. The digital image processor 20 provides the means for processing the digital images to produce pleasing looking images on the intended output device or media or alternatively, for analyzing the content of processed digital images. Although a single image output devices 30 is shown, the present invention may be used in conjunction with a variety of image output devices 30 which may include, but not limited to, a digital photographic printer. Those skilled in the art will recognize that the present invention is not limited to just these mentioned image processing modules.

[0026] The general control processor 40 shown in FIG. 2 may store the present invention as a computer program stored in a computer readable storage medium. The computer readable storage medium may comprise, for example: magnetic storage media such as a magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM). The associated computer program implementation of the present invention may also be stored on any other physical device or medium employed to store a computer program indicated by offline memory device 46.

[0027] It should also be noted that the present invention implemented in a combination of software and/or hardware is not limited to devices that are physically connected and/or located within the same physical location. One or more of the devices illustrated in FIG. 2 may be located remotely and may be connected via a wireless connection.

Description of a Image Processor

[0028] The digital image processor 20 shown in FIG. 2 is illustrated in more detail in FIG. 3. The rendering module 210 receives a source digital image 201 and produces two or more rendered digital images from the source digital image 201. These rendered digital images represent the image content of the source digital image 201 with some image characteristic, such as brightness or color, having been modified. The rendered digital images are then displayed on the monitor device 42. The user of the system then makes selections with an input control device 44 of two or more of the rendered digital images in step 220. The rendered digital image selection information is then passed on to the enhancement rendition module 230. Associated with the rendered digital images are corresponding image processing parameters that were used by the rendering module 210 to produce the rendered digital images. The image processing parameters corresponding to the selected rendered digital images are also passed on to the enhancement rendition module 230. The enhancement rendition module 230 uses the image processing parameters to generate one or more enhanced image processing parameters. The enhanced image processing parameters are used by the enhancement rendition module 230 to process the source digital image 201 to produce an enhanced digital image 204. Alternatively, the enhanced image processing parameters may be attached to, or stored with, the source digital image 201 as meta-data, i.e. non-pixel information associated with a digital image. The source digital image 201 with the attached meta-data can then be transmitted via the communications network 110 to a remote computer that uses the source digital image 201 and the attached meta-data to produce an enhanced digital image 204.

[0029] The rendering module 210 shown in FIG. 3 is illustrated in more detail in FIG. 4. The source digital image 201 is received by the multiple rendition module 212 which produces more than one rendered digital image. The multiple rendered digital images 203a–203i differ from one another is at least one visual characteristic. For example, the preferred embodiment of the present invention varies the overall brightness characteristic of the multiple rendered digital images. However, it should be recognized by those skilled in the art that the present invention can be practiced in a form where the multiple rendered digital images vary in some other visual characteristic, such as but not limited to, color, tone scale, or contrast. The multiple rendered digital images 203a–203i are then displayed on a monitor device 42.

[0030] The multiple rendition module 212 shown in FIG. 4 is illustrated in more detail in FIG. 5. The source digital image 201 is received by the scene balance module 214. The scene balance module 214 calculates an adjustment for the overall brightness and color of the source digital image 201. The present invention may be practiced with any scene balance module such as the one described by Cok et al. in U.S. Pat. No. 4,945,406. The scene balance module calculates overall balance values, i.e. pixel values corresponding to a theoretical 20% gray card reflector. The present invention may also be practiced with a scene balance module that involves the user to manually adjust balance of the source digital image 201. For this alternative embodiment of the present invention the scene balance module 214 substitutes a manual user interface for the automatic calculation logic.

[0031] The balance sequencer 216 receives the overall balance values corresponding to the source digital image 201 and generates a set of additional balance values. Each additional balance value will correspond to a specific rendered digital image 203. An equal number of additional balance values are calculated which are numerically higher and lower than the overall balance value in equal increments of brightness. For example, if the overall balance value for the source digital image 201 is code value 1000, the additional balance values of 900, 800, 700, and 600 are calculated corresponding to progressively lighter rendered digital images 203. Similarly, the additional balance values of 1100, 1200, 1300, and 1400 are calculated corresponding to progressively darker rendered digital images 203. The overall balance value and the additional balance values are received by the balance applicator 218 which produces a rendered digital image 203a–203i corresponding to each balance value.

[0032] The details of the image selection module 220 shown in FIG. 3 is described in more detail below. For the example given above, the nine rendered digital images 203a–203i are displayed on a monitor device 42 as illustrated in FIG. 1. The digital images are arranged from darkest to lightest with the rendition corresponding to the overall balance value displayed in the middle. The user is allowed to interact with the display by making one or more selections of the rendered digital images 203a–203i with a selection input control device 44 such as a keyboard or mouse. The user is instructed be make a first selection based on the rendition for which the main subject of the source digital image is best rendered. This can be accomplished in a variety of ways. The preferred embodiment of the present invention uses a help message that is displayed on the monitor device 42. The user views all of the rendered digital images 203a–203i and makes a first selection by placing the display cursor over the intended rendered digital image 203a–203i and clicking with the mouse. The user is also instructed to select another rendered digital image 203a–203i for which non-subject regions of the source digital image 201 is better rendered than in the first selection. The user then makes a second selection. The user is allowed to select the OK button 209 at any time. If the use determines that the first selected rendered digital image 203a–203i is a also the best rendition for the non-subject regions of the image, the user can select the OK button 209 which results in no second selection. The selections made by the user are recorded by the digital image processor.

[0033] In the preferred embodiment of the present invention the user is requested to make a maximum of two selections but is allowed to make only one selection. If the user does make two selections, the system automatically terminates the selection process. The preferred embodiment of the present invention is primarily designed to work well in a semi-automated environment. The user, or operator, of the system, can achieve a better overall productivity of producing enhanced digital images if the next source digital image to be processed is automatically displayed on the monitor device 42 in the multiple rendition mode after the selection process associated with the last source digital image has been completed.

[0034] In an alternative embodiment of the present invention the user must select a button which is displayed on the monitor device 42 before making selections of the multiple rendered digital images. Referring to FIG. 1, the user can select the subject mode button 207 or the background mode button 208 by clicking with the mouse when the cursor is positioned over the display button. In this alternative embodiment the order of selecting rendered digital images is controlled by the user. However, the user must first select one of the mode buttons and then make a selection of a rendered digital image related to the mode. In another alternative embodiment the user must only select a mode button once. For example, the user selects the background mode button 208 first and then makes a rendered digital image selection. The use then makes another rendered digital image selection which is automatically interpreted by the digital image processor as relating to the other mode.

[0035] In another alternative embodiment of the present invention the user is instructed to make multiple selections of the same rendered digital image. For this embodiment, the enhanced digital image is produced by interpolating between two selections of rendered digital images. In the example illustrated in FIG. 1, the nine rendered digital images represent different renditions of the source digital image which vary in their in brightness characteristic. If the user prefers a rendered digital image with a brightness characteristic which is in between those displayed on the monitor device 42, the user can select one of the rendered digital images more than once to indicate that the preferred rendition is closer to that of the rendered digital image which was selected multiple times. A weighted average of the image processing parameters is calculated using the number of times a rendered digital image is selected as the weighting factor. Thus with this alternative embodiment of the present invention the user can control the rendition of the final enhanced digital image with greater precision than merely having to select from the choices presented automatically.

Enhancement Rendition Module.

[0036] The enhancement rendition module 230 shown in FIG. 3 is illustrated in more detail in FIG. 6. The balance values corresponding to the first and second selected rendered digital images 203a-203i constitute a set of selection parameters 205 that is received by the tone scale function generator 232. The set of selection parameters 205 is used by the tone scale function generator 232 to calculate a rendering tone scale function. The tone scale function applicator 234 applies the rendering tone scale function to the source digital images 201 to produce an enhanced digital image 204.

[0037] The preferred embodiment of the present invention uses a tone scale function to produce the enhanced digital image 204. The simplest form of a tone scale function is a single valued function that is used to map input pixel values to output pixel values. The shape of the rendering tone scale function applied to the source digital image 201 determines the visual differences between the source digital image 201 and the enhanced digital image 204. It should be noted that although the preferred embodiment of the present invention uses a single valued function in the form of a one-dimensional look-up-table (LUT), those skilled in the art will recognize that other forms of tone scale functions, such as three-dimensional LUT's can also be constructed and practiced with the present invention.

[0038] The mathematical formulation of the LUT values that constitute the rendering tone scale function is as follows. Let y represent the output LUT value associated with the LUT index value x. The range in x values, i.e. the length of the and extent of the numerical range of the LUT, depends on the type of image data to be processed. For 12-bit digital imaging applications the range in x values is from 0 to 4095. Let the value ρ represent the reference gray value of the digital imaging application system that corresponds to a reference, or calibrated gray point. The mathematical formula for the numerical value y placed in the LUT at index x is divided into three zones depending on the index value x. and whether the second balance value is higher or lower than the overall balance value. For the case in which the second balance value is numerically lower than the overall balance value, the formula is given by:
y=x+(ρ−B_o)+(B_o−B₂)γ  (1)
and
γ=0 for x>=φ+ε
γ=1 for x<=φ−ε
γ=(x−φ−ε)/(2ε) for φ−ε<x<φ+ε
where B_o represents the overall balance value, B₂ represents the second balance value, and φ represents a the midpoint of the transition zone and E represents the half width of the transition zone. The rendering tone scale function y(x) is shown in graphical form in FIG. 7. In similar form, if the second balance value B₂ is numerically higher than the overall balance value B_o, the mathematical form is given by:
y=x+(ρ−B_o)+(B_o−B₂)γ  (2)
and
γ=0 for x>=φ−ε
γ=1 for x>=φ+ε
γ=(φ+ε−x)/(2ε) for φ−ε<x<φ+ε

[0039] The mathematical formulation of the rendering tone scale function employed by the preferred embodiment of the present invention is designed to compress the dynamic range of the pixel image data. More photographed natural scenes are too high in dynamic range than too low. When the user selects the first rendered digital image the overall balance value is determined. This process sets the overall brightness for the main subject of the image. When the user selects the second rendered digital image it is assumed that source digital image is too high in dynamic range. Therefore, a compressive tone scale function is calculated from the overall balance value and the second balance value. When the rendering tone scale function illustrated in FIG. 7 is applied to the source digital image, different regions of the image will experiment different amounts of brightness changes depending on the code value of the image pixels. The darkest regions of the source digital image 201 will experience more brightness change than the lightest regions based on the shape of the rendering tone scale function. The resultant enhanced digital image 204 generated will have a reduced dynamic range relative to the source digital image 201, i.e. the shadow or dark regions of the image will appear lighter.

[0040] The rendering tone scale function is applied to the source digital image 201 by the tone scale function applicator 234 as shown in FIG. 6. The preferred embodiment of the present invention applies the rendering tone scale function to a luminance digital image channel constructed from the source digital image 201. The red, green, blue digital image channels composing the source digital image 201 are transformed into a luminance-chrominance domain. The rendering tone scale function is applied to the luminance channel producing a tone scaled luminance digital image channel. The a tone scaled luminance digital image channel and the chrominance digital image channels are then transformed back into the red-green-blue domain constituting the enhanced digital image 204. Those skilled in the art will recognize that the present invention can be practiced with many different forms of luminance-chrominance domain transforms and their corresponding inverse transforms.

[0041] The luminance-chrominance transform employed by the present invention utilizes a linear 3 by 3 matrix transform. Let R_ij, G_ij, and B_ij refer to the pixel values corresponding to the red, green, and blue digital image channels located at the i^th row and j^th column. Let L_ij, C1_ij, and C2_ij refer to the transformed pixel values of the modified digital image. The 3 by 3 matrix transformation relating the input and output pixel values is as follows:
L_ij=τ₁₁R_ij+τ₁₂G_ij+τ¹³B_ij  (3)
C1_ij=τ₂₁R_ij+τ₂₂G_ij+τ₂₃B_ij
C2_ij=τ₃₁R_ij+τ₃₂G_ij+τ₃₃B_ij
where the τ_mn terms are the coefficients of the 3 by 3 LCC matrix denoted by [τ]. The constants employed by the present invention for τ₁₁, τ₁₂ and τ₁₃ are 0.333, 0.333 and 0.333 respectively. It is important to note that the present invention may be practiced with other luminance chrominance transformation and still yield good results. For example, a matrix with τ₁₁, τ₁₂ and τ₁₃ values set to 0.30, 0.59, and 0.11 respectively also works well.

[0042] The preferred embodiment of the present invention applies the rendering tone scale function, in the form of a look-up-table, directly to the pixels of the luminance digital image channel of the source digital image 201. This method is preferred primarily for its computational efficiency properties. An alternative embodiment of the present invention employs a method of applying a tone scale function disclosed by Lee et al. in U.S. Pat. No. 5,012,333 for improved image quality results. Although Lee et al. describe a method for interactively modifying image attributes, the present invention employs the method of applying tone scale functions to digital images based on spatial filtering techniques. This method involves spatially filtering the luminance digital image channel resulting two spatial frequency components, (high and low components), applying the tone scale function to the low spatial frequency component, and combining the tone scale modified low spatial frequency component with the high spatial frequency component. The resulting processed digital image has enhanced brightness and contrast attributes with improved spatial detail content.

[0043] A computer program product may include one or more storage medium, for example; magnetic storage media such as magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such as optical disk, optical tape, or machine readable bar code; solid-state electronic storage devices such as random access memory (RAM), or read-only memory (ROM); or any other physical device or media employed to store a computer program having instructions for practicing a method according to the present invention.

[0044] Each computer readable storage medium can comprise, for example; magnetic storage media such as magnetic disc (such as a floppy disc) or magnetic tape; optical storage media such as optical disk, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any other physical device or medium employed to store a computer program.

[0045] The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.