Adaptive processing for images captured with flash

[0001] This application claims priority under 35 U.S.C. Section 119(e)(1) of provisional application No. 60/743,378, filed Feb. 28, 2006 and entitled “Adaptive Processing for Images Captured with Flash”.

[0002] In outdoor photography, the sun at noon will typically come straight from above, causing a face to self-shadow which results in unflattering dark rings under eyes and the neck. If the sun comes from behind the subject, the subject may appear too dark in comparison to the bright background. A common solution to this problem is to use a flash to brighten shadows in a face, or to bring foreground and background to similar brightness. This kind of flash is called a fill-flash as it fills in dark areas of an image with extra light.

[0003] Fill-flash, however, has a few shortcomings. Finding the right amount of flash to use, so that the foreground receives just enough light to match the background, is difficult. Too much light causes the foreground to be overexposed (for example, the face becomes too bright), whereas too little flash renders the technique useless and causes details in the darkened areas to be lost. Furthermore, it is hard to estimate the right flash intensity to use as different skin colors reflect more or less light. Similarly, oily patches of skin or flat surfaces such as the forehead cause hotspots which can emphasize sweaty or oily skin and wrinkles.

[0004] Another problem with using fill-flash is that the color of the flash and the color of the ambient light often do not agree. Photographers measure light color in terms of a warm or cool temperature expressed in Kelvin (K). For example, indoor lighting is very warm or red in nature (2800 K) compared to a cool flash (5500 K) or outdoor sun (9000+ K). As a result, the color of a subject lit by flash in certain indoor situations looks unnatural. Professional photographers add colored filters to their flash to offset this effect. However, this requires them to carry around a collection of color filters. Additionally, determining the proper filter and using it is often too difficult for the novice photographer.

[0005] The present adaptive fill-flash technique describes a photo acquisition and manipulation technique for digital photography. The adaptive fill-flash technique improves the look and simplifies the use of fill-flash photography, in particular in the case of, but not limited to, portrait photography.

[0006] The adaptive fill-flash technique is a technique for digital photography that employs flash and no-flash image pairs. In one exemplary embodiment, a fill-flash image, which is captured using a flash of known characteristics, and an ambient light image, which is taken without flash, are captured for the same image subject matter. Preferably these two images are acquired within a short time interval so that there is minimum motion of the camera and the subject matter of the image. A difference image, a fill-flash only image, is then computed by extracting the known ambient light from the flash image. The fill-flash only difference image and the ambient light image are then each white balanced to form two white-balanced images. These two white-balanced images are then combined to form a white balanced image, I, in which the flash and ambient color agree.

[0007] It is also possible to reduce the hot spots in image I. In one embodiment, this is done by replacing its mid-frequency with that of the white-balanced ambient image. Additionally, the fill-flash strength can then be adjusted by blending I and the white-balanced ambient image.

[0008] It is noted that while the foregoing limitations in existing techniques for overcoming flash photography issues described in the Background section can be resolved by a particular implementation of the adaptive fill-flash technique described, this technique is in no way limited to implementations that just solve any or all of the noted disadvantages. Rather, the present technique has a much wider application as will become evident from the descriptions to follow.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0019] In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present disclosure.

2.0 Adaptive Fill-Flash Technique.

[0020] The present adaptive fill-flash technique describes a photo acquisition and manipulation technique for digital photography. The adaptive fill-flash technique improves the look and simplifies the use of fill-flash photography, in particular in the case of, but not limited to, portrait photography.

[0021] The adaptive fill-flash technique is a technique for digital photography that employs flash and no-flash image pairs. This image pair provides a few interesting properties. First, the flash will separate the image into foreground and background, because the flash power falls off with the square of the distance. Hence only nearby objects receive flash light. Second, the flash color is known and provides a reference point for estimating scene illuminants. Third, the flash adds more light which leads to a cleaner image signal. Lastly, the image pair provides a photographer with choice as to which image to use.

2.1 Exemplary Adaptive Fill-Flash System and Operating Environment.

[0022] A high level exemplary block diagram of an exemplary adaptive fill-flash system and overall operating environment is shown in FIG. 1. As shown in FIG. 1, the adaptive fill-flash technique in this embodiment includes a camera 102, that captures images of a subject 104, such as, for example, a person. One or more image pairs are captured, one image of each pair being captured with flash 106 and the other being captured without flash 108. These images are input into a computing device 110 that performs the adaptive fill-flash processing and outputs a corrected image 120. The computing device may run program modules including an image pair input module 112, a white balancing module 114, a hot spot reduction module 116 and other processing modules such as a flash strength adjustment module 118.

2.2 Adaptive Fill-Flash Technique/Overview.

[0023] In general, the adaptive fill-flash technique uses images captured with different lighting conditions. In one exemplary implementation, multiple images of the same scene but with different lighting conditions are acquired. For example, two images may be captured for the same subject. The first image may be captured with ambient lighting (where ambient lighting is typically generated by a light source that remains constant during the capture process, or may also be no light at all) and the second image may be captured with the addition of a known light source, such as a flash. The contribution of the illuminant by the known light source in the second image is extracted to form a third image. For example, the third image may be extracted by subtracting the first image from the second image. Separate processing steps are performed on the extracted image and the first image. For example, the extracted image may be white-balanced for the color of the known light source and the first image may be white-balanced for an estimated color of the ambient light. The processed images are then combined to form the final image. Other corrective processing may also take place.

[0024] More specifically, one embodiment of the adaptive fill-flash technique described herein is shown in FIG. 2. As shown in process actions 202, 204, a fill-flash image, F+A, acquired using a flash of known characteristics and ambient light, and an ambient light only image, A, of the same image subject matter are acquired. Images F+A and A are preferably captured within a short time interval so that there is little motion of the camera and the subject matter of the image between images of the set. A fill-flash only image, F, which is a difference image computed by subtracting image A from image F+A, is computed, as shown in process action 206. The ambient light color W_ba is estimated from the ambient light image A using conventional methods (process action 208). Image F and the ambient light image, A, are then white balanced, as is shown in process actions 210, and 212, to form images F^wb and A^wb, where F^wb is the white-balanced fill-flash only difference image, and A^wb is the white-balanced ambient light image. In a preferred implementation, F^wb can be used as a surface color estimate for the white-balancing process for image A^wb. For example, a large white surface in the image F^wb should still appear white in the image A^wb. Images F^wb and A^wb are then combined to form a white balanced image I in which the color of flash and ambient illuminant agree (process action 214). It is also optionally possible to reduce the hot spots, as shown in process action 216. In one embodiment, this is done by replacing its mid-frequency with that in A^wb, by a process to be described later. Additionally, the fill-flash strength can also optionally be adjusted by blending I and A^wb (as shown in process action 218), which will also be discussed in greater detail below. It should be noted that the above process can be performed for multiple images taken with flash (as indicated by the dotted line from process action 214 to 202) to generate multiple white balanced images in which the color of the flash and ambient light agree. Subsequent processing can also be performed on these images.

[0025] Although the present examples are described and illustrated herein as being implemented in a system for white-balancing an image using separate processing for a flash and ambient image pair, the technique described is provided as an example and not a limitation. As those skilled in the art will appreciate, the present examples are suitable for application in a variety of different types of systems for image enhancement by separately processing multiple images captured under different lighting conditions.

[0026] Additionally, it should be noted that the order in which the fill-flash image and the ambient light image are captured is not important. The previously described process is shown here for illustration. In actual implementation, more, less, or different process actions may be implemented. For instance, it is not necessary to correct the image I for hot spots, or to adjust the fill-flash strength by blending. Any and all of these process actions can be performed as stand alone image enhancements or can be combined into different hybrid embodiments.

[0027] The following sections provide additional details and variations of the adaptive fill-flash technique described above.

2.3 Image Acquisition.

[0028] As discussed above, the adaptive fill-flash technique acquires and stores an image set, typically composed of two images, with a camera. Typically, these images will depict the same static scene. One picture is taken without flash and the other picture is taken with a flash. These pictures in the image set ideally should be acquired in a short time interval, so that there is a minimum of motion of the camera and the subject matter of the image. However, if there is significant misalignment between the initially acquired images they can be aligned using conventional image alignment methods.

[0029] While the acquisition of a fill-flash, F+A, and ambient light image, A, pair is performed by a camera, the processing of the image pair can occur on the imaging device (camera, video, etc.) or other computation devices such as, for example, a desktop PC. In one embodiment a camera acquires the image pair and the processing occurs on a PC. However, other embodiments exist were processing occurs on a camera, cell-phone cameras or flash assisted video cameras, for example.

[0030] More specifically, as shown in FIG. 3, in one embodiment, the acquisition steps for the image pair is as follows. Initially, in order to acquire the non-flash or ambient image, the camera is focused on the subject, and then the focal length and aperture are locked, as shown in process action 302. The exposure time and ISO sensitivity are also set for a good exposure (process action 304). Ideally, no image element should be overexposed. The ambient light image, A, is captured (process action 306). Then the flash is turned on and the flash image, F+A, is taken (process action 308), ideally not overexposing any imaging element. In taking the flash image, the flash is preferably set to a conservative low energy amount thereby avoiding overexposing the image. In one working embodiment of the adaptive fill-flash technique this is accomplished by using a Through-the-Lens (TTL) metered type of flash, or by acquiring multiple flash images while reducing the flash power until no pixel is overexposed.

[0031] Finally there should be sufficient ambient light available to ensure fairly low-noise images. While this is not a strict requirement, it enhances the look of the correct image that is output noticeably.

2.4 Bronzing Lighting Variation.

[0032] In another exemplary embodiment of the present adaptive fill-flash technique, instead of correcting the flash color to white, a colored tone can be intentionally added to the flash light. For example, a bronze tone can be added to a model's skin color to create a warm white tone. This is similar to the effect of using golden reflective foil, often used by professional photographers, without the trouble of carrying additional equipment along.

[0033] More specifically, one exemplary embodiment of the adaptive fill-flash technique described herein, wherein a bronzing or tanning effect (or other desired color) is accomplished, is shown in FIG. 4. As shown in process actions 402, 404, a fill-flash image, F+A, acquired using a flash of known characteristics and ambient light, and an ambient light only image, A, of the same image subject matter are acquired, preferably captured within a short time interval. A fill-flash only image, F, which is a difference image computed by extracting image A from image F+A is computed, as shown in process action 406. The ambient light color W_ba is estimated from the ambient light image A (process action 408). Image F and the ambient light image, A, are then white balanced, as is shown in process actions 410, and 412, to form images F^wb and A^wb, where F^wb is the white-balanced fill-flash only difference image, and A^wb is the white-balanced ambient light image. The foreground of image F^wb is then white-balanced to make it more amber (or other specified color), resulting in image F^wb_bronze. Images F^wb_bronze and A^wb are then added to form a white balanced image I in which foreground objects are more bronze or other specified color (process actions 416, 418).

[0034] Additionally, it should be noted that the ambient image A could be adjusted using the same techniques. For example, rather than correcting image A to a white tone, the image could be colored in such a way to create a pronounced color contrast between the areas exposed by flash an those without.

[0035] While this embodiment describes the application of adaptive white-balancing to the images A and F respectively, other operations such as sharpening, blurring, adjustments to color saturation or contrast could be made as well. That is, one could apply any image editing operation to the regions carved out by the flash. For example, just like color-balancing the flash only area separately, the background could be blurred for a depth-of-field effect or the background could be desaturated to emphasize the foreground via conventional methods.

2.5 Slow Sync Flash/Night Shot Mode Variation.

[0036] Many digital cameras come with a night shot mode, also referred to as slow sync flash. Slow sync flash is useful in dark indoor environments such as evening soirées, art-openings or bars. In this mode, an image is taken with flash and the shutter is then kept open to expose for the background. As a result part of the image is exposed by flash, tinting that area blue, while other parts are lit by the ambient illumination, which tints those areas orange or other tones. A traditional approach is to correct for an average color tint, but this means part of the scene will be either too blue or orange.

[0037] As discussed previously, the adaptive fill-flash technique described herein allows for adaptive white balancing by using two images: one with the flash, the other without. After each image has been corrected for flash and ambient light respectively, they can be added to create an image that has the correct colors for a night setting.

[0038] More specifically, one embodiment of the adaptive fill-flash technique for a slow sync or night shot image acquisition mode is shown in FIG. 5. As shown in process action 502, a fill-flash image, F+A, is acquired using a flash of known characteristics and ambient light. Since there is little ambient light, the fill-flash image, F+A, generally approximates a flash only image, F. An ambient light only image, A, of the same image subject matter is also acquired. The ambient image, A, is acquired taking a long exposure (process action 504). The ambient light color W_ba is estimated from the ambient light image A (process action 506). Image F and the ambient light image, A, the one that is taken with ambient light only, are then white balanced, as is shown in process actions 508 and 510, to form images F^wb and A^wb, where F^wb is the white-balanced fill-flash only difference image, and A^wb is the white-balanced ambient light image. Images F^wb and A^wb are then added (process action 512) to form a white balanced image I in which flash and ambient color agree (process action 514).

2.6 Hot Spot Reduction.

[0039] Hot spots are areas illuminated by flash that appear too bright. They occur in the image, because the flash is a very bright focused light source and certain areas, such as the oily unflattering T-zone in the face, reflect more light than the surrounding areas of the face. As a result typical portraits of a person's face in fill-flash photography suffer from hot spots.

[0040] Images can be expressed in the frequency domain, as is known to those with ordinary skill in the art. The frequency domain is a space in which each image value at image position P represents the amount that the intensity values in image I vary over a specific distance related to P. In the frequency domain, changes in image position P correspond to changes in the spatial frequency, or the rate at which image intensity values are changing in the spatial domain image. In many cases, the Fourier transform is used to convert images from the spatial domain into the frequency domain and vice-versa. A related term used in this context is spatial frequency, which refers to the inverse of the periodicity with which the image intensity values change. Image features with high spatial frequency (such as edges) are those that change greatly in intensity over short image distances.

[0041] Typically hot spots will be fairly large and uniform across the face. In signal processing terms, the hot spots can be expressed as the mid-frequency band of the image. In one embodiment, the present adaptive fill-flash technique replaces the mid-frequency band from of image I with the mid-frequency band of the white balance corrected ambient image A^wb. The replacement of the mid-frequency band can be constrained to the area affected by the flash only, which is captured by F, the flash only image. The mid-frequency band can be found through a band-pass filter that is of the approximate width of the hot spot. Automatic estimation of the hot spot areas can be performed using face recognition techniques that are well known in the art. With user interaction a user could specify the size of the face by drawing a bounding rectangle around it. In one exemplary implementation the band-pass is a bi-lateral filter, with the domain size set to about half a face (for example, plus or minus 10%) and the range large enough to cover the color change caused by the flash. Domain denotes the width in the spatial domain (or frequency), and range the level of color to be affected. The resulting image after white-balancing and hot spot correction is called I* for purposes of this description.

[0042] More specifically, an exemplary embodiment of hot spot reduction using the adaptive fill-flash technique is shown in FIG. 6. As shown in process action 602, a pair of images, one with flash, F+A, and one without flash, A, are acquired and input into the system. An image, I, is computed that is white-balanced for both flash and ambient light using the white-balanced version of image A, and a white-balanced version of a flash-only image, F, in the same manner as was discussed with respect to FIG. 2, as shown in process action 604. Image I, corresponding image F and the white-balanced version of image A, are converted from the image domain into the frequency domain, as shown in process action 606. Image F is searched to find any hot spots, such as would be found on faces. This can be done by using a band pass filter that is the approximate width of the hot spot or be estimated by using manual or automatic face detection methods which are well known in the art. Any hot spots, such as, for example, any detected faces in image F, are designated as the location of the mid-frequency band (e.g., a hot spot) (process action 608). The portion of the mid-frequency band of image I that corresponds to the mid-frequency band in image F is replaced by the mid-frequency band of white-balanced image A to obtain an image with reduced hot spots, I* (process action 610).

2.7 Continuous Fill-Flash/Fill-Flash Strength Adjustment.

[0043] Image I* represents the fill-flash image that is properly white-balanced and has reduced hot spots, A^wb is the white-balanced ambient light image. These two images can be blended, for example, by using a technique described by Hoppe in 2004. The Hoppe system and method, named the continuous flash system and method, facilitates user-selectable illumination of a photograph or other image ranging from full artificial flash illumination to ambient light illumination and every gradation between, based on a set of initially acquired images of identical or similar subject matter. Additionally, it is even possible to extrapolate beyond the initially acquired images to achieve other illumination effects.

[0044] The improvement to the Hoppe continuous flash technique provided by the adaptive fill-flash technique is that the input images are correctly white-balanced and hot-spots are corrected for in order to improve the appearance of the flash adjusted image output by the Hoppe continuous flash technique. In Hoppe, the images white-points may not have been aligned causing unnatural colorations when extrapolating between the images.

[0045] More specifically, one embodiment of the adaptive fill-flash technique that is an improvement to the continuous flash system and method is shown in FIG. 7. The adaptive fill-flash technique acquires and stores an image set, typically composed of two images, with a camera (process action 702). Typically, these images will depict the same static scene. One picture is taken without flash and the other picture is taken with a flash. It should be noted that if the originally acquired image set includes more than one image with flash (while still retaining the image taken at ambient light conditions) better interpolation/extrapolation results can be obtained. These pictures in the image set ideally should be acquired in a short time interval, so that there is little motion of the camera and the subject matter of the image between the images of the set. Once the initial set of images is acquired, and optionally aligned, the system interpolates and/or extrapolates the amount of flash, as shown in process action 704, 706. In such a case, additional images are generated. Essentially, in the case of interpolation, image processing is employed to interpolate the flash intensity between input images to generate an image or images having flash intensity value(s) in-between those of the input image set. Whereas, in the case of extrapolation, extrapolation techniques are used to extrapolate the flash intensity to generate images having a flash intensity outside those of the input image set. The resulting images are referred to as flash adjusted images. One or more of these flash adjusted images can be selected as the desired image, the image with the preferred amount of flash, as shown in process action 706. The selected flash-adjusted image can also be white-balanced as was discussed with respect to FIG. 2. To this end, the ambient light image is subtracted from the selected image in order to obtain a flash only image, F, as shown in process action 708. Image F is white-balanced for the known flash color to create F^Wbf, a white-balanced flash only image corresponding to the selected image (process action 710). The ambient light color is estimated from the ambient light image and is used to white-balance the ambient light image, (process actions 712, 714) in a manner similar to that discussed with respect to FIG. 2. Finally, the white-balanced flash only image and the white-balanced ambient image are combined to create a version of the selected image that is white-balanced for flash and ambient light (process action 716). Other variations, such as reducing hot spots and applying the bronzing effect, are also possible.

3.0 The Computing Device/Environment.

[0046] The technique is operational with numerous general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the process include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0047] FIG. 8 illustrates an example of a suitable computing system environment. The computing system environment is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the present system and process. Neither should the computing environment be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment. With reference to FIG. 8, an exemplary system for implementing the present process includes a computing device, such as computing device 800. In its most basic configuration, computing device 800 typically includes at least one processing unit 802 and memory 804. Depending on the exact configuration and type of computing device, memory 804 may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. This most basic configuration is illustrated in FIG. 8 by dashed line 806. Additionally, device 800 may also have additional features/functionality. For example, device 800 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in FIG. 8 by removable storage 108 and non-removable storage 810. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Memory 804, removable storage 808 and non-removable storage 810 are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by device 800. Any such computer storage media may be part of device 800.

[0048] Device 800 may also contain communications connection(s) 812 that allow the device to communicate with other devices. Communications connection(s) 812 is an example of communication media. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable media as used herein includes both storage media and communication media.

[0049] Device 800 may also have input device(s) 814 such as keyboard, mouse, camera, microphone, pen, voice input device, touch input device, etc. In particular, one such input device is a camera. Output device(s) 816 such as a display, speakers, printer, etc. may also be included. All these devices are well know in the art and need not be discussed at length here.

[0050] The present technique may be described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types. The technique may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

[0051] It should also be noted that any or all of the aforementioned alternate embodiments may be used in any combination desired to form additional hybrid embodiments. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.