Dynamic range enhancement for imaging sensors

[0002] The present invention herein resides in the art of digital imaging. More particularly, the present invention relates to digital imaging of a scene which has extreme lighting level differences. Specifically, the present invention relates to digital imaging of a video signal wherein the video signal is segmented according to respective gain level settings and wherein the segmented signals are recombined into a composite output signal presenting the most important information from the video signal.

[0003] In current video cameras and digital imaging capture systems, a gain setting or exposure level is set to enhance the presentation of the viewed image. These systems work well with scenes that are highly illuminated or where plenty of ambient light is present. For example, daytime or well lit rooms provide adequate lighting for viewing the scene with the video camera or sensing device. However, nighttime scenes are not easily viewed due to the lack of illumination. In order to enhance viewing of nighttime or poorly illuminated scenes, the gain setting or exposure time is increased to capture any ambient light available.

[0004] It is known that current systems are capable of adjusting the exposure or gain of the optical/electrical components within the sensor to provide viewable imaging as long as the scene's ambient lighting remains within the sensitivity range of the sensor. Current sensors provide either overexposed or underexposed imagery if the ambient conditions exceed the dynamic range of a sensor at a given gain level. This results in the loss of useful scene information and over-illuminated or under-illuminated areas of the scene. In other words, a low gain video image only presents viewable information very near the predominate light source, whereas a high gain video image only presents useful image information away from the predominant light source.

[0005] It will be appreciated that poor lighting conditions are especially frustrating in military or aviation applications where a pilot is viewing a video monitor of an approaching target, at night, where the scenery includes bright city lights. Accordingly, the target locations within or near the city are not easily detected because of the lost video information. A wide range of lighting levels and their associated problems may also occur in night-time surveillance systems or police helicopter surveillance of urban areas.

[0006] In light of the foregoing, it is evident that there is a need to view a scene with a wide dynamic range and to present the scene in an image wherein the highly illuminated areas do not over-saturate the dimly illuminated areas so as to generate a composite image of the scene.

[0007] In light of the foregoing, it is a first aspect of the present invention to provide a dynamic range enhancement system.

[0008] Another aspect of the present invention is to provide a dynamic range enhancement system with a sensor to view an optical image, wherein the sensor has a video attenuator coupled thereto to adjust gain levels or exposure times to predetermined settings.

[0009] Yet another aspect of the present invention, as set forth above, is to provide a controller that is attached to both the sensor and the video attenuator to adjust the gain levels or exposure times at predetermined intervals.

[0010] Still another aspect of the present invention, as set forth above, is to time multiplex on a field basis between unattenuated (high gain) and attenuated (low gain) video signals such that both high and low gain video signals are available simultaneously for analysis.

[0011] A further aspect of the present invention, as set forth above, is to provide a combiner which receives the high gain and low gain video signals and combines them into a composite signal with minimal saturation.

[0012] Yet a further aspect of the present invention, as set forth above, is to provide a normalizer which receives the composite video signal to reduce the dynamic range of the signal while preserving important scene information content for viewing by the human eye.

[0013] The foregoing and other aspects of the present invention which shall become apparent as the detailed description proceeds are achieved by a system for enhancing the dynamic range of an imaging signal, comprising a sensor for viewing a scene having a wide dynamic range and generating a digital image signal of the scene; an attenuator coupled to the sensor for selectively attenuating the digital image signal; a demultiplexer for receiving the digital image signal and grouping like attenuated signals into at least a first gain signal and a second gain signal; and a combiner for receiving the first and second gain signals and producing a combined image signal with minimum saturation.

[0014] Other aspects of the present invention are attained by a method for enhancing an imaging signal, comprising the steps of viewing a scene with a sensor that generates a digital image signal representative of the scene; coupling an attenuator to the sensor to generate different exposure levels while generating the digital image signal; demultiplexing the digital image signal into at least a first digital image signal and a second digital image signal corresponding to attenuation levels set by the attenuator; combining the first and second digital image signals to produce a composite image signal containing both low and high illuminated areas of the scene with minimal saturation.

[0015] For a complete understanding of the objects, techniques and structure of the invention, reference should be made to the following detailed description and accompanying drawings wherein:

[0016] FIG. 1 is a block diagram of the dynamic range enhancement system of the present invention;

[0017] FIGS. 2A and 2B are graphical representations of one embodiment showing how the high gain and low gain signals are combined into a composite output signal;

[0018] FIGS. 3A and 3B is an alternative embodiment for combining the high gain and low gain signals into a composite output signal;

[0019] FIG. 4 is a representation of a low gain signal;

[0020] FIG. 5 is a representation of a high gain signal; and

[0021] FIG. 6 is a representation of a video output signal utilizing the low gain and high gain signals shown in FIGS. 4 and 5.

[0022] Referring now to the drawings and particularly to FIG. 1, it can be seen that a dynamic range enhancement system according to the present invention, is designated generally by the numeral 10. As shown, the system 10 is employed to capture an optical image 12 which is typically any scene to be viewed and analyzed in a real-time image processing environment. It will be appreciated that the optical image 12 may be any day-time or night-time scene with either dimly-illuminated or highly-illuminated characteristics. The present invention is especially useful in viewing dimly-illuminated scenes with discrete highly-illuminated light sources.

[0023] An attenuator 14 is included in the system 10 and generates a gain signal 16 received by a sensor 18. It will be appreciated that the sensor 18 may be a charge-coupled device (CCD), an image intensified CCD or other imaging array. Moreover, the sensor 18 may be capable of either capturing a video image or a still image, depending upon the particular application. Additionally, the sensor 18 may be any source of digital imaging with different gain settings utilized for capturing the optical image 12. In the preferred embodiment, the sensor 18 is used to capture video images.

[0024] A controller 19 is connected to the attenuator 14 and the sensor 18 by control signal lines 20 for the purpose of setting or adjusting the gain variations of the attenuator 14. In embodiments where a CCD is used for the sensor 18, the controller 19 adjusts the integration time. In embodiments where a digital camera is used for the sensor 18, the controller 19 adjusts the iris setting. Depending upon the particular application, the type of scene being viewed and the sensor 18, the controller 19 adjusts the gain setting in alternating frames of video. In other words, one frame of video is captured at a high gain setting and the subsequent frame of video is captured at a low gain setting. This process is repeated accordingly. When the sensor 18 is a still digital camera, it will be appreciated that any number of high gain-set frames can be taken with a corresponding number of low gain-set frames. The controller 19 in the preferred embodiment is a field programmable gate array, such as manufactured by Xilinx, Inc.

[0025] The sensor 18 generates a digital image signal 24 which is received by a demultiplexer 26, which in turn stores frames of the digital image signal 24 in predetermined groups. In particular, the demultiplexer 26 segments the digital image signal 24 on a field basis between unattenuated (high gain) and attenuated (low gain) signals. The demultiplexer 26 then stores the appropriate field of video, corresponding to the gain signal applied to the sensor 18 by the attenuator 14, and outputs a corresponding low gain signal (V_LG) 28 and a high gain signal (V_HG) 30 which are available simultaneously at the output of the demultiplexer 26. For video images it is preferred that only two signals be multiplexed. If more than two video signals are taken, the output image tends to be smeared and detracts from the appearance thereof. Operation of the demultiplexer 26 is sequenced by the controller 19 via the signal line 20 to coordinate actions between the demultiplexer, the sensor 18 and the attenuator 14. Alternatively, the demultiplexer 26 could be in the form of two sensors which are. co-boresighted. In other words, both sensors would be looking at exactly the same scene but utilizing different gain signals to capture the optical image 12. Accordingly, each sensor would generate its appropriate gain signal.

[0026] A combiner 32, which is in communication with the controller 19 via the signal line 20, receives the low gain signal 28 and the high gain signal 30. Utilizing one of various algorithms, the combiner 32 generates a composite image signal 40 from the signals 28 and 30 by taking the most pertinent information from each. It will be appreciated that when the sensor 18 operates under night-time conditions, information contained in an image can span 15 bits (32,768). Most light amplified electro-optical sensors can cover at the most an 8 bit to 10 bit dynamic range. Furthermore, the human eye can only distinguish locally 6 bits. The combiner 32 allows for an 8 to 10 bit light amplified electrical optical sensor to cover the scene's entire dynamic range. Ultimately, this allows the human eye to view the resulting image with a minimum loss in scene information content. Generally, the combiner 32 merges the low and high gain signals 28 and 30 and produces an output with minimum saturation. To simplify discussion of these various methods, the equations below employ the following variables.

[0027] LG=low gain

[0028] HG=high gain

[0029] C=constant

[0030] F(·) and G(·) are corresponding functions of LG and HG.

[0031] HTHR=high gain threshold

[0032] LTHR=low gain threshold

[0033] V_HG=intensity value for the high gain signal

[0034] V_LG=intensity value for the low gain signal

[0035] P_LG=points for which V_HG≧HTHR

[0036] AT=attenuation factor

[0037] V_ZERO=video black value

[0038] One algorithm used by the combiner 32 is a linear combination wherein the composite image signal 40 is defined by the variable OUT. Thus, a representative equation is:

[0039] OUT=C*LG+HG  (1)

[0040] In the linear combination, the constant C can be set to the attenuation factor AT or to an arbitrary value dictated by the hardware of the system 10.

[0041] Another method of combining the low gain signal 28 and the high gain signal 30 is embodied in the following non-linear equation:

[0042] OUT=C*F(LG,HG)*LG+G(LG,HG)*HG  (2)

[0043] This method takes a function of the low gain and high gain and multiplies it by a corresponding low gain or high gain intensity value.

[0044] Another method of combining the low gain and high gain signals 28 and 30, which is referred to as a cookie-cutter method, is graphically represented in FIGS. 2A and 2B. This method removes all signal information from V_HGthat is greater than or equal to HTHR and retains all signal information of V_HGless than HTHR. Next, the method retains all signal information from V_LGthat is greater than LTHR. Accordingly, the composite image signal 40 (OUT) is defined by the following equations.

[0045] OUT=V_HGif V_HG≦HTHR  (3)

[0046] OUT=HTHR−LTHR+V_LGCif V_HG>HTHR  (4)

[0047] In other words, the high gain signal 30 is represented by the shaded area A in FIG. 2A and the low gain signal 28 is represented by the shaded area B as defined by equations 3 and 4 above. Shaded area B is shifted by HTHR-LTHR and combined with shaded area A as shown in FIG. 2B.

[0048] From the above, it will be appreciated that a portion of the high gain signal 30 and a portion of the low gain signal 28 are combined with one another to generate the composite image signal 40. This method quickly and easily combines the two image signals in an attempt to generate a “seamless” image. However, it will be appreciated that the values HTHR and LTHR may not be constant values throughout the image generation process. This results in discontinuities in the image and may generate an image that is not seamless.

[0049] Another method of combining the low gain and high gain image signals 28 and 30, which is referred to as an attenuator-based merge method, is presented in FIGS. 3A and 3B. Given the values V_ZERO, HTHR1, and HTHR2 and the function H(V_HG), OUT may be defined as follows.

[0050] H(V_HG)=0if V_HG≦HTHR1  (5)

[0051] H(V_HG)=1 if V_HG≦HTHR2  (6)

[0052] 0≦H(V_HG)≦1 if HTHR1<V_HG≦HTHR2  (7)

[0053] OUT=(1−H(V_HG))*(V_HG−V_ZERO)+H(V_HG)*AT*(V_LG−V_ZERO)  (8)

[0054] Note if Image did not saturate V_HG−V_ZERO=AT*(V_LG−V_ZERO)  (9)

[0055] As seen from the above equations and FIG. 3A, portions of the high gain signal V_HGare represented by the areas shaded with an A designation and that a portion of the low gain signal V_LGis represented by the area shaded with a B designation. Portions A and B are off-set from one another by interpolating data between the functions of the high gain signal 30 and the low gain signal 28. This results in the composite image signal 40 as represented in FIG. 3B, wherein a smooth transition is presented between the high gain and low gain signals.

[0056] The composite image signal 40 may also be defined as a combination of selected portions of the low gain signal 28 and the high gain signal 30. Referring to FIG. 3A, a portion of the low gain signal 28 may be defined along the y-axis with a lower limit of V_ZEROand no upper limit. The portion of the low gain signal 28 may be defined along the x-axis as those points bounded by H(V_HG)=1. In regard to the high gain signal 30, portions are defined along the y-axis as any point between V_ZEROand HTHR1 and along the x-laxis as any point defined by H(V_HG)=0. In combining all these portions to define a seamless composite image signal 40, FIG. 3B illustrates that those points defined by 0≦H(V_HG)≦1 provide a transition or interpolation therebetween. In this interpolation area, it will be appreciated that a linear monotomically increasing function is employed with weighting toward the method in which the signal is directed.

[0057] Based upon the foregoing methods of utilizing the combiner 32, it will be appreciated that considerations such as processing time and efficiency are considered when selecting which combining method to use. In any event, it will be appreciated that the composite image signal 40 generated by the combiner 32 typically has a much larger dynamic range than the original video signal. Accordingly, only certain portions of the composite image signal 40 are viewable by the human eye. In order to properly display the composite image signal 40, it is received by a normalizer 42 which reduces the original video signal range while preserving the scene's information content to generate a viewable range dynamic video output signal 44.

[0058] To properly understand the normalization steps, the following definitions are required:

[0059] V_COMB=video of the composite image signal 40

[0060] V_CAVG=n×m average of V_COMB, where n×m defines a pixel matrix or block within the composite image signal 40

[0061] I(·) and J(·) are monotomically increasing functions

[0062] One easily employed method for normalizing the composite image signal 40 is to employ a nonlinear mapping of V_COMB. This method employs a “static” lookup table 41 in the normalizer 42 which emphasizes the low intensity values while de-emphasizing the large intensity values in the composite image signal 40 for all video fields. In other words, the nonlinear mapping employs algorithmic functions to emphasize the portions of the composite image signal 40 which contain viewable scene information therein.

[0063] Still another method to normalize the composite image signal 40 is to employ a “dynamic” table look-up function in which the table values are calculated from global image information obtained by calculating a histogram of the composite image signal for each video field. The normalization is then accomplished by applying a non-linear mapping function, which is determined using the histogram for each field of video, for the composite image signal 40, which in turn generates the video output signal 44. In other words, each video field generates a non-linear mapping function that is then applied to the composite image signal 40 to generate the video output signal 44.

[0064] Still another method of normalizing the composite image signal 40 to generate the video output signal 44 is to apply a monotomically increasing function wherein a high pass filtering equation

[0065] OUT=V_COMB−I(V_CAVG)  (10).

[0066] is employed. This equation removes the oversaturated points within the composite image signal 40.

[0067] Another filtering method employs a geometric equation based upon low pass filtering

[0068] OUT=V_COMB/J(V_CAVG)  (11)

[0069] that also employs a monotomically increasing function. This equation passes portions of the composite image signal 40 that provide useful scene information.

[0070] It will be appreciated that the output of the combiner 32 may be embodied in a second video output signal 46 which is used for further processing. In particular, signal 46 may be used for tracking targets by using optical flow detection or other methods of image processing.

[0071] As best seen in FIGS. 4-6, a representation of the low gain signal 28 is shown in FIG. 4 and a representation of the high gain signal 30 is shown in FIG. 5. The low gain signal 28 shown in FIG. 4 only provides useful scene information near the light source. The high gain signal 30 shown in FIG. 5 only provides useful scene information away from the light source. By employing an attenuation factor of {fraction (1/32)}with the attenuator-based merge method in the combiner 32 and by using the filtering equation (11), the video output signal 44 shown in FIG. 6 is generated. Clearly, display of a viewable video output signal 44 is shown which provides the most useful scene information from both the low gain and high gain signals.

[0072] In light of the foregoing, it will be appreciated that the structures and methods of use of the present invention provide numerous advantages over the known art. In particular, use of the present invention allows for the combination of two or more image fields of different gains to produce a composite image which simultaneously contains useful scene information of both poorly illuminated and highly illuminated scenes. A further advantage of the present invention is that it allows for use of video cameras in low-light areas. These advantages allow for nighttime viewing of scenes with discrete bright light sources. In military applications, this enhances the success of a mission by increasing the amount of useful scene information. These advantages could also be utilized with hand-held video cameras to enhance presentation of the optical image viewed.

[0073] Thus, it can be seen that the objects of the invention have been satisfied by the structure and use of the invention as presented above. While in accordance with the patent statutes, only the best mode and preferred embodiment of the invention has been presented and described in detail, it is to be understood that the invention is not limited thereto or thereby. Accordingly, for an appreciation of the true scope and breadth of the invention, reference should be made to the following claims.