Adaptive setting of wavelet sub-band thresholds for low-light image denoising

[0001] The present invention relates to denoising, and, in particular, to a method and apparatus for selecting a threshold for each sub-band for wavelet-based denoising.

[0002] A typical film-type camera includes a mechanical shutter. Light is reflected off of a scene from either artificial or natural light conditions. The camera receives the reflected light when the mechanical shutter is opened. While the shutter is opened, the reflected light is focused on a film that contains light sensitive components. The light sensitive components of the film create a chemical recording of the received light. The image, or scene, that is recorded on the film corresponds to the total amount of light that directed onto the film during exposure to the reflected light. The total amount of light directed onto the film is related to the aperture of the shutter, and the exposure time. The exposure time of the film corresponds to the length of time that the mechanical shutter is open. The exposure time is a fixed time interval that is manually adjusted by an operator.

[0003] Digital cameras are similar to film-type cameras in that light is reflected off of the scene and received by a light sensitive component when the shutter is open. The light sensitive component may be a charge coupled device (CCD), or a complementary metal oxide semiconductor (CMOS) device. The image sensors convert light into electronic signals when exposed. The signals are provided to an image processor to store the image of the recorded scene. In digital cameras, the shutter may be a mechanical shutter that is similar to that found in the film-type camera, or an electronic shutter mechanism.

[0004] A typical image sensor includes an array of image capturing points, or pixels. Each pixel is arranged to provide a signal that is responsive to light. The signal level that is provided by each pixel is related to the intensity of the light that is received, and the length of time that the pixel is exposed. A CMOS array is arranged in rows and columns that may be read individually, while a CCD array typically provides a serial read out of the electronic signals. The electronic signals are typically converted into digital signals that are stored in a memory circuit, such that the digitized image is captured. The digitized image can be digitally manipulated for color adjustment, image contrast, image scaling, and image rotation functions.

[0010] Various embodiments of the present invention will be described in detail with reference to the drawings, where like reference numerals represent like parts and assemblies throughout the several views. Reference to various embodiments does not limit the scope of the invention, which is limited only by the scope of the claims attached hereto. Additionally, any examples set forth in this specification are not intended to be limiting and merely set forth some of the many possible embodiments for the claimed invention.

[0011] Throughout the specification and claims, the following terms take at least the meanings explicitly associated herein, unless the context clearly dictates otherwise. The meanings identified below are not intended to limit the terms, but merely provide illustrative examples for the terms. The meaning of “a,” “an,” and “the” includes plural reference, the meaning of “in” includes “in” and “on.” The term “connected” means a direct electrical connection between the items connected, without any intermediate devices. The term “coupled” means either a direct electrical connection between the items connected, or an indirect connection through one or more passive or active intermediary devices. The term “circuit” means either a single component or a multiplicity of components, either active and/or passive, that are coupled together to provide a desired function. The term “signal” means at least one current, voltage, charge, temperature, data, or other signal.

[0012] Briefly stated, the invention is related to a wavelet-based denoising system and process. A corrected reference image is acquired and stored. A test image is adjusted using a selected gain setting such that the mean pixel value of the adjusted test image matches the mean pixel value of the reference image. The adjusted image is decomposed into sub-bands using a wavelet transformation. Thresholding is applied to a selected sub-band with a selected threshold setting. An inverse wavelet transformation is then applied to the sub-bands to provide a denoised image. This process is repeated such that each sub-band is tested with multiple threshold settings. The denoised images are analyzed to locate a threshold setting that is approximately optimal for each sub-band. This process is repeated for multiple light settings. A gain-threshold map is constructed by mapping each selected gain setting to the approximately optimal threshold settings for each sub-band at the selected gain setting.

[0013] The process for constructing the gain-threshold map may be performed using an image processing system. The gain-threshold map may be used in a process for image denoising. The process for image denoising may also be performed using an image processing system.

Example System

[0014] FIG. 1 illustrates an image processing system (100) that is arranged in accordance with aspects of the present invention. System 100 comprises an image sensor (102), a row control logic block (104), a column control logic block (106), a column amplifier block (108), a front end processing block (110), an image processing and control block (112), and a shutter (114). Image processing and control block 112 includes an analog-to-digital conversion (A/D) block (116), a processing block (118), and a memory (120). Programs (122) may be embodied in memory 120.

[0015] Image sensor 102 comprises a plurality of pixels that are organized into rows and columns. The pixels are responsive to incident light when shutter 114 is open. Shutter 114 is responsive to a shutter control signal (SCTL). Row control logic block 104 is configured to select rows in response to a row control signal (RCTL). Column control logic block 106 is configured to select columns in response to a column control signal (CCTL). Column amplifier block 108 is configured to adjust the output signals from selected pixels to provide a raw analog signal (A1). Front end processing block 110 is configured to process signal A1 to provide a processed analog signal (A2). In one example, front end processing block 110 is arranged to provide offset and gain correction for the raw analog signal (A1). In another example, front end processing block 110 is a correlated double sampling circuit (CDS) that provides offset and/or gain correction.

[0016] Image processing and control block 112 is configured to provide signals SCT, RCTL, CCTL, and a digital image signal (OUT). Image processing and control block 112 is responsive to signal A2. A/D block 116 is arranged to convert signal A2 into a digital signal (Dig) in response to signal A2. A/D block 116 is controlled by various timing signals (Timing) that may be provided by processing block 118. Processing block 118 is arranged to process the digital signal (Dig) to perform image denoising. Processing block 118 is arranged to communicate with memory 120, which receives the processed digital signal.

[0017] In one example, programs 122 are stored in memory 120, where the programs are employed by processing block 118 to perform wavelet-based image denoising in accordance with aspects of the present invention. In another example, the processing block (118) and memory block (120) are included in a single functional block such as a specially designed integrated circuit (e.g., an ASIC). In still another example, the functions provided in the processing block (118) are provided by a microprocessor, a microcontroller, or some other general purpose processing unit. In yet another example, the functions provided by the processing block (118) are provided by an electronic circuit that is specially configured such as a programmable logic device (PLD), a programmable gate array (PGA), a programmable logic array (PLA), as well as any other appropriate device or devices.

Example Process for Constructing a Gain-Threshold Map

[0018] FIGS. 2A–2C illustrate a process (200) for determining threshold settings in accordance with aspects of the present invention. Processing begins at start block 202, and proceeds to block 204. Processing proceeds from block 204 to block 216. According to one example, block 204 comprises blocks 206, 208, 210, and 212. At block 206, a reference image is acquired using good lighting conditions. According to one example, the shutter (114) is opened. Light is reflected off of the scene and received by image sensor 102. Column amplifier block 108 amplifies the output signals from selected pixels to provide a raw analog signal (A1). Front end processing block 110 processes signal A1 to provide a processed analog signal (A2). A/D block 116 is arranged to convert signal A2 into a digital signal (Dig) in response to signal A2. The digital image (Dig) is stored in memory 120.

[0019] Processing then proceeds from block 206 to block 208. At block 208, a dark image is acquired (e.g., from system 100). Processing then proceeds from block 208 to block 210. At block 210, the locations of the bad pixels identified from the dark image are stored in memory. According to one example, bad pixel correction may be performed by processing block 118 under the control of a program 122. According to one example, each good pixel location has a corresponding value of approximately zero when exposed by a dark (e.g., black) image. A pixel value greater than approximately zero corresponds to a bad pixel value. In one example, a bad pixel value may result from speckle noise that is caused by leakage current in the pixel. Higher non-zero pixel values indicate worse pixels. A predetermined number (N) of bad pixels are stored in a memory to map the bad pixel locations. According to one example, N corresponds to 1024. According to another example, the bad pixel locations are ordered from worst to best. Processing continues from block 210 to block 212.

[0020] At block 212, the reference image that is stored in memory is corrected for bad pixels. For example, the values associated with bad pixel locations (as determined at block 210) may be replaced by the average of the values associated with the good neighboring pixels. Processing then proceeds from block 212 to block 216.

[0021] At block 216, a test image is acquired at a newly selected light setting. The test image should be matched to the reference image such that the only difference between the reference image and the test image is the light setting. Processing flows from block 216 to block 218. At block 218, the test image is corrected for bad pixels such that the values associated with the bad pixel locations (as determined at block 210) are replaced. As described previously, the values associated with bad pixel locations may be replaced by the average value of the good neighboring pixels. Processing then proceeds from block 218 to block 220.

[0022] At block 220, the corrected test image is adjusted by amplifying the value associated with the pixels according to a gain setting. The gain setting is related to the selected light setting. According to one example, the gain setting is selected such that the mean pixel value of the adjusted test image matches the mean pixel value of the reference image. In another example, the maximum pixel values may be matched rather than the mean pixel values. According to one example, adjusting the corrected test image is accomplished by processing block 118 under the control of a program 122. Processing proceeds from block 220 to block 222.

[0023] At block 222, a wavelet transformation is applied to the adjusted image such that the adjusted image is decomposed into a plurality of sub-bands. According to one example, a three-level Haar wavelet transform is performed on the adjusted image. According to one example, the wavelet transformation is performed by processing block 118 under the control of programs 122. Processing proceeds from block 222 to block 226.

[0024] At block 226, one of the sub-bands is selected. Processing continues from block 226 to block 228. At block 228, approximately optimal threshold settings for the selected sub-band are determined. Processing flows from block 228 to decision block 240.

[0025] At decision block 240, the system determines whether all of the sub-bands have been tested for the selected test image. Processing proceeds from decision block 240 to block 226 when additional sub-bands should be tested. Alternatively, processing flows from decision block 240 to decision block 242 when all of the sub-bands have been tested. At decision block 242, the system determined whether all of the light settings have been tested. Processing continues from decision block 242 to block 216 when additional lights settings need to be tested. Alternatively, processing flows from block 242 to end block 244 when all of the light settings have been tested.

[0026] According to one example, block 228 comprises blocks 230, 232, 234, 238, and decision block 236. At block 230, threshold parameters are selected for the selected sub-band. According to one example, the threshold parameters are selected according to a threshold percentage (P %). A threshold value is selected such that P % of the wavelet coefficients have a magnitude that is less than the threshold value. The coefficients may be stored in histogram bins for thresholding when threshold percentages are utilized. At block 232, thresholding is applied to the selected sub-band using the selected threshold parameters. Processing continues from block 232 to block 234 where an inverse wavelet transformation is performed on each of the sub-bands to provide a denoised test image. According to one example, the denoised test image is output to a display for viewing by an operator. Processing continues from block 234 to decision block 236.

[0027] At decision block 236, the system determines whether all of the threshold parameters have been tested for the selected sub-band. Processing flows from decision block 236 to block 230 when additional threshold parameters should be tested. Alternatively, processing proceeds from decision block 236 to block 238 when all of the threshold parameters have been tested. At block 238, approximately optimal threshold settings are determined for the selected sub-band. For example, an operator may view each of the denoised test images that were tested for the current sub-band. The operator may select one of the denoised test images as the subjectively best image. The threshold settings used for the subjectively best image may be selected as the approximately optimal threshold settings for the selected sub-band.

[0028] Each of the threshold parameter sets is tested in block 228. According to one example, for each sub-band, a threshold percentage of 5% is first selected at block 230. The threshold percentage is incremented by 5% each time block 230 is reached until a new sub-band is selected. At block 238, the system evaluates the results and determines which threshold percentage provided an approximately optimal denoising of the selected sub-band.

[0029] The results of process 200 may be used to construct a gain-threshold map in memory 120. The gain-threshold map may be organized as a lookup table. The lookup table links each of a plurality of gain settings to the associated set of threshold parameters for each sub-band for the gain setting (as determined by process 200). According to one example, the threshold parameters correspond to threshold percentages.

[0030] According to one example, process 200 may process the entire image such that the entire image is stored in memory. According to another example, process 200 may process a part of the image such that memory requirements are reduced. For example, process 200 may be applied to two lines of data at a time. The quality of the denoised image is degraded when the number of lines of data is reduced.

[0031] Process 200 may be applied to color images in addition to grey-scale images. Each color channel is processed separately according to process 200. White balancing is performed on the reference image such that each color channel may have a different gain setting.

[0032] Process 200 may be used to test threshold percentages for each sub-band at each of a plurality of gain settings. Process 200 may also be used to optimize other parameters such as wavelet filter type and length, the number of decomposition levels, and the thesholding type. For example, other types of wavelet types may be tested, such as Daubechies wavelet transformations and Symmlet wavelet transformations. Haar wavelet transformations may be preferable under typical operating conditions. Daubechies wavlet transformations may be preferable in applications where a blocky output image is undesirable and the original noise level is high.

Example of Wavelet Thresholding

[0033] FIG. 3 illustrates example system functions (300) for wavelet thresholding that is arranged in accordance with aspects of the present invention. System 300 comprises a forward wavelet transformation block (302), a thresholding block (304), and an inverse wavelet transformation block (306). According to one example, system 300 is included in block 118 and is controlled by programs 122.

[0034] Forward wavelet transformation block 302 is configured to perform a wavelet transformation on an input signal (x[n]). According to one example, wavelet transformation block 302 is configured to perform a three level Haar wavelet transformation on x[n] to provide ten sub-bands as follows. At each level, an input signal is filtered by low pass and high pass decomposition filters to provide filtered outputs. The filtered outputs are sub-sampled by two (i.e. every other sample is discarded). Each sub-band can be determined by the equation:

[0035] $z\left[n\right]=\sum _{n}x\left[n\right]·h\left[2k-n\right],$
where z[n] is the sub-band obtained after filtering by a decomposition filter with a frequency response of h[n] and then subsampling the filter output by two. Low and high frequency bands are provided in the horizontal direction and low and high frequency bands are provided in the vertical direction such that four resulting frequency sub-bands (low—low, low-high, high-low, and high—high) are provided. The low—low sub-band is the input signal to each subsequent level, because most of the image data is concentrated in the low—low sub-band. According to one example, forward wavelet transformation block 302 may be used in block 222.

[0036] Threshold block 304 is arranged to threshold the decomposition output coefficients. Coefficients having a magnitude that is less than the threshold level are set to a value of zero. The coefficients having a magnitude that is greater than the threshold value are not changed when hard thresholding is implemented. The threshold value is added to coefficients that are less than zero and have a magnitude that is less than the threshold value when soft thresholding is implemented. The threshold value is subtracted from coefficients that are greater than the threshold when soft thresholding is implemented. According to one example, threshold block 304 may be used in block 232.

[0037] Inverse wavelet transformation block 306 is configured to perform an inverse wavelet transformation on the sub-bands to provide an output image signal (y[n]). Each threshold output is filtered with reconstruction wavelet filters. According to one example, each filtered output is then upsampled by two, first through rows and then through columns. According to one example, inverse wavelet transformation block 306 may be used in block 234.

Example of Image Denoising Using Gain-Threshold Map

[0038] FIG. 4 illustrates a process (400) for providing a denoised image, in accordance with aspects of the present invention. Processing begins at a start block (402). Processing then proceeds from start block 402 to block 404. At block 404, an image is captured and stored in memory. According to one example, the shutter (114) is opened. Light is reflected off of the scene and received by image sensor 102. Column amplifier block 108 amplifies the output signals from selected pixels to provide a raw analog signal (A1). Front end processing block 110 processes signal A1 to provide a processed analog signal (A2). A/D block 116 is arranged to convert signal A2 into a digital signal (Dig) in response to signal A2. The digital image (Dig) is stored in memory 120.

[0039] Processing then proceeds from block 404 to block 405, where the captured image is processed. For example, the capture image may be processed in processing block 118 under the control of a program (122). Processing proceeds from block 405 to block 418. At block 418, the denoised image is stored or output. Processing continues from block 418 to block 420 where processing is terminated.

[0040] According to one example, block 405 comprises block 406, block 408, block 410, block 412, block 414, and block 416. At block 406, the captured image is corrected for bad pixels. According to one example, correction of the bad pixels is performed by processing block 118 under the control of a program 122. Processing then proceeds from block 406 to block 408. At block 408, the corrected image is adjusted by amplifying the pixel values according to a gain setting. The gain setting may be selected manually by a user, or automatically according to the lighting conditions. Processing proceeds from block 408 to block 410. At block 410, threshold settings are retrieved from a gain-threshold map according to the selected gain setting. The gain-threshold map may be provided as a lookup table that links each of a plurality of gain settings to associated threshold parameters for each sub-band for the selected gain setting. According to one preferred embodiment, the optimal set of threshold parameters comprises an optimal threshold percentage for each sub-band for the selected gain setting. The threshold percentage corresponds to a threshold value. The threshold percentage represents the percentage of wavelet coefficients that have a magnitude that is less than the threshold value.

[0041] Processing flows from block 410 to block 412. At block 412, a wavelet transformation is applied to the adjusted image to provide sub-bands. According to one example, the wavelet transformation is performed by wavelet transformation block 302. Processing then proceeds from block 412 to block 414, where thresholding is applied to the sub-bands. According to one example, thresholding is performed by threshold block 304. The thresholding is applied according to the threshold parameters that were retrieved from the gain-threshold map for the selected gain setting. According to one example, soft thresholding is applied. Processing then proceeds form block 414 to block 416. At block 416, an inverse wavelet transformation is applied to each of the sub-bands to provide the denoised image. According to one example, the inverse wavelet transformation is accomplished by block 306. Processing then proceeds from block 416 to block 418.

[0042] The gain-threshold map may be constructed from results obtained from applying process 200. Different threshold values for each sub-band are preferable when the gain setting is high and also when the noise level is high. The threshold settings may be preferably set to the median value of wavelet composition when noise levels are low.

[0043] Process 400 may be performed on color images and grey-scale images. Each color channel is processed separately according to process 400. Each color channel is adjusted differently for white balancing. The settings for each color channel are retrieved separately on the gain-threshold map since the gain amount is different for each channel.

[0044] The above specification, examples and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.