Method for modification of non-image data in an image processing chain

[0002] This invention relates to a digital imaging system wherein noise and other non-image data associated with the input devices and media is propagated through a series of imaging transforms. Particularly, this invention relates to the methods for propagating the noise and other non-image data through an image processing chain.

[0003] Many image processing operations require non-image information in order to properly adapt their parameters to the expected conditions. Non-image information refers to data that is carried along with each image and provides information about the capture conditions, devices, expected noise statistics, etc. Non-image data will herein be also referred to as metadata. An example of this type of image processing operation is a noise reduction algorithm that uses a table of noise root-mean-square (rms) values for each color record for every signal level. The rms value of a nominally uniform image area is a measure of the pixel-to-pixel variation, and may be computed from a set of N values, as: $\begin{array}{cc}{\sigma }_x=\sqrt{\frac{\underset{i=1}{\sum ^N}{\left(x_i-{\mu }_x\right)}^2}{N-1}}& \text{Eq. (1)}\end{array}$

[0004] where the sample mean is ${\mu }_x=\frac{\underset{i=1}{\sum ^N}{x}_i}{N},$

[0005] x_iis the pixel value of the i^th-location and σ_xis the rms pixel-to-pixel variation. Thus if the rms value is computed for uniform areas of various signal levels, the set of these values can be seen to characterize the amplitude of image noise in actual scenes acquired using the same image source, as a function of image signal.

[0006] The rms statistics could be used to adaptively discern texture from noise in localized regions of an image. Another example is image dependent sharpening where a measure of sharpness loss could be used along with rms statistics to adaptively change the level of sharpening in the image. During the processing of image information in a multistage imaging system, however, the metadata, including noise statistics, may be changed by every operation or transformation applied to the signal. If this transformation of metadata is not taken into account, then subsequent adaptive operations will not operate as intended and system performance, usually in terms of image quality, will suffer.

[0007] One way to account for the transformation of the non-image information is to estimate it directly at every step in the imaging system. For instance, U.S. Pat. No. 5,641,596 issued Jun. 24, 1997 to Gray et al., entitled “Adjusting Film Grain Properties in Digital Images”, discloses a method for measuring image noise statistics based on the scanning of several uniform step patches that are not usually present in actual scenes. This estimation step could be used in imaging systems where the image processing operations are deterministic and there exists flexibility to process the set of uniform patches. However, it does not account for adaptive transformation of the noise statistics after the estimation step.

[0008] Another approach is U.S. patent application Ser. No. 08/822,722 filed Mar. 24, 1997, by Snyder et al., now allowed as of Feb. 1, 1999, which teaches how to automatically create tables consisting of noise rms values as a function of signal level from real images. Again, the method does not teach how to transform the noise statistics based on image processing operations thereafter.

[0009] Image noise propagation through several common imaging operations is addressed in the article by Peter D. Bums and Roy S. Berns, in Color Research and Application, entitled “Error Propagation Analysis in Color Measurement and Imaging”. The analysis was not, however, applied to the transformations of noise statistics in image processing systems for use by adaptive algorithms.

[0010] Similarly, it is very useful to provide data associated with the image capture devices or capture conditions, that could be smartly used by image processing algorithms to deliver enhanced image quality. U.S. Pat. No. 5,461,440 issued Oct. 24, 1995 to Toyoda et al., entitled “Photographing Image Correction System,” discloses a photographic image correction system to correct an image on the basis of degradation information inherent in a camera body or lens. The degradation information is inferred from the information of the camera and lens that is recorded in the film. This method does not teach, however, how to adjust the degradation information after subsequent image processing.

[0011] U.S. Pat. No. 5,694,484 issued Dec. 2, 1997 to Cottrell et al., entitled “System and Method for Automatically Processing Image Data to Provide Images of Optimal Perceptual Quality”, teaches a method for selecting proper image transforms to a given set of metadata. However, this method does not teach the adjustment of the metadata specific to the image transform.

[0012] The prior art presented acknowledges the usefulness of noise statistics and other metadata in image processing applications. However, no prior art addresses the propagation of such data in an imaging system after its collection.

[0013] The object of the invention is to provide a method for modification of noise and other metadata along an image processing chain to improve the utility of data in subsequent operations and thereby improve the system performance and the image quality. It is necessary that the metadata be transformed after each image processing operation, so that it reflects the changes made to the image. The object is accomplished by:

[0014] a) providing metadata corresponding to characteristics of a specific digital image;

[0015] b) generating a metadata transformation related to at least one specific image transformation; and

[0016] c) modifying the metadata according to the metadata transformation.

[0017] A further object of the present invention is to provide an alternative method for modification of metadata in an image processing chain to improve the output of the captured image data after processing. The metadata should be applied only on demand. The object is accomplished by a method that comprises the steps of:

[0018] a) providing metadata corresponding to characteristics of a specific digital image;

[0019] b) generating an image target whose characteristics are described by the provided metadata;

[0020] c) providing an image transformation;

[0021] d) processing the image target through the image transformation applied to the digital image; and

[0022] e) calculating the modified metadata from the processed image target.

[0023] It is an object of the invention to provide an image reproduction system for modifying metadata during several steps of image processing. This improves image data output when compared to the original digital image data. The object is accomplished by a system that comprises:

[0024] a) an imaging capture device for providing digital image data;

[0025] b) memory device for storing noise and other metadata associated with the digital image data;

[0026] c) an image processing chain for carrying out image transformations; and

[0027] d) a metadata processing chain being connected to the image processing chain in cases where the image transformations are metadata sensitive.

[0028] A further object of the invention is to provide an image reproduction system for modifying noise and other metadata during several steps of image processing. The metadata should only be applied on demand. The object is accomplished by a system that comprises:

[0029] a) an imaging capture device for providing digital image data;

[0030] b) memory device for storing noise and other metadata associated with the digital image data;

[0031] c) a first processing chain for carrying out image transformations;

[0032] d) a second processing chain for carrying out transformations on a target image; and

[0033] e) a target image evaluator for determining metadata from the target image in the second processing chain and thereby providing input to a metadata sensitive image transformation in the first processing chain.

[0034] No prior art provides the details of metadata propagation in an imaging system, with application to adaptive algorithms. Clearly, the metadata needs to be transformed after each image processing operation, so that it reflects the changes made to the image. Although stochastic process theory provides the background for generating such transformations, the theory has not been applied in a digital imaging system to derive needed input data for image transformations.

[0035] The subject matter of the invention is described with reference to the embodiment shown in the drawings.

[0036] FIG. 1 is a schematic description of an image acquisition and rendering system;

[0037] FIG. 2 is a schematic description of an image processing chain;

[0038] FIG. 3 illustrates an image processing chain wherein one of the image transform utilizes one or more elements of metadata associated with the digital image;

[0039] FIG. 4 illustrates the process by which a metadata transform is generated;

[0040] FIG. 5 is a typical sensitometry correction look-up table;

[0041] FIG. 6 is a typical signal dependent noise table for a scanned film, wherein the noise level is shown for the colors red, green and blue;

[0042] FIG. 7 is a modified noise table to compensate for sensitometry correction look-up table, wherein the standard deviation of the noise is shown for the colors red, green and blue;

[0043] FIG. 8 is a schematic description of the propagation of localized maps for adaptive algorithms; and

[0044] FIG. 9 illustrates a further embodiment of the present invention wherein the image processing chain makes use of evaluated target image transformations.

[0045] It facilitates understanding to note that the present invention operates on a digital image. As such, a digital image typically consists of a number of image channels. A common number of image channels is three, one channel for each of the red, green, and blue image channels. A pixel value within the digital image consists of the values at a specific location for each channel of the digital image. Thus the value of a pixel in a digital image may be represented as a function of location. For instance 𝑓₀(x,y) refers to a triad of pixel values, for instance {19 34 19} respectively referring to the red, green, and blue values of the pixel located at position (x,y) in the digital image.

[0046] With reference to FIG. 1, an image acquisition system 8 and an image rendering system 12 are the input and the output side respectively of an image processing chain 10. The image acquisition system 8 can be a film-scanner, a document or print scanner, or a digital camera. The image rendering system 12 may be a conventional display, a digital printer, a photographic printer or a digital copier. The image processing chain 10 comprises a series of image transforms 20₁, 20₂, . . . ,20_i, . . . ,20_Nas shown in FIG. 2. The image transforms 20₁, 20₂, . . . ,20_i, . . . ,20_Noperate on the digital image data, provided by the image acquisition system 8, for producing transformed digital image data. The first image transform 20₁is applied to the input digital image data 𝑓₀(x,y), generating a first transformed image 𝑓₁(x,y). Each remaining image transform i (where 2≦i<N) operates upon the output of the previous image transform 𝑓_i−1(x,y) in order to generate a transformed image 𝑓_i(x,y). The final image transform 20_Nof the image processing chain 10 produces a final image 𝑓_N(x,y).

[0047] The image transforms 20₁, 20₂, . . . ,20_i, . . . 20_Nof the image processing chain 10 may comprise simple operations like one-dimensional look-up-tables (1D LUTs), multidimensional look-up-tables (for example a 3-dimensional LUT, as described in U.S. Pat. No. 4,500,919 by William F. Schreiber), matrices that combine the existing channels of the digital image to generate new channels, spatial operations that involve linear convolution of one or more image channels with a linear filter, or other more complicated transforms, like noise reduction and adaptive sharpening.

[0048] Data associated with the digital image, which describe attributes of the digital image not described by the pixel values may be useful in the image processing chain 10. The data associated with attributes of a digital image not described by the pixel values of the digital image will herein be referred to as metadata. The set of elements of metadata associated with the digital image 𝑓_i(x,y) will herein be referred to as M_i. A non-inclusive list of possible elements of metadata may include any of the following: image capture time, shutter speed, image color encoding metric, source of image (digital camera or film camera, for example), focal length of lens, lens aperture, Modulation Transfer Function (MTF) of capture lens, film type, detector type, subject location, subject distance, camera flash information, camera flash fire information, image resolution, focus information and image noise characteristics.

[0049] With reference to FIG. 2, it may be beneficial for an image transform to have access to the image metadata. For example, in the case of a transform that sharpens an image, it is advantageous to tailor the level of sharpening by the metadata that describes the MTF of the capture lens. Any i^thimage transform that may be improved with accurate metadata describing the image 𝑓_i−1(x,y) input to the i^thimage transform will herein be referred to as a metadata sensitive image transform. An overall image quality improvement can be made by applying such image transforms with knowledge of the image metadata.

[0050] Furthermore, it has also been recognized that the application of certain transforms to a digital image will generate a transformed digital image for which the previously acquired metadata is no longer accurate. Stated another way, if image 𝑓_i−1(x,y) is transformed by the i^thimage transform generating image 𝑓_i(x,y), the metadata M_i−1of the original image is not necessarily equivalent to the metadata of the transformed image M_i. In essence, for each i^thimage transform, there must be found the corresponding i^thmetadata transform in order to maintain the correspondence between the metadata and the image. For example, consider again the case where the image transform sharpens the digital image. The metadata of the digital image may contain MTF information relating to a blurred image. A transform then applies a sharpening to the digital image, generating a transformed digital image. The metadata (with respect to MTF) of the original digital image is then no longer accurate with respect to the transformed digital image. The advantage of the present invention is to create a metadata transform, related to the image transform that will modify the metadata of the digital image. When the metadata of the digital image is transformed by the metadata transform, the transformed metadata will accurately describe the attributes of the transformed digital image. Any subsequent i^thimage transform that utilizes metadata will then utilize metadata that is accurate for the digital image 𝑓_i−1(x,y).

[0051] As another example, the input digital image data 𝑓₀(x,y) may have associated metadata M₀which notes that the input digital image data 𝑓₀(x,y) were captured with a narrow aperture. Suppose the first image transform 20₁simulates depth of field, by blurring the non-subject regions of the image, generating image 𝑓_i(x,y). The appearance of image 𝑓₁(x,y) is that of an image captured with a wide aperture. Thus, the metadata M₀must be transformed by a first metadata transform 30₁to create metadata M₁, which then contains data associating image 𝑓₁(x,y) with a wide aperture. Any subsequent image transforms i (wherein i>1) that make use of the aperture element of the metadata will then recognize that the input digital image data 𝑓₀(x,y) have been modified with regard to aperture.

[0052] FIG. 3 illustrates an image processing chain 10, of which one of the image transforms 20₁, 20₂, . . . ,20_N(for example the third image transform 20₃) utilizes one or more elements of metadata associated with the digital image 𝑓₂(x,y). The first and second metadata transforms 30₁and 30₂are generated corresponding to the first and second image transforms 20₁and 202. Thus, M₁is generated by applying the first metadata transform 20₁to metadata M₀. Likewise, M₂is generated by applying the second metadata transform 30₂to M₁. Metadata M₂describes attributes of the image 𝑓₂(x,y) output from the second image transform 20₂which are not described by the individual pixels. Thus the third image transform 20₃, which requires as an input some element of metadata associated with the input digital image, inputs the image 𝑓₂(x,y) and the associated metadata M₂which accurately describes the digital image input to the third image transform 20₃.

[0053] FIG. 4 is an illustration of the present invention with regard to the (i+1)^thimage transform 20_i+1. The specific (i+1)^thimage transform 20_i+1is input to the metadata transform generator 40_i+1for generating a (i+1)^thmetadata transform corresponding to the (i+1)^thimage transform. The metadata M₁is then modified by the (i+1)^thmetadata transform 40_i+1for producing metadata M_i+1, which accurately describes the characteristics of image 𝑓_i+1(x,y). In addition, the (i+1)^thimage transform 20_i+1is applied to 𝑓_i(x,y) thereby generating 𝑓_i+1(x,y).

[0054] The remainder of the present invention will be described with regard to one specific element of metadata, the image noise characteristics. Although the present invention is described with reference to noise characteristics as the metadata element under consideration, those skilled in the art will recognize that similar metadata transforms may be generated to modify other elements of metadata. The following description describes the operation of the metadata transform generator, which generates for each image transform an appropriate metadata transform for modification of the noise data.

[0055] In the embodiment, shown in FIG. 4, the metadata M_iis data describing the noise statistics of the i^thimage 𝑓_i(x,y). These noise statistics are described by a noise table that include the covariance between the noise in related (color) signals. In the following, the symbol, σ_rindicates the rms value (standard deviation) for the random variable or process, r. The variance and covariance terms will be written as, e.g., σ_rr, σ_rg. The symbols Σ_iand Σ_i+1, will herein indicate the covariance matrices as a function of signal level corresponding to the noise present in the images 𝑓_i(x,y) and 𝑓_i+1(x,y), respectively. For notation convenience we will refer to the image 𝑓_i(x,y) as 𝑓_iin the following discussion. For three color record (r, g, b) images, the covariance matrices contain the following six quantities: σ_rr, σ_gg, σ_bb, σ_rg, σ_rb, σ_gbfor image data, 𝑓_i ${\sum }_i=\left[\begin{array}{ccc}{\sigma }_{\mathrm{rr}}& {\sigma }_{\mathrm{rg}}& {\sigma }_{\mathrm{rb}}\\ {\sigma }_{\mathrm{rg}}& {\sigma }_{\mathrm{gg}}& {\sigma }_{\mathrm{gb}}\\ {\sigma }_{\mathrm{rb}}& {\sigma }_{\mathrm{gb}}& {\sigma }_{\mathrm{bb}}\end{array}\right].$

[0056] For each such image transform 20_i+1, the corresponding transformation of the image noise statistics, representing the i^thmetadata transform 30_i+1, is determined.

[0057] Matrix

[0058] With regard to FIG. 4, if the image transform 20_i+1is a linear matrix A, then

[0059] 𝑓_i+1=A𝑓_i,  Eq. (2)

[0060] then the corresponding (metadata) noise covariance transformation is

[0061] Σ_i+1=AΣ_iA^T,  Eq. (3)

[0062] where the superscript ^Tindicates the transpose.

[0063] One-Dimensional Continuous Transformation

[0064] With regard to FIG. 4, if the image transform 20_i+1is a one-dimensional function. It is defined by a continuous function g_ifor each image channel. If it is denoted as G and operates as

[0065] 𝑓_i+1=G𝑓_i  Eq. (4)

[0066] and G can be expanded, for a three-color system, so Equation (4) becomes $\begin{array}{cc}\left[\begin{array}{c}s\\ t\\ u\end{array}\right]=\left[\begin{array}{c}{g}_1\left(p\right)\\ g_2\left(q\right)\\ g_3\left(r\right)\end{array}\right],& \text{Eq. (5)}\end{array}$

[0067] where the sets of input and output signals are {p, q, r} and {s, t, u}, respectively. The corresponding covariance matrix transform is described in the article by Peter D. Burns and Roy S. Bems, “Error Propagation Analysis in Color Measurement and Imaging.”

[0068] Σ_i+1=J_GΣ_i+1J_G^T  Eq. (6)

[0069] where $J_G={\left[\begin{array}{ccc}\frac{\partial g_1}{\partial p}& 0& 0\\ 0& \frac{\partial g_2}{\partial q}& 0\\ 0& 0& \frac{\partial g_3}{\partial r}\end{array}\right]|}_{{\mu }_G}$

[0070] where J_Gis evaluated at the average input signal level, μ_G.

[0071] One-Dimensional Look-up Table

[0072] With regard to FIG. 4, if the image transform 20_i+1is a one-dimensional LUT, it can be seen as a discrete form of the above one-dimensional function. If so, the partial derivative elements of J_Gcan be approximated by $\begin{array}{cc}\frac{\partial g_1}{\partial p}={\mathrm{LUT}}_1\left[p\right]-{\mathrm{LUT}}_1\left[p-1\right]& \text{Eq. (7)}\\ \mathrm{or}& \\ \frac{\partial g_1}{\partial p}=\frac{{\mathrm{LUT}}_1\left[p+1\right]-{\mathrm{LUT}}_1\left[p-1\right]}{2},& \text{Eq. (8)}\end{array}$

[0073] or other similar digital filter, where the discrete signals take on integer values, {0, 1, 2, . . . }.

[0074] FIG. 5 is a typical film sensitometry correction LUT. It shows the relation between the input signal level and the output signal level which can be regarded as a tone scale transformation. The output signal level is not a linear function of the input signal level. An example for the original signal dependent noise table for a scanned film is shown in FIG. 6. Using the sensitometry correction LUT shown in FIG. 5 as the image transform under consideration, the metadata transform for noise characteristics is represented by equation 8. Application of this metadata transform to the original noise data yields the noise table in FIG. 7. The table in FIG. 7 is a modified noise table to compensate for the sensitometry correction LUT, wherein the standard deviation of the noise is shown for the colors red, green and blue. This modified noise table (FIG. 7) is appropriate to input to algorithms applied to the image after the sensitometry correction LUT. It can be noticed that FIG. 7 shows extremely high noise at low input signal levels. This is a direct result of the noise amplification resulting from the high slope at low input signal levels in FIG. 5.

[0075] Multi-dimensional Continuous Transformation

[0076] With regard to FIG. 4, if the image transform 20_i+1is a three-dimensional function, it is defined by a continuous function for each image record. If it is denoted as G, and operates as

[0077] 𝑓_i+=G𝑓_i  Eq. (9)

[0078] where $\left[\begin{array}{c}s\\ t\\ u\end{array}\right]=\left[\begin{array}{c}{g}_1\left(p,q,r\right)\\ g_2\left(p,q,r\right)\\ g_3\left(p,q,r\right)\end{array}\right],$

[0079] where the sets of input and output signals are {p, q, r} and {s, t, u}, respectively. The corresponding covariance matrix transform is described in the article by Peter D. Bums and Roy S. Berns, in Color Research and Application, 22:280-289 1997.

[0080] Σ_i+1=J_GΣ_i+1J_G^T  Eq. (10)

[0081] where $J_G={\left[\begin{array}{ccc}\frac{\partial g_1}{\partial p}& \frac{\partial g_1}{\partial q}& \frac{\partial g_1}{\partial r}\\ \frac{\partial g_2}{\partial p}& \frac{\partial g_2}{\partial q}& \frac{\partial g_2}{\partial r}\\ \frac{\partial g_3}{\partial p}& \frac{\partial g_3}{\partial q}& \frac{\partial g_3}{\partial r}\end{array}\right]|}_{{\mu }_G}$

[0082] and J_Gis evaluated at the average input signal level, μ_G.

[0083] Multi-Dimensional Look-up Table

[0084] With regard to FIG. 4, if the image transform 20_i+1is a multi-dimensional LUT, it can be seen as a discrete form of the above continuous function. If so, the partial derivative elements of J_Gcan be approximated by discrete differences, e.g., $\begin{array}{cc}\frac{\partial g_1}{\partial p}={\mathrm{LUT}}_1\left[p,q,r\right]-{\mathrm{LUT}}_1\left[p-1,q,r\right]& \text{Eq. (11a)}\\ \mathrm{or}& \\ \frac{\partial g_1}{\partial p}=\frac{{\mathrm{LUT}}_1\left[p+1,q,r\right]-{\mathrm{LUT}}_1\left[p-1,q,r\right]}{2},& \text{Eq. (11b)}\end{array}$

[0085] or other similar digital filter.

[0086] The noise propagation can be accomplished by estimating the six unique elements of the matrix. Thus, the element of metadata M_i+1regarding the image noise statistics may be determined.

[0087] Spatial Filter

[0088] Digital filtering either by a fixed-coefficient kernel or by a locally adaptive kernel is usually accomplished by a discrete convolution. For a general (n×m) filter with coefficients h_k,l, wherein k=1,2, . . . n and l=1,2, . . . m, the variance transformation is $\begin{array}{cc}{\sigma }_{\mathrm{ss}}=\left[\stackrel{n}{\sum _{k=1}}\underset{l=1}{\sum ^m}{h}_{k,l}^2\right]{\sigma }_{\mathrm{pp}},& \text{Eq. (12)}\end{array}$

[0089] if the p color-record is filtered. Note that the filter operation, becomes an identity operation for color records where no spatial filtering is performed. The above equations can be expanded to include pixel-to-pixel and record-to-record covariance if needed.

[0090] Localized Metadata Maps

[0091] Some image transforms behave in such a way that their properties are adapted based on the characteristics in localized regions of the image. In this regard, these algorithms are referred to as adaptive algorithms. This results in varying of the metadata, dependent upon the original pixel value within any given region. As an example, an adaptive noise reduction algorithm could consist of a kernel sliding across the image and aggressively blurring regions where it senses that the region does not contain textures or edges, while conservatively removing noise in regions where there are textures or edges present. Similarly an adaptive sharpening algorithm may boost the high frequencies in busy or edge regions of the image while avoiding the boost in the uniform backgrounds to avoid amplification of the noise.

[0092] The metadata corresponding to images that have been processed using adaptive algorithms will be affected in the localized regions, rather than on a global basis. That is, different regions of the image will exhibit different noise statistics after an adaptive noise reduction algorithm has been applied. Also, different regions of the image will exhibit different effective sharpness after an adaptive sharpening algorithm has been applied. Another example of localized operations is lens falloff compensation. Lens falloff in the image capture device causes the image to exhibit different sharpness and exposure characteristics from the center of the image towards the corners. The lens light falloff could be compensated using an image masking transform, but the noise characteristics after application of the masking transform will be location dependent. FIG. 8 presents a schematic diagram with the preferred method to deal with the propagation of localized metadata in the present embodiment. Referring to FIG. 8, a first image transform 20₁is applied to the original image 𝑓₀. A second image transform 20₂is then applied. This transform is adaptive or localized as defined above. The metadata M₀corresponding to the original image is first transformed by a global metadata transformation producing metadata M₁. Due to the adaptive nature of the second image transform 20₂a two dimensional map, L₂(x,y), is created by a localized metadata transform 30₂. This map carries the local metadata information of the image on a pixel by pixel basis. In this regard the maps, L_i(x,y), are generally referred to as localized metadata maps, or localized noise data maps in the case where the element of metadata is image noise. The localized map is transformed after each metadata transformation 30₂, 30₃, . . . , 30_Ncorresponding to image transforms 20₂, 20₃, . . . , 20_N. Noise transforms such as the ones described before (LUT's, Matrices, spatial filters, etc.) are applied to the localized metadata maps. The next example provides a detailed description of this method as it relates to the application of an adaptive noise reduction algorithm to a digital image.

[0093] A popular adaptive noise reduction algorithm is considered here: Lee's sigma filter (see, Lee, J. Digital Image Smoothing and the Sigma Filter. Computer Vision, Graphics, and Image Processing 24, 255-269, 1983). It is not the purpose of the authors to make claims on this particular algorithm, but to use it for demonstration purposes. The sigma filter consists of a rectangular sampling window that is convolved with the digital image. A sliding window is moved across the image, and at every center pixel position the image noise is smoothed by averaging only those neighborhood pixels that have intensities within a fixed range from the center pixel intensity. Consequently, image edges are preserved and subtle details are retained.

[0094] It is assumed that the expected noise as a function of signal level is provided in the form of a noise table as the one described in U.S. Pat. No. 5,641,596,

[0095] σ=N[𝑓(x,y)]

[0096] where σ is the noise standard deviation, and N is the noise table.

[0097] FIG. 6 presents an example of the noise table for scanned negative film. The sigma filter will select those pixels in the local window that fall inside some pre-defined number of intensity standard deviations from the center pixel intensity. The center pixel is then replaced by the average of the selected pixels. That is, if 𝑓_i(k,l) identifies the pixels in a rectangular window of size m×n, at row k, column l in the digital image 𝑓_i, and 𝑓_i+1(x,y) is the center pixel value at row x, column y, in the digital image 𝑓_i+1we can express the local filter transform of size m×n by the following equations:

[0098] Let $\begin{array}{cc}\begin{array}{c}{\delta }_{k,l}=1,\mathrm{if}\left[f_i\left(x,y\right)-\Delta \right]\le f_i\left(k,l\right)\le \left[f_i\left(x,y\right)+\Delta \right]\\ =0,\mathrm{otherwise}\end{array}& \text{Eq. (13)}\end{array}$

[0099] where Δ is an intensity range based on the expected noise standard deviation given by the noise table (e.g. Δ=2σ). Then, $\begin{array}{cc}{f}_{i+1}\left(x,y\right)=\frac{\underset{k=x-n}{\sum ^{n+x}}\underset{l=y-m}{\sum ^{m+y}}{\delta }_{k,l}{f}_i\left(k,l\right)}{\underset{k=x-n}{\sum ^{n+x}}\underset{l=y-m}{\sum ^{m+y}}{\delta }_{k,l}}& \text{Eq. (14)}\end{array}$

[0100] After applying the adaptive noise reduction algorithm to the image, the previously acquired metadata elements corresponding to noise characteristics, provided in the noise table, are no longer valid. Therefore, a metadata transform needs to be determined to modify the noise elements in the metadata. Let σ_pp(d) denote the noise variance as a function of signal level d corresponding to image 𝑓_iin Equation 14; and let σ_ss(x,y,d) denote the noise variance as a function of position (x,y) and signal level d corresponding to image 𝑓_i+1in Equation 14 after applying an n×m sigma filter; due to the adaptive nature of the filter the new variance σ_sswill be also a function of position, and is given by

[0101] σ_ss(x,y,d)=α(x,y)·σ_pp(d)  Eq. (15)

[0102] where the location dependent coefficient α is derived from Equation 12, and is given by $\begin{array}{cc}\alpha \left(x,y\right)=\stackrel{n}{\sum _{k=1}}\underset{l=1}{\sum ^m}{h}_{k,l}^2& \text{Eq. (16)}\end{array}$

[0103] where $\begin{array}{cc}{h}_{k,l}=\frac{{\delta }_{k,l}}{\stackrel{n}{\sum _{k=1}}\underset{l=1}{\sum ^m}{\delta }_{k,l}}& \text{Eq. (17)}\end{array}$

[0104] and δ is given by equation 13.

[0105] Then, the noise at every image plane could be propagated via the two dimensional function, α(x,y), where each location (x,y) corresponds to a location (x,y) in the digital image 𝑓_i+1(x,y). In essence, α(x,y) along with the original noise variance statistics σ_pp(d), become the localized metadata or noise data map L(x,y). The entries at every (x,y) location in L(x,y) when multiplied by the original noise variance statistics σ_pp(d), correspond to the new noise variance statistics as a function of signal level, in the form of a noise table. Also, the map L(x,y) could be decimated or compressed to a manageable size in order to avoid demanding memory storage and to improve system's throughput. Subsequent noise sensitive transformations will read the map, interpolate or decompress it and calculate the expected noise at every image location (x,y). The method could be easily extended to provide any metadata information such as Noise Power Spectrum, covariance, MTF, etc. on a localized basis.

[0106] Alternative Embodiment Using Generated Targets

[0107] FIG. 9 illustrates an alternative embodiment of the present invention in which a target image g₀(x,y) is generated exhibiting the characteristics of the metadata M₀. In this regard, the metadata M₀is input to the target image generator 50. In the preferred embodiment, the metadata M₀includes noise data. This noise data is density dependent and may be described in a noise table. Such a table consists of a list of mean intensities and corresponding covariances. Table 1 presents an example of such table with noise standard deviation as a function of mean intensity.

[0108] The target image generator 50 then generates a target image g₀(x,y) with the noise characteristics described by M₀. In the preferred embodiment, the target image g₀(x,y) is an image consisting of several patches that span the signal range, each of which has dimensions A pixels by B lines and statistics similar to those of the actual image. Each patch of A×B pixels is generated by randomly assigning each of the pixel values a value x, where x is a random variable with probability distribution as in the equation (18): $\begin{array}{cc}{f}_x\left(x\right)=\frac{e^{\frac{-0.5{\left(x-\mu \right)}^2}{{\sigma }^2}}}{\sigma \sqrt{2\pi }}& \text{Eq. (18)}\end{array}$

[0109] The value of μ for a given patch n of the target image g₀(x,y) may be selected as the μ value for row n of the noise table in Table 1. For example, for the first line of the target image g₀(x,y), μ=500, and σ=20.

[0110] Again referring to FIG. 9, there is illustrated an image processing chain 10 for the image 𝑓₀(x,y) and a corresponding target image processing chain 52 for the target image g(x,y). In the preferred embodiment, each image transform 52₁, 52₂, . . . , 52_Nof the target image processing chain 52 is identical to the image transform 20₁, 20₂, . . . , 20_Nin the image processing chain 10. The first and second image transform 20₁and 20₂are then applied successively to the image 𝑓₀(x,y). The third image transform 20₃is a metadata sensitive image transform. The required inputs for the third image transform 20₃are the image 𝑓₂(x,y) and the metadata M₂. The metadata M₂is determined by applying the first two image transforms 52₁and 52₂of the target image processing chain 52 to the target image g₀(x,y). The target image g₂(x,y) output from the second target image transform 52₂is then passed to a target image evaluator 54 for generating metadata M₂. The target image evaluator 54 computes the mean μ and covariance elements of each line in the target image g₂(x,y), thereby generating the required information for noise table of metadata M₂. The metadata M₂may then be passed to the third image transform 20₃along with the image 𝑓₂(x,y) for generating the image 𝑓₃(x,y). Those skilled in the art will recognize that this process may be continued for any length of image processing chain.

[0111] The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.

[0112] 8 image acquisition system

[0113] 10 image processing chain

[0114] 12 image rendering system

[0115] 20₁, 20₂, . . . ,20_i, . . . ,20_Nimage transforms

[0116] 30₁, 30₂, . . . ,30_Nmetadata transform

[0117] 40_i+1metadata transform generator i+1

[0118] 50 target image generator

[0119] 52 target image processing chain

[0120] 52₁, 52₂, 52₃, . . . , 52_Ntarget image transforms

[0121] 54 target image evaluator

[0122] 𝑓₀(x,y) input digital image data

[0123] 𝑓₁(x,y) first transformed image

[0124] 𝑓_N(x,y) final image

[0125] M_iset of elements of metadata i

[0126] 𝑓_i−1(x,y) input to the image transform i

[0127] L_i(x,y) localized metadata map i

[0128] g₀(x,y) target image