Joint denoising and demosaicking method for color RAW images guided by monochrome images

[0001] The present application is a continuation of International Application No. PCT/CN2023/084429, filed on Mar. 28, 2023, which claims priority to Chinese Application No. 202310277581.2, filed on Mar. 21, 2023, the contents of both of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates to the field of image processing, and in particular, to a joint denoising and demosaicking method for color RAW images guided by monochrome images.

[0003] Image denoising and demosaicking is an essential part of the color camera image processing process. The existing methods generally first denoise the image in the RAW domain, and then use the demosaicking algorithm to convert the image into the RGB domain. With the development of deep learning technology, a neural network-based algorithm model has achieved better results in both denoising and demosaicking tasks. With the support of a large amount of training data and model parameters, it is possible to model multiple degradation types simultaneously using neural networks. Therefore, a joint denoising and demosaicking network can model both processes simultaneously to prevent error accumulation and reuse image features. For example, according to the paper titled “Beyond Joint Demosaicking and Denoising: An Image Processing Pipeline for a Pixel bin Image Sensor” presented at the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, various mechanisms, such as attention modules and adversarial training, are introduced to enhance the joint denoising and demosaicking effects. However, while neural networks can learn the image restoration process well, in low signal-to-noise ratio dark scenes, single-image restoration algorithms encounter performance bottlenecks in noise reduction and detail recovery.

[0004] On the other hand, with the development of multi-sensor devices, multi-image fusion processing has become a crucial means to break through the bottleneck of single-image algorithms. The monochrome-color binocular camera system is widely used in devices such as smartphones. Compared to the color cameras, the imaging process of monochrome cameras is not affected by color filters and has a higher amount of light, thus providing a significant advantage in imaging quality. Guiding color cameras for denoising and demosaicking based on monochrome cameras can leverage the strengths of multiple sensors combinations. However, the existing guided restoration neural network methods cannot directly deal with non-aligned scenes. According to the paper titled “Robust Joint Image Reconstruction from Color and Monochrome Cameras” presented at the 2019 British Machine Vision Conference, binocular images are first registered at the pixel level based on optical flows to mitigate the effects of parallax, and then the restoration results are solved iteratively by using traditional optimization methods. The method involves a significant computational load and faces challenges in achieving precise registration, especially in low signal-to-noise ratio scenes, which further impacts the subsequent joint denoising and demosaicking processes.

[0005] The present disclosure proposes a joint denoising and demosaicking method for a color RAW image guided by a monochrome image, starting from the imaging advantages of monochrome cameras. In order to eliminate non-aligned factors caused by parallax between the monochrome image and the color RAW image, the present disclosure constructs an aligned guidance image generation module based on an attention mechanism, and trains the aligned guidance image generation module based on a perceptual loss function to generate a high-quality aligned guidance image for guiding the color RAW image denoising and demosaicking. Meanwhile, the present disclosure trains a joint guided denoising and demosaicking network module based on a color-structure joint loss function, resulting in better visual effects of restoration results. The method of the present disclosure can process non-aligned guided joint denoising and demosaicking scenes containing parallax by generating high-quality aligned guidance images, while allowing monochrome images to be contaminated by noises without affecting the guiding effect. The impact of binocular parallax and the noises of the monochrome image is reduced based on a parallax attention mechanism, and the information of monochrome image in the monochrome-color binocular system is utilized to assist in the joint denoising and demosaicking process of the color image.

[0006] The present disclosure provides a joint denoising and demosaicking method for a color RAW image guided by a monochrome image. The method performs denoising and demosaicking on the color RAW image under the guidance of the monochrome image in parallax scenarios, including the following steps:

• • [0007] S1, a synthetic noisy dataset of a monochrome-color binocular camera system in a dark scene is constructed.
  • [0008] S2: a joint denoising and demosaicking model is constructed based on the dataset in the step S1.

[0009] The joint denoising and demosaicking model includes an aligned guidance image generation module and a guided denoising and demosaicking module.

• • [0010] S21, the aligned guidance image generation module is constructed: computing a feature correlation between the monochrome image and a RAW image in a parallax direction based on a parallax attention mechanism by the aligned guidance image generation module, and constructing an aligned guidance image feature by aggregating the relevant structural feature to obtain an aligned guidance image; and using a clean grayscale image ground truth corresponding to the RAW image as supervision, and training the aligned guidance image generation module based on a perceptual loss function until convergence, so that the aligned guidance image generated by the aligned guidance image generation module is structurally similar to the RAW image as much as possible.
  • [0011] S22, the guided denoising and demosaicking module is constructed: extracting a feature of the RAW image and a feature of the aligned guidance image, respectively, by the guided denoising and demosaicking module, performing feature fusion on the feature of the RAW image and the feature of the aligned guidance image through channel concatenation among the feature dimension after up-sampling the feature of the RAW image, generating a clean RGB image corresponding to the RAW image through the feature decoder, and completing a guided denoising process and a guided demosaicking process simultaneously; and training the guided denoising and demosaicking module based on a color-structure joint loss function until convergence, so that the denoising and demosaicking result has both an accurate color reconstruction and a sharp detail structure.

[0012] The aligned guidance image generation module fuses the non-local information of the monochrome camera image with the information of the color camera image based on the parallax attention mechanism to generate high-quality guidance images aligned with color images; and the guided denoising and demosaicking module guides the joint denoising and demosaicking process of the color RAW images based on high-quality aligned guidance images.

• • [0013] S3: the color RAW image is denoised and demosaicked based on the joint denoising and demosaicking model constructed in the step S2. By using the aligned guidance image generation module, an additional high signal-to-noise ratio information provided by the monochrome camera image is utilized to reduce the impact of non-aligned factors. Meanwhile, the denoising and demosaicking process of the color RAW image is guided to output a color image with a good visual effect.

[0014] In the step S1, a large number of synthetic noisy datasets of monochrome-color binocular camera system in dark scenes is generated for training and testing of the joint denoising and demosaicking model based on the existing color binocular image dataset. The construction method for the synthetic noisy datasets of the monochrome-color binocular camera system is as follows:

• • [0015] S11, each pixel in the normal images captured by a color binocular camera system is divided by K in brightness to simulate the dark scene.
  • [0016] S12, for a left view, a value of a color channel is sampled based on a Bayer pattern of RGGB to generate the RAW image; and for a right view, a complete amount of light entering the monochrome camera is simulated by adding values of three color channels to generate the monochrome image.
  • [0017] S13, the Poisson-Gaussian noise with same parameters is added to the monochrome image and the RAW image.

[0018] The construction of the joint denoising and demosaicking model in the step S2 includes two parts: one is to construct the aligned guidance image generation module, and the other is to build the guided denoising and demosaicking module:

[0019] Computing a feature correlation between the monochrome image and a RAW image in a parallax direction based on a parallax attention mechanism by the aligned guidance image generation module, and constructing an aligned guidance image feature by aggregating the relevant structural feature further includes:

• • [0020] firstly demosaicking the RAW image based on a traditional demosaicking algorithm, and transforming the RAW image into a single channel grayscale image by adding the three color channels; next, extracting a RAW image feature F_raw and a monochrome image feature F_mono from the left view and the right view through a same feature extractor, which can reduce noise interference and enhance structural information; subsequently, computing a correlation weight matrix M in the parallax direction for the RAW image feature F_raw and the monochrome image feature F_mono based on the parallax attention mechanism, and using a correlation in a row direction to construct the aligned guidance image feature: {circumflex over (F)}_raw=f(F_mono)*M, where f represents a convolution operation, and {circumflex over (F)}_raw represents the aligned guidance image feature, obtained by weighted fusion of the feature of the guidance monochrome image based on parallax correlation; and finally, decoding the aligned guidance image feature {circumflex over (F)}_raw into the aligned guidance image, which is structurally consistent with a target color image, by using a feature decoder.

[0021] The correlation weight matrix M is computed as follows: for the RAW image feature {circumflex over (F)}_raw and the monochrome image feature F_mono from the left view and the right view, both tensor dimensions are H×W×C, where H, W, and C represent a height, a width, and a number of channels of a feature tensor, respectively; first, passing the RAW image feature F_raw from the left view through a convolutional layer of 3×3 to generate a query tensor, passing the monochrome image feature F_mono from the right view through two unshared convolutional layers of 3×3, respectively, to generate a key tensor and a value tensor, wherein the query tensor, the key tensor and the value tensor all have a dimension of H×W×C; rearranging and converting the dimension of the key tensor to H×C×W; performing a matrix multiplication on the query tensor and the rearranged key tensor to obtain a correlation calculation result matrix with a dimension of H×W×W, and obtaining the correlation weight matrix M in the row direction by a SoftMax operation.

[0022] The perceptual loss function is used as the loss function during the training of the aligned guidance image generation module. The perceptual loss function in the step S2 is defined as: L_P=∥F_VGG(Ĝ)−F_VGG (G)∥₂, where L_P represents the perceptual loss function; Ĝ represents an output result of the aligned guidance image generation module; G represents a reference ground truth for supervision, and is set to the clean grayscale image corresponding to the RAW image; and F_VGG (⋅) represents an image feature extracted by a pre-trained VGG model. The perceptual loss function is used to calculate a distance between the output result and the ground truth in VGG feature space. By optimizing the distance during training, the generated aligned guidance image can focus more efficiently on structure reconstruction.

[0023] The guided denoising and demosaicking module utilizes the generated high-quality aligned guidance image to replace the non-aligned monochrome noisy image, and guides the joint denoising and demosaicking process of color RAW image. The guided denoising and demosaicking module extracts the features of the RAW image with noises and the features of the high-quality aligned guidance image, respectively, and performs feature fusion through feature concatenation among the channel dimension. Based on the color-structure joint loss function, the guided denoising and demosaicking module is trained to directly decode and generate the clean RGB image corresponding to the RAW image, and completing the guided denoising process and the guided demosaicking process simultaneously.

[0024] The neural network mainly learns structural information from the aligned guidance image and color information from the RAW image, and employs the color-structure joint loss function as the loss function for training the guided joint denoising and demosaicking module. The color-structure joint loss function in the step S2 is defined as: L_pc=∥F_VGG(Ŷ)−F_VGG (Y)∥₂+∥F_gaussian (Ŷ)−F_gaussian (Y)∥₂, where L_pc represents the color-structure joint loss function; Ŷ represents a joint denoising and demosaicking result; Y represents a reference ground truth, which is set to a clean RGB image corresponding to the RAW noisy image. The first term of the loss function is a structural loss, F_VGG(⋅) represents an image feature extracted by a pre-trained VGG model, constraining the structural information of the output results through VGG spatial features; and the second term of the loss function is a color loss, F_gaussian (⋅) represents image low-frequency information extracted through Gaussian filtering, and the color loss function calculates a loss in low-frequency space of the image by F_gaussian(⋅). The joint denoising and demosaicking result is constrained in terms of both structural and color aspects, so that the structural information can be fully transferred, while maintaining accurate color fidelity in the output results.

[0025] The beneficial effect of the present disclosure are as follows: the non-aligned factors in the guiding process are eliminated based on the parallax attention mechanism, allowing the guidance information of monochrome camera images to be accurately assigned to the denoising and demosaicking process of color camera images in parallax scenarios. Meanwhile, through the design and combination of the perceptual loss function and the color-structure joint loss function, accurate high-frequency detail structure is reconstructed after denoising, while the demosaicking process achieves an accurate color interpolation effect. The present disclosure has significant application value for binocular imaging devices in the dark scene.

[0034] The present disclosure will be further explained below in conjunction with specific embodiments and accompanying drawings.

[0035] Firstly, the synthetic monochrome-color binocular image dataset is constructed based on the color binocular image dataset.

[0036] In an embodiment, the normal images captured by the color binocular cameras are divided by K in brightness to simulate the dark scene (the maximum pixel value of a dark scene image is less than 40). For the left view, the value of the color channel is sampled based on the Bayer patter of RGGB to generate the Bayer RAW image. For the right view, the complete amount of light entering the monochrome camera is simulated by adding values of three color channels to generate the monochrome image. Finally, since the monochrome-color cameras are inevitably contaminated by real noises in actual scenes, the Poisson-Gaussian noise with same parameters is added to the monochrome image and the RAW image. As a result, a noisy RAW-monochrome binocular image pair is obtained for subsequent network training in the dark scene.

[0037] The network architecture of the method of the present disclosure is shown in FIG. 1, mainly divided into two modules: the aligned guidance image generation module and the guided denoising and demosaicking module.

[0038] The aligned guidance image generation module, as shown in FIG. 2, eliminates non-aligned factors between the monochrome image and the RAW image based on the parallax attention mechanism to generate the aligned guidance image. Meanwhile, the monochrome ground truth of the RAW image is used as supervised information, and the perceptual loss is used as a loss function to train the aligned guidance image generation module separately until convergence.

[0039] The specific description of the aligned guidance image generation process based on the parallax attention mechanism is as follows:

[0040] Firstly, for the left camera image and the right camera image, the same feature encoder is used for feature encoding. The specific structure of the feature encoder is shown in FIG. 3. The RAW image is demosaicked based on the traditional demosaicking algorithm DDFAPD, and is transformed into the single channel grayscale image by adding the three color channels. The feature encoding process includes two important roles: the noise of the RAW image and the monochrome image are suppressed through feature encoding, reducing interference to structural similarity judgment; and the target RAW image and the guidance monochrome image are converted into the same feature space simultaneously, preserving the structural correlation between the target RAW image and the guidance monochrome image.

[0041] Second, the RAW image feature F_raw and the monochrome image feature F_mono from the left view and the right view are extracted through a same feature extractor.

[0042] Subsequently, the left view feature and the right view feature are fused based on the parallax attention mechanism, and the specific structure is shown in FIG. 4. Due to the non-aligned factors caused by parallax generally appearing in the same row, the correlation between the left view feature and the right view feature on the same row is calculated based on the parallax attention mechanism. The correlation weight matrix M is computed in the parallax direction for the RAW image feature F_raw and the monochrome image feature F_mono, and the correlation in a row direction is used to construct the aligned guidance image feature: {circumflex over (F)}_raw=f(F_mono)*M, where f represents convolution operation, and {circumflex over (F)}_raw represents the aligned guidance image feature; and decoding the aligned guidance image feature {circumflex over (F)}_raw into the aligned guidance image, which is structurally consistent with a target color image, by using a feature decoder. The aligned guidance image feature is computed by linearly combining the monochrome image feature value on the corresponding row based on correlation.

[0043] The correlation weight matrix M is computed as follows: for the RAW image feature F_raw and the monochrome image feature F_mono from the left view and the right view, both tensor dimensions are H×W×C, where H, W, and C represent a height, a width, and a number of channels of a feature tensor, respectively. The RAW image feature F_raw from the left view is passed through the convolutional layer of 3×3 to generate the query tensor, the monochrome image feature F_mono from the right view is passed through two unshared convolutional layers of 3×3, respectively, to generate the key tensor and the value tensor. The query tensor, the key tensor and the value tensor all have a dimension of H×W×C. The dimension of the key tensor is rearranged and converted to H×C×W. The matrix multiplication is performed on the query tensor and the rearranged key tensor to obtain the correlation calculation result matrix with a dimension of H×W×W, and the correlation weight matrix M in the row direction is obtained by a SoftMax operation.

[0044] The visualization results of feature correlation of the parallax attention mechanism are shown in FIGS. 5(a), 5(b), 5(c), 5(d) and 5(e). FIG. 5 (a) shows the visualization results of the left camera target image, and FIG. 5 (b) shows the monochrome image of the right camera. For the L1, L2, and L3 rows of pixels indicated in FIG. 5 (a) and FIG. 5 (b), the feature correlation results are shown in FIG. 5 (c), FIG. 5 (d), and FIG. 5 (e). The feature correlation matrix forms a clear peak value in the diagonal direction, and shows that the dislocation structure caused by parallax is well captured under the parallax attention mechanism.

[0045] Finally, the aligned guidance image is decoded and generated based on the aligned guidance image feature {circumflex over (F)}_raw. In order to generate the aligned guidance images with low noise and clear structure, the algorithm employs the ground truth grayscale image corresponding to the RAW image as training supervision, and employs the perceptual loss as the loss function to train the aligned guidance image generation module. The perceptual loss function in the step S2 is defined as: L_p=|F_VGG(Ĝ)−F_VGG(G)∥₂, where L_p represents the perceptual loss function; Ĝ represents the output result of the aligned guidance image generation module; G represents the reference ground truth for supervision, and is set to the clean grayscale image corresponding to the RAW image; and F_VGG (⋅) represents the image features extracted through the pre-trained VGG model. The perceptual loss function optimizes the generated results of the aligned guidance image generation module by calculating the distance of the feature space of the pre-trained VGG network, making the structure more consistent and visually appealing to the human eye.

[0046] The guided denoising and demosaicking module, shown in FIG. 6, extracts the RAW image feature and the aligned guidance image feature, respectively, and up-samples the resolution of the RAW image feature. The information of the RAW image feature and the aligned guidance image feature is fused through feature concatenation among the channel dimension, and finally the joint denoising and demosaicking results are outputted through the feature decoder. Based on the color-structure joint loss function, the guided denoising and demosaicking module is trained to achieve an accurate color reconstruction and a sharp detail structure in the joint denoising and demosaicking results. The specific network structure of the guided denoising and demosaicking module is shown in FIG. 7.

[0047] The color-structure joint loss function is defined as:

[0048] $L_{\mathrm{pc}}={F_{\mathrm{VGG}}\left(\hat{Y}\right)-F_{\mathrm{VGG}}\left(Y\right)}_2+{F_{\mathrm{gaussian}}\left(\hat{Y}\right)-F_{\mathrm{gaussian}}\left(Y\right)}_2$
where L_pc represents the color-structure joint loss function, the first term being the perceptual loss function, and the second term being the color loss function; F_VGG(⋅) represents the image features extracted through the pre-trained VGG model, and the perceptual loss function calculates the loss in the deep feature space of the image through F_VGG (⋅) function. F_gaussian (⋅) represents the image low-frequency information extracted through Gaussian filtering, and the color loss function calculates the loss in low-frequency space of the image by F_gaussian (⋅). The color-structure joint loss function constrains the various structures of image content in the VGG feature space, while constrains the overall brightness of image content in the low-frequency space of the image. The combination of the two helps to fully transfer various structures in the guidance image to the target image during the training process, while preventing the influence of guidance image brightness on color reconstruction, thus obtaining the guided joint denoising and demosaicking results with clear structure and accurate color.

[0049] FIG. 8(a), 8(b), 8(c), 8(d), 8(e) and 8(f) show the operation results of a specific embodiment based on the method of the present disclosure. In FIG. 8(d), the aligned guidance image, generated by the method of the present disclosure, has low noise, clear structure, and is consistent with the structure of the target color image. FIG. 8(e) shows the joint denoising and demosaicking results generated by the method of the present disclosure. Compared with the reference target image in FIG. 8(f), the structure and color are consistent.

[0050] The present disclosure proposes a novel guided denoising and demosaicking method based on the monochrome-color binocular camera system, which generates the high-quality aligned guidance image in non-aligned scenes containing parallax. Based on the color-structure joint loss function, the joint denoising and demosaicking results with good visual perception are obtained. When the method of the present disclosure is deployed in a monochrome-color binocular system, the advantage of the monochrome camera in imaging, unaffected by color filters, is utilized to assist in the demosaicking process of the color camera, and the advantage of the monochrome camera in imaging having a larger amount of light when simultaneously imaging in the binocular system is utilized to assist in the denoising process of the color camera.