Method for automatically determining whether to perform denoising

[0001] The present disclosure claims the priority to the Chinese patent application with the filing No. 202411047872.3, entitled “METHOD FOR AUTOMATICALLY DETERMINING WHETHER TO PERFORM DENOISING” and filed on Aug. 1, 2024 with the Chinese Patent Office, the contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to the technical field of light field microscopy systems, and more particularly to a method for automatically determining whether to perform denoising.

[0003] As an emerging three-dimensional imaging technique, light field microscopy system may not only obtain high-resolution 3D reconstructed images, but also achieve rapid and non-destructive imaging, thus boasting broad application prospects in the biomedical field.

[0004] However, due to the complexity of the scanned samples and the limitations of the light field microscopy system itself, the final reconstructed 3D image is often affected by various noises, such as scattering noise, background noise, and detector noise. Relatively severe noise not only may reduce the quality of the 3D reconstructed image, but also may cause distortion of the 3D reconstructed image, thus affecting subsequent analysis and processing. Therefore, it is particularly important to quickly and accurately determine whether denoising needs to be performed on the light field microscopic images obtained by scanning samples with a light field microscopy system.

[0005] At present, whether denoising needs to be performed on the light field microscopic images obtained by scanning samples with a light field microscopy system depends entirely on the judgement based on the system user's personal experience, which is not only inefficient but also highly subjective.

[0006] Therefore, the problem to be urgently solved by those skilled in the art is how to provide a method for automatically determining whether to perform denoising, which is not only efficient but also provides a more objective determination result.

[0007] In view of this, a purpose of the present disclosure is to provide a method for automatically determining whether to perform denoising, which not only is efficient but also provides a more objective determination result.

[0008] In order to achieve the above purpose, the present disclosure adopts the following technical solutions.

[0009] A method for automatically determining whether to perform denoising includes the following steps:

• • [0010] S1: scanning a sample by using a light field microscopy system to obtain a light field microscopic image containing 4D information, where the sample is a sample to be 3D reconstructed;
  • [0011] S2: rearranging the light field microscopic image to obtain a multi-view rearranged image; and
  • [0012] S3: calculating a cleanliness score of the multi-view rearranged image; and
  • [0013] determining whether to perform a denoising process on the multi-view rearranged image, based on the cleanliness score of the multi-view rearranged image.

[0014] Preferably, S3 specifically includes:

• • [0015] S31: selecting a central view image, an image adjacent to the left side of the central view image, and an image adjacent to the right side of the central view image from the multi-view rearranged image by using the microlens array parameters used by the scanning light field microscopy system;
  • [0016] S32: obtaining the cleanliness score of the multi-view rearranged image based on the central view image, the image adjacent to the left side of the central view image, and the image adjacent to the right side of the central view image;
  • [0017] S33: comparing the cleanliness score of the multi-view rearranged image with a preset threshold:
  • [0018] if the cleanliness score of the multi-view rearranged image is greater than or equal to the preset threshold, the denoising process does not need to be performed on the multi-view rearranged image; and
  • [0019] if the cleanliness score of the multi-view rearranged image is less than the preset threshold, S34 is executed;
  • [0020] S34: dividing each of rearranged images of three views which are the central view image, the image adjacent to the left side of the central view image, and the image adjacent to the right side of the central view image, into n*n sub-images, where

[0021] $\frac{H}{200}\le n\le \frac{H}{100},$

• • [0022]  H indicates the pixel height of the central view image, the image adjacent to the left side of the central view image, and the image adjacent to the right side of the central view image;
  • [0023] S35: calculating cleanliness scores of corresponding sub-images of the images of three views, to obtain an image cleanliness score matrix; and
  • [0024] S36: comparing the maximum value in the image cleanliness score matrix with the preset threshold:
  • [0025] if the maximum value in the image cleanliness score matrix is greater than or equal to the preset threshold, the denoising process does not need to be performed on the multi-view rearranged image; and
  • [0026] if the maximum value in the image cleanliness score matrix is less than the preset threshold, the denoising process needs to be performed on the multi-view rearranged image.

[0027] Preferably, S32 specifically includes the following steps:

• • [0028] S321: calculating a structural similarity index measure between the central view image and the image adjacent to the left side of the central view image; and
  • [0029] calculating a structural similarity index measure between the central view image and the image adjacent to the right side of the central view image; and
  • [0030] S322: calculating the average of the two structural similarity index measures in S321, and using the average as the cleanliness score of the multi-view rearranged image.

[0031] Preferably, the following steps are adopted in S35 to calculate the cleanliness score of corresponding m-th sub-images of the images of three views:

• • [0032] S351: calculating a structural similarity index measure between the m-th sub-image in the central view image and the m-th sub-image in the image adjacent to the left side of the central view image, where 1<=m<=n²; and
  • [0033] calculating a structural similarity index measure between the m-th sub-image in the central view image and the m-th sub-image in the image adjacent to the right side of the central view image; and
  • [0034] S352: calculating the average of the two structural similarity index measures in S351, and using the average as the cleanliness score of the corresponding m-th sub-images of the images of three views.

[0035] Preferably, the structural similarity index measures in S321 and S351 are obtained by using the following formula:

[0036] $\mathrm{SSIM}\left(x,y\right)={\left[l\left(x,y\right)\right]}^{\alpha }·{\left[c\left(x,y\right)\right]}^{\beta }·{\left[s\left(x,y\right)\right]}^{\gamma };$

• • [0037] where x and y indicate the images for which structural similarity index measures are to be calculated; SSIM(x, y) indicates the structural similarity index measure between image x and image y; l(x, y) indicates the brightness comparison function value of image x and image y; c(x, y) indicates the contrast comparison function value of image x and image y; s(x,y) indicates the structure comparison function value of image x and image y; α, β and γ indicate the relative importance of brightness, contrast and structure respectively; α>0; β>0; and γ>0.

[0038] Preferably, the calculation formulas of l(x, y), c(x, y) and s(x,y) are as follows:

[0039] $l\left(x,y\right)=\frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_y^2+C_1};$ $c\left(x,y\right)=\frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_y^2+C_2};$ $s\left(x,y\right)=\frac{{\sigma }_{\mathrm{xy}}+C_3}{{\sigma }_x{\sigma }_y+C_3};$

• • [0040] where l(x, y) indicates the brightness comparison function value of image x and image y; c(x, y) indicates the contrast comparison function value of image x and image y; s(x,y) indicates the structure comparison function value of image x and image y; μ_x indicates the average value of all pixel values in image x; μ_y indicates the average value of all pixel values in image y; σ_x indicates the standard deviation of all pixel values in image x; σ_y indicates the standard deviation of all pixel values in image y; σ_xy indicates a structure-aware operator; and C₁, C₂, C₃ are constants.

[0041] Preferably, the calculation formulas of μ_x and μ_y are as follows:

[0042] ${\mu }_x=\frac{1}{N}{\sum }_{i=1}^N{x}_i;$ ${\mu }_y=\frac{1}{N}{\sum }_{i=1}^N{y}_i;$

• • [0043] where μ_x indicates the average value of all pixel values in image x; μ_y indicates the average value of all pixel values in image y; N indicates the total number of pixels in image x and the total number of pixels in image y; x_i indicates the pixel value of the i-th pixel in image x; and y_i indicates the pixel value of the i-th pixel in image y.

[0044] Preferably, the calculation formulas of σ_x and σ_y are as follows:

[0045] ${\sigma }_x={\left(\frac{1}{N-1}{\sum }_{i=1}^N{\left(x_i-{\mu }_x\right)}^2\right)}^{\frac{1}{2}};$ ${\sigma }_y={\left(\frac{1}{N-1}{\sum }_{i=1}^N{\left(y_i-{\mu }_y\right)}^2\right)}^{\frac{1}{2}};$

• • [0046] where σ_x indicates the standard deviation of all pixel values in image x; σ_y indicates the standard deviation of all pixel values in image y; μ_x indicates the average value of all pixel values in image x; μ_y indicates the average value of all pixel values in image y; N indicates the total number of pixels in image x and the total number of pixels in image y; x_i indicates the pixel value of the i-th pixel in image x; and y_i indicates the pixel value of the i-th pixel in image y.

[0047] Preferably, the calculation formula of σ_xy is as follows:

[0048] ${\sigma }_{\mathrm{xy}}=\frac{1}{N-1}{\sum }_{i=1}^N\left(x_i-{\mu }_x\right)\left(y_i-{\mu }_y\right);$

• • [0049] where σ_xy indicates the structure-aware operator; μ_x indicates the average value of all pixel values in image x; μ_y indicates the average value of all pixel values in image y; N indicates the total number of pixels in image x and the total number of pixels in image y; x_i indicates the pixel value of the i-th pixel in image x; and y_i indicates the pixel value of the i-th pixel in image y.

[0050] It can be seen from the above technical solutions that compared with the prior art, the present disclosure discloses a method for automatically determining whether to perform denoising, which can achieve the following beneficial technical effects.

[0051] 1) The present disclosure can automatically determine whether to perform denoising process, which compared with the manual judgment method, is not only efficient but also more objective.

[0052] 2) The present disclosure assists in realizing the fully automatic three-dimensional reconstruction process of the scanned sample by the light field microscopy system, unlike that in the background art in which the three-dimensional reconstruction is performed after manual judgment on whether to perform denoising.

[0055] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described examples are only some of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by a person ordinarily skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0056] As shown in the sole FIGURE, an embodiment of the present disclosure discloses a method for automatically determining whether to perform denoising, including the following steps:

• • [0057] S1: scanning a sample by using a light field microscopy system to obtain a light field microscopic image containing 4D information, where the sample is a sample to be 3D reconstructed;
  • [0058] S2: rearranging the light field microscopic image to obtain a multi-view rearranged image with an unknown noise level; and
  • [0059] S3: calculating a cleanliness score of the multi-view rearranged image; and
  • [0060] determining whether to perform a denoising process on the multi-view rearranged image based on the cleanliness score of the multi-view rearranged image.

[0061] It may be understood that images of adjacent views have small parallax, and similar structures, and the noise of the image of each view is random and independent.

[0062] It may be understood that determining whether the multi-view rearranged image needs to be subjected to the denoising process is equivalent to determining whether the light field microscopic image needs to be subjected to denoising process, because the multi-view rearranged image and the light field microscopic image include the same noises and differs only in their forms of presentation on the compute.

[0063] Further, S3 specifically includes:

• • [0064] S31: selecting a central view image, an image adjacent to the left side of the central view image, and an image adjacent to the right side of the central view image from the multi-view rearranged image by using the microlens array parameters used by the scanning light field microscopy system; and
  • [0065] S32: obtaining the cleanliness score of the multi-view rearranged image based on the central view image, the image adjacent to the left side of the central view image, and the image adjacent to the right side of the central view image.

[0066] Further, S32 specifically includes the following steps:

• • [0067] S321: calculating a structural similarity index measure between the central view image and an image adjacent to the left side of the central view image; and
  • [0068] calculating a structural similarity index measure between the central view image and an image adjacent to the right side of the central view image,
  • [0069] where further, the structural similarity index measure in S321 is obtained by the following formula:

[0070] $\mathrm{SSIM}\left(x,y\right)={\left[l\left(x,y\right)\right]}^{\alpha }·{\left[c\left(x,y\right)\right]}^{\beta }·{\left[s\left(x,y\right)\right]}^{\gamma };$ $l\left(x,y\right)=\frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_x^2+C_1};$ $c\left(x,y\right)=\frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_x^2+C_2};$ $s\left(x,y\right)=\frac{{\sigma }_{\mathrm{xy}}+C_3}{{\sigma }_x{\sigma }_y+C_3};$ ${\mu }_x=\frac{1}{N}{\sum }_{i=1}^N{x}_i;$ ${\mu }_y=\frac{1}{N}{\sum }_{i=1}^N{y}_i;$ ${\sigma }_x={\left(\frac{1}{N-1}{\sum }_{i=1}^N{\left(x_i-{\mu }_x\right)}^2\right)}^{\frac{1}{2}};$ ${\sigma }_y={\left(\frac{1}{N-1}{\sum }_{i=1}^N{\left(y_i-{\mu }_y\right)}^2\right)}^{\frac{1}{2}};$ ${\sigma }_{\mathrm{xy}}=\frac{1}{N-1}{\sum }_{i=1}^N\left(x_i-{\mu }_x\right)\left(y_i-{\mu }_y\right);$

• • [0071] where x and y indicate images for which the structural similarity index measures are to be calculated (specifically: when calculating the structural similarity index measure between the central view image and the image adjacent to the left side of the central view image, image x indicates the central view image, and image y indicates the image adjacent to the left side of the central view image; when calculating the structural similarity index measure between the central view image and the image adjacent to the right side of the central view image, image x indicates the central view image, and image y indicates the image adjacent to the right side of the central view image); SSIM(x, y) indicates the structural similarity index measure between the image x and image y; l(x, y) indicates the brightness comparison function value of the image x and image y; c(x, y) indicates the contrast comparison function value of image x and image y; s(x,y) indicates the structure comparison function value of image x and image y; α, β, and γ indicate the relative importance of brightness, contrast, and structure respectively; α>0; β>0; γ>0; μ_x indicates the average value of all pixel values in image x, which represents the brightness level of image x; μ_y indicates the average value of all pixel values in image y, which represents the brightness level of image y; σ_x indicates the standard deviation of all pixel values in image x, which represents the contrast level of image x; σ_y indicates the standard deviation of all pixel values in image y, which represents the contrast level of image y; σ_xy indicates a structure-aware operator; C₁, C₂ and C₃ are constants used to ensure stability when the denominator is 0; N indicates the total number of pixels of image x and the total number of pixels of image y (where the total number of pixels of image x is the same as the total number of pixels of image); x_i indicates the pixel value of the i-th pixel in image x; and y_i indicates the pixel value of the i-th pixel in image y; and
  • [0072] S322: calculating the average of the two structural similarity index measurement values in S321, and using the average as the cleanliness score of the multi-view rearranged image;
  • [0073] S33: comparing the cleanliness score of the multi-view rearranged image with a preset threshold:
  • [0074] if the cleanliness score of the multi-view rearranged image is greater than or equal to the preset threshold, the denoising process does not need to be performed on the multi-view rearranged image; and
  • [0075] If the cleanliness score of the multi-view rearranged image is less than the preset threshold, S34 is executed;
  • [0076] S34: dividing each of rearranged images of three views which are the central view image, the image adjacent to the left side of the central view image, and the image adjacent to the right side of the central view image, into n*n sub-images, where

[0077] $\frac{H}{200}\le n\le \frac{H}{100},$

• • [0078]  where H indicates the pixel height of the central view image, the image adjacent to the left side of the central view image, and the image adjacent to the right side of the central view image;
  • [0079] S35: calculating the cleanliness scores of corresponding sub-images of the images of three views, to obtain an image cleanliness score matrix;
  • [0080] where further, the following steps are adopted in S35 to calculate the cleanliness scores of corresponding m-th sub-images of the images of three views:
  • [0081] S351: calculating a structural similarity index measure between the m-th sub-image in the central view image and the m-th sub-image in image adjacent to the left side of the central view image, in the above, 1<=m<=n²;
  • [0082] calculating a structural similarity index measure between the m-th sub-image in the central view image and the m-th sub-image in image adjacent to the right side of the central view image;
  • [0083] where further, the structural similarity index measure in S351 is obtained by using the following formula:

[0084] $\mathrm{SSIM}\left(x,y\right)={\left[l\left(x,y\right)\right]}^{\alpha }·{\left[c\left(x,y\right)\right]}^{\beta }·{\left[s\left(x,y\right)\right]}^{\gamma };$ $l\left(x,y\right)=\frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_x^2+C_1};$ $c\left(x,y\right)=\frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_x^2+C_2};$ $s\left(x,y\right)=\frac{{\sigma }_{\mathrm{xy}}+C_3}{{\sigma }_x{\sigma }_y+C_3};$ ${\mu }_x=\frac{1}{N}{\sum }_{i=1}^N{x}_i;$ ${\mu }_y=\frac{1}{N}{\sum }_{i=1}^N{y}_i;$ ${\sigma }_x={\left(\frac{1}{N-1}{\sum }_{i=1}^N{\left(x_i-{\mu }_x\right)}^2\right)}^{\frac{1}{2}};$ ${\sigma }_y={\left(\frac{1}{N-1}{\sum }_{i=1}^N{\left(y_i-{\mu }_y\right)}^2\right)}^{\frac{1}{2}};$ ${\sigma }_{\mathrm{xy}}=\frac{1}{N-1}{\sum }_{i=1}^N\left(x_i-{\mu }_x\right)\left(y_i-{\mu }_y\right);$

• • [0085] where x and y indicate images for which the structural similarity index measures are to be calculated (specifically: when calculating the structural similarity index measure between the m-th sub-image in the central view image and the m-th sub-image in the image adjacent to the left side of the central view image, image x indicates the m-th sub-image in the central view image, and image y indicates the m-th sub-image in the image adjacent to the left side of the central view image; when calculating the structural similarity index measure between the m-th sub-image in the central view image and the m-th sub-image in the image adjacent to the right side of the central view image, image x indicates the m-th sub-image in the central view image, and image y indicates the m-th sub-image in the image adjacent to the right side of the central view image); SSIM(x,y) indicates the structural similarity index measure between image x and image y; l(x, y) indicates the brightness comparison function value of image x and image y; c(x, y) indicates the contrast comparison function value of image x and image y; s(x,y) indicates the structure comparison function value of image x and image y; α, β, and γ indicate the relative importance of brightness, contrast, and structure respectively; α>0; β>0; γ>0; μ_x indicates the average value of all pixel values in image x, which represents the brightness level of image x; μ_y indicates the average value of all pixel values in image y, which represents the brightness level of image y; σ_x indicates the standard deviation of all pixel values in image x, which represents the contrast level of image x; σ_y indicates the standard deviation of all pixel values in image y, which represents the contrast level of image y; σ_xy indicates a structure-aware operator; C₁, C₂ and C₃ are constants used to ensure stability when the denominator is 0; N indicates the total number of pixels of image x and the total number of pixels of image y (where the total number of pixels of image x is the same as the total number of pixels of image); x_i indicates the pixel value of the i-th pixel in image x; and y_i indicates the pixel value of the i-th pixel in image y; and
  • [0086] S352: calculating the average of the two structural similarity index measurement values in S351, and using the average as the cleanliness score of corresponding m-th sub-images of the images of three views.

[0087] It may be understood that as the parallax between adjacent views increases with the increase of sample thickness, the structural similarity then decreases. In order to prevent the influence of excessive parallax caused by excessive sample thickness, the present disclosure performs a secondary judgment when the cleanliness score of the multi-view rearranged image is less than the preset threshold.

• • [0088] S36: comparing the maximum value in the image cleanliness score matrix with the preset threshold:
  • [0089] if the maximum value in the image cleanliness score matrix is greater than or equal to the preset threshold, the denoising process does not need to be performed on the multi-view rearranged image; and
  • [0090] if the maximum value in the image cleanliness score matrix is less than the preset threshold, the denoising process needs to be performed on the multi-view rearranged image.

[0091] It may be understood that: when the method of the present disclosure is used for three-dimensional reconstruction of a sample, the multi-view rearranged image is directly subjected to three-dimensionally reconstructing when it is automatically determined that the multi-view rearranged image does not need to be subjected to the denoising process; and when it is automatically determined that the multi-view rearranged image needs to be subjected to the denoising process, the multi-view rearranged image is input into the denoising network for denoising and then subjected to three-dimensional reconstruction. Obviously, the present disclosure realizes the fully automatic three-dimensional reconstruction process of the scanned sample by the light field microscopy system, unlike that in the background art in which the three-dimensional reconstruction is performed after manual judgment on whether to perform denoising.

[0092] In this specification, the embodiments are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments may be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method parts.

[0093] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments shown herein, but rather will conform to the widest scope consistent with the principles and novel features disclosed herein.