Signal readout method of solid-state imaging device and image signal processing method

[0001] The present invention relates to a signal readout method of solid-state imaging device which simultaneously readouts image signals for generating multiple images from one solid-state imaging device, and also relates to an image signal processing method for generating an image having high added value based on multiple images that are generated based on the image signals readout by the signal readout method of solid-state imaging device.

[0002] In recent years, the number of the pixels of a solid-state imaging device such as a CCD image sensor is heightened. As a method which readouts signals from such a high-pixel solid-state imaging device at a high speed, a pixel mixture readout method or a down sampling readout method is utilized conventionally.

[0003] As one example of the conventional pixel mixture readout method, for example, a pixel mixture readout method that mixes nine pixels and readouts signals is disclosed in a color solid-state imaging device according to Patent Document 1. The conventional pixel mixture readout method as disclosed in Patent Document 1, performs pixel mixture of the equal number of pixels and readouts signals after pixel mixture.

[0004] In other words, in all readout signals that are readout by the conventional pixel mixture readout method, pixel mixture of the equal number of pixels is performed.

[0005] However, since the signals that are readout by the conventional pixel mixture readout method are the signals obtained by performing pixel mixture of the same number of pixels, the properties and features of these signals are equal. That is to say, there is a problem that the signals readout by the conventional pixel mixture readout method can generate only one kind of image.

[0006] Therefore, it is impossible to generate an image having high added value (for example, the image such as a wide dynamic range image or a super-resolution image obtained by a super-resolution processing) based on the signals readout by the conventional pixel mixture readout method.

[0007] The present invention has been developed in view of the above described circumstances, and an object of the present invention is to provide a signal readout method of solid-state imaging device which can simultaneously readouts the signals having different properties that can generate multiple images obtained by capturing the same subject under different capturing conditions from one solid-state imaging device, and also provide an image signal processing method for generating an image having high added value based on the multiple images that are generated based on the signals having different properties readout by the signal readout method of solid-state imaging device.

[0008] The present invention relates to a signal readout method of solid-state imaging device. The above object of the present invention is effectively achieved by the construction that a signal readout method of solid-state imaging device that is applied to a solid-state imaging device with a color filter array (CFA) and multiple pixels, said method characterized in that a pixel mixture of a different number of pixels is performed for every readout signal in said solid-state imaging device, and a signal after said pixel mixture is readout. The above object of the present invention is also effectively achieved by the construction that said signal after said pixel mixture means a first signal after said pixel mixture and a second signal after said pixel mixture, said first signal after said pixel mixture is obtained by performing said pixel mixture of predetermined L pixels (L≦n×m) within (n×m) pixels (n and m are natural numbers) consisting of n pixels in a vertical direction and m pixels in a horizontal direction of said solid-state imaging device, said second signal after said pixel mixture is obtained by down sampling pixel signals of said (n×m) pixels without performing said pixel mixture of said (n×m) pixels.

[0009] Further, the present invention also relates to an image signal processing method. The above object of the present invention is effectively achieved by the construction that an image signal processing method characterized in that a wide dynamic range image is generated based on said first signal after said pixel mixture and said second signal after said pixel mixture that are readout by the signal readout method of solid-state imaging device according to the present invention described above. The above object of the present invention is also effectively achieved by the construction that an image signal processing method characterized in that a registration processing of images is performed based on said first signal after said pixel mixture that is readout by the signal readout method of solid-state imaging device according to the present invention described above, and a super-resolution image is generated by performing a super-resolution processing based on displacement information obtained by said registration processing and said second signal after said pixel mixture that is readout by the signal readout method of solid-state imaging device according to the present invention described above.

[0015] The following is a description of preferred embodiments for carrying out the present invention, with reference to the accompanying drawings.

[0016] The greatest characteristic of the present invention is that the present invention can simultaneously readouts the signals obtained by capturing the same subject under different capturing conditions from one solid-state imaging device by changing the number of pixels to be mixed for every readout signal in one solid-state imaging device, in other words, by performing pixel mixture of the different number of pixels for every readout signal in one solid-state imaging device.

[0017] In a word, as the greatest feature of the present invention is that the pixel mixture technique is applied to one solid-state imaging device with a color filter array (CFA) and multiple pixels, and the signals having different properties are readout. This greatest feature of the present invention realizes that the signals having different properties are readout by performing the pixel mixture of the different number of pixels for each readout signal in one solid-state imaging device.

[0018] In the present invention, it is possible to simultaneously generate (obtain) multiple images obtained by capturing the same subject (the same scene) under different capturing conditions based on the signals having different properties obtained by performing the pixel mixture of the different number of pixels. Further, for example, in the present invention, it is possible to generate a wide dynamic range image by using the generated multiple images having the different capturing conditions, and is applied to the super-resolution processing by utilizing the difference of the property of the spatial direction and generate a super-resolution image by the super-resolution processing. According to the present invention, for example, it is possible to generate an image having high added value such as a wide dynamic range image and a super-resolution image (hereinafter, called simply as a high-added-value image).

[0019] The signal readout method of solid-state imaging device according to the present invention is applied to a solid-state imaging device with a color filter array (CFA) and multiple pixels. In the signal readout method of solid-state imaging device according to the present invention, a pixel mixture of a different number of pixels is performed for every readout signal in the solid-state imaging device and a signal after the pixel mixture is readout.

[0020] In the signal readout method of solid-state imaging device according to the present invention, for example, the signal after the pixel mixture means a first signal after the pixel mixture and a second signal after the pixel mixture, the first signal after the pixel mixture is obtained by performing the pixel mixture of predetermined L pixels (L≦n×m) within (n×m) pixels (n and m are natural numbers) consisting of n pixels in a vertical direction and m pixels in a horizontal direction of the solid-state imaging device, the second signal after the pixel mixture is obtained by down sampling pixel signals of the (n×m) pixels without performing the pixel mixture of the (n×m) pixels.

[0021] Next, in the image signal processing method according to the present invention, a wide dynamic range image is generated based on the first signal after the pixel mixture and the second signal after the pixel mixture that are readout by the signal readout method of solid-state imaging device according to the present invention described above.

[0022] Further, in the image signal processing method according to the present invention, a registration processing of images is performed based on the first signal after the pixel mixture that is readout by the signal readout method of solid-state imaging device according to the present invention, and a super-resolution image is generated by performing a super-resolution processing based on displacement information obtained by the registration processing and the second signal after the pixel mixture that is readout by the signal readout method of solid-state imaging device according to the present invention.

[0023] The present invention is described in more detail below through the following specific embodiment.

[0024] FIG. 1 is an explanatory diagram illustrating readout positions of an eight-pixel mixed readout and a one-pixel mixed readout in the signal readout method of solid-state imaging device according to the present invention. Since the one-pixel mixed readout can be regarded as a down sampling readout, hereinafter, the one-pixel mixed readout is simply called as the down sampling readout.

[0025] In a preferred embodiment of the signal readout method of solid-state imaging device according to the present invention shown in FIG. 1, the solid-state imaging device has a CFA with Bayer arrangement, an eight-pixel mixed readout signal is obtained by performing the pixel mixture of predetermined eight pixels (in this embodiment, the eight pixels having the same channel of color space) within twenty-five pixels consisting of five pixels in a vertical direction and five pixels in a horizontal direction of the solid-state imaging device, and a down sampling readout signal is obtained by down sampling pixel signals of the twenty-five pixels without performing the pixel mixture of the twenty-five pixels.

[0026] In other words, in the embodiment of FIG. 1, the signal readout method of solid-state imaging device that combines the eight-pixel mixed readout and the down sampling readout, is carried out. In addition, in FIG. 1, in order to avoid the complication of the drawing, only an R channel and a B channel of RGB color space are illustrated.

[0027] Here, an image generated by the eight-pixel mixed readout of FIG. 1 is called as an eight-pixel mixed image, and an image generated by the down sampling readout of FIG. 1 is called as a down sampling readout image.

[0028] Therefore, in the eight-pixel mixed image, since the pixel values of the eight pixels are mixed, the eight-pixel mixed image has sensitivity which is eight times as high as that of the down sampling readout image.

[0029] In the image signal processing method according to the present invention, one wide dynamic range image is generated based on the multiple images captured (obtained) under different sensitivity conditions, namely, based on the eight-pixel mixed image and the down sampling readout image that are obtained by the signal readout method of solid-state imaging device according to the present invention.

[0030] As a method for generating the wide dynamic range image based on the multiple images captured under different sensitivity conditions, for example, a method disclosed in Non-Patent Document 1 can be utilized. Also an image converting method for effectively displaying the wide dynamic range image is disclosed in Non-Patent Document 1.

[0031] FIG. 2 is a diagram illustrating the down sampling readout image and the eight-pixel mixed image described schematically in FIG. 1, and a wide dynamic range image generated based on the down sampling readout image and the eight-pixel mixed image by using the image signal processing method according to the present invention. In FIG. 2, the method disclosed in Non-Patent Document 1 is used for the generation processing of the wide dynamic range image and the image conversion for effectively displaying the generated wide dynamic range image.

[0032] FIG. 3 is a block diagram illustrating a processing flow for generating the wide dynamic range image shown in FIG. 2 in the image signal processing method according to the present invention. As shown in FIG. 3, first, the eight-pixel mixed image and the down sampling readout image are generated by applying the signal readout method of solid-state imaging device according to the present invention to the solid-state imaging device with a CFA and multiple pixels, and then the wide dynamic range image is generated by for example using the method disclosed in Non-Patent Document 1 based on the generated eight-pixel mixed image and the down sampling readout image.

[0033] Applying the multiple images obtained by the signal readout method of solid-state imaging device according to the present invention to the super-resolution processing is described below through a specific embodiment.

[0034] The super-resolution processing is known as a technique for generating one high-resolution image based on multiple low-resolution images. The super-resolution processing is performed based on displacement information between low-resolution images. Further, the displacement information between low-resolution images is estimated by registration processing. In the case where the displacement information is accurately obtained, a high-resolution image with higher-definition is generated so as to the influence of aliasing included in the low-resolution images is larger.

[0035] However, in the case where the low-resolution images include the large influence of aliasing, an error is included in the displacement information estimated by the registration processing. There is a problem that the high-resolution image with high definition is not generated due to the error of the displacement information.

[0036] Since a high-resolution image with high-definition can be generated by performing the super-resolution processing based on the multiple images obtained by the signal readout method of solid-state imaging device according to the present invention, therefore the above problem can be solved.

[0037] As a specific embodiment, as shown in FIG. 1, the eight-pixel mixed readout and the down sampling readout are combined to be used. Since the pixel mixture is a processing corresponding to spatial averaging processing, the eight-pixel mixed image has less influence of aliasing. On the other hand, the down sampling readout image includes a lot of influences of aliasing. In the present invention, the features of the eight-pixel mixed image and the down sampling readout image are utilized so that the high-resolution image with high definition is generated by the super-resolution processing.

[0038] That is to say, in the image signal processing method according to the present invention, first, the displacement information is estimated by using the eight-pixel mixed image obtained by the signal readout method of solid-state imaging device according to the present invention, and then the high-resolution image with high definition (hereinafter, simply the super-resolution image) is generated by performing the super-resolution processing that utilizes the down sampling readout image obtained by the signal readout method of solid-state imaging device according to the present invention and based on the estimated displacement information.

[0039] The generation processing of the super-resolution image by using the image signal processing method according to the present invention, is specifically described with reference to the block diagram of FIG. 4.

[0040] As shown in FIG. 4, first, the multiple eight-pixel mixed images and the multiple down sampling readout images are readout continuously by applying the signal readout method of solid-state imaging device according to the present invention to the solid-state imaging device with a CFA and multiple pixels, and then the multiple eight-pixel mixed images and the multiple down sampling readout images that are readout, are saved in a memory.

[0041] Next, the displacement amount (the displacement information) between the multiple eight-pixel mixed images saved in the memory, is estimated by the registration processing. Then, the super-resolution image is obtained by performing the super-resolution processing based on the displacement information obtained by the registration processing and the multiple down sampling readout images saved in the memory.

[0042] As to details of the registration processing and the super-resolution processing, see Non-Patent Document 2, for example.

[0043] FIG. 5 is a diagram illustrating one down sampling readout image and one eight-pixel mixed image described schematically in FIG. 1, and the super-resolution processed image generated based on the down sampling readout image and the eight-pixel mixed image along the procedure of the block diagram of FIG. 4. In FIG. 5, the eight-pixel mixed image has sensitivity which is eight times as high as that of the down sampling readout image, but its gain is adjusted so that its sensitivity becomes equal with the sensitivity of the down sampling readout image.

[0044] It can be confirmed from FIG. 5 that the super-resolution image generated by the image signal processing method according to the present invention has higher definition than those of the down sampling readout image and the eight-pixel mixed image.

[0045] As described above, according to the signal readout method of solid-state imaging device of the present invention, since the signals having different properties can simultaneously readout from one solid-state imaging device, so the present invention plays an excellent effect such that the multiple images obtained by capturing the same subject under different capturing conditions can be generated.

[0046] Further, according to the image signal processing method of the present invention, the present invention plays an excellent effect such that the image having high added value such as a wide dynamic range image and a super-resolution image obtained by the super-resolution processing can be generated based on the multiple images having the different capturing conditions that are obtained by the signal readout method of solid-state imaging device of the present invention.

• [0047] Patent Document 1:
• [0048] Japanese Patent Publication No. 2005-109968
• [0049] Non-Patent Document 1:
• [0050] “Recovering High Dynamic Range Radiance Maps From Photographs” written by P. E. Debevec and J. Malik, Proc. ACM SIGGRAPH' 97, p. 369-378
• [0051] Non-Patent Document 2:
• [0052] “Image Mosaicing and Super-Resolution” written by David Capel, Springer, 2004