Image pickup apparatus and image pickup method using plural photoelectric conversion sections per pixel

[0001] 1. Field of the Invention

[0002] The present invention relates to an image pickup apparatus and image pickup method for picking up an object using a solid-state image pickup element.

[0003] 2. Related Background Art

[0004] There is an image pickup apparatus such as a digital camera for electronically or magnetically recording and reproducing on various mediums still images or moving images picked up by a solid-state image pickup element such as a CCD or MOS sensor.

[0005] In order to reduce price or improve performance of such an image pickup apparatus, there is an image pickup element (refer to FIG. 8) as described in U.S. Pat. No. 5,965,875 where it is noted that a light absorption coefficient of a semiconductor substrate used in an image pickup element is different according to a light wavelength.

[0006] In this patent, there are described a principle of an image pickup element having a three-layer photodiode structure where photodiodes are formed with a triple-well structure, and a pixel circuit. According to the patent, the photodiodes are sequentially diffused from a surface of a p-type silicon substrate, and an n-type layer, a p-type layer, and an n-type layer are deeply formed in this order, so that pn-junction diodes are formed over three layers in the depth direction of the silicon in the same pixel. As a light incident into the diode from the surface has a longer wavelength, it penetrates more deeply. Since the incident light wavelength and the light absorption coefficient indicate a value specific to the silicon, a current is separately detected from the three-layer diodes, so that an optical signal having a different wavelength band can be detected.

[0007] The depth of the pn-junction of the three-layer diode is designed so as to cover the wavelength band of a visible light. Three signals are calculated so that signals where an object image is color-separated into three colors of R color, G color, and B color can be obtained.

[0008] By the use of such an image pickup element, it is possible to eliminate a color filter for performing color separation of the object image or an optical low-pass filter for preventing moiré due to spatial sampling. Further, in the respective colors obtained by color-separating the object image, centers of gravity of sensitivity match with each other, so that color moiré is advantageously difficult to be produced.

[0009] However, in the image pickup element for performing color separation in the depth direction of a semiconductor, when a noise is present, whose occurrence position is unstable in the depth direction of the semiconductor like a dark current, a different noise occurs for each color even in the same pixel, thereby deteriorating the image.

[0010] Further, in the image pickup element as described above, among the three photodiodes corresponding to the respective colors in the same pixel, two photodiodes adjacent in the vertical direction are capacitive-coupled to each other via the pn-junction. Further, as charges generated in photoelectric conversion are accumulated in the photodiode, the capacity of the photodiode changes. Therefore, a potential of a photodiode in one layer is influenced even by the amount of charges accumulated in other photodiodes. Further, when the photodiode in the uppermost layer is saturated, the excess charges in this diode are flowed into the n-type area of the photodiode in the lowermost layer beyond a potential barrier made of a second p-type layer from the top. Therefore, when there is a white defect or the like due to a local crystal defect of a semiconductor, even a color signal at the depth where there is no defect is influenced, thereby deteriorating the image.

[0011] The present invention is directed for solving the above problems, and it is therefore an object to provide an image pickup apparatus and image pickup method capable of obtaining a good object image even when a noise such as a dark current different for each color occurs.

[0012] As one aspect of the present invention for achieving the above object, the following structure is provided.

[0013] There is provided an image pickup apparatus comprising an image pickup area where a plurality of pixels each having at least first and second photoelectric conversion sections are arranged in a depth direction, and a first correction section for performing differential processing between a signal obtained in the first photoelectric conversion section in case that the image pickup area is light-shielded, and a signal obtained in the first photoelectric conversion section by picking up an object image in the image pickup area, and differential processing between a signal obtained in the second photoelectric conversion section in case that the image pickup area is light-shielded and a signal obtained in the second photoelectric conversion section by picking up an object image in the image pickup area.

[0014] It is another object of the present invention to provide an image pickup apparatus and image pickup method capable of obtaining a good object image even when there is a pixel defect which also influences a color signal at the depth where there is no defect.

[0015] As one aspect of the present invention for achieving the above object, the following structure is provided.

[0016] There is provided an image pickup apparatus comprising an image pickup area where a plurality of pixels each having a plurality of photoelectric conversion sections are arranged in a depth direction, and a second correction unit for correcting each of signals from the plurality of photoelectric conversion sections arranged in the depth direction in the same pixel to correct a defective pixel.

[0017] It is still another object of the present invention to provide an image pickup apparatus and image pickup method capable of obtaining a good object image without holding data for defect correction in advance even when there is a pixel defect which also influences a color signal at the depth where there is no defect.

[0018] As one aspect of the present invention for achieving the above object, the following structure is provided.

[0019] There is provided an image pickup apparatus comprising an image pickup area where a plurality of pixels each having at least first and second photoelectric conversion sections are arranged in a depth direction, a first correction unit for performing differential processing between a signal obtained in the first photoelectric conversion section in case that the image pickup area is light-shielded and a signal obtained in the first photoelectric conversion section by picking up an object image in the image pickup area, and differential processing between a signal obtained in the second photoelectric conversion section in case that the image pickup area is light-shielded and a signal obtained in the second photoelectric conversion section by picking up an object image in the image pickup area, a second correction unit provided in the rear stage of the first correction unit for correcting each of signals from the first and second photoelectric conversion sections arranged in the depth direction in the same pixel to correct a defective pixel, and a detection unit for detecting a defective pixel in the image pickup area, wherein, when it is detected that a signal from one photoelectric conversion section in the same pixel requires correction, the detection unit controls the second correction unit so that signals from other photoelectric conversion sections in the same pixel are also corrected, and detects a defective pixel on the basis of a signal before being input into the first correction unit.

[0020] The aforementioned objects of the present invention will be apparent from the following drawings and detailed description on the basis of the drawings.

[0029] Hereinafter, embodiments according to the present invention will be described referring to the drawings.

[0030] First embodiment

[0031] FIG. 1 shows a configuration of an image pickup apparatus (digital camera) according to a first embodiment of the present invention.

[0032] In FIG. 1, numeral 1 denotes an image pickup optical system, numeral 2 denotes a shutter, numeral 3 denotes an image pickup element, numeral 4 denotes an amplifying circuit, numeral 5 denotes an A/D converter, numeral 6 denotes a signal pre-processing unit, numeral 7 denotes a color separation switch, numeral 8 denotes a changeover switch, numeral 9 denotes an image memory, numeral 10 denotes a subtractor, numeral 11 denotes a defect data memory, numeral 12 denotes a defect correction processing unit, numeral 13 denotes a signal post-processing unit, numeral 14 denotes a recording unit, and numeral 15 denotes a display unit.

[0033] Correction processing according to the first embodiment of the present invention will be described using FIG. 1 and a flow chart of FIG. 2.

[0034] The image pickup element 3 is an image pickup element having an image pickup area in which a plurality of pixels each having photodiodes which are a plurality of photoelectric conversion sections are arranged in the depth direction of a semiconductor as shown in FIG. 8. Defect data (position of defective pixel) of the image pickup element 3 detected in advance is stored in the defect data memory 11. This defect data is obtained by detecting a position of a pixel which exceeds a certain threshold value, from an image picked up, for example, in the light-shielded state.

[0035] In the present embodiment, at first, an object is picked up (S101). The object image formed by the image pickup optical system 1 is converted into an electric signal by the image pickup element 3 via the shutter 2 controlled and driven so as to have the appropriate amount of exposure. The object image is amplified or level-shifted so as to be appropriate for an input range of the A/D converter 5 in the amplifying circuit 4. An analog signal is digitized in the A/D converter 5. A black level adjustment such as clamp is performed on the digitized image data by the signal pre-processing unit 6. At this time, three changeover switches 8 are connected to the image memories 9, respectively.

[0036] The color separation switch 7 is driven so as to store the image signals output from the image pickup element 3 into the image memories corresponding to the respective colors. In other words, a signal from a photodiode at the deepest position in the depth direction is stored in the image memory R, a signal from a photodiode at the second deepest position in the depth direction is stored in the image memory G, and a signal from a photodiode at the shallowest position in the depth direction is stored in the image memory B. Thereby, the image data of the color-separated object image is stored in the image memories 9 (S102).

[0037] Thereafter, correction processing using the subtractors 10 is performed (S103). Here, at first, image pickup of the same accumulation time as that of image pickup of the object is performed under a condition that the shutter 2 is closed, that is, the image pickup element 3 is light-shielded. At this time, the three changeover switches 8 are connected to the direction opposite to the image memories 9, that is, the subtractors 10. Further, the image memories 9 output the image data of the object image at the same position as that of the light-shielded image data to the subtractors 10. Thereby, the image data of the object image whose unevenness or slight defect resulting from a dark current of the image pickup element has been corrected is output from the subtractors 10. Further, the correction is independently performed for each color separated in the image pickup element 3. In other words, a dark current signal from the photodiode at the deepest position in the depth direction is subjected to differential processing relative to the signal stored in the image memory R, a dark current signal from the photodiode at the second deepest position in the depth direction is subjected to differential processing relative to the signal stored in the image memory G, and a dark current signal from the photodiode at the shallowest position in the depth direction is subjected to differential processing relative to the signal stored in the image memory B.

[0038] As described above, the image pickup apparatus according to the present embodiment has said image pickup area (image pickup element 3) where a plurality of pixels each having at least first, second, and third photoelectric conversion sections (corresponding to BLUE PHOTON COLLECTION, GREEN PHOTON COLLECTION, and RED PHOTON COLLECTION, respectively, in FIG. 8) are arranged in the depth direction of a semiconductor, and a first correction unit consisting of the image memories 9 and the subtractors 10 for performing differential processing between a signal obtained in the first photoelectric conversion section in case that said image pickup area is light-shielded and a signal obtained in the first photoelectric conversion section by picking up an object image in said image pickup area, differential processing between a signal obtained in the second photoelectric conversion section in case that said image pickup area is light-shielded and a signal obtained in the second photoelectric conversion section by picking up an object image in said image pickup area, and differential processing between a signal obtained in the third photoelectric conversion section in case that said image pickup area is light-shielded and a signal obtained in the third photoelectric conversion section by picking up an object image in said image pickup area, so that correction can be performed even when different noises are present for the respective colors.

[0039] In the defect correction processing unit 12, defect correction processing is performed on the basis of the defect data stored in the defect data memory 11 in advance (S104). The defect-corrected image data is subjected to color processing and the like in the signal post-processing unit 13, and then transmitted to the recording unit 14 or the display unit 15 so as to be recorded or displayed therein.

[0040] The defect correction processing in step S104 will be described with reference to FIGS. 3, 4, and 5. FIG. 3 shows a vicinity of m-th column and n-th row of the R color of the image signal of the object image color-separated and stored in the image memory. In this example, the R color pixel at m-th column and n-th row is detected as a defect in advance and stored in the defect data memory 11. In the defect correction processing unit 12, position information of the defect is read by the defect data memory 11, and the image data of the defect position is replaced and interpolated by the image data of the same color as the neighboring pixels. In the case of FIG. 3, since the R color pixel at m-th column and n-th row is defective, for example, the image data Ra of the R color pixel at m−1-th column and n-th row and the image data Rc of the R color pixel at m+1-th column and n-th row are used, so that the average value therebetween is assumed as the image data of the R color pixel at m-th column and n-th row.

[0041] FIGS. 4 and 5 show vicinities of m-th column and n-th row of the image signals of the object image of the G color and the B color, respectively. The pixels Gb and Bb at m-th column and n-th row in FIGS. 4 and 5 have not been detected as defects in advance. However, in the present invention, when other colors in the same pixel are defective as in FIG. 3, defect correction is similarly performed on the image data of all the colors. Therefore, the image data Gb of the G color is replaced with the average value between Ga and Gc. Similarly, the image data Bb of the B color is replaced with the average value between Ba and Bc. In this manner, when the image signal of any one of colors of the same pixel has been detected as a defect in advance, similar defect correction processing is performed on the image signals of all the colors, so that image deterioration resulting from influence in terms of the structure on a color signal at the depth where there is no defect can be prevented.

[0042] As described above, the image pickup apparatus according to the present embodiment has said image pickup area where a plurality of pixels each having a plurality of photoelectric conversion sections are arranged in the depth direction of a semiconductor, and a second correction unit consisting of the defect data memory 11 and the defect correction processing unit 12 for correcting each of signals from the plurality of photoelectric conversion sections in the depth direction in the same pixel to correct defective pixel, so that defect correction can be accurately performed.

[0043] In the embodiment described above, the front stage of the image memory 9 is configured so that signals are transmitted with time-series, but the amplifying circuit 4, the A/D converter 5, and the signal pre-processing unit 6 may be provided in parallel so that the signals are output from the image pickup element to the image memory R, the image memory G, and the image memory B in parallel. In this case, the signals from the three photoelectric conversion sections of the same pixel arranged in the depth direction of a semiconductor are read from the image pickup element in parallel.

[0044] Second embodiment

[0045] FIG. 6 shows a configuration of an image pickup apparatus according to a second embodiment of the present invention. In FIG. 6, same components as those in FIG. 1 are denoted with the same reference numerals, and description thereof will be omitted. Numeral 16 denotes a defect detection circuit, and numeral 17 denotes a defect data mixing circuit.

[0046] Correction processing according to the second embodiment of the present invention will be described with reference to FIG. 6 and a flow chart of FIG. 7.

[0047] The image pickup element 3 is an image pickup element having a three-layer photodiode structure described as a conventional example. The defect detection circuit 16 detects a defect resulting from a dark current, from image data picked up in case that the shutter 2 is closed, that is, the image pickup element 3 is light-shielded. The method therefor is to determine as a defect a pixel whose ratio or difference between the image data value of a detection target pixel and the average value of the image data of the neighboring pixels is not less than a predetermined value, and to output a result as defect data.

[0048] Even when any one of colors of the same pixel is defective, the defect data detected for each color is combined into the data of defect pixel position, by the defect data mixing circuit 17.

[0049] In the present embodiment, at first, an object is picked up (S201). The object image formed in the image pickup optical system 1 is converted into an electric signal in the image pickup element 3 via the shutter 2 controlled and driven so as to have the appropriate amount of exposure. The object image is amplified or level-shifted so as to be appropriate for an input range of the A/D converter 5 in the amplifying circuit 4. An analog signal is digitized in the A/D converter 5. A black level adjustment such as clamp is performed for the digitized image data in the signal pre-processing unit 6. At this time, the three changeover switches 8 are connected to the image memories 9, respectively.

[0050] The color separation switch 7 is driven so as to store the image signals output from the image pickup element 3 into the image memories corresponding to the respective colors. Thereby, the image data of the color-separated object image is stored in the image memories 9 (S202).

[0051] Thereafter, correction processing using the subtractors 10 is performed (S203). Here, at first, image pickup of the same accumulation time as that of image pickup of the object is performed under a condition that the shutter 2 is closed, that is, the image pickup element 3 is light-shielded. At this time, the three changeover switches 8 are connected to the direction opposite to the image memories 9, that is, the subtractors 10. At the same time, the light-shielded imaged data is input into the defect detection circuits 16 for each color.

[0052] The defect detection circuit 16 performs the aforementioned defect detection to output the defect data for each color (S204). Further, the image memories 9 output to the subtractors 10 the image data of the object image at the same position as that of the light-shielded image data. Thereby, the image data of the object image whose unevenness or slight defect resulting from a dark current of the image pickup element has been corrected is output from the subtractors 10. Further, the correction is independently performed for each color separated in the image pickup element 3, so that correction can be performed even when different noises are present for the respective colors.

[0053] The defect correction processing unit 12 performs defect correction processing on the basis of the defect data after the defect data output from the three defect detection circuits 16 is combined in the defect data mixing circuit 17 (S205). The defect-corrected image data is subjected to color processing and the like in the signal post-processing unit 13, and then transmitted to the recording unit 14 or the display unit 15 so as to be recorded or displayed therein. Since, after the defect data of all the colors is combined in the defect data mixing circuit 17, correction is performed by the defect correction processing units 12, image deterioration resulting from influence in terms of the structure on a color signal at the depth where there is no defect can be prevented. Furthermore, defect correction can be performed without preparing the defect data in advance.

[0054] As described above, the image pickup apparatus according to the present embodiment has said image pickup area where a plurality of pixels each having a plurality of photoelectric conversion sections are arranged in the depth direction of a semiconductor, and a second correction unit consisting of the defect detection circuits 16, the defect data mixing circuit 17, and the defect correction processing units 12 for performing correction of each signal from the plurality of photoelectric conversion sections in the depth direction in the same pixel to correct a defective pixel, so that defect correction can be accurately performed.

[0055] In the aforementioned embodiment, the front stage of the image memory 9 is configured so that signals are transmitted with time-series, but the amplifying circuit 4, the A/D converter 5, and the signal pre-processing unit 6 may be provided in parallel so that the signals are output from the image pickup element to the image memory R, the image memory G, and the image memory B in parallel. In this case, the signals from the three photoelectric conversion sections of the same pixel, arranged in the depth direction of a semiconductor are read from the image pickup element in parallel. Furthermore, the defect data detection is performed after the object image pickup, but may be performed, for example, after the image pickup apparatus is turned on, alternatively, before or after the object is picked up, or when there is time for control of the image pickup apparatus.

[0056] In addition, in the first and second embodiments described above, the number of colors of the image pickup element to be separated in the depth direction of a semiconductor is three of R, G, and B, but may be two or more. In other words, the image pickup element is configured to have at least first and second photoelectric conversion sections arranged in the depth direction of a semiconductor.

[0057] Further, in the above first and second embodiments, there is shown the circuit having both a circuit for correcting a dark current (first correction unit) and a circuit for correcting a defective pixel (second correction unit), but the circuit may be configured to have either one circuit.

[0058] Other embodiments

[0059] In the above embodiments, there is described an example where the present invention is applied to a digital camera, but the present invention is not limited thereto, and can be applied to other apparatuses such as digital video camera, as long as the image pickup element used in the above embodiments is available.

[0060] Further, in the above embodiments, the first correction unit and the second correction unit for performing processings according to the present invention are configured with circuits, but the present invention is not limited thereto, and all or part of these units may be configured on the memory by software.

[0061] In other words, it goes without saying that the objects of the present invention can be accomplished by supplying a storage medium storing a program code of a software for realizing the functions of the above embodiments therein to a system or an apparatus so that a computer (or CPU or MPU) in the system or the apparatus reads the program code stored in the storage medium.

[0062] In this case, the program code itself read from the storage medium realizes the functions of the above embodiments, so that the storage medium storing the program code therein configures the present invention.

[0063] As a storage medium for supplying a program code, there may be employed, for example, a flexible disk, a hard disk, an optical disk, a magneto optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, and the like.

[0064] Further, it goes without saying not only that the program code read by the computer is executed so that the functions of the above embodiments can be realized but also that an OS (operating system) running on the computer performs part of or all the actual processings, so that the functions of the above embodiments can be realized.

[0065] Furthermore, it goes without saying that, after the program code read from the storage medium is written into a memory provided in a function extension board inserted into the computer or a function extension unit connected to the computer, the CPU or the like provided in the function extension board or the function extension unit performs part of or all the actual processings on the basis of an instruction of the program code, so that the functions of the above embodiments can be realized.

[0066] Hereinbefore, the present invention is described by the preferred embodiments, but the present invention is not limited to the above embodiments, and can be variously modified within the range described in the claims.