Image pickup apparatus

[0002] 1. Field of the Invention

[0003] The present invention relates to an image pickup apparatus having an image sensor which is provided with color filters, and more particularly to an image pickup apparatus having an improved signal processing for white-balance adjustment.

[0004] 2. Description of Related Art

[0005] FIG. 10 shows in a block diagram the signal processing system of a conventional single-plate type image pickup apparatus. In this case, the image pickup apparatus is assumed to be provided with complementary color filters having such an arrangement as shown in FIG. 2.

[0006] Referring to FIG. 10, an analog image signal outputted from an image sensor 401 is supplied through a preprocessing circuit 402 to an A/D converting circuit 403, where the analog signal is converted into a digital signal. The black level of the digital signal is uniformalized by an OB circuit 404 and is then temporarily stored in a memory 405. It is to be noted that the digital signal as mentioned above is composed of signals Wr, Wb, Gr and Gb obtained by performing a generalized addition read-out on the image sensor 401 and adding together respective signals corresponding to color filters Ma, G, Cy and Ye.

[0007] The image signal read out from the memory 405 is gain-adjusted by a pixel gain adjusting circuit 406 according to a gain adjustment value obtained from a white balance circuit 420. The gain-adjusted image signal from the pixel gain adjusting circuit 406 is sent to a color-separation and color-interpolation circuit 407, where the image signal is then sent out as signals Wr, Wb, Gr and Gb to a color processing part situated at the next stage, and at the same time, the gain-adjusted image signal from the pixel gain adjusting circuit 406 is also sent to a luminance processing part without altering the arrangment of signals read out from the image sensor 401.

[0008] At the color processing part, the output of the color-separation and color-interpolation circuit 407 is sent to a vertical low-pass filter circuit 408. The output of the vertical low-pass filter circuit 408 is sent to a matrix circuit 409 to be converted into color signals R, G and B. The color signals R, G and B thus obtained are sent to a horizontal low-pass filter circuit 410 to have their frequency band restricted. The output of the horizontal low-pass filter circuit 410 is sent to a gamma converting circuit 411 to be subjected to a gamma correcting process. The output of the gamma converting circuit 411 is sent to a pure-color color-difference converting circuit 412 to be converted into a signal Y and color-difference signals R-Y and B-Y. These color-difference signals R-Y and B-Y are sent to the white balance circuit 420 and a C-SUP circuit 413. At the C-SUP circuit 413, the color-difference signals R-Y and B-Y are subjected to a process of killing the color of their saturated areas. The output of the C-SUP circuit 413 is sent to a Y compensating circuit 419, and is also sent to a color compensating circuit 414, which then performs fine color adjustment.

[0009] On the other hand, at the luminance processing part, the output of the pixel gain adjusting circuit 405 is processed to correct a difference in luminance brought about by the color filters. This process is carried out through a vertical low-pass filter circuit 415 and a horizontal low-pass filter circuit 416. Then, an edge emphasizing process is performed on the output of the horizontal low-pass filter circuit 416 by a luminance aperture correcting (APC) circuit 417. The output of the APC circuit 417 is sent to a gamma correcting circuit 418 to be subjected to a gamma correction process. A luminance signal thus obtained is sent from the gamma correcting circuit 418 to the Y compensating circuit 419 to have its luminance finely adjusted by using the color-difference signals sent from the C-SUP circuit 413.

[0010] The color-difference signals processed by the color processing part and the luminance signal processed by the luminance processing part are temporarily stored in a memory 423. In a case where the image sensor 401 is subjected to the field reading, the field signals are added together into one frame signal by an addition circuit 421. The frame signal is sent to a compressing circuit 422 to be compressed in accordance with the JPEG system or the like and, after that, is recorded in a flash memory or the like.

[0011] The white balance circuit 420 has a clipping circuit for receiving the color-difference signals R-Y and B-Y and for limiting inputs having a large amplitude exceeding a predetermined value with respect to the amplitude of the received color-difference signals R-Y and B-Y. The white balance circuit 420 is controlled by a system control CPU to adjust the gain of output values of the image sensor 401 as follows. While one picture is divided into a plurality of blocks, the color-difference signals and the luminance signal which have passed through the clipping circuit are integrated for each of the divided blocks. Then, in a coordinate system formed by the color-difference signals R-Y and B-Y as shown in FIG. 11, a color-temperature varying range of white color is set as a white determining range 501 on the coordinate system. It is determined whether the integrated value obtained at each of the above-stated blocks is within the white determining range 501 on the coordinate system. Then, gains for the R signal and the B signal are decided by using the integrated values which are within the white determining range 501. From the gains, a gain control value for control over the output of the image sensor 401 is obtained. Thus, the gain of output values of the image sensor 401 is adjusted on the basis of the gain control value.

[0012] In an image pickup apparatus using a conventional white balance adjusting method in which a white determining range is set on a coordinate system formed by the color difference signals R-Y and B-Y and an integrated value included within the white determining range is used for white balance adjustment, as mentioned above, if the area of an object of red system of low saturation is large at the time of photography under illumination of high color temperature, the illumination of high color temperature would be misjudged as illumination of low color temperature, because the object of red system of low saturation is included in the same area as that of white of low color temperature in the white determining range. Similarly, if the area of an object of blue system of low saturation is large at the time of photography under illumination of low color temperature, the illumination of low color temperature would be misjudged as illumination of high color temperature, because the object of blue system of low saturation is included in the same area as that of white of high color temperature in the white determining range.

[0013] Further, in the conventional white balance adjusting method, any color evaluation value that is not included within the white determining range is not used. Otherwise, the color evaluation value that is not included within the white determining range is replaced with the value of an edge part of the white determining range located nearest to that color evaluation value, and the thus-obtained value is integrated as a new color evaluation value. This arrangement necessitates the image pickup apparatus to be provided with an additional computing circuit.

[0014] Further, in a case where the conventional image pickup apparatus uses an image sensor which is provided with complementary color filters, the color reproducibility would degrade unless a matrix arithmetic operation is performed in an optimum manner according to color temperature in converting the image signal after white balance adjustment into a pure color signal through the matrix arithmetic operation.

[0015] According to the conventional arrangement, the color evaluation value varies with the method of reading from the image sensor. Therefore, it has been necessary to set the white determining range for one method separately from the range set for another method. For example, with the image sensor having the complementary color filters arranged as shown in FIG. 2, the color evaluation value to be used is a value expressed as (R-Y, B-Y) or (((Wr−Gb)−(Wb−Gr))/(Wr+Gb), ((Wr−Gb)+(Wb−Gr))/(Wb+Gr)) in the case of addition read-out, and is a value expressed as ((Ye−Cy)/Y1, (Ma−G)/Y1) in the case of non-addition read-out. Thus, there exist a plurality of color evaluation values, which necessitate a plurality of white determining ranges to be prepared.

[0016] Further, according to the arrangement of the conventional image pickup apparatus described, if the output of the image sensor is inputted to the white balance adjusting part as it is, the number of sample points might become too small in the event of an image having many edges. Besides, in that event, these edges might be misjudged to be white in color. Under such a condition, accurate white detection is impossible.

[0017] In view of the above-stated problems, it is a general object of the invention to provide an image pickup apparatus which is capable of accurately detecting color temperature independently of the level of a luminance signal, appositely performing white balance control and reproducing colors in an optimum manner.

[0018] To attain the above object, in accordance with an aspect of the invention, there is provided an image pickup apparatus, which comprise an image sensor having a plurality of kinds of color filters, color signal evaluating means for evaluating a color signal outputted from the image sensor, determining means for determining whether the color signal evaluated by the color signal evaluating means is included within a predetermined evaluation range, white balance adjusting means for adjusting white balance on the basis of the color signal determined by the determining means to be included within the predetermined evaluation range, and varying means for varying the evaluation range of the determining means according to a predetermined condition.

[0019] In accordance with another aspect of the invention, there is provided an image pickup apparatus, which comprises an image sensor having a plurality of kinds of color filters, first color signal evaluating means for evaluating a color signal outputted from the image sensor, first determining means for determining whether the color signal evaluated by the first color signal evaluating means is included within a predetermined evaluation range, first arithmetic operation means for performing an arithmetic operation to obtain a first control signal on the basis of the color signal determined by the first determining means to be included within the predetermined evaluation range, second color signal evaluating means for dividing a signal for one picture outputted from the image sensor into a plurality of areas, averaging color signals obtained in each of the divided areas and evaluating the averaged color signals, second determining means for determining whether each of the color signals evaluated by the second color signal evaluating means is included within the predetermined evaluation range, second arithmetic operation means for performing an arithmetic operation to obtain a second control signal on the basis of the color signals determined by the second determining means to be included within the predetermined evaluation range, and white balance adjusting means for adjusting white balance on the basis of the first control signal and the second control signal.

[0020] In accordance with a further aspect of the invention, there is provided an image pickup apparatus, which comprises an image sensor having a plurality of kinds of color filters, color signal evaluating means for evaluating a color signal outputted from the image sensor, determining means for determining whether the color signal evaluated by the color signal evaluating means is included within one of a plurality of evaluation ranges, weighted-correction means for, if the color signal is determined by the determining means to be included within one of the plurality of evaluation ranges, performing a weighted correction on the color signal according to the evaluation range within which the color signal is included, and white balance adjusting means for adjusting white balance on the basis of the color signal corrected by the weighted-correction means.

[0021] The above and other objects and features of the invention will become apparent from the following detailed description of preferred embodiments thereof taken in connection with the accompanying drawings.

[0022] FIG. 1 is a block diagram showing the structure of the signal processing system of an image pickup apparatus according to a first embodiment of the invention.

[0023] FIG. 2 is a diagram for explaining an example of arrangement of color filters.

[0024] FIG. 3 is a diagram for explaining a white determining range in the first embodiment.

[0025] FIGS. 4(a), 4(b) and 4(c) are diagrams for explaining the shifting of the white determining range in the first embodiment.

[0026] FIG. 5 is a diagram for explaining the division of a white determining range by color temperature in a second or third embodiment of the invention.

[0027] FIGS. 6(a) and 6(b) are diagrams for explaining the graphs of correction gains relative to color temperature and the weighting of the white determining range in the third embodiment.

[0028] FIG. 7 is a block diagram showing the structure of the signal processing system of an image pickup apparatus according to a fourth embodiment of the invention.

[0029] FIG. 8 is a block diagram showing the structure of the signal processing system of an image pickup apparatus according to a fifth embodiment of the invention.

[0030] FIGS. 9(a), 9(b), 9(c) and 9(d) are diagrams for respectively explaining the white determining ranges.

[0031] FIG. 10 is a block diagram showing the structure of the signal processing system of a conventional image pickup apparatus.

[0032] FIG. 11 is a diagram for explaining a white determining range of the conventional image pickup apparatus.

[0033] FIG. 12 is a conceptual diagram showing the structure of pixels for one picture in each of the embodiments of the invention.

[0034] Hereinafter, preferred embodiments of the invention will be described in detail with reference to the drawings.

[0035] FIG. 1 is a block diagram showing the structure of the signal processing system of an image pickup apparatus according to a first embodiment of the invention. In this embodiment, an image sensor 101 of the image pickup apparatus is assumed to be provided with complementary color filters having such an arrangement as shown in FIG. 2.

[0036] An analog image signal outputted from the image sensor 101 is supplied through a preprocessing circuit 102 to an A/D converting circuit 103, where the analog signal is converted into a digital signal. The black level of the digital signal is uniformalized by an OB circuit 104 and is then temporarily stored in a memory 105. The digital signal read out from the memory 105 is sent to a white balance circuit 106 to have its color signal gain adjusted according to a gain adjustment value determined in a manner as will be described later. The output of the white balance circuit 106 is sent to a color-separation and color-interpolation circuit 108. Then, four signals of colors Ma, G, Cy and Ye are simultaneously sent to a color processing part. Meanwhile, to a luminance processing part, signals having the same arrangement as the filter arrangement of the image sensor 101 are sent.

[0037] In the color processing part, the band in the vertical direction of each input signal is limited by a vertical low-pass filter circuit 109. The output of the vertical low-pass filter circuit 109 is sent to a matrix circuit 110 to be converted into color signals of colors R, G and B. The R, G and B signals are sent to a gamma converting circuit 111 to be subjected to a gamma correction process. The output of the gamma converting circuit 111 is sent to a pure-color color-difference converting circuit 112 to be converted into a signal Y and color-difference signals R-Y and B-Y. These color-difference signals R-Y and B-Y are sent to a C-SUP circuit 113. At the C-SUP circuit 113, the color-difference signals R-Y and B-Y are subjected to a process of killing the color of their saturated areas. The output of the C-SUP circuit 113 is sent through a horizontal low-pass filter circuit 114 to a Y compensating circuit 120 and also to a color compensating circuit 115.

[0038] Meanwhile, at the luminance processing part, the output of the white balance circuit 106 is processed to correct a difference in luminance brought about by the color filters. This process is carried out through a vertical low-pass filter circuit 116 and a horizontal low-pass filter circuit 117. Then, an edge emphasizing process is performed by a luminance aperture correcting (APC) circuit 118. The output of the APC circuit 118 is sent to a gamma correcting circuit 119 to be subjected to a gamma correction process. After that, a luminance signal thus obtained is sent from the gamma correcting circuit 119 to the Y compensating circuit 120 to have its luminance finely adjusted by using the color-difference signal coming from the horizontal low-pass filter circuit 114.

[0039] The color-difference signals processed by the color processing part and the luminance signal processed by the luminance processing part are sent to a compressing circuit 121 to be compressed in accordance with the JPEG system or the like and, after that, are stored in a memory 122 such as a flash memory.

[0040] Next, the action of the white balance circuit 106, which presents a main feature of the first embodiment, is described as follows.

[0041] In a case where the complementary color filters of the image sensor 101 are arranged as shown in FIG. 2, for example, an array of eight pixels shown in FIG. 2 constitutes one unit block. The whole picture is composed of such unit blocks which are repeatedly arranged. The upper or lower four pixels Ma, G, Cy and Ye of each block of eight pixels are used as a white evaluating sample point. A color evaluation value is computed for each of the white evaluating sample points in a manner as expressed below:

[0042] (Ma−G)/Y, (Ye−Cy)/Y(Y: a luminance signal)

[0043] Then, any point that is included within a white determining range set beforehand is determined to be white and is subjected to an integrating action.

[0044] The above-mentioned white determining range is set in the following manner. Shots of a white paper are taken at a plurality of color temperatures existing in the natural world. The color evaluation value is thus obtained at each color temperature. Then, the white determining range is set by plotting the color evaluation values on a graph with (Ye−Cy)/Y taken on the X axis and (Ma−G)/Y on the Y axis, as shown in FIG. 3.

[0045] A flow of processes to be executed at the white balance circuit 106 is next described as follows.

[0046] (1) The white determining range is corrected according to photo-taking conditions of an object and/or conditions of an object. This process prevents detection of such a signal that is included within the white determining range but has a color temperature which greatly differs from its actual color temperature, and also improves the white determining accuracy in a case where a flash device is used for shooting under a daylight condition.

[0047] The above-stated photo-taking conditions of an object and/or conditions of an object include an EV value indicative of brightness, a shutter speed value TV, an F-number which is an aperture value, a distance to the object of shooting, information on the on-state or off-state of a flash device, a degree of effect (influence) of external light and flash light on the object of shooting, etc.

[0048] When the flash light is off in shooting, if the EV value is larger than a certain value, the shooting operation is assumed to have been performed outdoors. Then, a limiter, which is a value on the X axis, i.e., in the direction of changes of color temperature, on the side of low color temperature in the white determining range is shifted toward the side of high color temperature, as shown on the right side part of FIG. 4(a). The outdoor light (normal light) is around 5000 K in most cases. According to experiences, in a case where the EV value is 10 or above, the probability of the outdoor shooting is great. On the basis of this probability, the limiter is shifted with the EV value used as an input, by a linear arithmetic operation as expressed by the following formula (1-1):

[0049] limit Low Temp=a1*EV+b1  (1-1)

[0050] Further, in a case where the EV value is a relatively large value (about 13 or above), a limiter on the high color temperature side, as shown on the left side part of FIG. 4(a), is also shifted toward the low color temperature side (around 6000 K) by using a formula (1-2) shown below. This shift is made for the purpose of preventing a light blue color of the sky from being misjudged to be white. Further, since the color temperature of an object in the shade sometimes exceeds 6000 K, the limit is preferably set at about 7000 K.

[0051] limit High Temp=a2*EV+b2  (1-2)

[0052] In the case of shooting with flash light, the effect of external light increases accordingly as the degree of effect of the flash light on the object decreases. In this case, therefore, the area of the white determining range is broadened so as to enable the white determining range to follow external light. Hence, the degree of effect of the flash light on the object “effectFlash” is obtained from the distance to the object, the EV value, the amount of emission of the flash light, etc. Then, according to the degree of effect “effectFlash”, the white determining range is changed by using the following formula (1-3), as shown in FIG. 4(b). Similar to the above-stated formulas (1-1) and (1-2), the formula (1-3) is a linear equation. In the formula (1-3), the degree of effect is used as an input and the limiter value as an output.

[0053] limit Flash Temp=a3*effectFlash+b3  (1-3)

[0054] Incidentally, the degree of effect of the flash light on the object is computed in a manner as disclosed, for example, in U.S. Pat. No. 5,568,187 (Okino), etc.

[0055] Further, in a case where the shutter speed value TV indicates a relatively long shutter time and the aperture F-number indicates a full aperture value, it can be judged that an indoor shooting is being performed. In that case, the limiter on the high color temperature side is shifted up to the color temperature of a fluorescent lamp light, as shown in FIG. 4(c). The limiter, in that case, is also set by carrying out a linear arithmetic operation according to the following formula with the TV value used as an input and the limiter value (the value on the X axis) as an output:

[0056] limit TV Temp=a4*TV+b4  (1-4)

[0057] Next, in order to control the gain of each color signal, a first gain control value and a second gain control value are computed.

[0058] The first gain control value is computed on the basis of the signal of such a sample point that is included within the white determining range determined by the process (1) described above. As mentioned above, the sample point is composed of four pixels Ma, G and Cy and Ye, which constitute one set, as shown in FIG. 12.

[0059] The second gain control value is computed on the basis of a signal obtained by averaging signals of such a small area (sample area) that is included within the white determining range determined by the process (1) described above. Herein, the small area means one area of m areas into which one picture is divided. The signal of the small area means a signal obtained by averaging each of the pixel signals Ma, G, Cy and Ye existing within the small area.

[0060] The method for computing and obtaining the first gain control value is described first as follows.

[0061] (2) A check is made to find if the sample point shown in FIG. 12 satisfies all of the following formulas:

[0062] Dark Threshold<(Ma+G+Cy+Ye)/4

[0063] Bright Threshold>Ma

[0064] Bright Threshold>G

[0065] Bright Threshold>Cy

[0066] Bright Threshold>Ye

[0067] (3) At a sample point which satisfies all the formulas of the process (2), color evaluation values wX and wY are computed and obtained by the following formulas:

[0068] wX=(Ye−Cy)/Ylow  (I)

[0069] wY=(Ma−G)/Ylow  (II)

[0070] where Ylow=(Ma+G+Cy+Ye).

[0071] (4) A check is made to find if the color evaluation values obtained at each sample point by the above process (3) are included within the white determining range determined by the above process (1). If so, an output signal from each pixel of the sample point is added to a adding register for each color. This process is carried out for all sample points on the picture, in a manner as expressed below:

[0072] reg1Ma=reg1Ma+Ma

[0073] reg1G=reg1G+G

[0074] reg1Cy=reg1Cy+Cy

[0075] reg1Ye=reg1Ye+Ye

[0076] (5) Upon completion of the above process (4) for all the sample points, the total number of the sample points determined to be white by the process (4) is substituted for SN1 in formulas which will be described later herein, and the total number of the sample points determined to be not-white is substituted for SN2 in these formulas.

[0077] (6) The first gain control value is obtained from the adding registers of the process (4), as follows:

[0078] gmg1=aveReg1/reg1Ma

[0079] gg1=aveReg1/reg1G

[0080] gcy1=aveReg1/reg1Cy

[0081] gye1=aveReg1/reg1Ye

[0082] where aveReg1=(reg1Ma+reg1G+reg1Cy+reg1Ye)/4.

[0083] Next, a method for computing the second gain control value is described. The second gain control value is computed in parallel with the process of computing the first gain control value.

[0084] (7) While one picture is divided into m small areas as shown in FIG. 12, an adding register arranged to store the sum value of each of pixel signals Ma, G, Cy and Ye for each of divided small areas (sample areas) is first initialized, in a manner as expressed below:

[0085] reg2Ma[m]

[0086] reg2G[m]

[0087] reg2Cy[m]

[0088] reg2Ye[m]

[0089] (8) A check is made to find if the value of level of an output signal from each pixel of the sample point, within each divided small area (sample area), satisfies all of the following formulas. In other words, the sample point is examined to find if it is within a brightness range in which white balance data can be adequately computed.

[0090] Dark Threshold<(Ma+G+Cy+Ye)/4

[0091] Bright Threshold>Ma

[0092] Bright Threshold>G

[0093] Bright Threshold>Cy

[0094] Bright Threshold>Ye

[0095] In these areas, the value of level of an output signal from each pixel of such sample points that satisfy these formulas is added to the adding register for each color, in a manner as expressed below:

[0096] reg2Ma[m]=reg2Ma[m]+Ma

[0097] reg2G[m]=reg2G[m]+G

[0098] reg2Cy[m]=reg2Cy[m]+Cy

[0099] reg2Ye[m]=reg2Ye[m]+Ye

[0100] (9) In each of the divided small areas, the values of the adding register for each color obtained by the process (8) are averaged. The values thus obtained by averaging are substituted into the above-stated formulas (I) and (II) to obtain the color evaluation values of the applicable small area (sample area). Then, the color evaluation values of each of the small areas are checked to find if they are included within the white determining range determined by the above process (1). Then, the value of the adding register of only the small area that is determined to be included within the white determining range is added to an area averaging register for each color, in a manner as expressed below:

[0101] reg2Ma=reg2Ma+reg2Ma[m]

[0102] reg2G=reg2G+reg2G[m]

[0103] reg2Cy=reg2Cy+reg2Cy[m]

[0104] reg2Ye=reg2Ye+reg2Ye[m]

[0105] (10) The second gain control value is obtained by the following formulas on the basis of the values of the area averaging registers obtained by the above process (9):

[0106] gmg2=aveReg2/reg2Ma

[0107] gg2=aveReg2/reg2G

[0108] gcy2=aveReg2/reg2Cy

[0109] gye2=aveReg2/reg2Ye

[0110] where aveReg2=(reg2Ma+reg2G+reg2Cy+reg2Ye)/4.

[0111] Incidentally, in a case where none of the color evaluation values of each small area obtained by the process (9) are determined to be included within the white determining range determined by the process (1), the area of each divided small area (sample area) of the picture is stepwise expanded until any one of the color evaluation values of the expanded area comes to be included within the white determining range determined by the process (1).

[0112] (11) White balance adjusting data (wmg1, wg1, wcy1 and wye1) are formed according to the ratio between the values SN1 and GN2 obtained by the process (5), the first gain control value obtained by the process (6) and the second gain control value obtained by the process (10), for example, as follows.

[0113] In the case of “(SN1/(SN1+SN2))>α”, the first gain control value is employed as white balance adjusting data as it is.

[0114] wmg1=gmg1

[0115] wg1=gg1

[0116] wcy1=gcy1

[0117] wge1=gye1

[0118] In the case of “β<(SN1/(SN1+SN2))<α”, a linear function of an output p defined by the input SN1 is prepared based on the following condition:

[0119] (Condition)

[0120] If SN1=α, then p=1.

[0121] If SN1=β, then p=0.

[0122] (Linear function)

[0123] p=a×SN1+b

[0124] where a=1/(α−β) and b=−β/(α−β),

[0125] which are calcualted from the condition. By using the value p obtained in the above equation, the white balance adjusting data (wmg1, wg1, wcy1 and wye1) are calculated as follows:

[0126] wmg1=p*gmg1+(1−p)*gmg2

[0127] wg1=p*gg1+(1−p)*gg2

[0128] wcy1=p*gcy1+(1−p)*gcy2

[0129] wye1=p*gye1+(1−p)*gye2

[0130] In the case of “(SN1/(SN1+SN2))≦β”),

[0131] wmg1=gmg2

[0132] wg1=gg2

[0133] wcy1=gcy2

[0134] wye1=gye2

[0135] (on condition that 0≦β<α≦1)

[0136] Then, these values are stored as white balance adjusting data.

[0137] As described above, the first embodiment is arranged to vary the white determining range according to the shooting condition and/or the object condition. By virtue of this arrangement, a white balance signal can be accurately obtained without the fear of detecting from within the white determining range any signal that shows a color temperature which greatly differs from its actual color temperature.

[0138] In the first embodiment described above, the image sensor is provided with complementary color filters arranged as shown in FIG. 2. However, the image sensor may be arranged to have pure color filters, instead of the complementary color filters. In that case, the color evaluation values can be set by using the color-difference signals R-Y and B-Y.

[0139] Further, while the first embodiment is arranged to automatically decide the width of area of the white determining range, the arrangement may be changed to manually set the width as shown in FIGS. 9(a) to 9(d). For example, in that case, the white determining range is limited as shown in FIG. 9(a) in the case of sunlight, as shown in FIG. 9(b) in the case of fluorescent light, as shown in FIG. 9(c) in the case of tungsten light, and as shown in FIG. 9(d) in the case of flash light.

[0140] A second embodiment of the invention is arranged to perform the white balance adjusting process in the following manner. In the case of the second embodiment, the white determining range as set in the first embodiment is divided by color temperature into a plurality of white determining areas. Then, from among the plurality of white determining ranges, an apposite white determining area is selected according to the shooting condition and/or the object condition. The white balance is controlled on the basis of sample points having color evaluation values included within the selected white determining area and also on the basis of areas obtained by dividing one picture into m areas. Further, the second embodiment is provided with complementary-color-to-pure-color converting matrixes for different values of color temperature. The color temperature is computed by averaging the color evaluation values included within the selected white determining area. Then, a matrix corresponding to the computed color temperature is selected from among the plurality of matrixes.

[0141] The flow of white balance adjusting processes of the second embodiment is described below. In the flow of processes, the method for obtaining the second gain control value is the same as in the first embodiment, and is, therefore, omitted from the following description.

[0142] (1) The white determining range is divided beforehand into a total of six white determining areas by color temperature within a range approximately from 8000 K to 2000 K at intervals of 1000 K.

[0143] In order to obtain images with an optimum color reproducibility, the matrixes used for conversion from complementary colors to pure colors must be changed according to the color temperature of the object. To attain the optimum color reproducibility, therefore, the number of the color conversion matrixes necessary within the color temperature range of an applicable illumination light is decided beforehand. The white determining range is then divided into k white determining areas in correspondence to the ranges of color temperature to be reproduced with the respective matrixes.

[0144] (2) To obtain the first gain control value, the sample point as shown in FIG. 12 is checked to find if its brightness is sufficient for computing white balance data, in the same manner as in the case of the first embodiment.

[0145] (3) Color evaluation values wX and wY are obtained at the sample point, as expressed below:

[0146] wX=(Ye−Cy)/Ylow

[0147] wY=(Ma−G)/Ylow

[0148] where Ylow=(ma+G+Cy+Ye)/4.

[0149] (4) The color evaluation values obtained by the process (3) are added to the values of the adding registers arranged for an applicable white determining area among the preset white determining areas, as expressed below:

[0150] reg1Ma[k]=reg1Ma+Ma

[0151] reg1G[k]=reg1G+G

[0152] reg1Cy[k]=reg1Cy+Cy

[0153] reg1Ye[k]=reg1Ye+Ye(0≦k≦5)

[0154] (5) Upon completion of the process (4) for all sample points, the total number of samples determined to be white by the process (4) is substituted for the value SN1 and the total number of samples determined to be not white is substituted for the value SN2.

[0155] (6) An apposite white determining area is selected from among the six white determining areas shown in FIG. 5 according to the shooting condition and/or the object condition. The first gain control value is obtained from the adding registers of the white determining area selected. Then, in the same manner as in the first embodiment, the white balance adjusting data is prepared on the basis of the first and second gain control values. The white balance is corrected according to the white balance adjusting data.

[0156] As in the case of the first embodiment, the shooting condition and/or the object condition in the second embodiment includes an EV value indicative of brightness, a TV value indicative of a shutter speed, an F-number indicative of an aperture value, a distance to the object, information on the on-state or off-state of a flash device, the degree of effect (influence) on the shooting object of external light or flash light, etc.

[0157] When the flash light is off in shooting, if the EV value is larger than a certain value, the shooting operation is assumed to have been performed outdoors. Then, a white determining area corresponding to the white determining range which is similar to that of the first embodiment is defined and restricted. If the number of points having color evaluation values included within the white determining area becomes zero as a result of the restriction, the number of white determining areas is increased by one.

[0158] If the EV value is a relatively large value (about 13 or above), use of any white determining area on the high color temperature side (around 6000 K) is inhibited to prevent the light blue color of the sky from being misjudged to be white.

[0159] When the light of a flash device is used, the degree of effect of the flash light on the object of shooting is obtained from a distance to the main object, the EV value, the amount of flash light emission, etc. One of the white determining areas is selected according to the degree of effect thus obtained. As in the first embodiment, in a case where the flash light has a high degree of effect on the object, the second embodiment uses only a white determining area which corresponds to the color temperature of the flash light. The selecting range for the white determining area is broadened up to the area of external light accordingly as the degree of effect of the flash light decreases.

[0160] In a case where the shutter speed value TV is set to a relatively long shutter time with the F-number (value) indicative of a maximum (full-open) aperture position, the shooting operation is considered to be the indoor shooting. In that case, the white determining area on the high color temperature side is gradually excluded from use, as in the case of the first embodiment.

[0161] (7) The color temperature of an applicable light source is computed and obtained from the average value of color evaluation values included within the white determining area selected by the process (6). A color conversion matrix corresponding to the color temperature as obtained is selected from among others. For example, referring to FIG. 5, if the color temperature of the light source is computed as 5100 K, a matrix 4 which corresponds to the color temperature of 5000 K, which is closest to the color temperature of 5100 K, is selected.

[0162] In the second embodiment, a white determining range is divided into a plurality of white determining areas by color temperature and an apposite white determining area is chosen from among these white determining areas according to the shooting condition and/or the object condition as described above. The arrangement effectively eliminates the possibility of detecting any sample point showing a color temperature which greatly differs from its actual color temperature, so that an accurate white balance signal can be obtained. Further, the second embodiment is arranged to select a color conversion matrix on the basis of a computed color temperature of the light source. For example, a matrix which emphasizes a red color is selected, if the color temperature of the light source is high, or a matrix which emphasizes a blue color is selected if the light source has a low color temperature, so that the color can be reproduced in an optimum manner at the color temperature.

[0163] In the second embodiment described above, the image sensor is provided with complementary color filters arranged as shown in FIG. 2. However, the image sensor may be arranged to be provided with pure color filters, instead of the complementary color filters. In that case, the color evaluation values can be set by using the color-difference signals R-Y and B-Y.

[0164] Further, while the second embodiment is arranged to automatically select the white determining area from among a plurality of areas shown in FIG. 5. However, as in the case of the first embodiment, this arrangement may be changed to manually select the white determining area from among the plurality of areas.

[0165] A third embodiment of the invention is arranged to perform the white balance adjusting process in the following manner. In this case, the white determining range in the first embodiment is divided by predetermined color temperatures into a plurality of white determining areas. The values of adding registers for applicable white determining areas are respectively corrected to attach weight to them according to the shooting condition and/or the object condition. Further, the third embodiment is provided with complementary-color-to-pure-color converting matrixes for the respective color temperatures in the same manner as in the case of the second embodiment. Color temperature is extracted from the computed white balance coefficients. Then, a matrix which corresponds to the extracted color temperature is selected from among the plurality of matrixes.

[0166] The flow of white balance adjusting processes of the third embodiment is described below. In the flow of processes, the method for obtaining the second gain control value is the same as in the first embodiment and is, therefore, omitted from the following description.

[0167] (1) The white determining range is divided beforehand into a total of six white determining areas by color temperature within a range approximately from 8000 K to 2000 K at intervals of 1000 K, as shown in FIG. 5.

[0168] (2) To obtain the first gain control value, the sample point is checked to find if its brightness is sufficient for computing white balance data, in the same manner as in the case of the first embodiment.

[0169] (3) Color evaluation values wX and wY are obtained at the sample point. The color evaluation values thus obtained are checked to find to which of the six white determining areas they apply.

[0170] wX=(Ye−Cy)/Ylow

[0171] wY=(Ma−G)/Ylow

[0172] where Ylow=(ma+G+Cy+Ye)/4.

[0173] (4) The color evaluation values obtained by the process (3) are added to the values of the adding registers of an applicable white determining area among the preset white determining areas, as expressed below:

[0174] reg1Ma[k]=reg1Ma+Ma

[0175] reg1G[k]=reg1G+G

[0176] reg1Cy[k]=reg1Cy+Cy

[0177] reg1Ye[k]=reg1Ye+Ye(0≦k≦5)

[0178] (5) Upon completion of the process (4) for all sample points, the total number of sample points determined to have color evaluation values included within the white determining area by the process (4) is substituted for the value SN1 and the total number of sample points determined to have no color evaluation values included within the white determining area is substituted for the value SN2.

[0179] (6) For the six white determining areas, correction gains applicable to the adding registers of the white determining areas are obtained, according to the shooting condition and/or the object condition, from a graph of FIG. 6(a) which shows the relation of correction gains to color temperatures. After that, the first gain control value is obtained from the averaged values of the adding registers of the white determining area obtained after gain correction. Then, in the same manner as in the first embodiment, white balance adjusting data is prepared on the basis of the first and second gain control values. The white balance is adjusted according to the white balance adjusting data.

[0180] In shooting without using any flash light, if the EV value is 13, for example, the shooting operation is assumed to be the outdoor shooting. In this case, the gain of the values of the adding registers of the white determining areas on the low color temperature side is lowered as shown in a color-temperature/correction-gain graph on the right side of FIG. 6(a). When the EV value is 13, for example, the gains of the white determining areas of color temperature ranges from 2000 K to 400 K and from 7000 K to 8000 K are “0”, so that the values of the applicable adding registers become “0”. Since the correction gain of the white determining area of a color temperature range from 4000 K to 5000 K is 0.25, in this instance, the values of the adding registers for this area are multiplied by 0.25. Further, since the correction gain of the white determining area of a color temperature range from 5000 K to 7000 K is 1.0, the values of the adding registers for this area are multiplied by 1.0. Then, the first gain control value is obtained from the averaged value of the values of the adding registers which have been gain-corrected in the above manner. Graphs showing such correction gains in relation to color temperature are prepared to cover an adequate number of EV values, using the color temperature as inputs and the correction gain as outputs. In the case of the third embodiment, four graphs of this kind are prepared for EV values from EV13 to EV10.

[0181] Meanwhile, as shown on the left side part of FIG. 6(a), the values of the adding registers of the white determining areas on the high color temperature side are also corrected. This correction is made for the purpose of preventing a light blue color of the sky from being misjudged to be white.

[0182] In the case of shooting with a flash light, the degree of effect of the flash light on the shooting object is obtained from a distance to the main object, an EV value, an amount of light emission from the flash device, etc. The gain of the white determining area is varied according to the degree of effect of the flash light thus obtained. As shown in FIG. 6(b), in a case where the flash light has a high degree of effect on the object, the weight to be attached to the white determining areas are restricted to the peripheral part of the color temperature of the flash light. The range of the attaching weight to the white determining areas is gradually broadened accordingly as the degree of effect of the flash light decreases, so as to enable the white determining areas to follow external light.

[0183] The process of attaching weight according to shutter speeds is also carried out in the same manner as in the case of the first embodiment. The amount of weight on the white determining areas on the high color temperature side is decreased in the event of a low shutter speed.

[0184] (7) The values of the adding registers of the white determining areas are corrected by the process (6). The color evaluation values are computed according to the formulas of the process (3) and obtained from the averaged value of each adding register obtained after the gain correction. The color temperature of the light source corresponding to the computed color evaluation values is obtained from the graph of FIG. 5. After that, a color converting matrix which corresponds to the color temperature of the light source is selected.

[0185] As described above, the white determining range is divided by color temperature into a plurality of white determining areas, and the values of the adding registers provided for these white determining areas are weighted according to the shooting condition and/or the object condition. A white balance signal can be accurately obtained by these processes. Further, since an apposite matrix is selected on the basis of the color temperature of the light source computed as described above, colors can be reproduced in an optimum manner at the color temperature.

[0186] In the third embodiment described above, the image sensor is provided with complementary color filters arranged as shown in FIG. 2. However, the image sensor may be arranged to be provided with pure color filters, instead of the complementary color filters. In that case, the color evaluation values can be set by using the color-difference signals R-Y and B-Y.

[0187] Further, the third embodiment may be arranged, as in the first embodiment, to manually set a condition of illumination for shooting and to set weight according to the illumination condition. White balance then can be adjusted more effectively by that arrangement.

[0188] FIG. 7 is a block diagram showing the structure of the signal processing system of an image pickup apparatus according to a fourth embodiment the invention. In this case, processes to be performed after adjustment of white balance are the same as those of the first embodiment and are, therefore, details of these processes are omitted from the following description. In the fourth embodiment, a white balance adjusting block arranged to be used with a non-addition read-out from an image sensor is arranged to be usable also with an addition read-out from the image sensor.

[0189] An output signal from the image sensor 201 is sent through a preprocessing circuit 202 to an A/D converting circuit 203 to be converted into a digital signal. The black level of the digital signal is uniformalized by an OB circuit 204. The output of the OB circuit 204 is temporarily stored in a memory 205. An image signal read out from the memory 205 is sent to an arithmetic operation part 206 to be restored into complementary color signals, which are sent to a white balance circuit 207 to have gain control values computed there. Then, the image signal is sent out in a state of having its white balance adjusted by the gain control values obtained by the white balance circuit 207.

[0190] The details of processes to be performed by the arithmetic operation part 206 are as described below. Signals Wr, Wb, Gr and Gb which are obtained by the addition read-out from the image sensor 201 are expressed as follows:

[0191] Wr=Ma+G  (4-1)

[0192] Wb=Ma+Cy  (4-2)

[0193] Gb=G+Cy  (4-3)

[0194] Gr=G+Ye  (4-4)

[0195] It is apparent that signal G-Cy is obtained from the formulas (4-1) and (4-2) and a signal G is obtained from the formula (4-3). Signals Ma, G, Cy and Ye are thus restored and are sent to the white balance circuit 207. The output of the white balance circuit 207 is sent to a color-separation and color-interpolation circuit 208.

[0196] As described above, in a case where a non-addition read-out is performed from the image sensor 201 having complementary color filters and the white determining range for the white balance circuit 2078 is set, for example, with gain control values “(Ma−G)/Y” and “(Ye−Cy)/Y”, output values (Wr, Wb, Gr and Gb) obtained by the addition read-out from the image sensor 201 are subjected to arithmetic operations to be restored into signals Ma, G, Cy and Ye, and the restored signals Ma, G, Cy and Ye are inputted to the white balance circuit 207. This arrangement enables the circuits in the white balance block to be usable both in the case of the addition read-out and the non-addition read-out. Besides, only one set value is sufficient for the white determining range.

[0197] An image pickup apparatus according to a fifth embodiment of the invention includes any of the white balance circuits in the first, second and third embodiments described in the foregoing. The fifth embodiment, however, differs from the other embodiments in that the image signal from the image sensor is subjected to color-separating and band-limiting processes before it is supplied to the white balance circuit.

[0198] In a case where an image picked up has many edges, problems arise in that (i) the number of samples to be used for white balance adjustment decreases as the color evaluation values of sample points in the edge parts are excluded from the white determining range and (ii) some of edge parts might be misjudged to be white of a different color temperature. To solve these problems, a low-pass filter is applied to the image signal to limit its frequency band in such a way as to eliminate edges before the signal is supplied to the white balance circuit.

[0199] FIG. 8 is a block diagram showing the structure of a signal processing system of the image pickup apparatus according to the fifth embodiment. Referring to FIG. 8, the output signal of an image sensor 301 is supplied through a preprocessing circuit 302 to an A/D converting circuit 303 to be converted into a digital signal. The black level of the digital signal is uniformalized by an OB circuit 304 and is then temporarily stored in a memory 305. The digital signal read out from the memory 305 is sent to a color-separation and color-interpolation circuit 306 to be processed by a light low-pass filter in the horizontal and vertical directions concurrently with a color-separation process. The output of the color-separation and color-interpolation circuit 306 is sent to a white balance circuit 307 to have signal gains corrected to adjust the white level of each color. Meanwhile, the signal from the memory 305 is sent also to a luminance processing part. In the luminance processing part, the image signal, after passing through a vertical low-pass filter circuit 315, has its white level corrected by a gain correcting circuit 316.

[0200] In the color processing part, the output of the white balance circuit 307 is sent to a matrix circuit 308 to be converted into signals R, G and B. The signals R, G and B are sent to a gamma correcting circuit 309 to be subjected to a gamma correcting process. The output of the gamma correcting circuit 309 is sent to a pure-color color-difference converting circuit 310 to be converted into a signal Y and color-difference signals R-Y and B-Y. These color-difference signals R-Y and B-Y are sent to a C-SUP circuit 311. At the C-SUP circuit 311, the color-difference signals R-Y and B-Y are subjected to a process of killing colors of saturated areas. The output of the C-SUP circuit 311 is sent through a vertical low-pass filter circuit 312 and a horizontal low-pass filter circuit 313 to a Y compensating circuit 320 and also to a color compensating circuit 314.

[0201] Meanwhile, at the luminance processing part, the image signal from which a stepped difference in luminance is removed by the horizontal low-pass filter circuit 317 is supplied to an APC circuit 318 to be subjected to an edge emphasizing process. The output of the APC circuit 318 is sent to a gamma correcting circuit 319 to be subjected to a gamma correction process. After that, a luminance signal thus obtained is sent from the gamma correcting circuit 319 to the Y compensating circuit 320 to have its luminance finely adjusted by using the color-difference signals supplied from the horizontal low-pass filter circuit 313.

[0202] The color-difference signals from the color processing part and the luminance signal from the luminance processing part are sent to a compressing circuit 321 to be compressed in accordance with the JPEG system or the like and, after that, are stored in a memory 322 such as a flash memory.

[0203] The arrangement described above enables the fifth embodiment to effectively prevent the decrease of samples and the color misjudgment due to many edge parts included in an image.

[0204] A white balance circuit employed in the signal processing system of an image pickup apparatus according to a sixth embodiment of the invention is the same as the white balance circuit of any of the first to fifth embodiments. However, the sixth embodiment differs from the other embodiments in the manner in which the matrix to be used for color conversion is obtained. In the case of the sixth embodiment, several matrixes are arranged beforehand for variations in color temperature as shown in FIG. 5. In the case of FIG. 5, seven matrixes are arranged for a color temperature range from 8000 K to 2000 K, which are divided at intervals of 1000 K.

[0205] When the non-addition read-out is performed on an image sensor having complementary color filters, gain control values wbMa, wbG, wbCy and wbYe are outputted from the white balance circuit. From these values, an X-axis value wX of color evaluation values, as shown in FIG. 5, is obtained as expressed below

[0206] wX=(1/wbYe−1/wbCy)  (6-1)

[0207] Next, a white determining area corresponding to this value wX is obtained. Then, a new matrix newMAT is prepared by the following linear arithmetic operation, using two matrixes matA and matB of color temperature limiters limA and limB of the obtained white determining area, the value wX and a distance between the limiters limA and limB:

[0208] dis=limB−limA

[0209] newMat=((dis(wX−limA))*matA+(dis(limB−wX))*matB)/dis  (6-2)

[0210] According to the arrangement of the sixth embodiment, a color evaluation value which indicates a color temperature is obtained by using a gain control value obtained from the white balance circuit. Then, a color conversion matrix can be obtained from this value by a linear arithmetic operation. By this arrangement, variations in color reproducibility due to change-over from one matrix to another can be smoothed.

[0211] As described above in the foregoing, according to one of the embodiments, in detecting the color temperature, a white determining range is varied according to the shooting condition and/or the object condition, so that the color temperature can be accurately detected without depending on the level of the luminance signal. This arrangement enables an accurate white balance signal to be obtained while effectively eliminating the possibility of detecting any signal existing within the white determining range in a state of greatly differing from its actual color temperature.

[0212] Further, according to one of the embodiments, a white determining range is divided beforehand into a plurality of white determining areas by color temperature. Electric charge accumulated at each sample point having color evaluation values included within each of the plurality of white determining areas is integrated for each color. From among the plurality of white determining areas, an apposite white determining area is selected according to the shooting condition and/or the object condition. A gain control value and color temperature are obtained by using the integrated value. Further, a plurality of matrixes for converting complementary colors into pure colors are prepared. The color temperature is obtained from the gain control value and the white determining area. One of the matrixes corresponding to the color temperature is selected. Accordingly, white balance can be appositely corrected. Further, since the matrix is selected on the basis of computed color temperature, colors can be reproduced most appositely to the color temperature.

[0213] Further, according to one of the embodiments, a white determining range is divided beforehand into a plurality of white determining areas by color temperature. Electric charge accumulated at each sample point having color evaluation values included within each of the plurality of white determining areas is integrated for each color. The gain of the integrated value of the white determining area is corrected according to the shooting condition and/or the object condition. A gain control value and color temperature are obtained by using the gain-corrected integrated value. A plurality of complementary-color/pure-color converting matrixes are prepared, and one of the matrixes corresponding to the computed color temperature is selected. Thus, the gain of any signal (integrated value) that is within the white determining area but greatly differs from its actual color is controlled in such a way as to lessen its influence, so that a more accurate white balance signal can be obtained. Further, since a matrix corresponding to the color temperature computed on the basis of the corrected integrated value, colors can be reproduced most appositely to the color temperature.

[0214] Further, according to one of the embodiments, a white determining area is set beforehand according to the shooting condition and/or the object condition. Electric charge accumulated at each sample point having color evaluation values included within the preset white determining area is integrated for each color to obtain a first gain control value. At the same time, while a picture is divided into a number of areas, an averaged color evaluation value is obtained for every one of the divided areas. A second gain control value is obtained on the basis of an integrated value corresponding to the color evaluation value included within the white determining area. Further, the number of pixels included within the white determining area and the number of pixels not included within the white determining area are detected. A mixing ratio between the detected numbers of pixels is computed. The first gain control value and the second gain control value are mixed according to the computed mixed ratio to obtain a gain control value for the output of the image sensor. Therefore, even if the first gain control value can not be accurately obtained due to a less number of or the absence of points included within the white determining area, the white balance can be saved from degrading too much, because the second gain control value is obtained on the basis of the integrated value corresponding to the color evaluation values included within the white determining area.

[0215] Further, according to one of the embodiments, even in a case where the non-addition read-out is performed on an image sensor having complementary color filters, the output values of the image sensor for the addition read-out are restored through an arithmetic operation to obtain a signal to be inputted to the white balance circuit. This arrangement permits use of one and the same circuit both for the addition read-out and the non-addition read-out.

[0216] Further, according to one of the embodiments, the signal outputted from the image sensor is divided by colors. Each of the divided signals is processed through a low-pass filter before it is inputted to the white balance circuit. This arrangement effectively prevents the decrease in number of samples and the misjudgment of colors due to edge parts existing in an image.

[0217] Further, according to one of the embodiments, the color evaluation value which indicates color temperature is obtained by using a gain control value obtained from the white balance circuit. Then, a color conversion matrix is obtained from this evaluation value by a linear arithmetic operation. That arrangement can effectively smooth variations of color reproduction caused by change-over from one matrix over to another.

[0218] The functions of each of the embodiments disclosed can be carried out by recording software program codes for the functions on a storage medium, by supplying the storage medium to a system or an apparatus, and by causing a computer (CPU or MPU) of the system or the apparatus to read the program codes from the storage medium and carry out the program codes stored there.

[0219] In this case, the functions of each of the embodiments are carried out by the program codes read out from the storage medium and the storage medium which stores the program codes represents each embodiment.

[0220] The storage medium to be used for supplying the program codes may be, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a DC-R, a magnetic tape, a nonvolatile memory card, a ROM, etc.

[0221] While the functions of each of the embodiments can be accomplished by carrying out the program codes read by a computer as mentioned above, it is also possible to have the actual processes of the functions carried out either in part or in their entirety by an OS (operating system) which operates on the computer according to instructions given by the program codes.

[0222] Further, the functions of each of the embodiments can be also accomplished by reading the program codes from the storage medium, by writing the program codes into a memory provided on a function-expanding board inserted into the computer or a function-expanding unit connected to the computer, and, after that, by having the actual processes of the functions carried out either in part or in their entirety by a CPU or the like provided on the function-expanding board or the function-expanding unit in accordance with instructions given by the program codes.

[0223] In a case where the arrangement of each of the embodiments disclosed is to be applied to a storage medium in the manners mentioned above, the program codes which correspond to the flow of operation described in the foregoing are stored in the storage medium. In short, in such a case, a module indispensable to the image pickup apparatus according to each of the embodiments disclosed is stored in the storage medium.