Image processing unit, image processing method, and image processing program

[0001] 1. Field of the Invention

[0002] The present invention relates to an image processing unit to be mounted on or connected to a device such as an electronic camera. Further, the present invention relates to an image processing method and an image processing program which are applied to the image processing unit.

[0003] 2. Description of the Related Art

[0004] An image processing circuit inside the conventional electronic camera automatically conducts white balance processing, color transformation processing, gray-scale transformation processing, and so on, on individual images.

[0005] Parameters, which are used for these processings, are decided for each of the processings according to color temperature, intensity, and the like of the individual images.

[0006] Therefore, if there exists even a slight difference between two images, even though they are obtained by photographing the same scene, different parameters are set for the above processings. This results in the individual images' having different finishes.

[0007] In general, however, when a user obtains two images by photographing the same scene twice consecutively, probably with slight framing changes, the two images will mostly have a little difference. In such a case, the user often hopes to have the two images with the same photographic atmosphere (particularly, color balance).

[0008] The aforesaid conventional electronic camera, however, may recognize a large difference between the two images, event though the user can see no difference therebetween, and as a result, these two images may have quite different photographic atmospheres.

[0009] Hence, it is an object of the present invention to provide an image processing unit, an image processing method, and an image processing program which realize creation of two images with a difference which the user expects to see.

[0010] An image processing unit of the present invention comprises a histogram creation unit for creating a color histogram of each image obtained by an imaging sensor; a correlation calculation unit for calculate color correlation between a processing object image and a comparison object image by comparing the color histograms of the two object images, the processing object image being subjected to a predetermined processing and the comparison object image being obtained prior to the processing object image; and a decision unit for deciding a parameter to be used for the predetermined processing according to the calculated correlation. In general, the human eyes easily recognize differences in colors of images when comparing a plurality of images. Deciding the parameter according to the color histogram makes it possible to differentiate the plurality of images with a difference which the user expects to see.

[0011] It is preferable that the histogram creation unit evaluates, when creating the color histogram, a color at each position on the image according to a hue area to which this color belongs. By way of this evaluation a plurality of images can be differentiated with a difference which the user expects to see.

[0012] The histogram creation unit preferably sets respective dimensions of the hue areas unequally. This realizes such image processing that a subtle difference in colors which appears a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

[0013] The histogram creation unit preferably evaluates, when creating the color histogram, a color at each position on the image according to a hue area and a chroma area to which this color belongs.

[0014] The histogram creation unit preferably sets respective dimensions of the hue areas and/or chroma areas unequally. This realizes such image processing that a subtle difference in colors which appears a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

[0015] The histogram creation unit preferably evaluates, when creating the color histogram, a frequency at which each color appears on the image. The evaluation is made for every area of the image consisting of a plurality of pixels. This shortens the time required for creation of the color histogram.

[0016] It is preferable that the image processing unit of the invention further comprises a tentative decision unit for tentatively deciding the parameter suitable for each of the images obtained by the imaging sensor, and the decision unit decide, according to the correlation, the parameter to be used by applying weighted average to the tentatively decided parameter for the processing object image and the tentatively decided parameter for the comparison object image. This makes it possible to reflect the colors of the processing object image and of the comparison object image, and the correlation between the two images in the parameter.

[0017] The correlation calculation unit preferably calculates, as an index of the correlation, a Euclidean distance between a vector of the frequency at which each color appears in the color histogram of the processing object image and a vector of the frequency at which each color appears in the color histogram of the comparison object image. This realizes reliable calculation of the correlation.

[0018] Moreover, the histogram creation unit preferably excludes an area of the image from objects of color evaluation for creation of the color histogram when the area has a color being outside a predetermined color area. This can prevent a distinctive color from having influence on the parameter even in a case only one of the processing object image and the comparison object image includes the distinctive color.

[0019] An image processing method of the present invention comprises: a histogram creation step of creating a color histogram of each image obtained by an imaging sensor; a correlation calculation step of calculating color correlation between a processing object image and a comparison object image by comparing the color histograms of the two object images, the processing object image being subjected to a predetermined processing and the comparison object image being obtained prior to the processing object image for comparison; and a decision step of deciding a parameter to be used for the predetermined processing according to the calculated correlation. The human eyes are generally sensitive to differences in colors of images when comparing a plurality of images. This way of deciding the parameter based on the color histogram makes it possible to differentiate the plurality of images with a difference which the user expects to see.

[0020] Preferably, in the histogram creation step, a color at each position on the image is evaluated for the creation of the color histogram, according to a hue area to which this color belongs.

[0021] In the histogram creation step, respective dimensions of the hue areas are preferably set unequally. This realizes such image processing that a subtle difference in colors which appear a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

[0022] In the histogram creation step, when the color histogram is created, a color at each position on the image is preferably evaluated based on a hue area and a chroma area to which this color belongs.

[0023] In the histogram creation step, respective dimensions of the hue areas and/or of chroma areas are preferably set unequally. This realizes such image processing that a subtle difference in colors which appears a large difference to the human eyes is discriminated, on the other hand, a large difference in colors which appears a small difference to the human eyes is judged as equivalent.

[0024] In the histogram creation step, when the color histogram is created, a frequency at which each color appears on the image is preferably evaluated for each area consisting of a plurality of pixels. This shortens the time required for creating the color histogram.

[0025] It is preferable that the image processing method further comprises a tentative decision step of tentatively deciding the parameter suitable for each of the images obtained by the imaging sensor. In the decision step, the parameter to be used is decided according to the correlation by applying weighted average to the tentatively decided parameter for the processing object image and the tentatively decided parameter for the comparison object image. This makes it possible to properly reflect the colors of the processing object image and of the comparison object image, and the correlation between the two images in the parameter.

[0026] Preferably in the correlation calculation step, as an index of the correlation calculated is a Euclidean distance between a vector of the frequency at which each color appears in the color histogram of the processing object image and a vector of the frequency at which each color appears in the color histogram of the comparison object image. This realizes reliable calculation of the correlation.

[0027] In the histogram creation step, an area on the image is preferably excluded from objects of color evaluation for creation of the color histogram when the area has a color outside a predetermined color area. This can prevent a distinctive color from affecting the parameter even in a case only one of the processing object image and the comparison object image includes the distinctive color.

[0028] An image processing program of the present invention causes an image processing unit to execute the steps of: creating a color histogram of each image obtained by an imaging sensor; calculating color correlation between a processing object image and a comparison object image by comparing the color histograms of the two images, the processing object image being subjected to a predetermined processing and the comparison object image being obtained prior to the processing object image for comparison; deciding a parameter to be used for the predetermined processing according to the calculated correlation. This image processing program is capable of causing the image processing unit to execute processing by the image processing method of the present invention.

[0029] As described hitherto, the present invention realizes an image processing unit which can differentiate a plurality of images with a difference which the user expects to see, and an image processing method and an image processing program thereof.

[0030] The nature, principle, and utility of the invention will become more apparent from the following detailed description when read in conjunction with the accompanying drawings in which like parts are designated by identical reference numbers, in which:

[0031] FIG. 1 is a view showing a schematic configuration of an electronic camera of an embodiment;

[0032] FIG. 2 is an operational flow chart of a setting circuit 13;

[0033] FIG. 3 is a view showing the outline of the embodiment;

[0034] FIG. 4 is a color diagram; and

[0035] FIG. 5B is a color histogram H_j of an image I_j and FIG. 5A is a color histogram H_j-1 of an image I_j-1.

[0036] Hereinafter, an embodiment of the present invention will be explained with reference to the drawings.

[0037] FIG. 1 is a view showing a schematic configuration of an electronic camera of this embodiment.

[0038] A camera body 1 is provided with an imaging sensor 11, an A/D converter 12, a white balance processing circuit 14, a color interpolation circuit 15, a gray-scale transformation circuit 16, a setting circuit 13, and so on.

[0039] An image of a subject is formed through a lens 2 on the imaging sensor 11. An output signal of the imaging sensor 11 is inputted to the A/D converter 12 via a not-shown analog gain controller to be converted to a digital signal. The digital signal undergoes white balance processing in the white balance processing circuit 14, a color interpolation processing in the color interpolation circuit 15, and a gray-scale transformation processing in the gray-scale transformation circuit 16, respectively.

[0040] Here, the setting circuit 13 sets for each image a gain (hereinafter, defined as Gr, Gb) by which the white balance processing circuit 14 multiplies the digital signals (hereinafter, defined as R/G, B/G). This setting circuit 13 corresponds to what characterizes the present invention.

[0041] Note that the reference symbol 13a in FIG. 1 denotes an operating circuit in the setting circuit 13, the reference symbol 13b denotes a RAM used for processing thereof, and the reference symbol 13c denotes a ROM used for this processing.

[0042] In the ROM 13c, various kinds of the gains (Gr, Gb) to be set for the white balance processing circuit 14 are stored in advance.

[0043] Various kinds of gains (Gr₁, Gb₁), (Gr₂, Gb₂), . . . are, for example, stored in advance in the ROM 13c in the form of color temperature-gain table such as (13c-1) in FIG. 1, in correspondence with color temperature of an image.

[0044] In the ROM 13c of this embodiment, an alienation-weight table (to be described later) such as (13c-2) in FIG. 1 is also stored in advance.

[0045] In the conventional processing, however, the color temperature of each image is obtained, thereafter its corresponding gain (Gr, Gb) for each image obtained in reference with the color temperature-gain table is only sent to the white balance processing circuit 14 for the setting.

[0046] FIG. 2 is an operational flow chart of the setting circuit 13.

[0047] Note that only processing for the jth (j2) obtained image I_j is shown in this operational flow chart.

[0048] The gain (Gr, Gb) is set for the first image I₁ similarly to the conventional art. Further, the gain (Gr₁, Gb₁) thereof and a color histogram H₁ (refer to Steps S1, S3 described below for the obtaining method thereof) of the image I₁ are stored in the RAM 13b as necessary information for the processing on a subsequent second image I₂.

[0049] In this embodiment, as shown in FIG. 3, the gain (Gr_j, Gb_j) for the image I_j is decided not only according to this image I_j (B in FIG. 3) but also according to the correlation between the image I_j and an image I_j-1, (A in FIG. 3).

[0050] In Step S1 in FIG. 2, each color (R_i, G_i, B_i) of small areas e_i (i1 to n, and in n35 in FIG. 3) in the image I_j is obtained. Here, since each of the small areas e_i consists of a plurality of pixels, the color (R_i, G_i, B_i) thereof can be defined as the total value (or the average value) of the respective pixel values in the small area e_i.

[0051] Further, in Step S2, the color temperature of the image I_j is obtained in a fashion similar to the conventional technique, and the gain (Gr_j, Gb_j) is tentatively decided based on the color temperature-gain table in the ROM 13c.

[0052] Further, in Step S3, the color histogram H_j of the image I_j is created based on the color (R_i, G_i, B_i) (i1 to 35) of each of the small areas e_i obtained in Step S1. In this embodiment, the color histogram H_j is used for judging the correlation between the image I_j and the Image_j-1.

[0053] At comparing a plurality of images, in general, the human eyes are sensitive to differences in colors, so that the judgment made based on the color histogram is close to that of the human eyes.

[0054] FIG. 4 is a color diagram.

[0055] For creation of the color histogram, which color area of the color diagram each color (R_i, G_i, B_i) of the respective small areas e_j belongs to is determined, and the frequency at which each color appears is counted for every color area.

[0056] Here, for example, the color areas are differently positioned in a hue direction of the color diagram (namely, a circumferential direction of a circle with (R/G, B/G)(1, 1) being its center).

[0057] The human eyes are also sensitive to a chroma direction to some degree, so that it is preferable to provide several color areas in the chroma direction (namely, a diameter direction of the circle with (R/G, B/G)(1, 1) being its center), as shown in FIG. 4.

[0058] In addition, it is preferable that the respective dimensions of the color areas in the hue direction and in the chroma direction are set unequally. This is because it is preferable to discriminate with reliability colors which have a subtle difference from each other but appears quite different to the human eyes, while colors having a large difference from each other but appearing slightly different to the human are preferably judged as equivalent.

[0059] In the example shown in FIG. 4, in the chroma direction provided are a plurality of color areas close to a green color (lower left of the color diagram) (consequently, each of the dimensions thereof in the chroma direction is relatively narrow).

[0060] This enables green with a low chroma (such as artificial color of a fluorescent lamp) and green with a high chroma (such as natural color of a plant) to be included in different color areas. As a result, the artificial green and the natural green are discriminated.

[0061] FIG. 5B is the color histogram H_j of the image I_j created in the manner described above.

[0062] Note that at this instant the RAM 13b already has the color histogram H_j-1 of the previous image I_j-1 and the gain (Gr_j-1, Gb_j-1) tentatively decided for this image I_j-1 stored therein.

[0063] FIG. 5A is the color histogram H_j-1 of the image I_j-1.

[0064] In Step S4 in FIG. 2 obtained is the correlation between the color histogram H_j (FIG. 5B) of the image I_j created in Step S3 and the color histogram H_j-1 (FIG. 5A) of the image I_j-1 stored in the RAM 13b.

[0065] An index of the correlation is, for example, a Euclidean distance (hereinafter, referred to as alienation) D between a vector of each frequency at which each color appears in the color histogram H_j and a vector of each frequency at which each color appears in the color histogram H_j-1 (Formula (1)). $\begin{array}{cc}D=\sqrt{\underset{c=1}{\overset{10}{}}{\left(H_{j-1}\left(c\right)-H_j\left(c\right)\right)}^2}& \left(1\right)\end{array}$

[0066] Here, c in the formula signifies the number (1 to 10) of the color areas in the color diagram.

[0067] The smaller this alienation D is, the higher the correlation between the color of the image I_j and the color of the previous image I_j-1 is.

[0068] Then, in Step S5 in FIG. 2, the gain (Gr, Gb) to be used for the image I_j is decided according to this alienation D, the gain (Gr_j, Gb_j) tentatively decided for the image I_j in Step S2, and the gain (Gr_j-1, Gb_j-1) tentatively decided for the previous image I_j-1.

[0069] At this time, the weighted average is applied to the gain (Gr_j, Gb_j) and the gain (Gr_j-1, Gb_j-1) according to the alienation D with reference to the alienation-weight table shown in FIG. 1 (13c-2). This table stores weights _j-1, _j by which the gain (Gr_j-1, Gb_j-1) and the gain (Gr_j, Gb_j) are to be multiplied, in correspondence with the respective alienations D.

[0070] Here, the contents of this table are determined in advance such that the smaller the alienation D becomes, the closer to the gain (Gr_j-1, Gb_j-1) the gain (Gr, Gb) after the averaging becomes, and that the larger the alienation D becomes, the closer to the gain (Gr_j, Gb_j) the gain (Gr, Gb) after the averaging becomes.

[0071] The setting circuit 13 decides the value of the gain (Gr, Gb) by applying the weighted average by the formula (2) (this corresponds to Step S5) and sets this value for the white balance processing circuit 14 (Step S6). $\begin{array}{cc}\left(\mathrm{Gr},\mathrm{Gb}\right)=\frac{{}_{j-1}\left({\mathrm{Gr}}_{j-1},{\mathrm{Gb}}_{j-1}\right)+{}_j\left({\mathrm{Gr}}_j,{\mathrm{Gb}}_j\right)}{{}_{j-1}+{}_j}& \left(2\right)\end{array}$

[0072] Note that at the time the processing hitherto is finished, the color histogram H_j and the gain (Gr_j, Gb_j) of the image I_j are stored in the RAM 13b to replace the color histogram H_j-1 and the gain (Gr_j-1, Gb_j-1) of the image I_j-1.

[0073] Thereafter, the white balance processing circuit 14 in FIG. 1 performs white balance processing on the image I_i, using the gain (Gr, Gb).

[0074] In this embodiment, as described above, the correlation between the image I_j and the image I_j-1 is determined according to the color histogram, and a parameter (gain (Gr, Gb)) for the white balance processing on the image I_j is decided according to this correlation.

[0075] As described above, since the human eyes are sensitive to color differences, this way of deciding based on the color histogram can differentiate the image I_j-1 and the image I_j in white balance in a similar way that the human eyes do.

[0076] Incidentally, the evaluation of the color of each of the small areas e_i in the aforesaid Step S1 in this embodiment may also be made in the following manner.

[0077] First, when there exists a pixel having a distinctive color (for example, a primary color) on the image, the pixel is excluded from objects of color evaluation.

[0078] Specifically, the color of the small area e_i including the pixel having the distinctive color is recognized as the total value (or the average value) of pixels except the pixel having the distinctive color.

[0079] In this way, the correlation will not be lower (the alienation D will not be higher) even when only one of the image I_j and the image I_j-1 includes the distinctive color, since the distinctive color is ignored.

[0080] In a case where a part of one of the image I_j and the image I_j-1 contains the distinctive color but most of the areas of the image I_j and the image I_j-1 are similar to each other, the human eyes usually recognizes the two images as equivalent. Therefore, it can be said that the correlation is determined in a similar way that the human eyes do.

[0081] Incidentally, in the above-described embodiment, the number of the combinations of the weights (_j-1, _j) is five, namely, (10, 0), (8, 2), . . . (0,10), in accordance with the levels of the alienation D as shown in FIG. 1 (13c-2), but it may be a different number (for example, 2). For reference, when the number is two and the alienation D has a value smaller than a predetermined value, (_j-1, _j), will be (1, 0), and when the alienation D has a value equal to or larger than the predetermined value, (_j-1, _j) will be (0, 1).

[0082] Further, in the above-described embodiment, the objects of the weighted average are the gain (Gr_j-1, Gb_j-1) of the image I_j-1 and the gain (Gr_j, Gb_j) of the image I_j, but they may be the color temperatures of the image I_j-1 and of the image I_j.

[0083] In this case, the gain (Gr, Gb) is stored in the color temperature-gain table in advance, in correspondence with the color temperature after the weight averaging. The gain (Gr, Gb) is set for the white balance processing circuit 14.

[0084] Moreover, in the above-described embodiment, the parameter set according to the correlation is the parameter (gain (Gr, Gb)) for the white balance processing, but it may be a parameter for the gray-scale transformation processing (gray-scale transformation curve) and a parameter for color transformation processing (color transformation matrix).

[0085] Further, in the above-described embodiment, the gain tentatively decided for a certain image is used for deciding the gain for its subsequent image but an actually set gain may be used.

[0086] In this case, especially when the electronic camera is in a continuous exposure mode, it is highly likely that the correlation between continuously obtained images is sufficiently high. Namely, the weights (_j-1, _j) are likely to be (1, 0) (in other words, the value of the tentatively decided gain (Gr_j, Gb_j) is not required for the operation of the weighted average). Hence, a part of the operational flow chart in FIG. 2 may be modified such that Step S2 (the step of tentatively deciding (Gr_j, Gb_j)) in FIG. 2 is executed only when necessary.

[0087] Moreover, the above-described embodiment describes a case where the present invention is applied to the electronic camera, but the present invention is applicable to other devices on which an image processing circuit is mounted, for example, a scanner.

[0088] It is also possible to constitute an image processing program comprising the steps (FIG. 2) described in the above embodiment (a program executable by an image processing device such as a computer).

[0089] The invention is not limited to the above embodiments and various modifications may be made without departing from the spirit and scope of the invention. Any improvement may be made in part or all of the components.