Method, apparatus and program for image processing

[0001] This nonprovisional application claims priority under 35 U.S.C. 119(a) on Patent Application No. 2002-041343 filed in Japan on Feb. 19, 2002, which is herein incorporated by reference.

[0002] 1. Field of the Invention

[0003] The present invention relates to a method and an apparatus for carrying out image processing on image data obtained by a digital camera. The present invention also relates to a program embodied on a computer-readable recording medium to cause a computer to execute the image processing method.

[0004] 2. Description of the Related Art

[0005] In a digital still camera (hereinafter referred to as a digital camera), an image obtained by photography can be recorded as digital image data in a recording medium such as an internal memory or an IC card, and can be reproduced by a printer or a monitor based on the digital image data. In the case where an image obtained by a digital camera is printed, the image is expected to have as high a quality as a print generated from a negative film.

[0006] When a print is generated from such image data, the quality thereof can be improved by carrying out various types of image processing such as density conversion processing, white balance processing, gradation conversion processing, saturation enhancement processing, and sharpness processing on the image data. Consequently, a method for obtaining a higher-quality print has been proposed U.S. Pat. No. 6,011,547. In this method, photography information such as an ON or OFF state of flash and the type of lighting is added as tag information to image data obtained by a digital camera. Appropriate image processing is carried out on the image data with reference to the photography information added to the image data.

[0007] In some cases, such tag information includes information useful for image processing, such as a distance to a subject, brightness of the subject, exposure, a photometric quantity at the time of photography, and presence or absence of counterlight, in addition to an ON or OFF state of flash and the type of lighting. For this reason, various methods have also been proposed for image processing to be carried out on image data with reference to photography information (U.S. Pat. Nos. 6,133,983, 5,739,924, 5,016,039, 5,596,387, 5,710,948, and Japanese Unexamined Patent Publication Nos. 8(1996)-307767, 11(1999)-88576, for example).

[0008] Furthermore, as methods of carrying out tone processing on image data, various methods have also been proposed for calculating a tone processing condition used in gradation conversion according to a histogram of image data (Japanese Unexamined Patent Publication No. 9(1997)-200531, and U.S. Pat. No. 5,812,286, for example). Moreover, another method of calculating a tone processing condition has also been proposed (Japanese Unexamined Patent Publication No. 2000-196890) so as to cause density of a main subject such as a human face included in an image to be kept in a target density range while keeping the highest density value and the lowest density value in the image as they are.

[0009] In such image processing methods described above, image processing is carried out on image data with use of photography information. However, in the case where a main subject in an image represented by the image data varies from people to scenery for example, the appropriate image processing condition cannot be found with accuracy if the photography information is used alone.

[0010] In the methods of finding the tone processing condition described above, the appropriate tone processing condition is difficult to find in accordance with various scenes such as people and scenery, although the histogram or the information on the density is used in the methods.

[0011] The present invention has been conceived based on consideration of the above circumstances. An object of the present invention is therefore to obtain a high-quality reproduced image by carrying out accurate tone processing on image data obtained by a digital camera.

[0012] An embodiment of an image processing method of the present invention is a method of obtaining processed image data by carrying out image processing on image data obtained by a digital camera, and the image processing method comprises the steps of:

[0013] obtaining photography information added to the image data;

[0014] obtaining main part information regarding an image represented by the image data;

[0015] inferring a photography scene regarding the image data, based on the photography information and the main part information;

[0016] setting a tone processing condition in accordance with the photography scene; and

[0017] obtaining the processed image data by carrying out the image processing on the image data according to the tone processing condition that has been set.

[0018] In the image processing method embodiment of the present invention, the processed image data may be obtained by carrying out the image processing on the image data according to a degree of tone change caused by density and/or color correction on the image data, in addition to the tone processing condition.

[0019] In the image processing method embodiment of the present invention, the tone processing condition in accordance with the photography scene may be set by selection from predetermined tone processing conditions.

[0020] One or more of a distance to a subject, brightness of the subject, exposure, a photometric quantity at the time of photography, information on a camera lens, and magnification, in addition to an ON or OFF state of flash and the type of lighting can be included in the “photography information”. The photography information is described in tag information of the image data. Information on the date and time of photography may be included in the photography information.

[0021] The “tag information” refers to information added to the image data obtained by a digital camera. “Baseline TIFF Rev.6.0.ORGB Full Color Image” adopted for a non-compressed file of an Exit file, for example can be used as a standard of the tag information.

[0022] The “main part information” refers to information regarding a main part such as a human face included in an image represented by the image data. More specifically, presence or absence of the main part, and a size of the main part in the case of presence thereof can be used as the main part information.

[0023] In the case where the main part is a human face, the image data are subjected to skin-color extraction processing or the like. By judging whether or not an area of the extracted skin-color is equal to or larger than a predetermined value, presence or absence of a human face in the image can be judged.

[0024] The “tone processing condition” refers to a parameter for correcting tones in the image data. The tone processing condition determines whether the tones in a highlight range, in an intermediate range, and in a shadow range are softened or hardened, or determines a degree of soft tones and hard tones. The tone processing condition may be set by correcting a predetermined reference tone characteristic according to the photography scene.

[0025] The “photography scene” refers to a target of photography or a combination of the target and a photography condition such as lighting. More specifically, portrait, scenery, and close-up flash photography are listed as the photography scene, which is not limited to these examples.

[0026] An image processing apparatus of the present invention is an apparatus for obtaining processed image data by carrying out image processing on image data obtained by a digital camera, and the image processing apparatus comprises:

[0027] photography information acquisition means for obtaining photography information added to the image data;

[0028] main part information acquisition means for obtaining main part information regarding an image represented by the image data;

[0029] scene inference means for inferring a photography scene regarding the image data, based on the photography information and the main part information;

[0030] tone processing condition setting means for setting a tone processing condition in accordance with the photography scene; and

[0031] image processing means for obtaining the processed image data by carrying out the image processing on the image data according to the tone processing condition that has been set.

[0032] In the image processing apparatus of the present invention, the image processing means may obtain the processed image data by carrying out the image processing according to a degree of tone change caused by density and/or color correction on the image data, in addition to the tone processing condition.

[0033] In the image processing apparatus of the present invention, the tone processing condition setting means may set the tone processing condition in accordance with the photography scene by selection from predetermined tone processing conditions.

[0034] The image processing method of the present invention may be provided as a program to cause a computer to execute the image processing method.

[0035] According to the present invention, the photography scene is inferred based on the photography information added to the image data and the main part information on the image represented by the image data. The tone processing is carried out on the image data by setting the tone processing condition according to the inferred photography scene. Therefore, the tone processing condition can be set more accurately according to the inferred photography scene than in the case of setting the tone processing condition based on the photography information alone. As a result, the processed image data can reproduce a high-quality image regardless of what the photography scene is.

[0036] In the case where color and/or density of image data are corrected, a highlight range and a shadow range of the image tend to be too light and too dark, respectively. For this reason, by carrying out the image processing on the image data according to the degree of tone change caused by color and/or density correction in addition to the tone processing condition, the highlight range and the shadow range can be prevented from being too light and too dark, respectively. Consequently, the processed image data can reproduce a higher-quality image.

[0037] Furthermore, by selecting the tone processing condition in accordance with the photography scene from the predetermined tone processing conditions, the tone processing condition can be set in a short time, which leads to efficient tone processing.

[0046] Hereinafter, embodiments of the present invention will be explained with reference to the accompanying drawings. FIG. 1 is a block diagram showing a configuration of an image processing apparatus as a first embodiment of the present invention. As shown in FIG. 1, an image processing apparatus 20 in the first embodiment reads image data S0 obtained by photography of a subject with a digital camera 10, and outputs a print P from a printer 30 by carrying out various types of image processing on the image data S0.

[0047] The image processing apparatus 20 in the first embodiment comprises the following components. Reading means 21 reads the image data S0 obtained by the digital camera 10 from a recording medium such as a memory card. Based on the image data S0, main part information acquisition means 22 obtains main part information M regarding a main part included in an image represented by the image data S0. Scene inference means 23 obtains scene information H by inferring a photography scene based on the main part information M obtained by the main part information acquisition means 22 and photography information T included in tag information added to the image data S0. A memory 24 stores various tone processing conditions in accordance with the scene information H. Tone processing condition setting means 25 reads a tone processing condition K0 corresponding to the scene information H from the memory 24, and sets a tone processing condition K1 for carrying out tone processing on the image data S0 according to the tone processing condition K0. Rough image generation means 26 generates rough image data SR by decreasing the number of pixels in the image data S0. Based on the rough image data SR, automatic set-up means 27 sets a basic tone processing condition, and conditions for density and color conversions and sharpness processing to be carried out on the image data S0. The automatic set-up means 27 then outputs a tone processing condition used as reference (hereinafter referred to as a reference tone processing condition B) based on the image data S0, and the conditions other than the reference tone processing condition B (such as the density and color conversion conditions), as image processing conditions G. Image processing means 28 obtains processed image data S1 by carrying out image processing on the image data S0 according to the image processing conditions G and the tone processing condition K1. The reading means 21 corresponds to photography information acquisition means.

[0048] In this embodiment, the photography information T includes any combination of flash information representing an ON or OFF state of flash, information on a distance to the subject, and information on magnification at the time of photography. Brightness of the subject (BV) may be included in the photography information T. Alternatively, information on date and time of photography may be included in the photography information T.

[0049] The magnification may be found by dividing a focal length by the distance to the subject. The subject brightness value may be found by (aperture value+shutter speed)-photographic sensitivity, and is found in an APEX unit. The focal length, the aperture value, and the photographic sensitivity may be included in the photography information T.

[0050] The main part information acquisition means 22 judges presence or absence of a human face in the image represented by the image data S0, and includes a result of the judgment in the main part information M. More specifically, a skin-color area is extracted from the image data S0, and the image is judged to include the human face in the case where an area of the skin-color area is equal to or larger than a predetermined threshold value Th0. In the case where the image has been judged to include the human face, the area of the skin-color area having an oval shape is found and compared with a predetermined threshold value Th1. In this manner, information is obtained on how large the area is, that is, information on how large a size of the human face is. This information may be included in the main part information M.

[0051] The scene inference means 23 infers the photography scene based on the main part information M and the photography information T. FIG. 2 shows a table representing photography scenes that can be inferred from the main part information M and the photography information T. The table shown in FIG. 2 may be stored in the scene inference means 23, and the scene inference means 23 infers the photography scene of the image with reference to the table in FIG. 2. More specifically, the photography scene may be inferred from the information on presence or absence of the human face included in the main part information M, the flash information, the information on distance to the subject and magnification, and information of another type in the photography information T. The information of another type may include information on the subject brightness value (BV). In the case where the main part information M includes the face size information, the face size information may be included in the information of another type.

[0052] The distance to the subject included in the photography information T is compared with a predetermined threshold value Th2, and the distance is judged to be long if the distance is longer than the threshold value Th2. Otherwise, the distance is judged to be short. The magnification found from the photography information T is also compared with a predetermined threshold value Th3. The distance to the subject is judged to be long in the case where the magnification is larger than the threshold value Th3. Otherwise, the distance is judged to be short.

[0053] The scene inference means 23 infers the photography scene with reference to the table in FIG. 2. In the case where the information on presence or absence of human face, the face size, and the flash information in the main part information M shows presence, a large face, and an OFF state of flash, the photography scene is judged to be a portrait scene shown in row No. 1 in the table. In the case where the information on presence or absence of human face, the flash information, and the distance to the subject in the main part information M shows presence, an ON state, and a short distance, the photography scene is judged to be a close-up flash photography scene shown in row No. 8 in the table T. The scene information H representing the inferred photography scene is output to the tone processing condition setting means 25.

[0054] The memory 24 stores the tone processing conditions according to the various types of the scene information H.

[0055] The tone processing condition setting means 25 reads the tone processing condition K0 in accordance with the scene information H from the memory 24, and sets the tone processing condition K1 by correcting according to the tone processing condition K0 the reference tone processing condition B that has been set by automatic set-up means 27 based on the image data S0. The tone processing condition K0 represents values to correct tones in a highlight range, an intermediate range, and a shadow range of the input image data S0. The reference tone processing condition B is corrected by the values of the tone processing condition K0.

[0056] For example, in the case where the scene information H represents portrait shown as No. 1 in the table of FIG. 2, the tone processing condition K0 is selected so as to soften tones in the highlight range and in the shadow range while modestly harden tones in the intermediate range, in consideration of reproducibility of skin color. In this case, if the values to correct the highlight range and the intermediate range are −5 and 2 respectively, the reference tone processing condition B shown in FIG. 3 is corrected to become a curve B1. In this manner, the tone processing condition K1 is set. The values for correction represent degrees of tone correction, and positive values harden tones and negative values soften tones. In the case where the scene information H represents a photography scene such as scenery or a still object shown in the row No. 3 in the table, the tones in the highlight range and in the shadow range are hardened and modestly softened, respectively, in consideration of reproducibility of the scenery or the still object. In this case, if the values to correct the highlight range and the shadow range are 8 and −2 respectively, the reference tone processing condition B is corrected to become a curve B2 in FIG. 3. In this manner, the tone processing condition K1 is set for this case. Furthermore, in the case where the scene information H represents a photography scene of close flash photography shown by No. 8 in the table, a color of human face tends to be too light. Therefore, the tone processing condition K0 that hardens the highlight range is selected.

[0057] For the size of human face, the tone processing condition K0 is selected to soften the highlight range more as the face size increases. In the case where the face size is small as in a group photograph, the tone processing condition K0 is selected to modestly harden the tones in the highlight range, so as to improve reproducibility of the highlight range. In the case of flash photography, the tone processing condition K0 is selected to soften the shadow range, so as to improve reproducibility of the shadow range where the light from a flash is not sufficient.

[0058] The rough image generation means 26 generates the rough image data SR by thinning pixels in the image data S0 at every several pixels.

[0059] The automatic set-up means 27 sets the reference tone processing condition B, the density conversion condition, the color conversion condition, and the sharpness processing condition according to the rough image data SR, and outputs these conditions excluding the reference tone processing condition B to the image processing means 28 as the image processing conditions G. The reference tone processing condition B is output to the tone processing condition setting means 25.

[0060] The operation of the first embodiment will be explained next. FIG. 4 is a flow chart showing the operation in the first embodiment. In this case, the rough image generation means 26 generates the rough image data SR, and the automatic set-up means 27 sets the reference tone processing condition B and the image processing conditions G, based on the rough image data SR.

[0061] The reading means 21 reads the image data S0 (Step S1). The image data S0 are input to the main part information acquisition means 22 where the main part information M is obtained (Step S2). The main part information M is input to the scene inference means 23 together with the photography information T, and the scene information H is obtained by inferring the photography scene (Step S3). The scene information H is input to the tone processing condition setting means 25 where the tone processing condition K0 in accordance with the scene information H is read from the memory 24. The reference tone processing condition B is corrected based on the tone processing condition K0, and the tone processing condition K1 is set (Step S4). The tone processing condition K1 is input to the image processing means 28 where the image processing is carried out on the image data S0 according to the image processing conditions G and the tone processing condition K1, to generate the processed image data S1 (Step S5). The printer 30 prints the processed image data S1 (Step S6) to end the operation.

[0062] As has been described above, according to this embodiment, the photography scene regarding the image data S0 is inferred based on the photography information T added to the image data S0 and the main part information M regarding the image represented by the image data S0. The tone processing condition K1 is set in accordance with the inferred photography scene, and used for tone processing. Therefore, compared to the case of setting the tone processing condition K1 by using the photography information T alone, the tone processing condition K1 is set appropriately and with accuracy in accordance with the photography scene. As a result, the processed image data S1 can reproduce a high-quality image.

[0063] In this embodiment, the tone processing condition K0 represents the correction in the tones in the highlight range, in the intermediate range, and in the shadow range. However, the tone processing condition K1 may represent correction of tones in various density ranges other than the combination of the above-described three ranges, such as a combination of the highlight range and the shadow range only, or a combination of the highlight range and other density ranges.

[0064] By enabling selection of the tone processing condition K0 in accordance with the inferred photography scene from the predetermined tone processing conditions stored in the memory 24, the tone processing condition K1 can be set in a short time and the image processing can be carried out efficiently.

[0065] A second embodiment of the present invention will be explained next. FIG. 5 is a block diagram showing a configuration of an image processing apparatus of the second embodiment of the present invention. In the second embodiment, the same components as in the first embodiments have the same reference numbers, and detailed explanations thereof will be omitted. In the second embodiment, input tone processing condition setting means 41 is used for setting an input tone processing condition K2 for solving too light and/or too dark colors in a highlight range and/or in a shadow range caused by density and/or color conversion according to a density conversion condition and/or a color conversion condition (hereinafter referred to as density color conversion conditions D) among image processing conditions G set by automatic set-up means 27. Tone processing condition setting means 25 in the second embodiment sets a tone processing condition K4, based on a tone processing condition K1 and the input tone processing condition K2.

[0066] How to convert density will be explained first. FIG. 6 shows how to set the input tone processing condition K2. In FIG. 6, a solid line represents a density conversion condition P0 for outputting density of an image represented by image data S0 (hereinafter referred to as image density) without a change. A broken line represents a density conversion condition P1 for lowering the image density. In FIG. 6, a reference point in a highlight range that represents the lowest density to be reproduced is shown as O1, while a reference point in a shadow range that represents the highest density to be reproduced is shown as O2. The reference point O1 in the highlight range can be obtained by adding 3% of a difference between a maximum value and a minimum value to the minimum value in a histogram of the image data S0. The reference point O2 in the shadow range can be obtained by subtracting 3% of the difference between the maximum value and the minimum value from the maximum value in the histogram of the image data S0.

[0067] In the case where the image density needs to be lowered due to a failure in AE processing or the like by a digital camera 10, the image data S0 are converted according to the density conversion condition P1. However, in this case, the highlight range becomes too light while the density in the shadow range becomes insufficient. For this reason, density conversion condition P2 for hardening the shadow range and softening the highlight range is found and used as the input tone processing condition K2. By carrying out density conversion according to the input density conversion condition K2, the highlight range does not become too light and the density in the shadow range can be reproduced sufficiently. In an intermediate range, the density conversion condition P2 is the same as the density conversion condition P1. Therefore, the density conversion condition P2 can convert the image data S0 so that the image data S0 have appropriate density in the intermediate range where a main subject tends to be included in many cases.

[0068] The tone processing condition setting means 25 obtains the tone processing condition K1 by correcting a reference tone processing condition B based on a tone processing condition K0 read from a memory 24. At the same time, the tone processing condition setting means 25 combines the input tone processing condition K2 set by the input tone processing condition setting means 41 and the tone processing condition K1, to obtain the tone processing condition K4. For example, as shown in FIG. 7, the tone processing condition K4 is obtained by combining the input tone processing condition K2 in the first quadrant and the tone processing condition K1 in the second quadrant.

[0069] In this case, combination with the tone processing condition K1 may change the density determined by the input tone processing condition K2, regarding a main part in an image represented by the image data S0. Therefore, it is preferable for the input tone processing condition K2 to be combined with the tone processing condition K1 in consideration of the density change in the main part.

[0070] A color conversion condition is found as in the case of density conversion condition, for each of RGB colors comprising the image data S0. Therefore, the color conversion condition is used as the input tone processing condition K2. In this case, the input tone processing condition K2 is combined with the tone processing condition K1 for each of the RGB colors, and used as the tone processing condition K4 for each of the RGB colors. In the case where both the density conversion condition and the color conversion condition are necessary, the density conversion condition and the color conversion condition are combined. In this manner, a condition for converting density and color is found for each of the RGB colors, and used as the input tone processing condition K2, . The input tone processing condition K2 is combined with the tone processing condition K1 for each of the RGB colors in this case, and used as the tone processing condition K4 for each of the RGB colors.

[0071] The operation of the second embodiment will be explained next. FIG. 8 is a flow chart showing the operation of the second embodiment. Rough image generation means 26 generates rough image data SR and the automatic set-up means 27 sets the reference tone processing condition B and the image processing conditions G, based on the rough image data SR.

[0072] Reading means 21 reads the image data S0 (Step S11). The image data S0 are input to main part information acquisition means 22 where main part information M is obtained (Step S12). The main part information M is input to scene inference means 23, together with photography information T. The scene inference means 23 infers a photography scene and obtains scene information H (Step S13). The scene information H is input to the tone processing condition setting means 25, and the tone processing condition setting means 25 reads the tone processing condition K0 in accordance with the scene information H from the memory 24, and sets the tone processing condition K1 by correcting the reference tone processing condition B according to the tone processing condition K0 (Step S14).

[0073] Among the image processing conditions G set by the automatic set-up means 27, the density color conversion conditions D are input to the input tone processing condition setting means 41 where the input tone processing condition K2 is set (Step S15).

[0074] The input tone processing condition K2 is input to the tone processing condition setting means 25, and the tone processing condition setting means 25 combines the input tone processing condition K2 with the tone processing condition K1. In this manner, the tone processing condition K4 is obtained (Step S16). The tone processing condition K4 is input to image processing means 28, and the image processing means 28 carries out image processing on the image data S0 according to the image processing conditions G and the tone processing condition K4. In this manner, processed image data S are obtained (Step S17). A printer 30 prints the processed image data S1 (Step S18) to end the operation.

[0075] As has been described above, in the second embodiment, the tone processing condition K4 is set based on the input tone processing condition K2 found in consideration of too light or too dark colors caused by density color correction of the image data S0, in addition to the tone processing condition K1. Therefore, in the case where density color correction has been carried out on the image data S0, the processed image data S1 can reproduce a high-quality image without too light or too dark colors.

[0076] In the above embodiments, the tone processing condition K0 is selected from the memory 24, based on the scene information H. However, the tone processing condition K0 may be calculated according to the scene information H.

[0077] In the above embodiments, the reference tone processing condition B is corrected based on the tone processing condition K0 to obtain the tone processing condition K1. However, the memory 24 may store the tone processing condition K1 in accordance with the various types of the scene information H (hereinafter referred to as the tone processing conditions K1) so that the tone processing condition K1 is read from the memory 24 according to the scene information H that has been obtained. In this case, if the tone processing conditions K1 corresponding any possible scenes are stored in the memory 24, the memory 24 is occupied largely by the tone processing conditions K1. Therefore, only the tone processing conditions used as references are stored in the memory 24 so that the tone processing condition K1 can be set by interpolating the tone processing conditions used as references, according to the scene information H.

[0078] In the above embodiments, the main part information M includes information on presence or absence of human face and the size thereof. However, since the main part such as a human face often appears in the center of an image, the density and colors in the center of the image represented by the image data S0 may be used as the main part information M. Furthermore, the density conversion condition in the image processing conditions G is a condition for converting the density of the main part in the image data S0 to become appropriate density. Therefore, the density conversion condition may be used as the main part information M, since the density conversion condition enables understanding of how the density appears in the main part.

[0079] In the above embodiments, in the case where the photography information T includes information on date and time of photography, the tone processing condition K1 maybe set according to the information on date and time of photography. For example, if the photography was carried out in daytime on a summer day, the image is inferred to be a high-contrast image, and the tone processing condition K1 is set to soften the intermediate range of the reference tone processing condition B.