US 2011/0249863 A1Application

INFORMATION PROCESSING DEVICE, METHOD, AND PROGRAM

Publication Date:2011-10-13

•10 Claims

•44 Drawing Sheets

Abstract

An information processing device includes a face detection unit that detects a face area from a target image, a feature point detection unit that detects a feature point of the detected face area, a determination unit that determines an attention area that is an area to which attention is paid in the face area based on the detected feature point, a reference color extraction unit that extracts a reference color that is color setting obtained from the target image in the determined attention area, an adjustment unit that adjusts the extracted reference color to a color setting for a modified image generated from the target image as a base, and a generation unit that generates the modified image from the target image by drawing the attention area using the color setting for the modified image.

Metadata

Assignee

Sony Corporation

Inventors

Takeshi OHASHI
Kohtaro Sabe

Application Information

Application Number:US 13/071,951

Filing Date:2011-03-25

Priority Date:2010-04-09

Classifications

IPC:

G06K9/00

Patent Drawings (44 sheets)

Description

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to an information processing device, method, and program, particularly to an information processing device, method, and program that aim to enhance the expressiveness of portrait images.

[0003] 2. Description of the Related Art

[0004] As the technology of face image recognition has been developing in recent years, a technology of generating a portrait image from a face image included in a captured image such as a photograph or the like has come into practical use.

[0005] When it comes to the generation of such portrait images, various methods are suggested in order to make the images closer to the original face images.

[0006] For example, Japanese Unexamined Patent Application Publication No. 2009-223500 discloses that, when an image is to be processed which has been obtained by replacing the face of a subject in a contributed photographic image with a portrait, outlines are drawn based on specific edges of the face or the hair of the head, and colors close to the original photographic image are set, in order not to disclose the photographic image of a person as it is (particularly, paragraph 0039 of Japanese Unexamined Patent Application Publication No. 2009-223500).

[0007] In addition, Japanese Unexamined Patent Application Publication No. 2000-311248 discloses that the skin color is extracted by using pixel values in the area around the nose based on coordinates of the right eye, the left eye and the mouth, and the information of the skin color is used in the latter process when a portrait image is created from an input image (particularly, paragraph 0172 of Japanese Unexamined Patent Application Publication No. 2000-311248).

SUMMARY OF THE INVENTION

[0008] However, in the technologies of the related art, there is a problem in that there is a possibility that unsuitable colors will be used in generating portrait images depending on the capturing conditions of the captured images.

[0009] In other words, Japanese Unexamined Patent Application Publication No. 2009-223500 states that colors close to the original photographic image are to be set for the color of the skin and the hair, but there is a case where an unsuitable color is used depending on the capturing conditions.

[0010] In addition, Japanese Unexamined Patent Application Publication No. 2000-311248 states that the average value of the pixel values around the nose and the value around the center value are used, but there is a case where an unsuitable color is used depending on the capturing conditions. Furthermore, there is a possibility that the influence of an area that is inappropriate for deciding a color, such as shadows of the lips, nose and the like, may arise.

[0011] The present invention takes the above circumstances into consideration, and it is desirable for the invention to generate a portrait image with colors adjusted suitably and without awkwardness, thereby enhancing the expressiveness of the portrait image.

[0012] According to an embodiment of the present invention, an information processing device includes a face detection unit that detects a face area from a target image, a feature point detection unit that detects a feature point of the detected face area, a determination unit that determines an attention area that is an area to which attention is paid in the face area based on the detected feature point, a reference color extraction unit that extracts a reference color that is color setting obtained from the target image in the determined attention area, an adjustment unit that adjusts the extracted reference color to a color setting for a modified image generated from the target image as a base, and a generation unit that generates the modified image from the target image by drawing the attention area using the color setting for the modified image.

[0013] The attention area may be composed of a face outline area, a hair area, and a part area including a predetermined part, the reference color extraction unit may extract a reference color in the face outline area, the hair area, and the part area, the adjustment unit may adjust each grayscale of the extracted reference color to the grayscales in the color setting for the modified image according to a predetermined grayscale converting condition, and the generation unit may draw the face outline area, the hair area, and the part area by each of the grayscales in the color setting for the modified image.

[0014] The adjustment unit may adjust each of the grayscales of the extracted reference color to the grayscale in the color setting for the modified image according to the grayscale converting condition based on the face attribute information of which the attributes are obtained from the face area.

[0015] The adjustment unit may adjust each of the grayscales of the extracted reference color to the grayscale in the color setting for the modified image according to the grayscale converting condition based on illumination information regarding an illumination state obtained from the target image or information added to the target image.

[0016] The adjustment unit may adjust each of the extracted reference color to the color setting for the modified image based on the face attribute information that is information regarding the attributes of a face or illumination information regarding an illumination state set by a user.

[0017] The information processing device includes a discrimination unit that discriminates a model image to be similar to a part image in the part area from among model images corresponding to modified part images expressed by modifying the part, and a selection unit that selects the modified part image corresponding to the model image according to the discrimination result, the adjustment unit adjusts the grayscale of the color setting for the selected modified part image to the grayscale in the color setting for the modified image, and the generation unit draws the modified part image of which the grayscale is adjusted.

[0018] The target image may be a captured image, the modified image may be a portrait image obtained by expressing a face image in a detected face area by a picture, and the modified part image may be an illustration image obtained by expressing the part by a picture.

[0019] The information processing device further includes a display unit that displays the generated modified image.

[0020] According to another embodiment of the invention, an information processing method includes the steps performed by an information processing device of detecting a face area from a target image, detecting a feature point of the detected face area, determining an attention area that is an area to which attention is paid in the face area based on the detected feature point, extracting a reference color that is a color setting obtained from the target image in the determined attention area, adjusting the extracted reference color to the color setting for a modified image generated from the target image as a base, and generating the modified image from the target image by drawing the attention area using the color setting for the modified image.

[0021] According to still another embodiment of the invention, there is provided a program that causes a computer to function as a face detection unit that detects a face area from a target image, a feature point detection unit that detects a feature point of the detected face area, a determination unit that determines an attention area that is an area to which attention is paid in the face area based on the detected feature point, a reference color extraction unit that extracts a reference color that is color setting obtained from the target image in the determined attention area, an adjustment unit that adjusts the extracted reference color to a color setting for a modified image generated from the target image as a base, and a generation unit that generates the modified image from the target image by drawing the attention area using the color setting for the modified image.

[0022] In an information processing device and method and a program according to the embodiments of the invention, a face area is detected from a target image, a feature point of the detected face area is detected, an attention area that is an area to which attention is paid in the face area is determined based on the detected feature point, a reference color that is a color setting obtained from the target image in the determined attention area is extracted, the extracted reference color is adjusted to the color setting for a modified image, the attention area is drawn by using the color setting for the modified image, and a modified image is generated from the target image.

[0023] As above, according to an embodiment of the invention, the expressiveness of a portrait image can be enhanced.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG. 1 is a diagram showing the composition of an embodiment of a portrait image generating device to which the present invention is applied;

[0025] FIG. 2 is a diagram showing an example of the composition of a part area processing unit;

[0026] FIG. 3 is a flowchart describing a portrait image generation process;

[0027] FIG. 4 is a flowchart describing a detailed skin reference color extracted color adjustment process;

[0028] FIG. 5 is a diagram illustrating feature points detected for extracting the skin reference color;

[0029] FIGS. 6A to 6D are diagrams illustrating details of a process for extracting the skin reference color from an input image;

[0030] FIGS. 7A to 7C are diagrams illustrating the principle of converting the reference color obtained from the input image into the color of a portrait image;

[0031] FIGS. 8A and 8B are diagram showing an example of specific numerical values of a color adjustment setting of the skin color;

[0032] FIG. 9 is a flowchart describing the details of a hair reference color extracted color adjustment process;

[0033] FIGS. 10A to 10C are diagrams illustrating the details of a hair area extraction process;

[0034] FIGS. 11A to 11D are diagrams illustrating the details of a drawing process;

[0035] FIG. 12 is a flowchart describing the details of a part reference color extracted color adjustment process;

[0036] FIG. 13 is a diagram showing an example of part area definition information;

[0037] FIG. 14 is a diagram illustrating the details of a process of selecting illustration images;

[0038] FIGS. 15A and 15B are diagrams illustrating the details of an illustration image drawing process;

[0039] FIGS. 16A to 16F are diagrams illustrating examples of portrait images;

[0040] FIG. 17 is a diagram showing the composition of an embodiment of a portrait image generating device to which the present invention is applied;

[0041] FIG. 18 is a flowchart describing the details of a skin reference color extracted color adjustment process;

[0042] FIG. 19 is a diagram illustrating the adjustment of color adjustment setting using face attribute information;

[0043] FIGS. 20A and 20B are diagrams showing specific values of the color adjustment setting using the face attribute information;

[0044] FIGS. 21A and 21B are schematic diagrams showing examples of skin colors that change according to capturing conditions;

[0045] FIG. 22 is a diagram showing the composition of an embodiment of a portrait image generating device to which the present invention is applied;

[0046] FIG. 23 is a flowchart describing the details of a skin reference color extracted color adjustment process;

[0047] FIG. 24 is a diagram illustrating the adjustment of color adjustment setting according to an illumination state during capturing;

[0048] FIG. 25 is a diagram showing an example of values of color adjustment setting according to illumination scores;

[0049] FIG. 26 is a diagram showing the composition of an embodiment of a portrait image generating device to which the present invention is applied;

[0050] FIG. 27 is a flowchart describing a portrait image generation process;

[0051] FIG. 28 is a flowchart describing the details of a skin reference color extracted color adjustment process;

[0052] FIGS. 29A and 29B are diagrams illustrating the adjustment (gender) of color adjustment setting by a user;

[0053] FIGS. 30A and 30B are diagrams illustrating the adjustment (capturing place) of color adjustment setting by a user;

[0054] FIG. 31 is a diagram showing an example of specific values of the color adjustment setting when a user adjusts the color adjustment setting;

[0055] FIG. 32 is a diagram showing an example of the composition of a pre-processing device;

[0056] FIG. 33 is a flowchart describing the pre-process;

[0057] FIG. 34 is a flowchart describing a learning process;

[0058] FIG. 35 is a diagram illustrating the details of a K-class classifier;

[0059] FIG. 36 is a diagram illustrating the details of a calculation procedure of a K-dimensional score vector;

[0060] FIG. 37 is a flowchart describing a generation process;

[0061] FIG. 38 is a diagram showing examples of model images of eyes;

[0062] FIG. 39 is a flowchart describing a setting process;

[0063] FIG. 40 is a diagram showing examples of illustrating images of eyes;

[0064] FIG. 41 is a diagram showing corresponding examples of the model images and the illustration images of eyes;

[0065] FIGS. 42A and 42B are diagrams illustrating examples when illustration images are extended and drawn; and

[0066] FIG. 43 is a diagram showing the composition of a computer.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0067] Hereinbelow, exemplary embodiments of the present invention (hereinafter, referred to as embodiments) will be described. In addition, the description will be provided in the following order.

[0068] 1. First Embodiment

[0069] 2. Second Embodiment

[0070] 3. Third Embodiment

[0071] 4. Fourth Embodiment

[0072] 5. Modified Example

[0073] 6. Composition Example of Pre-processing device

1. FIRST EMBODIMENT

[0074] First, the first embodiment of the present invention will be described with reference to FIGS. 1 to 16F.

[Composition Example of Portrait Image Generating Device]

[0075] FIG. 1 is a diagram showing the composition of an embodiment of a portrait image generating device to which the invention is applied.

[0076] The portrait image generating device 1 of FIG. 1 generates a modified image such as a portrait image, and the like from a target image (for example, an input image) including a face image, and causes a display device 2 such as Liquid Crystal Display (LCD), or the like to display the image on the screen thereof.

[0077] The portrait image generating device 1 is constituted by a target image acquisition unit 11, a face detection unit 12, a feature point detection unit 13, a face outline area determination unit 14, a hair area determination unit 15, a part area processing unit 16, a reference color extraction unit 17, a color adjustment unit 18, and a drawing creation unit 19 as shown in FIG. 1.

[0078] The target image acquisition unit 11 acquires an input image input as a target image and supplies the image to the face detection unit 12.

[0079] When the target image acquisition unit 11 is a camera unit constituted by imaging devices such as a lens, charge coupled devices (CCD), complementary metal oxide semiconductors (CMOS), or the like, the target image is an image captured by the camera unit. In addition, the target image may be acquired by reading image data recorded on a recording medium such as a memory care or the like, or acquired from equipment connected to a network such as the Internet, or the like via a network. In other words, the target image may be an image including a face image, and the acquiring method is arbitrary.

[0080] The face detection unit 12 makes the input image supplied from the target image acquisition unit 11 be subjected to a predetermined image process for detecting the face area and supplies information pertaining to the face image in thus-obtained face area to the feature point detection unit 13.

[0081] The feature point detection unit 13 makes the face image in the face area supplied from the face detection unit 12 be subjected to a predetermined image process in order to detect feature points for specifying organs (parts) such as eyes, eyebrows, nose, mouth, and the like and outlines, and supplies information pertaining to thus-obtained feature points to the face outline area determination unit 14, the hair area determination unit 15, and the part area processing unit 16.

[0082] The face outline area determination unit 14 determines the inner area of the face outline (hereinafter, referred to as a face outline area) based on the information pertaining to the feature points supplied from the feature point detection unit 13, and supplies the determined result to the reference color extraction unit 17.

[0083] The hair area determination unit 15 determines the area of hair in the face (hereinafter, referred to as a hair area) based on the information pertaining to the feature points supplied from the feature point detection unit 13, and supplies the determined result to the reference color extraction unit 17.

[0084] The part area processing unit 16 determines and extracts the area of each part (hereinafter, referred to as a part area) corresponding to a predetermined part such as eyes, eyebrows, nose, mouth, and the like in the face area based on the information pertaining to the feature points supplied from the feature point detection unit 13, and supplies the extracted result to the reference color extraction unit 17. In addition, the part area processing unit 16 selects an illustration image of eyes, eyebrows, nose, mouth, and the like corresponding to the extracted part area, and supplies the image to the color adjustment unit 18.

[0085] Furthermore, these face outline area, hair area, part area can be said as attention areas because attention is paid to the areas in the face area. In addition, the details of the part area processing unit 16 will be described later with reference to FIG. 2.

[0086] The reference color extraction unit 17 is constituted by a skin reference color extraction unit 17A, a hair reference color extraction unit 17B, and a part reference color extraction unit 17C. In addition, corresponding to each reference color extraction unit constituting the reference color extraction unit 17, the color adjustment unit 18 is constituted by a skin color adjustment unit 18A, a hair color adjustment unit 18B, and a illustration image color adjustment unit 18C.

[0087] The skin reference color extraction unit 17A is supplied with a result from the determination of the face outline area from the face outline area determination unit 14. The skin reference color extraction unit 17A extracts the skin reference color based on the face outline area determined by the face outline area determination unit 14, and supplies the extracted color to the skin color adjustment unit 18A.

[0088] The hair reference color extraction unit 17B is supplied with a result from the determination of the hair area from the hair area determination unit 15. The hair reference color extraction unit 17B extracts the hair reference color based on the hair area determined by the hair area determination unit 15 and supplies the extracted color to the hair color adjustment unit 18B.

[0089] The part reference color extraction unit 17C is supplied with an extracted result of the part area for each part from the part area processing unit 16. The part reference color extraction unit 17C extracts the part reference color for each part based on the part area of each part extracted by the part area processing unit 16, and supplies the extracted color to the illustration image color adjustment unit 18C.

[0090] Furthermore, in the embodiment, the “reference color” indicates color setting which is setting for colors obtained from the target image (input image). Then, description will be provided by collectively referring the reference color obtained from the face outline area to the “skin reference color”, the reference color obtained from the hair area to the “hair reference color”, and the reference color obtained from the part area to the “part reference color”.

[0091] The skin color adjustment unit 18A adjusts the skin reference color in the face outline area supplied from the skin reference color extraction unit 17A to a color for portrait image (color setting) according to predetermined converting conditions, and supplies information obtained from the adjustment to a drawing creation unit 19.

[0092] The hair color adjustment unit 18B adjusts the hair reference color in the hair area supplied from the hair reference color extraction unit 17B to a color for portrait image (color setting) according to predetermined converting conditions, and supplies information obtained from the adjustment to the drawing creation unit 19.

[0093] The illustration image color adjustment unit 18C is supplied with the illustration images corresponding to each part from the part area processing unit 16 and the part reference color for each part from the part reference color extraction unit 17C. The illustration image color adjustment unit 18C adjusts the color of the corresponding illustration images to the color for the portrait image (color setting) based on predetermined converting conditions and each part reference color, and supplies information obtained from the adjustment to the drawing creation unit 19.

[0094] The drawing creation unit 19 is supplied with the information obtained from the adjustment of the color adjustment setting from the skin color adjustment unit 18A, the hair color adjustment unit 18B, and the illustration image color adjustment unit 18C.

[0095] Furthermore, in the description below, the corresponding relationship of color setting (color range) in the reference color obtained from the input image and color setting (color range) used in the portrait image will be referred to as “color adjustment setting”.

[0096] The drawing creation unit 19 fills the union area of the face outline area and the hair area with the skin color subjected to the adjustment of the color adjustment setting based on the information pertaining to the face outline area and the hair area, and after that, fills the hair area with the hair color subjected to the adjustment of the color adjustment setting to draw an image formed only of the skin area and the hair area (hereinafter, referred to as a base image).

[0097] After that, the drawing creation unit 19 performs a predetermined process such as rotation, enlargement, reduction (re-sizing), or the like for the illustration images subjected to the adjustment of the color adjustment setting so that a part point and an anchor point defined based on anchor point defining information coincide with each other, and draws and arranges thus-obtained an illustration image on the base image to generate a portrait image. Furthermore, the anchor point defining information is information, for example, acquired from the part area processing unit 16, and will be described later in detail.

[0098] The portrait image generated as such is displayed on a display device 2 by the drawing creation unit 19.

[0099] Furthermore, for the sake of convenience in description below, the face outline area determination unit 14, the skin reference color extraction unit 17A, and the skin color adjustment unit 18A are referred to as a skin reference color extracted color adjustment unit 20A. In the same manner, the hair area determination unit 15, the hair reference color extraction unit 17B, and the hair color adjustment unit 18B are referred to as a hair reference color extracted color adjustment unit 20B, and the part area processing unit 16, the part reference color extraction unit 17C, and the illustration image color adjustment unit 18C are referred to as a part reference color extracted color adjustment unit 20C.

[0100] The portrait image generating device 1 of FIG. 1 is constituted as above.

[Composition Example of Part Area Processing Area]

[0101] FIG. 2 is a diagram showing a composition example of the part area processing unit 16 of FIG. 1.

[0102] The part area processing unit 16 is constituted by a part area determination unit 31, a K-class classifier 32, a similarity discrimination unit 33, an illustration image selection unit 34, and a database 35.

[0103] The part area determination unit 31 is supplied with the information pertaining to the feature points from the feature point detection unit 13 and the part area defining information from the database 35. The part area determination unit 31 determines and extracts a part area including a predetermined part from the face image in the face area based on the feature points from the feature point detection unit 13, based on the part area defining information, and supplies the extraction result to the part reference color extraction unit 17C (FIG. 1) and the K-class classifier 32.

[0104] Furthermore, the part area defining information is, for example, registered in the database 35 or the like in advance, and acquired therefrom by the part area determination unit 31.

[0105] The K-class classifier 32 obtains a K-dimensional score (hereinafter, referred to as a K-dimensional score vector) for a part image in the part area supplied from the part area determination unit 31, and supplies the score to the similarity discrimination unit 33.

[0106] Furthermore, the K-class classifier 32 is prepared for each part, obtained by a pre-processing device 101 in FIG. 32 to be described later, and detailed description thereof will be provided later.

[0107] The similarity discrimination unit 33 specifies a K-dimensional score vector that is most similar to the K-dimensional score vector supplied from the K-class classifier 32 out of K-dimensional score vectors of model images corresponding to part correspondence information registered in the database 35, discriminates the most similar model image, and supplies the discrimination result to the illustration image selection unit 34.

[0108] Furthermore, the part correspondence information is information that gives correspondence between the illustration image and the model image, and registered in the database 35. The illustration image is an example of a modified part image expressed by modifying each part, and an image shown by expressing each part with a picture (illustration). In addition, the model image is an image generated such that a part image of an input image is classified into a plurality of prototypes, and an average image of a part image group included in each prototype is obtained based on a multi-dimensional score vector for the input image output from a multi-class classifier (K-class classifier 32 of FIG. 32 to be described later) generated by AdaBoost ECOC (Error Correct Output Coding) learning that uses a image feature amount of a plurality of sample images.

[0109] The part correspondence information will be described later in detail since the information is obtained by the pre-processing device 101 of FIG. 32.

[0110] The illustration image selection unit 34 selects an illustration image corresponding to the model image that has been discriminated to be the most similar from the part correspondence information registered in the database 35 based on the discrimination result supplied from the similarity discrimination unit 33, and supplies the image to the illustration image color adjustment unit 18C.

[0111] The database 35 stores various information such as anchor point defining information necessary for generating a portrait image in addition to the part area defining information and the part correspondence information described before. Furthermore, the anchor point defining information defines a point (hereinafter, referred to as an anchor point) on an illustration image corresponding to the location of each part (hereinafter, referred to as a part point) detected from the face image included in the input image. The anchor point defining information is used when a portrait image is generated by the drawing creation unit 19, and supplied to the drawing creation unit 19 when the generation of a portrait image is performed. The anchor point defining information will be described later in detail since the information is obtained by the pre-processing device 101 of FIG. 32.

[Description of Portrait Image Generation Process]

[0112] Next, a process of generating a portrait image (portrait image generation process) executed by the portrait image generating device 1 of FIG. 1 will be described with reference to the flowchart of FIG. 3.

[0113] In Step S11, the target image acquisition unit 11 acquires an input image including the face image such as a captured image.

[0114] In Step S12, the face detection unit 12 detects the location (x, y, w, h) of the face area in the input image by scanning the face pattern in the input image supplied from the target image acquisition unit 11.

[0115] In Step S13, the feature point detection unit 13 detects the location (x, y) of feature points by scanning the pattern of each feature point of the face image in the face area detected by the face detection unit 12. With the feature points, the face outline area, the hair area, and each part area are specified.

[0116] In Step S14, a skin reference color extracted color adjustment process is performed by the skin reference color extracted color adjustment unit 20A.

[0117] Here, the details of the skin reference color extracted color adjustment process corresponding to the process of Step S14 in FIG. 3 will be described with reference to the flowchart of FIG. 4.

[0118] In Step S31, the face outline area determination unit 14 determines the face outline area based on the feature points detected by the feature point detection unit 13.

[0119] In other words, since points in the outline of the face are detected as feature points, for example, as in FIG. 5, the inside of the outline of the face is determined as the face outline area by the face outline area determination unit 14 based on these feature points.

[0120] In Step S32, the skin reference color extraction unit 17A extracts the skin reference color in the face outline area determined by the face outline area determination unit 14.

[0121] FIGS. 6A to 6D are diagrams showing an example of the flow of a series of a process from the acquisition of the input image to the extraction of the skin reference color in the face outline area in the input image.

[0122] First of all, when a color segmentation process is performed for the input image of FIG. 6A, an image shown in FIG. 6B is obtained. Herein, for example, clustering is performed which uses k-means algorithm for an RGB value of the input image of FIG. 6A. Next, as shown in FIG. 6C, the range to extract the skin reference color is designated for the image of FIG. 6B obtained as a result of the color segmentation process by using the feature points detected by the feature point detection unit 13. For the designation of the range, for example, a method of connecting the detected feature points with splined curves is used.

[0123] When the range of extracting the skin reference color of FIG. 6C is designated, as shown in FIG. 6D, the skin reference color is extracted from the area in the designated range. For the extraction of the skin reference color, it is desirable to select a color occupying a relatively wide range in the designated range. For example, in the designated range of FIG. 6C, the color of a predetermined area occupying 50% or more, 30% or more, or the like is extracted as the skin reference color. FIG. 6D illustrates an example in which two kinds of the skin reference color are extracted, but in order to have one kind of the skin reference color, an average value of the colors may be adopted, or a color occupying the widest area in the designated range may be selected. In addition, a plurality of reference colors may be reflected in each area of a portrait image, and a plurality of color information pieces may be used as setting values of a gradation process.

[0124] Returning to the flowchart of FIG. 4, in Step S33, the skin color adjustment unit 18A adjusts the skin reference color extracted by the skin reference color extraction unit 17A according to predetermined grayscale converting conditions, and converts the color into the skin color to be used in the portrait image.

[0125] Herein, with reference to FIGS. 7A to 7C, a method of converting the skin reference color into the skin color to be used in the portrait image will be described.

[0126] Furthermore, as an example of the converting method, color information is treated as an RGB value of 8 bit, and a method of adjusting each value will be shown. FIGS. 7A to 7C illustrate the correspondence relationship between the reference color obtained from the input image (indicated by “input image” in the left side of each color component in the drawings) and the color range of a color for the portrait image (indicated by “portrait image” in the right side of each color component in the drawings) for each of the red component (FIG. 7A), the green component (FIG. 7B), and the blue component (FIG. 7C). The color range is the range surrounded by thick frames in grayscale values of RGB having the range from 0 to 255 in the drawings, and corresponds to the “input image” and the “portrait image” as shown by arrows in the drawings. Furthermore, the relationship is the same as other corresponding drawings to be described later.

[0127] As shown in FIG. 7A, the color range of the reference color obtained from the input image for the red component is set to [RImin, RImax]. The color range may be obtained by statistically acquiring a range in an image that has a possibility of being input.

[0128] Herein, conversion is performed such that a case when the R value of the reference color is smaller than RImin is set to RImin, and a case where the R value is greater than RImax is set to RImax so that an input value is in the range of [RImin, RImax]. In addition, in the same manner for the green component and the blue component, [GImin, GImax] and [BImin, BImax] are defined as shown in FIGS. 7B and 7C.

[0129] Furthermore, in the description below, the color range of the reference color obtained from the input image is indicated by [XImin, XImax] for generalization of RGB.

[0130] As shown in FIG. 7A, for the red component, the color range used in the portrait image is set to [RCmin, RCmax]. The color range is determined, for example, according to face attribute, ambience, theme or the like of the generated portrait image.

[0131] In addition, in the same manner for the green component and the blue component, [GCmin, GCmax] and [BCmin, BCmax] are defined as shown in FIGS. 7B and 7C.

[0132] Furthermore, in the description below, the color range of a color used in the portrait image is indicated by [XCmin, XCmax] for generalization of RGB.

[0133] As a simple example of a method of converting (grayscale of) the reference color into (grayscale of) the color used in the portrait image, linear conversion from [XImin, XImax] to [XCmin, XCmax] can be considered.

[0134] When the reference color is set to [Rin, Gin, Bin] and the color used in the portrait image is set to [R, G, B] in a method of independently converting each of RGB values, the following conversion formula (1) can be used.

[0135] if (Rin<RImin){Rin=RImin}

[0136] if (RImax<Rin){Rin=RImax}

R=(RCmax−RCmin)*(Rin−RImin)/(RImax−RImin)+RCmin

[0137] if (Gin<GImin){Gin=GImin}

[0138] if (GImax<Gin){Gin=GImax}

G=(GCmax−GCmin)*(Gin−GImin)/(GImax−GImin)+GCmin

[0139] if (Bin<BImin){Bin=Bimin}

[0140] if (BImax<Bin){Bin=Bimax}

B=(BCmax−BCmin)*(Bin−BImin)/(BImax−BImin)+BCmin (1)

[0141] However, since the RGB values are independently adjusted in the case of the conversion formula (1) above, there is a case where an intended color is not shown as the balance of the RGB values are broken depending on cases. In order to avoid the problem, the following conversion formula (2) that adjusts the RGB values in an interlocking manner can be considered.

RGBin=(Rin+Gin+Bin)/3

RGBImax=(RImax+GImax+BImax)/3

RGBImin=(RImin+GImin+BImin)/3

[0142] if (RGBin<RGBmin){RGBin=RGBmin}

[0143] if (RGBmax<RGBin){RGBin=RGBmax}

R=(RCmax−RCmin)*(RGBin−RGBImin)/(RGBImax−RGBImin)+RCmin

G=(GCmax−GCmin)*(RGBin−RGBImin)/(RGBImax−RGBImin)+GCmin

B=(BCmax−BCmin)*(RGBin−RGBImin)/(RGBImax−RGBImin)+BCmin (2)

[0144] With the above conversion formulas (grayscale converting conditions), (grayscale of) the skin reference color obtained from the input image can be converted into (grayscale of) the skin color used in the portrait image.

[0145] FIGS. 8A and 8B are diagrams showing an example of specific numerical values of color adjustment setting for the skin color.

[0146] FIG. 8A shows [XImin, XImax] as the color range in the skin reference color of the RGB values, and FIG. 8B shows [XCmin, XCmax] as the color range in the portrait image of the RGB values.

[0147] In other words, the color range of the skin reference color designated by [RImin, RImax]=[90, 170] is converted into [RCmin, RCmax]=[217, 255] as a color range in a portrait image.

[0148] In the same manner, [GImin, GImax]=[60, 140] is converted into [GCmin, GCmax]=[191, 231], and [BImin, BImax]=[40, 120] is converted into [BCmin, BCmax]=[172, 212].

[0149] As such, the color adjustment setting of the skin color is adjusted, and the process returns to the process of Step S14 of FIG. 3.

[0150] In Step S15, a hair reference color extracted color adjustment process is performed by the hair reference color extracted color adjustment unit 20B.

[0151] Here, the details of the hair reference color extracted color adjustment process corresponding to the process of Step S15 of FIG. 3 will be described with reference to the flowchart of FIG. 9.

[0152] In Step S51, the hair area determination unit 15 determines the hair area based on the feature points supplied from the feature point detection unit 13.

[0153] For example, the hair area determination unit 15 performs clustering for the RGB values obtained from the image of the area of the upper half (the area of the top of the head) of the face image in the face area (or the input image) detected by the face detection unit 12 by using, for example, k-means algorithm, and determines a predetermined area as the hair area.

[0154] FIGS. 10A to 10C are diagrams illustrating the details of a hair area determination process executed by the hair area determination unit 15.

[0155] As shown in FIG. 10A, as the hair area determination process, first, the upper half of the upper face area A_Uin the face area is determined. Then, if all pixels (R, G, and B) in the upper face area A_Uis subjected to clustering with three colors by using k-means algorithm, all the pixels belong to any of three classes. Next, when it is determined whether the labels of adjacent pixels coincide with each other or not, and area division is performed, the area is divided into three areas of the background, hair, and skin as shown in FIG. 10B.

[0156] Furthermore, in the division process, the area is divided into three or more areas, but the hair area determination unit 15 may determine the largest area as the hair area among areas closest to, for example, black color, as shown in FIG. 10C.

[0157] Returning to the flowchart of FIG. 9, in Step S52, the hair reference color extraction unit 17B extracts the hair reference color from the hair area determined by the hair area determination unit 15.

[0158] The above description is of the method of extracting the hair reference color, but in the same manner as the method of extracting the skin reference color of FIGS. 6A to 6D described above, for example, the color in the area occupying 50% of more of the total area is extracted as the hair reference color in the designated range of the hair area of FIG. 10C.

[0159] In Step S53, the hair color adjustment unit 18B adjusts the hair reference color extracted by the hair reference color extraction unit 17B and converts the color into the hair color used in the portrait image according to predetermined grayscale converting conditions.

[0160] The above description is of the adjustment of color adjustment setting of the hair reference color, but in the same manner as the color adjustment setting (FIGS. 7A to 7C and FIGS. 8A and 8B) of the skin reference color described above, for example, the adjustment is performed such that a color range of the hair reference color designated by [XImin, XImax] is converted into [XCmin, XCmax] as a color range in the portrait image.

[0161] As such, the color adjustment setting of the hair color is adjusted, and the process returns to the process of Step S15 of FIG. 3.

[0162] In Step S16, the drawing creation unit 19 fills the area of the union of the face outline area and the hair area with the skin color subjected to the adjustment of the color adjustment setting, and then fills the area with the hair color subjected to the adjustment of the color adjustment setting to draw a base image of the face.

[0163] FIGS. 11A to 11D are diagrams illustrating the details of a drawing process of the base image executed by the drawing creation unit 19.

[0164] As shown in FIG. 11A, when the feature points on the detected outline are linked by a predetermined interpolation process such as splined curve interpolation, and an area surrounded by the curve is obtained, the face outline area is formed as shown in FIG. 11B. Then, as shown in FIG. 11C, after the face outline area and the inside the hair area are filled with the skin color subjected to the adjustment of the color adjustment setting, and the hair area is filled with the hair color subjected to the adjustment of the color adjustment setting, the base image of FIG. 11D is obtained.

[0165] Furthermore, herein, for the sake of simplicity of the description, description on a re-sizing process for an image size of the input image is omitted, but actually, an expanded area mainly from the face area detected by the face detection unit 12 is cut out from the input image, and a process of re-sizing is performed for the area in accordance with the image size of the generated portrait image. When it comes to the extent of the re-sizing, since an area projecting a little from the entire face in the face area detected by the face detection unit 12 is extracted, for example, an area with 1.6 times thereof mainly from the area is cut out from the input image. At this time, for example, if the image size of a desired portrait image is set to 300×300 pixels, the face image in the detected face area is re-sized to the image size. In addition, at this time, the locations of the feature points of the face area detected by the feature point detection unit 13 are converted into the locations of the cut-out image.

[0166] Returning to the flowchart of FIG. 3, in Step S17, a part reference color extracted color adjustment process is performed by the part reference color extracted color adjustment unit 20C.

[0167] Here, the part reference color extracted color adjustment process of Step S17 in FIG. 3 will be described in detail with reference to the flowchart of FIG. 12. Furthermore, in order to make the description easily understood, an example where the illustration image of the eyes is selected from parts such as eyes, eyebrows, nose, mouth, and the like to perform the adjustment of the color adjustment setting will be mainly described.

[0168] In Step S71, the face area of the input image is re-sized to a predetermined image size in the part area processing unit 16, and a part area according to part area defining information is cut out from the face area that become the predetermined image size by the part area determination unit 31 in Step S72.

[0169] FIG. 13 is a diagram showing an example of the part area defining information.

[0170] As shown in FIG. 13, when each part area is cut out from the face area that has been re-sized to 64×64 pixels, a square area that has opposing corners of a point (x₁, y₁) and a point (x₂, y₂), which are arbitrary locations in the face area, is extracted. For example, since the eye part area is a square area that has opposing corners of a point (10, 23) and a point (55, 30), the area is cut out as the eye part area by the part area determination unit 31.

[0171] The eye part area that has been cut out as above is supplied to the part reference color extraction unit 17C and the K-class classifier 32.

[0172] Returning to the flowchart of FIG. 12, in Step S73, the K-class classifier 32 for the eye part area obtains a K-dimensional score vector corresponding to the part image in the eye part area.

[0173] In Step S74, the similarity discrimination unit 33 specifies a K-dimensional score vector that is most similar to the K-dimensional score vector obtained from the part image in the eye part area computed by the K-class classifier 32 for the eye part area out of K-dimensional score vectors of the model images corresponding to the illustration images based on part correspondence information registered in the database 35, and discriminates the most similar model image. Furthermore, for example, Euclidean distance is employed for the similarity of the K-dimensional score vectors.

[0174] In Step S75, the illustration image selection unit 34 selects an illustration image corresponding to the model image that has been determined to be most similar from the part correspondence information registered in the database 35 based on the discrimination result of the similarity discrimination unit 33.

[0175] FIG. 14 is a diagram illustrating the details of a process of selecting an illustration image.

[0176] As shown in FIG. 14, when a part image in the eye part area is extracted from the face area of the input image by the part area determination unit 31, the similarity discrimination unit 33 discriminates similarity of the part image and a plurality of model images from the computation result of the K-class classifier 32, and discriminates a model image with the highest similarity. Since the model images and the illustration images correspond to each other in advance in the part correspondence information, the illustration image selection unit 34 can select one illustration image paired with the model image with the highest similarity.

[0177] As such, one illustration image is selected that is paired with the model image having the highest similarity in the part image in the eye part area extracted from the face area out of the plurality of illustration images prepared in advance.

[0178] Returning to the flowchart of FIG. 12, in Step S76, the part reference color extraction unit 17C extracts the eye part reference color in the eye part area determined by the part area determination unit 31.

[0179] The above description is of a method of extracting the reference color in the eye part area, but in the same manner as the method of extracting the skin reference color of FIGS. 6A to 6D described above, for example, the color of the area occupying 50% or more of the total area is extracted as the eye part reference color in the designated range of the eye part area.

[0180] In Step S77, the illustration image color adjustment unit 18C adjusts the color of the illustration image of the eyes selected by the illustration image selection unit 34 based on the eye part reference color extracted by the part reference color extraction unit 17C according to the predetermined grayscale converting conditions, and converts the color into a color for the eyes to be used in the portrait image.

[0181] The above description is of the adjustment of the color adjustment setting for the eye part reference color, but in the same manner as the color adjustment setting (FIGS. 7A to 7C and FIGS. 8A and 8B) of the skin reference color described above, the adjustment is performed such that, for example, the color range of the eye part reference color designated by [XImin, XImax] is converted into [XCmin, XCmax] as the color range in the portrait image.

[0182] As described above, the adjustment of the color adjustment setting for the illustration image is performed, and the process returns to the process of Step S17 of FIG. 3.

[0183] In Step S18, the drawing creation unit 19 performs an image process such as rotation, enlargement, reduction, or the like for the illustration image subjected to the adjustment of the color adjustment setting by the illustration image color adjustment unit 18C so that anchor points defined by anchor point defining information and corresponding part points coincide with each other, and the illustration image thus obtained is drawn to the base image obtained from the process of Step S16.

[0184] FIGS. 15A and 15B are diagrams illustrating an illustration image drawing process in detail.

[0185] As shown in FIGS. 15A and 15B, the correspondence of an anchor point P_Aof the illustration image of the eyes subjected to the adjustment of the color adjustment setting by the illustration image color adjustment unit 18C and a part point P_Pon the portrait image (base image) corresponding to the feature points in the face area detected by the feature point detection unit 13 based on anchor point defining information is as described before.

[0186] In other words, as shown in FIG. 15A, since an anchor point P_A1and a part point P_P1correspond to each other on one hand and an anchor point P_A2and a part point P_P2correspond to each other on the other hand, the drawing creation unit 19 performs an image process such as rotation, enlargement, reduction or the like for the illustration image so as to draw an illustration image for the eyes on the base image by making those points coincide with each other, and draws so as to coincide the corresponding anchor point P_Aand part point P_P. If drawing is performed as above, the illustration image for the right eye is drawn on the base image as shown in FIG. 15B.

[0187] Furthermore, at this time, since the skin area has already been drawn as the base image, only pixels corresponding to the illustration image are drawn over. In addition, the part point P_Pmay be a point that coincides with a detected feature point, and may be a point set different from the feature point.

[0188] Returning to the flowchart of FIG. 3, in Step S19, the part reference color extracted color adjustment unit 20C determines whether or not all illustration images such as the eyes, eyebrows, nose, mouth, forehead, and the like are to be drawn on the base image.

[0189] In Step S19, when all illustration images are determined not to be drawn, the process returns to Step S17, and the part reference color extracted color adjustment/drawing process from Steps S17 to S19 described above is repeated.

[0190] In other words, by repeating the part reference color extracted color adjustment/drawing process, the eyebrows part area (point (8, 15)-point (57, 22)), the nose part area (point (21, 31)-point (44, 45)), the mouth part area (point (18, 46)-point (47, 59)), and the forehead part area (point (1, 1)-point (64, 14) each are extracted in addition to the eye part area defined in the part area defining information of FIG. 13, the similarity to the model images is discriminated by using the K-class classifier 32 prepared for every part area, and an illustration image corresponding to the model image with the highest similarity is each selected. Then, the selected illustration image is subjected to the adjustment of the color adjustment setting based on the corresponding part reference color, further subjected to the image process according to the anchor point defining information, and drawn and arranged at a predetermined location on the base image.

[0191] On the other hand, when drawing of all illustration images is determined to end, all parts are turned out to be drawn, and therefore, the part reference color extracted color adjustment/drawing process ends.

[0192] The exemplification of the portrait image generated as above is as follows. In other words, FIGS. 16A to 16F are diagrams showing examples where the portrait image (modified image) generated by the drawing creation unit 19 is displayed on the screen of the display device 2.

[0193] As shown the pairs of input images (the left side) and portrait images (the right side) of FIGS. 16A to 16F, the illustration images corresponding to each part area of the face image included in the input images of the left side are selected individually by executing the portrait image generation process, and after the adjustment of the color adjustment setting is performed for the selected illustration images, the images are drawn on the base image and the portrait images of the right side are each generated.

[0194] In other words, since the color of the face skin, the color of the hair, and the colors of each part such as the eyes and lip are appropriately adjusted in the portrait images, and the color reflecting the person of the face image in the input image is formed, the image looks like the original face image, and as a result, the quality of the portrait image is improved.

[0195] In addition, when the reference color is acquired from the input image, clustering is performed as a pre-process by a method of k-means algorithm or the like, and a color of an area that belongs to the same cluster is used to acquire the reference color with avoiding a place with a color other than the skin color such as surroundings of the eyes, lip, or the like, for example, in the case of the skin color. As a result, since a color that is more suitable for the objective can be obtained, the color of the face skin, the color of the hair, and the color of each part such as the eyes, lip, or the like can be turned into a suitable color that has no awkwardness in the portrait image.

[0196] In addition, since the portrait image is generated from the illustration images selected based not on the similarity between the part images in each part area and the illustration images but on the similarity between the part images and the model images, the illustration image paired with the model image most similar to each part image of the input image is directly selected from the view of the face image, and the portrait image having the features of the face image can be generated. In other words, it is difficult to directly have the similarity between the part image and the illustration image, but in the embodiment, the model image and the illustration image are paired in advance, the similarity between the part image and the model image is calculated, and the illustration image paired with the model image that is most similar to the part image is selected.

[0197] As above, the portrait image generation process is executed by the portrait image generating device 1 of FIG. 1.

2. SECOND EMBODIMENT

[0198] Next, a second embodiment of the invention will be described with reference to FIGS. 17 to 20B.

[Adjustment Method of Color Adjustment Setting Using Fact Attribute Information]

[0199] Incidentally, in the portrait image, if, for example, black color is used for the skin color of a man and white color is used for the skin color of a woman, there is a case where the impression of the portrait image improves. In addition, there are various elements for improving the impression of a portrait image such as using pink color for the skin color of a smiling face, and aqua color for the skin color of a nervous face, for example.

[0200] In order to perform adjustment of color adjustment setting as above, for example, a method can be considered in which attributes of a face image such as man/woman attribute pertaining to the gender of a person, facial expression attribute pertaining to facial expressions are recognized, and the color adjustment setting is adjusted using the recognition result. Thus, as the second embodiment, an adjustment method of color adjustment setting using face attribute information will be described.

[Composition Example of Portrait Image Generating Device]

[0201] FIG. 17 is a diagram showing the composition of another embodiment of the portrait image generating device to which the present invention is applied.

[0202] The portrait image generating device 1 of FIG. 17 has a difference in that the device is provided with a face attribute recognition unit 21 for recognizing face attribute information from a face image, in comparison to the portrait image generating device 1 of FIG. 1. Furthermore, in FIG. 17, the same portions as in FIG. 1 are given with the same reference numerals, and description on portions with the same process will be appropriately omitted.

[0203] The face attribute recognition unit 21 is supplied with information pertaining to the face image in the face area from the face detection unit 12. The face attribute recognition unit 21 performs a predetermined image process for recognizing face attributes with respect to the face image, and recognize the face attributes.

[0204] The recognition result of the face attributes is supplied to the color adjustment unit 18 as face attribute information. As the face attribute, for example, the gender of a person, race, facial expression, presence of glasses, and the like are discriminated, and recognized.

[0205] The color adjustment unit 18 adjusts a reference color to a color for a portrait image according to a grayscale converting condition based on the face attribute information supplied from the face attribute recognition unit 21, and supplies the information obtained from the adjustment of the color adjustment setting to the drawing generation unit 19.

[0206] In other words, the skin color adjustment unit 18A adjusts the skin reference color to the skin color used in the portrait image with the face attribute information. In the same manner, the hair color adjustment unit 18B adjusts the hair reference color to the hair color used in the portrait image with the face attribute information. In addition, the illustration image color adjustment unit 18C adjusts the color of a corresponding illustration image to the color for the portrait image based on each part reference color and the face attribute information.

[0207] The drawing creation unit 19 fills the area of the union of the face outline area and the hair area with the skin color subjected to the adjustment of the color adjustment setting using the face attribute information, and further, fills the hair area with the hair color subjected to the adjustment of the color adjustment setting using the face attribute information thereon to draw a base image. Furthermore, the drawing creation unit 19 performs a predetermined image process for the illustration image subjected to the adjustment of the color adjustment setting using the face attribute information, and draws and arranges the thus-obtained illustration image on the base image to generate a portrait image.

[0208] The portrait image generating device 1 of FIG. 17 is constituted as above.

[Description on Portrait Image Generation Process]

[0209] Next, a portrait image generation process executed by the portrait image generating device 1 of FIG. 17 will be described.

[0210] Furthermore, the portrait image generation process executed by the portrait image generating device 1 of FIG. 17 is basically the same as the portrait image generation process of FIG. 3 described above, but has a difference in that the color adjustment setting is adjusted using the face attribute information in the skin reference color extracted color adjustment process (the process of Step S14), the hair reference color extracted color adjustment process (the process of Step S15), and the part reference color extracted color adjustment process (the process of Step S17). Therefore, herein, the skin reference color extracted color adjustment process corresponding to the process of Step S14 of FIG. 3 will be mainly described with reference to the flowchart of FIG. 18.

[0211] In Steps S91 and S92, the face outline area is determined by the face outline area determination unit 14 and the skin reference color is extracted by the skin reference color extraction unit 17A in the same manner as in Steps S31 and S32 of FIG. 4.

[0212] In Step S93, the face attribute recognition unit 21 performs a predetermined image process for recognizing face attributes for the face image, and acquires face attribute information that is thus obtained. The face attribute information is supplied to the skin color adjustment unit 18A.

[0213] In Step S94, the skin color adjustment unit 18A adjusts the skin reference color according to grayscale converting conditions based on the acquired face attribute information, and converts the reference color into the skin color used in the portrait image.

[0214] Herein, the adjustment of the color adjustment setting using the face attribute information will be described in detail with reference to FIG. 19.

[0215] FIG. 19 shows that the color range of the reference color shown by [XImin, XImax] is converted into [XCmin, XCmax] as the color range in the portrait image, but as shown by arrows in the drawing, values of XCmin and XCmax are adjusted according to score values obtained from the face attribute information.

[0216] To be more specific, as an example of a method of adjusting the color adjustment setting using the face attribute information, the description of the method for adjusting the skin color using the face attribute information indicating the gender of a person is as follows.

[0217] Herein, in man/woman attribute recognition of a face image, a score S is assumed to be obtained in the range of [0.0 (man), 1.0 (woman)]. If the color range in the portrait image when the man/woman attribute score S is 0.0 (maximum man score) is set to [XCFmin, XCMmax] (X is any one of RGB), and the color range in the portrait image when the man/woman attribute score S is 1.0 (maximum woman score) is set to [XCFmin, XCFmax], the color range [XCmain, XCmax] in the portrait image by the following Formula (3) can be obtained from the man/woman attribute score S.

RCmax=(RCFmax−RCMmax)*S+RCMmax

RCmin=(RCFmin−RCMmin)*S+RCMmin

GCmax=(GCFmax−GCMmax)*S+GCMmax

GCmin=(GCFmin−GCMmin)*S+GCMmin

BCmax=(BCFmax−BCMmax)*S+BCMmax

BCmin=(BCFmin−BCMmin)*S+BCMmin (3)

[0218] By using thus-obtained [XCmin, XCmax], colors [R, G, B] to be used in the portrait image are obtained.

[0219] FIGS. 20A and 20B show examples of specific values necessary when the color adjustment setting for the skin color is adjusted by using scores of man/woman attributes and facial expression attributes as face attribute information.

[0220] FIG. 20A shows an example of values of the color adjustment setting by man/woman attributes, and while the base setting is [RCmin, RCmax]=[217, 255], [GCmin, GCmax]=[191, 231], and [BCmin, BCmax]=[172, 212], the case where man/woman attribute scores are used is as follows. So to speak, on one hand, when the man/woman attribute is of woman, the setting is [RCmin, RCmax]=[227, 255], [GCmin, GCmax]=[214, 254], [BCmin, BCmax]=[205, 245], on the other hand, when the man/woman attribute is of a man, the setting is [RCmin, RCmax]=[197, 237], [GCmin, GCmax]=[142, 182], and [BCmin, BCmax]=[101, 141].

[0221] The average color expressed by those RGB values is shown in the field of “average color” of FIG. 20A, but it is obvious that the average color of a woman is white, and the average color of a man is black, in comparison to the average color of the base setting. Accordingly, since the black color is used for the skin color of a man, and the white color is used for the skin color of a woman, the impression of the portrait image can be improved.

[0222] FIG. 20B shows an example of values of the color adjustment setting by the facial expression attributes, but while the base setting is [RCmin, RCmax]=[217, 255], [GCmin, GCmax]=[191, 231], and [BCmin, BCmax]=[172, 212], the case where facial expression attribute scores are used is as follows. So to speak, on one hand, when the facial expression attribute is of a smiling face, the setting is [RCmin, RCmax]=[220, 255], [GCmin, GCmax]=[171, 211], [BCmin, BCmax]=[170, 210], on the other hand, when the facial expression attribute is of a depressed face, the setting is [RCmin, RCmax]=[177, 217], [GCmin, GCmax]=[212, 252], and [BCmin, BCmax]=[213, 253].

[0223] The average color expressed by those RGB values is shown in the field of “average color” of FIG. 20B, but it is obvious that the average color of a smiling face is pink, and the average color of a depressed face is aqua, in comparison to the average color of the base setting. Accordingly, since the pink color is used for the skin color of a smiling face, and the aqua color is used for the skin color of a nervous face, the impression of the portrait image can be improved.

[0224] As such, the adjustment of the color adjustment setting using the face attribute information is performed.

[0225] In other words, if the color of the input image is used as it is to generate the portrait image, the color may be unnatural depending on the face attributes or the ambiance (for example, the gender, the facial expression, or the like) of the portrait image, and a color to be used is different. Therefore, in the second embodiment, in order to avoid the color being unnatural in the portrait image, the range of a color to be used in the portrait image is determined in advance, and a color to be used according to color information of the input image and face attributes is determined within the range.

[0226] Accordingly, for example, man/woman attributes pertaining to the gender of a person are acquired as face attribute information from the input image, and the color adjustment setting can be changed according to the face attribute, for example, by making the skin color be white in case of a woman.

[0227] Hereinabove, the skin reference color extracted color adjustment process has been described, but the adjustment of the color adjustment setting using the face attribute information is performed in the same manner as in the hair reference color extracted color adjustment process (the process of Step S15) and the part reference color extracted color adjustment process (the process of Step S17) as described above. Since the adjustment of the color adjustment setting using the face attribute information performed in the hair reference color extracted color adjustment process and the part reference color extracted color adjustment process is the same as the adjustment of the color adjustment setting using the face attribute information performed in the skin reference color extracted color adjustment process described above, the description thereof will not be repeated.

[0228] In addition, in the same manner as in the above-described method, the color of the portrait image can be adjusted by using the face attribute information such as the race, age, and the like obtained from the face image. For example, face attribute information regarding the race is acquired from the input image, and the eyes can be painted with blue color if, for example, the race is turned out to be a westerner. Furthermore, when such face attribute information is obtained from a monochrome face image, a colorful portrait image is generated from the monochrome face image (input image).

[0229] Furthermore, herein, an example where the value of the color adjustment setting is consecutively changed according to score values of man/woman attribute recognition has been described, but color adjustment setting with a fixed value may be used, for example, for a man when the man/woman attribute score is less than 0.5, and for a woman when the score is 0.5 or more.

[0230] As such, the portrait image generation process is executed by the portrait image generating device 1 of FIG. 17.

3. THIRD EMBODIMENT

[0231] Next, a third embodiment of the present invention will be described with reference to FIGS. 21A to 25.

[Method of Adjusting Color Adjustment Setting According to Capturing Conditions During Capturing]

[0232] Incidentally, if capturing conditions (for example, an illumination state, or the like) of a captured image are different, the extracted reference color is largely different for the face image of the same person.

[0233] FIGS. 21A to 21B show images captured indoors and outdoors and the relationship between a skin color area (face outline area) and the skin reference color of faces in those images.

[0234] When the skin reference color ([R, G, B]=[131, 62, 32]) extracted from the image captured indoors in FIG. 21A is compared to the skin reference color ([R, G, B]=[175, 138, 132]) extracted from the image captured outdoors in FIG. 21B, it is notable that the indoor image is darker overall than the outdoor image.

[0235] Originally, it is desirable that a similar reference color is extracted even if capturing conditions are different in images of a same person. In order to perform the adjustment of such color adjustment setting, a method can be considered in which color adjustment setting is adjusted by using information regarding capturing conditions obtained during capturing (for example, illumination information regarding an illumination state). Thus, as the third embodiment, a method of adjusting the color adjustment setting using illumination information during capturing will be described.

[Composition Example of Portrait Image Generation Device]

[0236] FIG. 22 is a diagram showing still another composition of an embodiment of the portrait image generating device to which the present invention is applied.

[0237] The portrait image generating device 1 of FIG. 22 has a difference in that there is provided an illumination condition recognition unit 22 for recognizing an illumination state from the input image, in comparison to the portrait image generating device 1 of FIG. 1. Furthermore, in FIG. 22, the same portions as those of FIG. 1 will be given with the same reference numerals, and description on portions for the same process will be appropriately omitted.

[0238] The illumination condition recognition unit 22 is supplied with an input image from the target image acquisition unit 11. The illumination condition recognition unit 22 performs a predetermined image process for recognizing the illumination state for the input image supplied from the target image acquisition unit 11 to recognize the illumination state.

[0239] The recognition result of the illumination state is supplied to the color adjustment unit 18 as illumination information. It is discriminated and recognized, for example, whether the input image has been captured outdoors or indoors as the illumination state.

[0240] The color adjustment unit 18 adjusts the reference color to the color for the portrait image according to grayscale converting conditions based on the illumination information from the illumination condition recognition unit 22, and supplies the information obtained by the adjustment of the color adjustment setting to the drawing creation unit 19.

[0241] In other words, the skin color adjustment unit 18A adjusts the skin reference color to the skin color to be used in the portrait image based on the illumination information. In the same manner, the hair color adjustment unit 18B adjusts the hair reference color to the hair color to be used in the portrait image based on the illumination information. In addition, the illustration image color adjustment unit 18C adjusts the color of the corresponding illustration image to the color for the portrait image based on each part reference color and the illumination information.

[0242] The drawing creation unit 19 fills the area of the union of the face outline area and the hair area with the skin color subjected to the adjustment of the color adjustment setting using the illumination information, and fills the hair area with the hair color subjected to the adjustment of the color adjustment setting using the illumination information thereon, to draw a base image. The drawing creation unit 19 further performs a predetermined image process for the illustration image subjected to the adjustment of the color adjustment setting using the illumination information, draws and arranges the thus-obtained illustration image on the base image, and generates the portrait image.

[0243] The portrait image generating device 1 of FIG. 22 is constituted as above.

[Description on Portrait Image Generation Process]

[0244] Next, a portrait image generation process executed by the portrait image generating device 1 of FIG. 22 will be described.

[0245] Furthermore, the portrait image generation process executed by the portrait image generating device 1 of FIG. 22 is basically the same as the portrait image generation process of FIG. 3 described above, but has a difference in that the color adjustment setting is adjusted using the illumination information in the skin reference color extracted color adjustment process (the process of Step S14), the hair reference color extracted color adjustment process (the process of Step S15), and the part reference color extracted color adjustment process (the process of Step S17). Therefore, herein, the skin reference color extracted color adjustment process mainly corresponding to the process of Step S14 of FIG. 3 will be described with reference to the flowchart of FIG. 23.

[0246] In Steps S111 and S112, the face outline area is determined by the face outline area determination unit 14, and the skin reference color is extracted by the skin reference color extraction unit 17A in the same manner as in Steps S31 and S32 of FIG. 4.

[0247] In Step S113, the illumination condition recognition unit 22 performs a predetermined image process for recognizing the illumination state for the input image, and acquires thus-obtained illumination information. The illumination information is supplied to the skin color adjustment unit 18A.

[0248] In Step S114, the skin color adjustment unit 18A adjusts the skin reference color according to grayscale converting conditions based on the acquired illumination information, and converts the color to the skin color to be used in the portrait image.

[0249] Herein, the adjustment of the color adjustment setting using the illumination information will be described in detail with reference to FIG. 24.

[0250] FIG. 24 shows that the color range of the reference color indicated by [XImin, XImax] is converted to [XCmin, XCmax] as the color range in the portrait image, but as shown by arrows in the drawing, values of XImin and XImax are adjusted according to score values obtained from the illumination information.

[0251] To be more specific, as an example of a method of adjusting the color adjustment setting using the illumination information, the description of a method using outdoor information is as follows.

[0252] Herein, in indoor/outdoor recognition of the input image, it is assumed that a score S is obtained in the range of [0.0 (indoor), 1.0 (outdoor)]. If the color range of the reference color when an indoor/outdoor recognition score S is 0.0 (maximum indoor score) is set to [XIImin, XIImax] (X is any one of RGB), and the color range of the reference color when an indoor/outdoor recognition score S is 1.0 (maximum outdoor score) is set to [XIOmin, XIOmax], the color range of the reference color [XImin, XImax] by the following Formula (4) can be obtained from the indoor/outdoor recognition score S.

RImax=(RIOmax−RIImax)*S+RIImax

RImin=(RIOmin−RIImin)*S+RIImin

GImax=(GIOmax−GIImax)*S+GIImax

GImin=(GIOmin−GIImin)*S+GIImin

BImax=(BIOmax−BIImax)*S+BIImax

BImin=(BIOmin−BIImin)*S+BIImin (4)

[0253] By using [XImin, XImax] obtained as above, colors [R, G, B] to be used in the portrait image are obtained.

[0254] FIG. 25 shows an example of specific values necessary when the color adjustment setting for the skin color is adjusted by using the indoor/outdoor recognition scores.

[0255] In the outdoor case, the setting is [RImin, RImax]=[155, 195], [GImin, Gimax]=[118, 158], and [BImin, BImax]=[112, 152], on the other hand, in the indoor case, the setting is [RImin, RImax]=[111, 151], [GImin, GImax]=[42, 82], and [BImin, BImax]=[12, 52]. The average color expressed by those RGB values is shown in the field of “average color” of FIG. 25, but it is obvious that the average color of the indoor setting is darker overall than the average color of the outdoor setting. Accordingly, a similar reference color is extracted even if the capturing conditions are different in the images captured indoors and outdoors in FIGS. 21A and 21B described above.

[0256] The adjustment of the color adjustment setting using the illumination information is performed as above.

[0257] In other words, if the color of the input image is used as it is to generate the portrait image, the color may be unnatural depending on the capturing conditions (for example, the illumination state or the like). Therefore, in the third embodiment, in order to avoid the color being unnatural in the portrait image, the range of a color to be used in the portrait image is determined in advance, and a color to be used according to color information of the input image and the illumination state or the like is determined within the range.

[0258] Accordingly, the illumination state during capturing, for example, capturing of a dark room, use of a flash, or the like can be presumed from image recognition of the input image, and the information can be reflected in the color adjustment setting. For example, if there is an image captured in a dark place, the illumination is made to be bright, if necessary, to reflect on the color for skin, or the like.

[0259] Hereinabove, the skin reference color extracted color adjustment process has been described, but the adjustment of the color adjustment setting using the illumination information is performed in the same manner as in the hair reference color extracted color adjustment process (the process of Step S15) and the part reference color extracted color adjustment process (the process of Step S17) as described above. Since the adjustment of the color adjustment setting using the illumination information performed in the hair reference color extracted color adjustment process and the part reference color extracted color adjustment process is the same as the adjustment of the color adjustment setting using the illumination information performed in the skin reference color extracted color adjustment process described above, the description thereof will not be repeated.

[0260] Furthermore, the color of the portrait image can be adjusted by using parameters such as information on lens aperture, or capturing with a flash added during capturing, for example, Exif (Exchangeable Image File Format), or the like, in the same manner as in the method described above.

[0261] Furthermore, as capturing conditions, information regarding weathers such as sunny, cloudy, rainy, or the like and information regarding time such as morning, afternoon, night (a night view), or the like can be used in addition to indoor/outdoor information as described above.

[0262] The portrait image generation process by the portrait image generating device 1 of FIG. 22 is executed as above.

4. FOURTH EMBODIMENT

[0263] Next, a fourth embodiment of the present invention will be described with reference to FIGS. 26 to 31.

[Method of Adjusting Color Adjustment Setting by User]

[0264] In the example described above, a method is described in which recognition of capturing conditions such as face attributes, the illumination state, or the like is automatically performed, and the color adjustment setting is performed using the scores. The information used in the color adjustment setting, for example, the fact attribute information such as the gender of a person, the facial expression, or the like, or the illumination information such as indoor or outdoor capturing may be set by a user. In other words, in such a case for example, a face attribute mode pertaining to the face attribute information and a illumination state mode pertaining to the illumination information are switched by a user. Hence, hereinbelow, as the fourth embodiment, a method of adjusting the color adjustment setting performed according to a mode set by a user will be described.

[Composition Example of Portrait Image Generating Device]

[0265] FIG. 26 is a diagram showing still another composition of an embodiment of the portrait image generating device to which the present invention is applied.

[0266] The portrait image generating device 1 of FIG. 26 has a difference in that there is provided a mode setting unit 23, in comparison to the portrait image generating device 1 of FIG. 1. In FIG. 26, the same portions as in FIG. 1 is given with the same reference numerals, and description on portions for the same process will be appropriately omitted.

[0267] The mode setting unit 23 sets various modes according to the operation of a user. Information regarding the set mode is supplied to the color adjustment unit 18. As such as mode, for example, a face attribute mode regarding the face attribute information such as the gender of a person, the facial expression, and the like, or an illumination state mode regarding the illumination information such as indoor or outdoor capturing is set.

[0268] The color adjustment unit 18 adjusts the reference color to a color for the portrait image according to the grayscale converting conditions based on the information regarding the mode supplied from the mode setting unit 23, and supplies the information obtained from the adjustment of the color adjustment setting to the drawing creation unit 19.

[0269] In other words, the skin color adjustment unit 18A adjusts the skin reference color to the skin color to be used in the portrait image based on the information regarding the mode. In the same manner, the hair color adjustment unit 18B adjusts the hair reference color to the hair color to be used in the portrait image based on the information regarding the mode. In addition, the illustration image color adjustment unit 18C adjusts the color of the corresponding illustration image to a color for the portrait image based on each part reference color and the information regarding the mode.

[0270] The drawing creation unit 19 fills the area of the union of the face outline area and the hair area with the skin color subjected to the adjustment of the color adjustment setting using the mode information, and fills the hair area with the hair color subjected to the adjustment of the color adjustment setting using the mode information thereon, to draw a base image. After that, the drawing creation unit 19 further performs a predetermined image process for the illustration image subjected to the adjustment of the color adjustment setting using the mode information, draws and arranges the thus-obtained illustration image on the base image, and generates the portrait image.

[0271] The portrait image generating device 1 of FIG. 26 is constituted as above.

[Description on Portrait Image Generation Process]

[0272] Next, a portrait image generation process executed by the portrait image generating device 1 of FIG. 26 will be described with reference to the flowchart of FIG. 27.

[0273] In Step S131A, an input image including a face image such as a captured image is acquired by the target image acquisition unit 11 as in Step S11 of FIG. 3.

[0274] The mode setting unit 23 receives setting of a mode by a user in Step S131B and sets a mode designated by the user in Step S131C.

[0275] For example, the user who recognized the input image sets a mode according to the target input image. After that, when the generation of a portrait image is designated at the mode set by the user, the process of Steps S132 and thereafter is executed. Basically in Steps S132 to S139, the same process as in Steps S12 to S19 of FIG. 3 is executed, but it is different that the adjustment of the color adjustment setting using the information regarding mode is performed in the skin reference color extracted color adjustment process (the process of Step S134), the hair reference color extracted color adjustment process (the process of Step S135), and the part reference color extracted color adjustment process (Step S137).

[0276] Therefore, herein, the skin reference color extracted color adjustment process corresponding to the process of Step S134 of FIG. 27 will mainly be described with reference to the flowchart of FIG. 28.

[0277] In Steps S151 and S152, the face outline area is determined by the face outline area determination unit 14, and the skin reference color is extracted by the skin reference color extraction unit 17A in the same manner as in Steps S31 and S32 of FIG. 4.

[0278] In Step S153, the skin color adjustment unit 18A acquires the information regarding mode set by the mode setting unit 23.

[0279] In Step S154, the skin color adjustment unit 18A adjusts the skin reference color according to the grayscale converting conditions based on the information regarding mode set by the user, and converts the color to the skin color to be used in the portrait image.

[0280] Herein, the adjustment of the color adjustment setting using the information regarding mode will be described in detail with reference to FIGS. 29A to 30B.

[0281] FIGS. 29A and 29B are diagrams describing the adjustment of the color adjustment setting when the gender of a person is set as a face attribute mode by a user.

[0282] FIGS. 29A and 29B show that the color range of the reference color indicated by [XImin, XImax] is converted to [XCmin, XCmax] as the color range in the portrait image, but values of XCmin and XCmax are adjusted according to the set mode. In other words, when a mode is within the color range in the portrait image, but the mode set for a woman is selected, the maximum grayscale value and the minimum grayscale value are increased together as shown in FIG. 29A, in comparison to the case where a mode set for a man of FIG. 29B is selected.

[0283] By using information regarding a mode for gender as above, when the mode set for a woman is selected by the user, the skin color of the portrait image can be close to white, on the other hand, when the mode set for a man is selected, the skin color of the portrait image can be close to black.

[0284] FIGS. 30A and 30B are diagrams describing the adjustment of the color adjustment setting when a capturing place such as indoor or outdoor is set as an illumination state mode by the user.

[0285] FIGS. 30A and 30B show that the color range of the reference color indicated by [XImin, XImax] is converted to [XCmin, XCmax] as the color range in the portrait image, but values of XImin and XImax are adjusted according to the set mode. In other words, when the mode in the color range of the reference color obtained from the input image but the mode set for outdoor capturing is selected, the maximum grayscale value and the minimum grayscale value are increased together, in comparison to a case where a mode set for indoor capturing is selected.

[0286] By using the information regarding mode for capturing place as above, for example, a similar reference color is extracted even if capturing conditions are different in the images captured indoors and outdoors.

[0287] FIG. 31 is a diagram showing an example of specific values of a color adjustment setting of a mode for gender and capturing place.

[0288] As shown in FIG. 31, as a default setting, the R value is converted from [RImin, RImax]=[90, 170] to [RCmin, RCmax]=[217, 255], the G value is converted from [GImin, GImax]=[60, 140] to [GCmin, GCmax]=[191, 231], and the B value is converted from [BImin, BImax]=[40, 120] to [BCmin, BCmax]=[172, 212].

[0289] As the mode for gender, when a mode set for a woman is selected, the R value is converted from [RImin, RImax]=[90, 170] to [RCmin, RCmax]=[227, 255], the G value is converted from [GImin, GImax]=[60, 140] to [GCmin, GCmax]=[214, 254], and the B value is converted from [BImin, BImax]=[40, 120] to [BCmin, BCmax]=[205, 245].

[0290] On the other hand, as the mode for gender, when the mode set for a man is selected, the R value is converted from [RImin, RImax]=[90, 170] to [RCmin, RCmax]=[197, 237], the G value is converted from [GImin, GImax]=[60, 140] to [GCmin, GCmax]=[142, 182], and the B value is converted from [BImin, BImax]=[40, 120] to [BCmin, BCmax]=[101, 141].

[0291] In addition, as the mode for capturing place, when the mode set for outdoor capturing is selected, the R value is converted from [RImin, RImax]=[155, 195] to [RCmin, RCmax]=[217, 255], the G value is converted from [GImin, GImax]=[118, 158] to [GCmin, GCmax]=[191, 231], and the B value is converted from [BImin, BImax]=[112, 152] to [BCmin, BCmax]=[172, 212].

[0292] On the other hand, as the mode for capturing place, when the mode set for indoor capturing is selected, the R value is converted from [RImin, RImax]=[111, 151] to [RCmin, RCmax]=[217, 255], the G value is converted from [GImin, GImax]=[42, 82] to [GCmin, GCmax]=[191, 231], and the B value is converted from [BImin, BImax]=[12, 52] to [BCmin, BCmax]=[172, 212].

[0293] As such, the adjustment of the color adjustment setting using the information regarding mode is performed.

[0294] In other words, if the color of the input image is used as it is to generate the portrait image, the color may be unnatural depending on the ambiance (for example, the gender, the facial expression, or the like) and capturing conditions (the illumination state or the like) of the portrait image, and a color to be used is different. Therefore, in the fourth embodiment, in order to avoid the color being unnatural in the portrait image, the range of a color to be used in the portrait image is determined in advance, and a color to be used according to the mode set by the user and the color information of the input image is determined within the range.

[0295] Hereinabove, the skin reference color extracted color adjustment process has been described, but the adjustment of the color adjustment setting using the information regarding mode is performed in the same manner as in the hair reference color extracted color adjustment process (the process of Step S135 of FIG. 27) and the part reference color extracted color adjustment process (the process of Step S137 of FIG. 27) as described above. Since the adjustment of the color adjustment setting using the information regarding mode performed in the hair reference color extracted color adjustment process and the part reference color extracted color adjustment process is the same as the adjustment of the color adjustment setting using the information regarding mode performed in the skin reference color extracted color adjustment process described above, the description thereof will not be repeated.

[0296] In addition, in the same manner as in the method described above, for example, a mode pertaining to facial expression attributes such as a smiling face, weather attributes such as sunny, cloudy, or the like, or time attributes such as a night view, or the like may be set by a user, and a plurality of color adjustment setting may be prepared for one kind of attribute. For example, a plurality of color adjustment settings regarding the attribute of the gender of a person may be prepared to be switched to one another.

[0297] The portrait image generation process by the portrait image generating device 1 of FIG. 26 is executed as above.

5. MODIFIED EXAMPLE

[0298] With the combination of the second to the fourth embodiments of the present invention, both of the face attribute information and the illumination information may be used to perform the adjustment of the color adjustment setting, and further, the information may be selected by a user as a mode.

[0299] In addition, in the above description, description has been provided for the example where the color adjustment setting is adjusted for all the face outline area, the hair area, and each part area, but the color adjustment setting may be adjusted only for a part of the areas, for example the color adjustment setting is adjusted only for the skin reference color in the face outline area. Furthermore, there is a method of determining those face outline area, hair area or part area, but the method is not limited thereto, and an existing method can be used.

[0300] In addition, in this embodiment, the color of the skin, the hair or the like of the portrait image may be a plurality of colors, not a single color, and gradation may be applied thereto. For the adjustment of a parameter for setting gradation, the above-described face attributes, capturing conditions and the like may be used. Furthermore, the adjustment of the color adjustment setting may be performed according to a predetermined theme for the portrait image. The theme of a portrait image is, for example, of an abstract icon, of a drama, or the like, and the color range of a color to be used in a portrait image is supposed to be determined by combining, for example, the theme of a portrait and each part to be used.

[0301] In addition, in the above description, an example where each part of a portrait image is selected from illustration images and model images corresponding to part correspondence information, but it is not limited to the method, and an image of each part may be selected and drawn by other methods. Furthermore, the presence of an eyeglass may be determined by the face attribute recognition unit 21 as a face attribute. Then, when the face attribute recognition unit 21 determines there is an eyeglass, the drawing creation unit 19 draws the image of the frame portion of the eyeglass on a predetermined position on a base image according to anchor point defining information.

[0302] Furthermore, in the above embodiments, the portrait image generating device 1 and the display device 2 are described as separate devices, but it can be understood that a display unit as a processing unit corresponding to the display device 2 is provided in the portrait image generating device 1. In this case, the drawing creation unit 19 causes the screen of the display unit to display a generated portrait image.

[0303] Moreover, the portrait image generated by the drawing creation unit 19 can be not only displayed on the screen of the display device 2, but also compressed in a predetermined compression format such as JPEG (Joint Photographic Experts Group), and stored in a predetermined recording medium as a file.

6. COMPOSITION EXAMPLE OF PRE-PROCESSING DEVICE

[Composition Example of Pre-Processing Device]

[0304] Next, as a prior process necessary for the portrait image generating device 1 of FIG. 1 to execute the portrait image generation process, description sill be provided on a learning process for generating the K-class classifier 32, a generation process for generating a model image, a setting process for setting various information such as part correspondence information, anchor point defining information, and the like. Such pre-processes are executed by a pre-processing device.

[0305] FIG. 32 is a diagram showing an example of the composition of the pre-processing device.

[0306] As shown in FIG. 32, the pre-processing device 101 is constituted by a learning system 111 for performing a learning process, a generation system 112 for performing a generation process, a setting system 113 for performing a setting process.

[0307] The learning system 111 includes a learning sample acquisition unit 121, an image pre-processing unit 122, a learning unit 123, and a K-class classifier 32. The generation system 112 includes an image-for-generation acquisition unit 124, an image pre-processing unit 125, a K-class classifier 32, a generation unit 126, and a database 35. The setting system 113 includes an illustration image acquisition unit 127, a setting unit 128, and a database 35.

[0308] Furthermore, in FIG. 32, portions corresponding to those of FIG. 1 are given with the same reference numerals, and the description thereof will be appropriately omitted. In other words, FIG. 32 illustrates the K-class classifier 32 and the database 35 same as in FIG. 1, but in the pre-processing device 101, the K-class classifier 32 constituting the portrait image generating device 1 is formed, and further, a process for setting and registering various information in the database 35 is performed.

[0309] First of all, the learning system 111 will be described.

[0310] The learning sample acquisition unit 121 acquires face images (hereinafter, referred to as sample image) prepared with a variety of K-number of sample persons (K=1, 2, . . . , K), and supplies the images to the image pre-processing unit 122.

[0311] The image pre-processing unit 122 detects a face area from a sample image supplied from the learning sample acquisition unit 121, and performs a re-sizing process to a predetermined image size. In the re-sizing process, the area is re-sized to a size in accordance with an image size defined in the above-described part area defining information (FIG. 13). Then, the image pre-processing unit 122 cuts out a part area from the face area subjected to the re-sizing according to part area defining information, and supplies the area to the learning unit 123.

[0312] The learning unit 123 obtains a image feature amount of a part image in the part area supplied from the image pre-processing unit 122, learns a plurality of weak classifiers by AdaBoost ECOC (Error Correct Output Coding), and generates the K-class classifier 32 formed by the plurality of weak classifiers.

[0313] Next, the generation system 112 will be described.

[0314] The image-for-generation acquisition unit 124 acquires a plurality of face images (hereinafter, referred to as images-for-generation) extracted at random for generating model images, and supplies the images to the image pre-processing unit 125.

[0315] The image pre-processing unit 125 cuts out a part area by re-sizing an image-for-generation from the image-for-generation acquisition unit 124 and supplies the area to the K-class classifier 32, in the same manner as in the process of re-sizing the sample image in the image pre-processing unit 122.

[0316] The K-class classifier 32 obtains a K-dimensional score vector corresponding to the part image in the part area supplied from the image pre-processing unit 125, and supplies the score vector to the generation unit 126.

[0317] The generation unit 126 classifies the part image into N-number (N=1, 2, . . . , N) prototypes based on the K-dimensional score vector supplied from the K-class classifiers 32, and generates a model image by obtaining an average image of a part image group that belong to each prototype. The model image is registered in the database 35.

[0318] Next, the setting system 113 will be described.

[0319] The illustration image acquisition unit 127 acquires illustration images for each part and supplies them to the setting unit 128.

[0320] The setting unit 128 sets part correspondence information by corresponding the illustration images supplied from the illustration image acquisition unit 127 and the model image registered in the database 35, and registers the information in the database. In addition, the setting unit 128 sets anchor point defining information and registers the information in the database 35.

[0321] As above, the pre-processing device 101 is constituted.

[Description on Pre-Process]

[0322] Next, a pre-process executed by the pre-processing device 101 of FIG. 32 will be described with reference to the flowchart of FIG. 33.

[0323] In Step S251, the learning system 111 of the pre-processing device 101 generates the K-class classifier 32 by performing a learning process.

[0324] In Step S252, the generation system 112 of the pre-processing device 101 generates a model image by performing a generation process and registers the image in the database 35.

[0325] In Step S253, the setting system 113 of the pre-processing device 101 sets each of part correspondence information and anchor point defining information by performing a setting process and registers the information in the database 35.

[0326] Hereinbelow, Steps S251 to S253 of FIG. 33 mentioned above will be described in detail.

[Details of Learning Process]

[0327] FIG. 34 is a flowchart describing the learning process in detail.

[0328] In Step S271, the learning sample acquisition unit 121 acquires sample images prepared with a variety of K-number of sample persons.

[0329] The image pre-processing unit 122 detects the face area from the sample images acquired by the learning sample acquisition unit 121 in Step S272, then performs re-sizing for the detected face area to a predetermined image size based on the part area defining information (a process of Step S273), and cuts out a part area from the face area subjected to re-sizing (a process of Step S274).

[0330] In the re-sizing process, the area is re-sized to the size of, for example, 64×64 pixels in accordance with the image size defined in the part area defining information of FIG. 13 described before, and for example, an eye part area (point (10, 23)-point (55, 30)) is cut out.

[0331] Then, in the learning system 123, an image feature amount of a part image is obtained in Step S275, and the K-class classifier 32 is generated by learning a plurality of weak classifiers by AdaBoost ECOC in Step S276.

[0332] Furthermore, the image feature amount of the sample images (part images) is used as the standard of classification in the plurality of weak classifiers for discriminating whether or the each class belongs thereto. As the image feature amount, for example, the Pixel Difference Feature (PixDif Feature) that the present inventors have already suggested can be used.

[0333] The Pixel Difference Feature is disclosed in “Learning of Real-Time Arbitrary Posture Face Detector using Pixel Difference Feature” by Sabe and Hidai, Proc. of 10^thImage Sensing Symposium, pp. 547-552, 2004, and Japanese Unexamined Patent Application Publication No. 2005-157679.

[0334] FIG. 35 is a diagram illustrating the K-class classifier 32 generated by the learning unit 123 in detail.

[0335] As shown in FIG. 35, a variety of sample images are acquired for K-number of sample persons including “Mr. A”, “Mr. B”, “Mr. C”, . . . and the like, and after their face areas are re-sized, each part area is cut out from them as described in the process of Steps S271 to S274 of FIG. 34.

[0336] In the learning unit 123, the image feature amount for each part image of the face images of the K-number of sample persons obtained as above is calculated by the pixel difference feature, a plurality of weak classifiers are learned by AdaBoost ECOC using the image feature amount, and the K-class classifier 32 is generated as a multi-class classifier.

[0337] In the K-class classifier 32 generated as above, a score that indicates how much the input face image is similar to the K-number of sample persons is calculated. Furthermore, the similarity is deemed to get higher as the value of a score gets higher. Accordingly, as described before, the K-class classifier 32 outputs a K-dimensional score vector that is a score of a K-dimension.

[0338] As such, a score space formed of the K-dimension is obtained for each part, and for example, if an input par images is of the same person, the distance in the K-dimensional score space is deemed to be close. Then, when the input face image is of “Mr. X” who is an unknown person, each part of “Mr. X” can be quantified with numbers indicating how much the parts are similar to each part of the sample persons including “Mr. A”, “Mr. B”, “Mr. C”, . . . , and the like. Accordingly, the similarity of each part can be determined with the distance in the K-dimensional score space for each part.

[0339] To be more specific, the process shown in FIG. 36 is performed. In other words, by calculating the difference (I₁−I₂) of pixel values (brightness values) I_Iand 1₂of two pixels on the face image (Face image A), the pixel Difference Feature (PixDif Feature) is obtained. In a binary classification weak classifier h(x) corresponding to the combination of two pixels, true (+1) or false (−1) is classified based on the pixel difference feature (I₁−I₂) and a threshold value Th, as shown in the following Formula (5).

h(x)=−1 if I₁−I₂≦Th

h(x)=+1 if I₁−I₂>Th (5)

[0340] When the h(x) obtained by Formula (5) is compared to the value in the K-th column (+1 or −1) scored in the ECOC bit (ECOC table of the 1^stline and K-th column) defined for each class, and the classification result coincides with the ECOC bit, the score of the class increases by the degree of reliability α, and on the contrary, when the classification result does not coincide with the ECOC bit, the score decreases by the degree of reliability α.

[0341] By repeating the process by the number of the image feature amount, H(1), H(2), H(3), H(4), and H(5) can be obtained as the K-dimensional score vector (Score of K-Class).

[0342] Returning to FIG. 34, the learning unit 123 determines whether or not the generation of the K-class classifier 32 ends for all parts in Step S277.

[0343] When it is determined that the generation of the K-class classifier 32 does not end for all parts in Step S277, the process returns to Step S274, and the above-described generation process (the process from Steps S274 to S277) is repeated.

[0344] In other words, by repeating the generation process, each of the part areas including the eye part area, eyebrow part area, nose part area, mouth part area, and forehead part area is extracted from the face area that has been subjected to the re-sizing according to the part area defining information of FIG. 13, and each K-class classifier 32 is individually generated for those part areas.

[0345] Then, the K-class classifier 32 is obtained for each part area defined by the part area defining information of FIG. 13, and the learning process ends.

[0346] With the K-class classifier 32 for each part generated as above, the image feature amount of the input face image (part image) can be expressed by a K-dimensional score vector. In other words, for example, if sample persons are set to “Mr. A”, “Mr. B”, “Mr. C”, . . . , and the like, each part of “Mr. X” who is an unknown person can be quantified with numbers indicating how much the part is similar to each part of the sample persons including “Mr. A”, “Mr. B”, “Mr. C”, . . . , and the like.

[Details of Generation Process]

[0347] FIG. 37 is a flowchart describing a generation process in detail.

[0348] In Steps S291 to S294, the image pre-processing unit 125 detects the face area from an image for generation acquired by an image-for-generation acquisition unit 124 and a part area is cut out from the face area that has been subjected to re-sizing, in the same manner as in Steps S271 to S274 of FIG. 34. Furthermore, as the image for generation, for example, images that includes 10000 sheets of face images extracted at random is used.

[0349] A part image that is obtained as above is input to the K-class classifier 32 for each part. In the K-class classifier 32, the pixel difference feature of the input part image is classified by the threshold value, and a K-dimensional score vector is obtained by comparing to an ECOC bit defined for each class in Step S295.

[0350] In Step S296, the generation unit 126 performs clustering for a set of the K-dimensional score vector corresponding to the part image computed by the K-class classifier 32 using, for example, k-means algorithm on the K-dimensional score space, and divides the set into N-number of subsets. Accordingly, since the part image is classified into N-number of prototypes, the generation unit 126 generates models images by obtaining an average image of a part image group where each of the classified prototypes belong, and registers the image in the database 35.

[0351] FIG. 38 is diagram showing examples of model images for eyes.

[0352] As shown in FIG. 38, for example, as a cluster of eyes, when the part image for eyes is classified into 24 prototypes, an image that has the average of the part image groups where each prototype belong becomes a model image for eyes. In examples of FIG. 38, it can be understood that, particularly, only the shapes of the eyes are different for each model image of each prototype.

[0353] Returning to the flowchart of FIG. 37, in Step S297, the generation unit 126 determines whether or not the generation of the model image has been completed for all parts.

[0354] When the generation of the model image is determined to be completed for all parts in Step S297, the process returns to Step S294, and the above-described generation process (the process of Steps S294 to S297) is repeated.

[0355] In other words, by repeating the generation process, each part area including the eyebrow part area, the nose part area, the mouth part area, and the forehead part area in addition to the above-described eye part area is extracted from the face area that has been subjected to re-sizing according to the part area defining information of FIG. 13, N-sheets of model images are individually generated for the part areas, and registered in the database 35.

[0356] Then, N-sheets of model images for each part area defined by the part area defining information of FIG. 13 are obtained (it is not necessary to make the number of model images for each part area be the same), thereby the generation process ends.

[Setting Process]

[0357] FIG. 39 is a flowchart describing a setting process in detail.

[0358] In Step S301, the illustration image acquisition unit 127 acquires illustration images of each part. When illustration images for eyes are to be set, for example, illustration images with a variety of shapes of eyes are prepared out of parts for constituting a portrait image as shown in (a) to (1) of FIG. 40 as illustration images.

[0359] In Step S302, the setting unit 128 acquires model images registered in the database 35. The model images are generated by the generation system 112 and registered in the database 35 (the generation process of FIG. 37), when an illustration image for eyes is to be set, for example, a model images for eyes of FIG. 38 is acquired out of the model images generated for each part area and registered in the database 35.

[0360] In Step S303, the setting unit 128 sets part correspondence information to register in the database 35 by making the acquired illustration image correspond to the model image.

[0361] As methods of correspondence, there are two kinds the first of which is a method of correspondence by executing an image process for making a pair of a certain model image and an illustration image with the highest similarity from a group of candidate illustration images, and the second of which is a method in which a user instructs correspondence after visually recognizing a model image and selecting an illustration image that is seemed to be similar to the model image from a group of candidate illustration images.

[0362] If the correspondence is performed with either of the methods, for example, illustration images (FIG. 40) and models images (FIG. 38) correspond to each other as shown in FIG. 41. In the example of FIG. 41, each of illustration images of 6×4 in accordance with the appearance of eyes corresponds to each of model images of 6×4. Accordingly, the similarity can be calculated not based on geometric size or ratio of a part but based on patterns of the appearance of part images and model images, and similarity that further matches with subjective notions of human beings can be defined.

[0363] Furthermore, when the user performs visual correspondence, the setting unit 128 can intentionally make correspondence of a model image with a characteristic in the mouth with an illustration image with a characteristic mouth as shown in FIG. 42A. In addition, the setting unit 128 intentionally makes correspondence of a model image with a characteristic in the eyes with an illustration image with characteristic eyes as shown in FIG. 42B based on the instruction of the user.

[0364] As such, by assigning illustration images with more exaggerated facial expression than actual appearance to model image when part correspondence information is set, a portrait image having a characteristic of the face that matches with subjective notions of human beings (a portrait image in which a characteristic of the face is further emphasized) can be generated during the generation of the portrait image.

[0365] Returning to the flowchart of FIG. 39, in Step S304, the setting unit 128 sets anchor point defining information for each illustration image and registers the information in the database 35, for example, based on the instruction of the user.

[0366] As described before on the anchor point defining information with reference to FIGS. 15A and 15B, by setting the locations of each of the anchor points P_A1and P_A2to further inside and making the interval of the two points narrower than the part points P_P1and P_P2, when the two points are arranged at the corresponding part points P_P1and P_P2, since an image, for example, an illustration image for eyes, is drawn after enlargement, the eyes can be expressed to be larger even in the same illustrations. On the other hand, if the locations of each of the anchor points P_A1and P_A2are set to further outside, since the interval of the two points is wider than that of the part points P_P1and P_P2on the contrary, and an image, for example, an illustration image for eyes, is drawn after reduction, the eyes can be expressed to be smaller even in the same illustrations.

[0367] As such, since a part can be expressed in a variety of display forms in the same illustration images only by changing the locations of the anchor points, the number of prepared illustration images can be reduced, and modified part images having characteristics of each part can be provided.

[0368] Furthermore, the anchor points are not limited to two, but can be set to a number in accordance with the number of corresponding part points.

[0369] In Step S305, the setting unit 128 determines whether or not setting of the part correspondence information and the anchor point defining information has been completed for all parts.

[0370] When the setting of the part correspondence information and the anchor point defining information for all parts is determined not to be completed in Step S305, the process returns to Step S301, and the above-described setting process (the process of Steps S301 to S305) is repeated.

[0371] In other words, with the repetition of the setting process, the illustration images are individually paired with the model images generated for each part area, and the anchor points of the illustration images are individually set.

[0372] The correspondence information set in the setting process is registered in the database 35 as part correspondence information. Then, the part correspondence information and anchor point defining information for all parts are set, thereby ending the setting process.

[0373] As above, the pre-processing device 101 generates the K-class classifier 32 for each part, furthermore, the part correspondence information and the anchor point defining information are set and registered in the database 35 in advance, and thus, the portrait image generation device 1 using the K-class classifier 32 and the database 35 can generate a portrait image from an input image that includes a face image without the operation of a user.

[Other Example of Classification]

[0374] Up until now, model images are simply made to correspond to illustration images based on part correspondence information, but attribute information of a sample person may be assigned to a sample image as a class label and learned for each in the K-class classifier 32. With regard to the attribute information of a sample person, the same sample person is deemed to belong to the same attribute such as race, age, gender, presence of eyeglass, or the like.

[0375] By using the K-class classifier 32 where each kind of attribute information is learned as a class label, a part image and a model image can be described in further detail, in comparison to a case where the similarity of the part image and the model image is simply obtained. Therefore, the similarity of the part image and the model image can be obtained more accurately, and a proper illustration image can be selected.

[0376] Furthermore, in the embodiment, the portrait image generation device 1 of FIG. 1 is described as a separate device from the pre-processing device 101 of FIG. 32 but the devices can be understood as one constituted by a portrait image generation unit as one processing unit corresponding to the portrait image generation device 1 of FIG. 1 and a pre-processing unit as one processing unit corresponding to the pre-processing device 101 of FIG. 32. In this case, the K-class classifier 32 and the database 35 commonly constituting the devices are generated (set) by a pre-processing unit and used in a portrait image generation process by a portrait image generation unit.

[Description on Computer Applied with Present Invention]

[0377] Incidentally, a series of processes described above can be executed by hardware and by software. When the series of processes is executed by software, a program that makes up the software is installed in a computer incorporated into dedicated hardware or, for example, a general-purpose computer that can execute a variety of functions by installing various programs, or the like from a program recording medium.

[0378] FIG. 43 is a diagram showing an example of the composition of a personal computer in which the above-described series of processes is executed by programs. A CPU (Central Processing Unit) 211 executes various processes according to programs recorded in a ROM (Read Only Memory) 212 or a recording unit 218. A RAM (Random Access Memory) 213 appropriately stores programs executed by the CPU 211, data, and the like. The CPU 211, the ROM 212, and the RAM 213 are connected to one another by a bus 214.

[0379] The CPU 211 is also connected to an input/output interface 215 via the bus 214. The input/output interface 215 is connected to an input unit 216 including a microphone and the like, and an output unit 217 including a display, a speaker, and the like. The CPU 211 executes a variety of processes responding to commands input from the input unit 216. In addition, the CPU 211 outputs results of the processes to the output unit 217.

[0380] A storage unit 218 connected to the input/output interface 215 is constituted by, for example, a hard disk, and stores various data and programs executed by the CPU 211. A communication unit 219 communicates with external devices via a network such as the Internet, a local area network, or the like.

[0381] Furthermore, a program may be acquired via the communication unit 219, and stored in the storage unit 218.

[0382] A drive 220 connected to the input/output interface 215 drives a mounted removable medium 221 such as a magnetic disc, an optical disc, a magneto-optic disc, a semiconductor memory, or the like, and acquires a program or data recorded thereon. The acquired program or data is transferred to a storage unit 218 and stored if necessary.

[0383] As shown in FIG. 43, a program recording medium for storing a program installed in a computer and to be in an activated state by a computer includes the removable medium 221 that is a package medium such as a magnetic disc (including a flexible disc), an optical disc (including CD-ROM (Compact Disc-Read Only Memory) and DVD (Digital Versatile Disc)), a magneto-optic disc, a semiconductor memory or the like, the ROM 212 where a program is stored temporarily or permanently, or a hard disk constituting the storage unit 218. The storage of a program into such a program recording medium is performed by using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via the communication unit 219 that is an interface of a router, a modem or the like if necessary.

[0384] Furthermore, in the present specification, steps to execute a program stored in a recording medium may include a process executed in a time series according to the described order, and are necessarily processed in a time series, but may include a process executed individually or in parallel.

[0385] The present application contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2010-090716 filed in the Japan Patent Office on Apr. 9, 2010, the entire contents of which are hereby incorporated by reference.

[0386] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

Claims

What is claimed is:

1. An information processing device comprising:

a face detection unit that detects a face area from a target image;

a feature point detection unit that detects a feature point of the detected face area;

a determination unit that determines an attention area that is an area to which attention is paid in the face area based on the detected feature point;

a reference color extraction unit that extracts a reference color that is color setting obtained from the target image in the determined attention area;

an adjustment unit that adjusts the extracted reference color to a color setting for a modified image generated from the target image as a base; and

a generation unit that generates the modified image from the target image by drawing the attention area using the color setting for the modified image.

2. The information processing device according to claim 1,

wherein the attention area is composed of a face outline area, a hair area, and a part area including a predetermined part,

wherein the reference color extraction unit extracts a reference color in the face outline area, the hair area, and the part area,

wherein the adjustment unit adjusts each grayscale of the extracted reference color to the grayscales in the color setting for the modified image according to a predetermined grayscale converting condition, and

wherein the generation unit draws the face outline area, the hair area, and the part area by each of the grayscales in the color setting for the modified image.

3. The information processing device according to claim 2, wherein the adjustment unit adjusts each of the grayscales of the extracted reference color to the grayscale in the color setting for the modified image according to the grayscale converting condition based on face attribute information of which the attributes are obtained from the face area.

4. The information processing device according to claim 2, wherein the adjustment unit adjusts each of the grayscales of the extracted reference color to the grayscale in the color setting for the modified image according to the grayscale converting condition based on illumination information regarding an illumination state obtained from the target image or information added to the target image.

5. The information processing device according to claim 2, wherein the adjustment unit adjusts each of the grayscale of the extracted reference color to the grayscale in the color setting for the modified image according to the grayscale converting condition based on the face attribute information that is information regarding the attributes of a face or illumination information regarding an illumination state set by a user.

6. The information processing device according to claim 2, further comprising:

a discrimination unit that discriminates a model image to be similar to a part image in the part area from among model images corresponding to modified part images expressed by modifying the part; and

a selection unit that selects the modified part image corresponding to the model image according to the discrimination result,

wherein the adjustment unit the grayscale of the color setting for the selected modified part image to the grayscale in the color setting for the modified image, and

wherein the generation unit draws the modified part image of which the grayscale is adjusted.

7. The information processing device according to claim 6,

wherein the target image is a captured image,

wherein the modified image is a portrait image obtained by expressing a face image in a detected face area by a picture, and

wherein the modified part image is an illustration image obtained by expressing the part by a picture.

8. The information processing device according to claim 1, further comprising a display unit that displays the generated modified image.

9. An information processing method comprising the steps performed by an information processing device of:

detecting a face area from a target image;

detecting a feature point of the detected face area;

determining an attention area that is an area to which attention is paid in the face area based on the detected feature point;

extracting a reference color that is a color setting obtained from the target image in the determined attention area;

adjusting the extracted reference color to the color setting for a modified image generated from the target image as a base; and

generating the modified image from the target image by drawing the attention area using the color setting for the modified image.

10. A program that causes a computer to function as:

a face detection unit that detects a face area from a target image;

a feature point detection unit that detects a feature point of the detected face area;

a determination unit that determines an attention area that is an area to which attention is paid in the face area based on the detected feature point;

a reference color extraction unit that extracts a reference color that is color setting obtained from the target image in the determined attention area;

an adjustment unit that adjusts the extracted reference color to a color setting for a modified image generated from the target image as a base; and

a generation unit that generates the modified image from the target image by drawing the attention area using the color setting for the modified image.

Abstract