Image processing device, image processing method, program recording medium

[0001] This application is based upon and claims the benefit of priority from the prior Japanese Patent Application No. 2014-057724, filed Mar. 20, 2014, and the entire contents of which are incorporated herein by reference.

[0002] Field of the Invention

[0003] The present invention relates to an image processing device, an image processing method, and a program recording medium.

[0004] Description of the Related Art

[0005] Conventionally, there is known a portrait creation device which creates a portrait by using feature points of face parts, such as eyes, a nose, a mouth, ears, and a facial contour (for example, JP 2004-145625 A).

[0006] There is also known another technique to create a portrait image from a photographic image by using a binarization process. However, when a portrait image is created based on an image in which a face is rotated, a sense of incongruity is caused unless lines expressing shades according to the rotation direction of the face are drawn. As a result, it is likely to degrade the expressiveness of the portrait image.

[0007] A purpose of the present invention is to provide an image processing device, an image processing method, and a program, which can improve the expressiveness of a face image.

[0008] To achieve the purpose described above, an image processing device according to an embodiment of the invention includes:

[0009] an acquiring unit configured to acquire a face image; and

[0010] a control unit configured to specify a face direction in the face image acquired by the acquiring unit and add, based on the specified face direction, a picture expressing a contour of a face component to the face image.

[0011] An image processing method according to an embodiment of the present invention is an image processing method using the image processing device which includes a control unit, the method including the steps of:

[0012] acquiring, by the control unit, a face image;

[0013] specifying, by the control unit, a face direction in the acquired face image; and

[0014] adding, by the control unit, a picture expressing a contour of a face component to the face image based on the specified face direction.

[0015] A non-transitory computer-readable recording medium according to an embodiment of the present invention stores a program for processing an image and causing a computer to execute:

[0016] acquiring, as an acquiring unit, a face image;

[0017] specifying, as a specifying unit, a face direction in the acquired face image; and

[0018] adding, as an adding unit, based on the specified face direction, a picture expressing a contour of a face component to the face image.

[0025] Specific aspects of the invention will be described below with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

[0026] FIG. 1 is a block diagram showing a schematic configuration of an image capturing device 100 according to an embodiment of the invention.

[0027] Specifically, as shown in FIG. 1, the image capturing device 100 according to the embodiment includes an image capturing unit 1, an image capturing control unit 2, an image data creating unit 3, a memory 4, an image recording unit 5, an image processing unit 6, a display control unit 7, a display unit 8, an operation input unit 9, and a central control unit 10.

[0028] Moreover, the image capturing unit 1, the image capturing control unit 2, the image data creating unit 3, the memory 4, the image recording unit 5, the image processing unit 6, the display control unit 7, and the central control unit 10 are connected to each other through a bus line 11.

[0029] Known devices can be applied to the image capturing device 100. The known devices include not only a digital camera which has an image capturing function as a major function as described in the embodiment, but also portable terminals, such as a mobile phone or a smart phone, which have an image capturing function as an auxiliary function.

[0030] The image capturing unit 1, as an image capturing unit, creates a frame image by capturing an image of a predetermined object.

[0031] Specifically, the image capturing unit 1 includes, although not shown, for example, a lens unit (e.g., a zoom lens and a focus lens), an electronic image capturing unit (e.g., a charge coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS)), and a lens driving unit which moves the zoom lens and the focus lens in the optical axis direction.

[0032] The image capturing control unit 2 controls capturing the image of the object by the image capturing unit 1.

[0033] Specifically, the image capturing control unit 2 includes, although not shown, for example a timing generator and a driver. The image capturing control unit 2 scans and drives the electronic image capturing unit by the timing generator and the driver. Then, the image capturing control unit 2 converts, by the electronic image capturing unit, an optical image having passed through the lens unit into a two-dimensional image signal at every predetermined cycle. Thereafter, the image capturing control unit 2 reads one screen portion of the frame images at a time from an image capturing region of the electronic image capturing unit and outputs the frame images to the image data creating unit 3.

[0034] Moreover, the image capturing control unit 2 may adjust and control the conditions for capturing images of the object, such as an automatic focusing process (AF), an automatic exposure process (AE), and an automatic white balance (AWB).

[0035] The image data creating unit 3 appropriately performs gain adjustment, for each of the color components of the RGB, to an analog value signal of the frame image transferred from the electronic image capturing unit of the image capturing unit 1. Thereafter, the image data creating unit 3 samples and holds the signal by a sample/hold circuit (not shown), and converts the signal to digital data by an A/D converter (not shown). Then, after performing color process treatment including a pixel interpolation process and a γ-correction process by a color processing circuit (not shown), the image data creating unit 3 creates a digital value luminance signal Y and color difference signals Cb and Cr (YUV data).

[0036] The digital value luminance signal Y and the color difference signals Cb and Cr which are output from the color processing circuit are DMA-transferred through a direct memory access (DMA) controller (not shown) to the memory 4 used as a buffer memory.

[0037] The memory 4 includes, for example, a dynamic random access memory (DRAM) and temporarily stores the data to be processed by, for example, the image processing unit 6 and the central control unit 10.

[0038] The image recording unit 5 includes, for example, a non-volatile memory (a flash memory) and records the image data to be recorded. The image data is encoded in accordance with a predetermined compression format (e.g., JPEG format) by an encoding unit (not shown) of the image processing unit 6.

[0039] The image recording unit 5 may detachably include, for example, a recording medium (not shown) and may also be configured to control reading/writing data from/to the attached recording medium.

[0040] The image processing unit 6 includes a first image acquiring unit 6a, a face component detecting unit 6b, a face direction specifying unit 6c, a face processing unit 6d, a second image acquiring unit 6e, a shade adding unit 6f, and a portrait image creating unit 6g.

[0041] Each unit of the image processing unit 6 includes, for example, a predetermined logic circuit. However, the configuration has been described as an example and is not limited to the example.

[0042] The first image acquiring unit 6a acquires an original image P1 to be processed in an image creating process.

[0043] In other words, the first image acquiring unit (first acquiring unit) 6a acquires image data of the original image (face image) P1. Specifically, the first image acquiring unit 6a acquires a duplication of the image data (RGB data or YUV data) of the original image P1 (e.g., a photographic image) from the memory 4 and also acquires a duplication of the image data of the original image P1 recorded in the image recording unit 5 (see FIG. 3A). The original image P1 is created by the image data creating unit 3 by capturing the image of the object by the image capturing unit 1 and the image capturing control unit 2.

[0044] Moreover, each process, which will be described later, by the image processing unit 6 may be performed to the image data of the original image P1 or to reduced image data in a predetermined size (e.g., VGA size) to which the image data of the original image P1 is reduced at a predetermined ratio as necessary.

[0045] The face component detecting unit 6b detects main face components of the face in the original image P1.

[0046] In other words, the face components detecting unit (detecting unit) 6b detects the face components, such as eyes, a nose, a mouth, eyebrows, and a facial contour, in the original image P1 acquired by the first image acquiring unit 6a. Specifically, the face component detecting unit 6b detects eyes, a nose, a mouth, eyebrows, and a facial contour in the original image P1 by a process using, for example, an active appearance model (AAM).

[0047] The AAM is a method for modeling a visual matter, and a process for modeling an image of an arbitrary face region. For example, the face component detecting unit 6b registers, in a predetermined registering unit, positions of predetermined feature parts (e.g., corners of eyes, tips of noses, and face lines) of sample face images, and statistical analysis results of pixel values (e.g., luminance values). Then, the face component detecting unit 6b sets, based on the positions of the feature parts, a shape model of a face shape and a texture model which expresses “appearance” in an average shape, and performs modeling of the image of the face region using these models. In this manner, the face components, such as eyes, a nose, a mouth, eyebrows, and a facial contour, in the original image P1 are modeled.

[0048] The process using AAM has been described as an example of detecting the face components. However, the process is not limited to the example and can be arbitrarily changed to an appropriate process, such as an edge extraction process, an anisotropic diffusion process, or a template matching.

[0049] The face direction specifying unit 6c specifies a face direction.

[0050] In other words, the face direction specifying unit (specifying unit) 6c specifies the face direction in the original image P1 acquired by the first image acquiring unit 6a. Specifically, the face direction specifying unit 6c specifies the right-left direction of the face where the axis is the straight line connecting the top of the head and the tip of the jaw in the original image P1. For example, the face direction specifying unit 6c estimates the visual line direction of the eyes based on the result of the process using AAM by the face component detecting unit 6b and specifies the right-left direction of the face according to the estimated visual line direction.

[0051] The method based on the result of the process using AAM by the face component detecting unit 6b has been described as an example of specifying the face direction. However, the method is not limited to the example and can be arbitrarily changed to an appropriate method which can specify the face direction. The above-described process of estimating the visual line direction is a known technique and the detailed description thereof is omitted.

[0052] The face processing unit 6d performs various processes to the original image P1.

[0053] In other words, the face processing unit 6d performs a binarization process to the original image P1 and creates a binarized image. Specifically, after performing a smoothing process and a blurring process to the original image (e.g., a photographic image) P1, the face processing unit 6d performs the binarization process using a first threshold and creates the binarized image. Moreover, the face processing unit 6d detects a face region from the binarized image by, for example, a predetermined face detecting process. Thereafter, the face processing unit 6d performs a detail extracting process on the face region using the detection results of the face components by the face component detecting unit 6b, and creates a face detail image (not shown) in which the face components, such as eyes, a nose, a mouth, eyebrows, hair, and a facial contour, are expressed with lines. Furthermore, the face processing unit 6d creates a binarized face image (processed face image) P2 in which the face components, including face component images A, such as eyes, a nose, a mouth, and eyebrows, inside and adjacent to the facial contour of the face detail image are expressed with lines (see FIG. 3B).

[0054] The above-described smoothing process, blurring process, binarization process, and face detecting process are known techniques, and the detailed description thereof is omitted.

[0055] The face processing unit 6d detects a nose ridge L from the face in the original image P1.

[0056] In other words, the face processing unit (detecting unit) 6d detects the nose ridge L from the original image P1 acquired by the first image acquiring unit 6a based on the right-left direction of the face specified by the face direction specifying unit 6c. Specifically, the face processing unit 6d performs a binarization process to the original image P1 using a second threshold and detects the nose ridge L (see FIG. 4A).

[0057] For example, the face processing unit 6d sets, to the threshold (the second threshold), a different value (for example, a smaller value) from that used for detecting the face components and detects the nose ridge L by performing, to the original image P1, the binarization process using the second threshold. The second threshold may be changeable, for example, within the range including a different value from that used for detecting the face components. Moreover, the second threshold may be set by taking into consideration, for example, the brightness and the contrast of the original image P1.

[0058] Furthermore, when the face direction specifying unit 6c specifies that the face direction is the left with respect to the face, the face processing unit 6d sets a detection range to easily detect the nose ridge L from the left region based on the nose position of the face. On the other hand, when the face direction specifying unit 6c specifies that the face direction is the right with respect to the face, the face processing unit 6d sets a detection range to easily detect the nose ridge L from the right region based on the nose position of the face. Then, the face processing unit 6d performs, to the pixels within the set detection range, the binarization process using the second threshold set to a different value from that used for detecting the face components. Thereafter, the face processing unit 6d detects the x-y coordinates of the pixels which constitute the nose ridge L and creates position data.

[0059] The method using the binarization process has been described as an example of detecting the nose ridge L. However, the method is not limited to the example and can be arbitrarily changed to an appropriate method which can detect the nose ridge L.

[0060] The second image acquiring unit 6e acquires the binarized face image P2.

[0061] In other words, the second image acquiring unit (second acquiring unit) 6e acquires the binarized face image (processed face image) P2 created by performing, to the original image P1, the binarization process using the predetermined threshold. Specifically, the second image acquiring unit 6e acquires the image data of the binarized face image P2 created by, for example, the face processing unit 6d.

[0062] The shade adding unit 6f adds the nose ridge L to the binarized face image P2.

[0063] In other words, the shade adding unit (adding unit) 6f adds lines expressing shades (e.g., the nose ridge L) to the binarized face image (processed face image) P2 acquired by the second image acquiring unit 6e based on the face direction specified by the face direction specifying unit 6c. Specifically, the shade adding unit 6f adds the nose ridge L detected by the face processing unit 6d to the binarized face image P2 acquired by the second image acquiring unit 6e. For example, the shade adding unit 6f specifies the position to superimpose the nose ridge L inside the binarized face image P2 (especially, inside the facial contour) based on the position data of the pixels which constitute the nose ridge L. Then, the shade adding unit 6f creates the image data of a binarized face image with ridge P3 obtained by superimposing the nose ridge L on the specified position (see FIG. 4A).

[0064] The portrait image creating unit 6g creates a portrait image P4 which schematically expresses the face.

[0065] In other words, the portrait image creating unit (creating unit) 6g creates the portrait image P4 which schematically expresses the face by using the binarized face image (processed face image) P2 to which the lines expressing the shades (e.g., the nose ridge L) are added by the shade adding unit 6f. Specifically, the portrait image creating unit 6g acquires the image data of the binarized face image with ridge P3 on which the nose ridge L is superimposed by the shade adding unit 6f and also acquires the image data of a predetermined hair style image corresponding to the facial contour detected by the face component detecting unit 6b. Then, the portrait image creating unit 6g specifies the positions to superimpose the face component images A and the nose ridge L of the binarized face image with ridge P3 inside the facial contour of the acquired predetermined hair style image. Thereafter, the portrait image creating unit 6g creates the image data of the portrait image P4 which schematically expresses the original image P1 with a portrait by superimposing the face component images A and the nose ridge L on the specified positions (see FIG. 4B).

[0066] The portrait image creating unit 6g may create the portrait image P4 in which predetermined parts (for example, the face components, such as eyes, a mouth and eyebrows) are expressed with predetermined colors.

[0067] The display control unit 7 controls reading the image data to be displayed which are temporarily stored in the memory 4 and displaying the image data on the display unit 8.

[0068] Specifically, the display control unit 7 includes, for example, a video random access memory (VRAM), a VRAM controller, and a digital video encoder. The digital video encoder periodically reads, from the VRAM, the luminance signal Y and the color difference signals Cb and Cr, which have been read from the memory 4 and stored in the VRAM (not shown), through the VRAM controller under the control of the central control unit 10. Then, the digital video encoder generates video signals based on these data and outputs the video signals to the display unit 8.

[0069] The display unit 8 is, for example, a crystal liquid display panel and displays, on the display screen, the images captured by the image capturing unit 1 based on the video signals from the display control unit 7. Specifically, in a still image capturing mode or a moving image capturing mode, the display unit 8 displays a live-view image while sequentially updating, at a predetermined frame rate, the frame images created by capturing the image of the object by the image capturing unit 1 and the image capturing control unit 2. Moreover, the display unit 8 displays images to be recorded as the still images (rec view image) and displays images being recorded as the moving images.

[0070] The operation input unit 9 is for performing predetermined operations of the image capturing device 100. Specifically, the operation input unit 9 includes operation units, such as a shutter button related to capturing instructions of the object, a selection deciding button related to selection instructions of image capturing modes, functions, and the like, and a zoom button related to adjustment instructions of zooming (which are not shown). The operation input unit 9 outputs, to the central control unit 10, predetermined operation signals according to the operation of the buttons of the operational unit.

[0071] The central control unit 10 controls each unit of the image capturing device 100. Specifically, the central control unit 10 includes, although not shown, a central processing unit (CPU) and the like, and performs various controlling operations according to various processing programs (not shown) for the image capturing device 100.

<Image Creating Process>

[0072] Next, an image creating process by the image capturing device 100 will be described with reference to FIGS. 2 to 4B.

[0073] FIG. 2 is a flowchart showing an exemplary operation according to the image creating process.

[0074] The image creating process is performed by each unit of the image capturing device 100, especially by the image processing unit 6 under the control of the central control unit 10, when an image processing mode is selected and instructed, by a user, from a plurality of operation modes displayed on a menu screen based on a predetermined operation of the selection deciding button of the operation input unit 9.

[0075] Moreover, it is assumed that the image data of the original image P1 to be processed in the image creating process is recorded in the image recording unit 5 in advance.

[0076] As shown in FIG. 2, first, the first image acquiring unit 6a reads, from the image recording unit 5, the image data designated, by a user, to be processed in the image creating process among the image data of a plurality of the original images P1 recorded in the image recording unit 5 based on the predetermined operation of the operation input unit 9 and acquires the designated data (step S1; see FIG. 3A).

[0077] Next, the face component detecting unit 6b performs the predetermined detecting process (e.g., AAM) to the image data of the original image P1 acquired by the first image acquiring unit 6a and detects the face components, such as eyes, a nose, a mouth, eyebrows, and a facial contour (step S2).

[0078] Then, the face direction specifying unit 6c estimates the visual line direction of the eyes based on the results of the process using AAM by the face component detecting unit 6b and specifies the right-left direction of the face according to the estimated visual line direction (step S3).

[0079] Next, after the face processing unit 6d performs the smoothing process, the blurring process and the like to the image data of the original image P1 acquired by the first image acquiring unit 6a, the face processing unit 6d performs the binarization process using the first threshold and creates the binarized face image P2 including the face component images A, such as eyes, a nose, a mouth, and eyebrows (step S4; see FIG. 3B).

[0080] Then, the face processing unit 6d performs the binarization process to the original image P1 using the second threshold based on the right-left direction of the face specified by the face direction specifying unit 6c, and detects the nose ridge L (step S5).

[0081] Next, the second image acquiring unit 6e acquires the image data of the binarized face image P2 created by the face processing unit 6d (step S6).

[0082] Then, the shade adding unit 6f specifies the position to superimpose the nose ridge L detected by the face processing unit 6d inside the binarized face image P2 (especially, inside the facial contour) acquired by the second image acquiring unit 6e based on the position data of the pixels which constitute the nose ridge L and creates the image data of the binarized face image with ridge P3 by superimposing the nose ridge L on the specified position (step S7; see FIG. 4A).

[0083] Next, the portrait image creating unit 6g acquires the image data of the binarized face image with ridge P3 on which the nose ridge L is superimposed by the shade adding unit 6f, also acquires the image data of the predetermined hair style image, superimposes the face component images A and the nose ridge L of the binarized face image with ridge P3 inside the facial contour of the predetermined hair style image, and creates the image data of the portrait image P4 which schematically expresses the original image P1 with the portrait (step S8; see FIG. 4B). Thereafter, the image recording unit 5 acquires and records the image data of the portrait image P4.

[0084] In this manner, the image creating process ends.

[0085] As described above, the image capturing device 100 according to the embodiments can specify the face direction in the face image. Then, by performing the binarization process to the face image based on the specified face direction, the image capturing device 100 can add lines expressing the shades on the created binarized face image (processed face image) P2. Accordingly, when creating the portrait image P4 based on the face image in which the face direction is not the front but rotated, the image capturing device 100 can express the shades according to the rotation direction of the face. Specifically, the image capturing device 100 can appropriately express the shades according to the right-left rotation direction of the face by adding the nose ridge L to the binarized face image P2 based on the right-left direction of the face where the axis is the straight line connecting the top of the head and the tip of the jaw, and can express the contour of the face component. In this manner, the image capturing device 100 can add a picture expressing the contour of the face component to the face image with a different brightness (color) from that of the periphery of the contour to the point of the face image where the contour of the face component cannot be recognized since the brightness of the contour is the same as that of the periphery of the contour. Similarly, when a nose cannot be recognized since a color of the nose is the same skin color as that of a cheek around the nose, the image capturing device 100 according to the embodiment can add the nose contour (ridge) to the face image with a different color from the skin color in order to recognize the nose.

[0086] Therefore, the expressiveness of the face image can be improved.

[0087] Moreover, the nose ridge L can be appropriately detected from the face image based on the right-left direction of the face. Furthermore, the nose ridge L can be more appropriately detected by the binarization process. The binarization process is performed, to the face image, using the predetermined threshold (especially, the second threshold which is different from the first threshold used for creating the binarized face image P2).

[0088] The portrait image P4 which schematically expresses the face is created by using the binarized face image with ridge P3 (processed face image) to which the lines expressing the shades are added. In other words, by using the binarized face image P2 in which the shades are expressed based on the rotation direction of the face, the portrait image P4 can be appropriately created more naturally with high completeness.

[0089] The present invention is not limited to the above-described embodiment, and various improvements and design changes can be made without departing from the scope of the invention.

[0090] For example, in the above-described embodiment, the nose ridge L has been described as an example of the lines expressing the shades. However, the method expressing the shades is not limited to the example, and can be arbitrarily changed. For example, a color and a shape expressing the shades as a shadow picture M of a face image P5 shown in FIG. 5 and a picture can be added in order to express the shades. In this manner, the nose ridge can be expressed by expressing the shades based on the face direction. Furthermore, the nose ridge L has been described as an example of the face component to be detected and added to the binarized face image P2. However, the face component to be detected and added to the binarized face image P2 is not limited to the example and whether the face processing unit 6d as a detecting unit to detect the nose ridge L is provided or not can be arbitrarily changed.

[0091] The right-left direction of the face has been described as an example of the face direction when the axis is the straight line connecting the top of the head and the tip of the jaw. However, the face direction is not limited to the example and may be, for example, an up-down direction when the axis is the straight line substantially orthogonal to the straight line connecting the top of the head and the tip of the jaw in the face image, or an oblique direction obtained by combining these right-left direction and up-down direction.

[0092] Furthermore, in the above-described embodiment, the portrait image P4 is created by using the binarized face image with ridge P3 to which the lines expressing the shades are added. However, the portrait image P4 is not necessarily created and whether the portrait image creating unit 6g is provided or not can be arbitrarily changed.

[0093] Moreover, in the above-described embodiment, the binarized face image P2 is created by performing the binarization process to the original image P1. However, the binarized face image P2 is not necessarily created in the main body of the device and whether the face processing unit 6d is provided or not can be arbitrarily changed. For example, the binarized face image P2 created by an external device may be acquired.

[0094] Furthermore, the configuration of the image capturing device 100 has been described as an example and is not limited to the example. The image capturing device 100 has been described as an example of the image processing device. However, the image processing device is not limited to the example and any configuration capable of executing the image process according to an embodiment of the invention may be used.

[0095] In the above-described embodiment, after specifying the face direction, the binarization process is performed to the face image, and then the lines expressing the shades are added to the face image created by performing the binarization process. However, the processing order of adding the lines expressing the shades is not limited to the example.

[0096] In addition, in the above-described embodiment, the functions of the first acquiring unit, the specifying unit, the second acquiring unit, and the adding unit are implemented by driving, respectively, the first image acquiring unit 6a, the face direction specifying unit 6c, the second image acquiring unit 6e, and the shade adding unit 6f under the control of the CPU of the central control unit 10. However, the configurations are not limited to these examples and may be implemented by, for example, executing a predetermined program by the CPU of the central control unit 10.

[0097] In other words, the programs including a first acquiring processing routine, a specifying processing routine, a second acquiring processing routine, and an adding processing routine are stored in a program memory (not shown) storing programs. Then, the CPU of the central control unit 10 may function, by the first acquiring processing routine, as a unit configured to acquire the face image. Furthermore, the CPU of the central control unit 10 may function, by the specifying processing routine, as a unit configured to specify the face direction in the acquired face image. The CPU of the central control unit 10 may function, by the second acquiring processing routine, as a unit configured to acquire the processed face image created by performing the binarization process to the face image. The CPU of the central control unit 10 may function, by the adding processing routine, as a unit configured to add, based on the specified face direction, the lines expressing the shades to the acquired processed face image.

[0098] Similarly, the detecting unit and the creating unit may be implemented by executing a predetermined program by the CPU of the central control unit 10

[0099] The control unit may have a configuration or a plurality of configurations.

[0100] Furthermore, as a non-transitory computer-readable recording medium storing a program to execute the above-described processes, a non-volatile memory such as a flash memory, and a portable recording medium such as a CD-ROM, other than a ROM and a hard disk can be applied. As a medium for providing the program data through a predetermined communication channel, a carrier wave may be applied.

[0101] The embodiments according to the invention have been described above. However, the scope of the invention is not limited to the above-described embodiments and includes the scope described in the claims and equivalents thereof.