Method for blond-hair-pixel removal in image skin-color detection

[0002] The current invention relates to the field of digital detection for skin color pixels in a digital image, and in particular to the field of hair-skin color pixel separation in a digital image.

[0003] Skin color detection is adopted as a preliminary step in automatic redeye detection and correction of consumer images for real-time use (see commonly assigned U.S. Ser. No. 08/919,560 filed Aug. 29, 1997 entitled “A Computer Program Product for Redeye Detection”). In this redeye detection application, skin color areas are first located and subsequent steps are employed to determine if the red dots within the skin color areas are true red eyes. The success of redeye detection largely depends on the success of having clean face skin color regions identified. Previous experiences tell that it is particularly difficult to obtain clean face regions in color images with the presence of blond hairs.

[0004] The algorithm designed in the aforementioned redeye detection application is used for locating and correcting redeye defects in consumer images without user intervention such as ROI (region-of-interest) selection. The major goal of the redeye detection algorithm is to detect redeye defects with a minimum number of false positives and within a minimum execution time without sacrificing performance. For this reason, face-region (with most of the hair regions eliminated) detection is performed so that unnecessary processing is not performed on red-dot candidates that are not located in detected face regions. The easiest and fastest way for face-region localization is the use of skin-color pixel detection that requires only pixel operations. It has been shown that difficulties arise when dealing with images having faces associated with blond hairs. In these cases, the skin-color detection process fails to produce satisfactory or desired results that would assist in the redeye detection procedure. FIG. 1 displays an example picture 500 that causes problems. In FIG. 1, objects 504, 505, 508, 509, and 510 are non-skin objects; 501 and 506 are blond hairs; 502, 503, and 507 are skin regions. In FIG. 2, picture 600 shows the example result of conventional skin detection algorithms. Clearly, the skin-color detection process does not separate the hairs from the face (see object 601) and therefore the subsequent redeye detection process will take the whole head plus the body as the face region and no redeye defects will be corrected.

[0005] There have been many publications recently addressing skin color detection for face recognition in color image processing, but only a few of them concern the issue of hair-face skin pixel identification. For instance, in Wu et al. (H. Wu, Q. Chen, and M. Yachida, “Face Detection From Color Images Using a Fuzzy Pattern Matching Method,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 21, No. 6, pp. 557-563, 1999), a hair model is used to assist face detection. The RGB color information in an image is first converted to CIE's XYZ color representation through a linear transformation resulting in one luminance component Y and two chromaticity components x=X/(X+Y+Z) and y=Y/(X+Y+Z). Then the two chromaticity components, x and y, are furthered converted to another space through a non-linear transformation, resulting in two new color components, u and v, that are perceptually uniformly-distributed in the new color space. The hair model is a function of three variables: the luminance Y and the chromaticities, u and v. Noticeably, this hair model works mainly for Asian faces with dark hairs. Moreover, the conversion from RGB to the corresponding CIE tristimulus values requires the knowledge of the color calibration that varies from imaging device to device.

[0006] There are a number of color spaces currently used by researchers for color image processing as described below.

[0007] Psychological space—Some researchers believe that the RGB basis is not a particularly good one to explain the perception of colors. Alternatively, a transformed non-RGB space, or a psychological space, is well accepted to describe ‘colors’. It is compatible to human color perception. This non-RGB space is composed of three components, hue (H), saturation (S) and brightness value (V). Instead of using three values (R,G,B) to distinguish color objects, a single component, H, is used to label a color pixel in this transformed space.

[0008] CIELab space-CIE in 1976 recommended the CIELab formula for color measurement. It is designed that colors in the CIELab space are perceptually more uniformly spread than are colors in RGB and psychological (e.g. HSV) spaces. Therefore, using the CIELab space enables the use of a fixed color distance in decision making over a wide range of colors.

[0009] Lst space—The Lst space is traditionally called T-space in graphic applications and is constructed with log exposures. L is the luminance component, and s and t are the two chrominance components. It is shown that the statistical distribution of color signals is more symmetrical in Lst space than in linear space.

[0010] YC_RC_Bspace—The YC_RC_Bspace is designed for television broadcasting signal conversion. Y is the luminance component, C_Rand C_Bare two chrominance components. Researchers working in video images prefer using this space.

[0011] Generalized R-G-B Space (gRGB)—This is also called normalized R-G-B space. This space is transformed from the native R-G-B space by normalizing each of the three elements of the original R-G-B by the summation of the three original elements. The resultant three new elements are linearly dependent so that only two elements are needed to effectively form a new space that is collapsed from three dimensions to two dimensions. So, it is also called a collapsed R-G-B space in some articles. This space does not provide an explicit luminance component like the other spaces. This generalization process reduces the illuminant effects on chromaticity components.

[0012] So far, there are no conclusive data showing that any one of the above color spaces is overwhelmingly superior to any others in terms of skin-color detection performance. The skin-color detector performance, rather, mostly depends on the structure of the detector itself. The reality is that space transformation from RGB to another color domain does not change the skin-pixel and non-skin-pixel distribution overlap in the original RGB space. This skin-pixel and non-skin-pixel distribution overlap is the major cause of FP (false positive) in skin-color detection. The selection of color space largely depends on designers' preference, practical use effectiveness, and model complexity.

[0013] Table 1 illustrates the computation expense for different transformation operations. Among them, gRGB transformation has the lowest computation expense. This is one of the advantages that gRGB transformation provides.

[0014] What is therefore needed is a way to provide an efficient skin-color detection method with which desirable clean face regions can be obtained with very low computation complexity prior to further image processing for applications such as the aforementioned redeye detection in the presence of blond hairs. The present invention describes a mechanism that overcomes the difficulty of separating blond hair pixels from skin color pixels by fusing two color detection strategies that work in generalized RGB (gRGB) space and the hue space respectively.

[0015] It is an objective of the present invention to automatically detect the skin color region in a digital color image.

[0016] It is a further object of the present invention to produce a clean skin color image in the presence of blond hairs.

[0017] The present invention is directed to overcoming one or more of the problems set forth above. Briefly summarized, according to one aspect of the present invention, a method and a computer program product for generating a clean skin color image comprises the steps of:

[0018] (a) converting an input RGB color image from the native RGB space to a preferred color space, the generalized RGB (gRGB) space, resulting in a GRGB color image;

[0019] (b) selecting two of the generalized RGB color components (generalized R and G in this case) to form a working space for skin color detection;

[0020] (c) classifying the pixels of the converted input image, the gRGB color image, into skin color and non-skin color pixels;

[0021] (d) eliminating non-skin color pixels and forming a binary skin color mask;

[0022] (e) masking the gRGB color image obtained in step (a) to form a masked gRGB image;

[0023] (f) converting the masked gRGB image to a hue image;

[0024] (g) separating blond hair color pixels and the skin color pixels in the hue space using the hue image;

[0025] (h) removing the blond hair color pixels in the hue image and forming a new skin color binary mask;

[0026] (i) masking the original RGB color image to form a masked color image that retains skin color pixels only.

[0027] These and other aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from a review of the following detailed description of the preferred embodiments and appended claims, and by reference to the accompanying drawings.

[0028] The present invention is effective in obtaining a clean skin color image in the presence of blond hair color pixels and has the advantages of:

[0029] (1) performing skin detection in the dimension-reduced space;

[0030] (2) conducting skin detection recfinement in one-dimension space;

[0031] (3) fusing two color classification strategies performed in two stages (skin detection and skin detection refinement);

[0032] (4) effectively using intermediate results of the first stage in the second stage;

[0033] (5) providing imaging applications such as redeye detection with cleaner skin color images.

[0034] FIG. 1 is a pictorial view of a human face with blond hair and other body parts and other non-skin color objects;

[0035] FIG. 2 is a pictorial view of the skin detection results of objects in FIG. 1 using traditional skin detectors;

[0036] FIG. 3 is the flow chart of the steps which are performed to effect formation of a clean skin color image in the presence of blond hairs in accordance with a preferred method of the present invention;

[0037] FIG. 4 is a pictorial view illustrating a normal skin color distribution in the generalized RG space and an ellipse classifier and its parameters used in this invention;

[0038] FIG. 5 is a pictorial view illustrating the blond hair and associated skin color distribution in the generalized RG space shown in FIG. 4;

[0039] FIG. 6 is an illustration of a histogram of blond hair and associated skin color in the hue space; and

[0040] FIG. 7 is a pictorial view of the skin detection results of objects in FIG. 1 using the present invention.

[0041] In the following description, a preferred embodiment of the present invention will typically be implemented as a software program. Those skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. Because image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance with the present invention. Other aspects of such algorithms and systems, and hardware and/or software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein may be selected from such systems, algorithms, components and elements known in the art. Given the method as described according to the invention in the following materials, software not specifically shown, suggested or described herein that is useful for implementation of the invention is conventional and within the ordinary skill in such arts.

[0042] Still further, as used herein, the computer program may be stored in a computer readable storage medium, which may comprise, for example; magnetic storage media such as a magnetic disk (such as a floppy disk or a hard drive) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any other physical device or medium employed to store a computer program.

[0043] Referring now to FIG. 3, block 1 represents the input (RGB) color image to be processed. For speedy processing, the input image is re-sampled to a default size of 252 by 320 if the input image size exceeds the default size. A simple pixel-averaging process is performed during the re-sampling procedure.

[0044] In block 2, the input RGB image, I_RGBwith size of M×N×3, is converted to the generalized RGB image, I_gRGB, using the formula: $\begin{array}{cc}{c}_j\left(m,n\right)=\frac{C_j\left(m,n\right)}{\sum _i{C}_i\left(m,n\right)}|j\in i=\left[1,2,3\right];m=\left[0,\dots M-1\right];n=\left[0,\dots N-1\right]& \left(1\right)\end{array}$

[0045] where C_iis the individual color channel (R, G, or B) of the input image I_RGB, c_jis the individual color channel of the resultant generalized R, G or B of the output image I_gRGB. This operation is not valid when $\sum _i{C}_i=0,$

[0046] and the output will be set to zero. The resultant three new elements are linearly dependent, that is, $\sum _j{c}_j=1,$

[0047] so that only two elements are needed to effectively form a new space that is collapsed from three dimensions to two dimensions. In most cases, as represented in block 3, c₁and c₂, that is, generalized R and G, are used in analyzing the skin color distribution of a large population of images and detecting skin color pixels in an image. Although only c₁and c₂need to be computed at this point, however, in this design, C₃is also computed since all of them are needed in a later stage, hue computation.

[0048] The skin-color region detection takes place in block 4. The generalized RGB image I_gRGBis passed to a classifier that is trained on a large population of images beforehand. c₁and c₂of the image I_gRGBare used in the classification process. Referring to FIG. 4, the pictorial view 800 presents an example where dark region 802 contains skin-color pixels in the (c₁,c₂) space. The classifier is an ellipse 801 in the (c₁,c₂) space. Parameters a, b, (o₁,o₂), and θ are the constants of the trained classifier, where a and b are the ellipse axes length, (o₁,o₂) is the center coordinates of the ellipse in the (c₁,c₂) space, and θ is the angle between the c₁axis and the longer axis of the ellipse. Exemplary parameter values for the ellipse are:

[0049] a=0.0223;

[0050] b=0.0443;

[0051] o₁=0.3872;

[0052] o₂=0.3180;

[0053] θ=−0.4323.  (2)

[0054] The resultant of the skin-color classifier is a skin-color mask (block 5) P_skin={p(i, j)}_M×N, that is, an M×N matrix with elements p(i, j). For each pixel of the generalized RGB image I_gRGB, define an evaluation function E(c₁(i, j), c₂(i, j)), where i=0, . . . M−1; j=0, . . . N−1, c₁(i, j)∈[0,1]; c₂(i, j)∈[0,1]. The classification is simply performed as: $\begin{array}{cc}p\left(i,j\right)=\{\begin{array}{ccc}1& \mathrm{if}& E\left(c_1,\left(i,j\right),c_2\left(i,j\right)\right)<1\\ 0& & \mathrm{else}\end{array}& \left(3\right)\end{array}$

[0055] where “1” indicates a skin-color pixel and “0” for non-skin color, and

[0056] E(c₁(i, j), c₂(i, j))=x_ij²/a²+y_ij²/b²  (4)

[0057] and

[0058] x_ij=(c₁(i, j)−o₁)cos(θ−π/4)−(c₂(i, j)−o₂)sin(θ−π/4)

[0059] y_ij=(c₁(i, j)−o₁)sin(θ−π/4)−(c₂(i, j)−o₂)cos(θ−π/4)  (5)

[0060] In FIG. 3, the skin-color mask, P_skin, is provided to terminal A, where it then becomes an input to AND gate a. The other input of AND gate a is the generalized RGB image I_gRGB. The output of gate a is the masked generalized RGB image (block 6) Î_gRGB=I_gRGB∩P_skin.

[0061] As was revealed in the study, the blond hair color distributes in the same region where the majority skin color occupies, as shown by the pictorial view 900 in FIG. 5. The blond hair color pixels are plotted in a region 903, skin color pixels in a region 902 for a person with blond hair, and a region 901 is for a larger population of skin color pixels. It is not a trivial task to parameterize the sub-regions in the (c₁,c₂) space so that the region 903 can be separated from the region 902. If the space dimension drops down to one, the separation of blond hair color pixels from the skin color pixels becomes fairly easy. This further reduction of dimension size is realized in block 7 where the masked generalized RGB image Î_gRGBis converted to a hue image H={h(m, n)}_M×N. A typical hue conversion is performed as: $\begin{array}{cc}\mathrm{if}\left(c_{\min }=\underset{i\in \left[1,2,3\right]}{\min }\left(c_i\left(m,n\right)\right)\right)\ne \left(c_{\max }=\underset{i\in \left[1,2,3\right]}{\max }\left(c_i\left(m,n\right)\right)\right)\mathrm{do}\mathrm{if}{c}_1\left(m,n\right)=c_{\max }\mathrm{do}h\left(m,n\right)=\left(c_2\left(m,n\right)-c_3\left(m,n\right)\right)/\left(c_{\max }-c_{\min }\right)\mathrm{elseif}{c}_2\left(m,n\right)=c_{\max }\mathrm{do}h\left(m,n\right)=2+\left(c_3\left(m,n\right)-c_1\left(m,n\right)\right)/\left(c_{\max }-c_{\min }\right)\mathrm{elseif}{c}_3\left(m,n\right)=c_{\max }\mathrm{do}h\left(m,n\right)=4+\left(c_1\left(m,n\right)-c_2\left(m,n\right)\right)/\left(c_{\max }-c_{\min }\right);h\left(m,n\right)=h\left(m,n\right)*{60}^{\circ };\mathrm{if}h\left(m,n\right)<0.0\mathrm{do}h\left(m,n\right)=h\left(m,n\right)+{360}^{\circ };& \left(6\right)\end{array}$

[0062] The separation of hair and skin and hair removal is conducted in blocks 8 and 9. Referring to the pictorial view 1000 in FIG. 6, a histogram of blond hair pixels and its associated skin pixels provides useful information for partitioning the blond hair color and the skin color in the hue space. A measured partition parameter T_His used to determine if an element h(m, n) is a skin pixel or a blond hair pixel: $\begin{array}{cc}\tilde{p}\left(m,n\right)=\{\begin{array}{cc}1& \mathrm{if}h\left(m,n\right)<T_H\\ 0& \mathrm{else}\end{array}& \left(7\right)\end{array}$

[0063] As shown in Equation (7), a new mask (block 10), {tilde over (P)}_skin={{tilde over (p)}(i, j)}_M×N, is formed; if h(m, n) is a skin pixel, then the corresponding element {tilde over (p)}(m, n)=1, else {tilde over (p)}(m, n)=0. In the preferred embodiment, the measured partition parameter T_H=15. The output skin-color RGB image Î_RGB(block 11) is the result of an AND operation (in gate b) of the new skin mask {tilde over (P)}_skinand the original RGB color image I_RGBinput through terminal B. In FIG. 7, the pictorial view 700 displays an exemplary skin mask {tilde over (P)}_skinfor the example image shown in FIG. 1. A clean face part 701 is separated from other body parts 702 and 703, and therefore, provides useful information for applications such as redeye detection.

[0064] The subject matter of the present invention relates to digital image understanding technology, which is understood to mean technology that digitally processes a digital image to recognize and thereby assign useful meaning to human understandable objects, attributes or conditions and then to utilize the results obtained in the further processing of the digital image.

[0065] The invention has been described with reference to a preferred embodiment. However, it will be appreciated that variations and modifications can be effected by a person of ordinary skill in the art without departing from the scope of the invention.

[0066] 0 detection phase

[0067] a block

[0068] b block

[0069] 1 block

[0070] 2 block

[0071] 3 block

[0072] 4 block

[0073] 5 block

[0074] 6 block

[0075] 7 block

[0076] 8 block

[0077] 9 block

[0078] 10 block

[0079] 11 block

[0080] 501 blind hair

[0081] 502 face

[0082] 503 body part

[0083] 504 non-skin object

[0084] 505 non-skin object

[0085] 506 blond hair

[0086] 507 body part

[0087] 508 non-skin object

[0088] 509 non-skin object

[0089] 510 non-skin object

[0090] 601 non-skin part

[0091] 602 skin part

[0092] 701 skin part

[0093] 702 skin part

[0094] 703 skin part

[0095] 801 ellipse classifier

[0096] 802 skin color pixels

[0097] 901 skin color pixels

[0098] 902 skin color pixels

[0099] 903 blond hair pixels

[0100] 1000 histogram