US 6,990,217 B1Grant
Gender classification with support vector machines
Issue Date:2006-01-24
•1 Claims
•3 Drawing Sheets
Abstract
A method classifies images of faces according to gender. Training images of male and female faces are supplied to a vector support machine. A small number of support vectors are determined from the training images. The support vectors identify a hyperplane. After training, a test image is supplied to the support vector machine. The test image is classified according to the gender of the test image with respect to the hyperplane.
Metadata
Assignee
- Mitsubishi Electric Research Labs. Inc.
Inventors
- Baback Moghaddam
- Ming-Hsuan Yang
Application Information
Application Number:US 09/444,689
Filing Date:1999-11-22
Priority Date:1999-11-22
Art Unit:2723
Classifications
IPC:
G06K9/00
Patent Drawings (3 sheets)
Description
Field of the Invention
[0001] The present invention deals with classifying facial images, and more particularly to classifying the facial images according to gender.
Background of the Invention
[0002] Although humans are extremely good at classifying gender from facial images, experiments have shown that most people have difficulty in classify gender from images were the faces are hairless. Furthermore, error rates in gender classification using low resolution facial images increases almost ten fold.
[0003] It would represent a major commercial advantage if computers could have reliable vision capabilities for classifying a person's gender, particularly from low resolution images. Although gender classification has been investigated from both a psychological and computational perspective, relatively few learning based vision methods are known for gender classification.
[0004] Golomb et al. in “SEXNET: A neural network identifies sex from human faces,” Advances in Neural Information Processing Systems, pp. 572–577, 1991, described a fully connected two-layer neural network to identify gender from human face images consisting of 30×30 pixel images.
[0005] Cottrell et al. in “Empath: Face, emotion, and gender recognition using holons,” Advances in Neural Information Processing Systems, pp. 564–571, 1991 also applied neural networks for face emotion and gender recognition. They reduced the dimensionality of a set of 4096×4096 images to 40×40 via an auto-encoder network. Those vectors were then given as inputs to another one layer network for training and recognition.
[0006] Brunelli et al, in “HyperBF networks for gender classification,” Proceedings of the DARPA Image Under-standing Workshop, pp. 311–314, 1992 developed HyperBF networks for gender classification in which two competing RBF networks, one for male and the other one for female, were trained using sixteen geometric features, e.g., pupil to eyebrow separation, eyebrow thickness, and nose width, as inputs.
[0007] Instead of using a raster scan vector of gray levels to represent face images, Wiskott et al. in “Face recognition and gender determination,” Proceedings of the International Workshop on Automatic Face and Gesture Recognition, pp. 92–97, 1995 described a system that used labeled graphs of two-dimensional views to describe faces. The nodes were labeled with jets which are a special class of local templates computed on the basis of wavelet transform, and the edges were labeled with distance vectors. They used a small set of controlled model graphs of males and females to encode the general face knowledge.
[0008] More recently, Gutta et al. in “Gender and ethnic classification of Face Images,” Proceedings of the IEEE International Automatic Face and Gesture Recognition, pp. 194–199, 1998 proposed a hybrid method which consists of ensemble of neural networks (RBFs) and inductive decision trees.
[0009] Using computer vision to determine the gender of subject faces can be used in various application areas, such as the gathering of population gender-statistics from patrons at entertainment/amusement/sports parks. Gender classification can also be used by television network viewer-rating studies. Gender specific computer vision can further have application in such fields as automated security/surveillance systems, demographic studies, safety monitoring systems, and human interfaces to computers.
Summary of the Invention
[0010] It is an object of the present invention to classify gender from thumbnail facial images in which only the main facial regions appear, i.e., the images have minimal or no hair information. The motivation for using such images is as follows. Humans can easily and frequently change their hair styles, therefore, facial images cropped to keep only the main facial regions, without hair, will provide a more robust gender recognition method.
[0011] The method according to the invention masks images of faces such that as little hair as possible appears in the images. The images are reduced down to thumbnail images to minimize storage requirements. A support vector machine is applied to training images. After a classification discriminator has been trained, faces in test images can be classified.
[0012] More specifically, the invention provides a method for classifying images of faces according to gender. Training images of male and female faces are supplied to a vector support machine. A small number of support vectors are determined from the training images. The support vectors identify a hyperplane. After training, a test image is supplied to the support vector machine. The test image is classified according to the gender of the test image with respect to the hyperplane.
Brief Description of the Drawings
[0013] FIG. 1 is a flow diagram of a gender classification method according to the invention; and
[0014] FIG. 2 is a diagrammatic of a support vector machine hyperplane;
Detailed Description of the Preferred Embodiment
[0015] FIG. 1 shows a method 100 for gender classifying images of human faces according to our invention. Our method takes training facial images 101 as input. The training images can be pre-processed using an automatic face-processing system. The face-processing system normalizes the training images for translation, scale, and rotations. Step 110 performs a multiscale 111 maximum-likelihood estimation to detect faces in the training images.
[0016] Step 120 performs a facial feature search with affine warping 121 for geometric shape alignment, and contrast normalization for ambient lighting changes. Step 130 applies a mask to produce full resolution (80×40 pixels) faceprints 131. Masking removes most of the extraneous features present in the faceprints such as background, and particularly hair. Step 140 sub-samples the full resolution faceprints to obtain a low resolution (21×21 pixels) or “thumbnail” image 141. This part of the face processing system is described in greater detail in U.S. Pat. No. 5,710,833 issued to Moghaddam, et al. on Jan. 20, 1998, incorporated herein by reference.
Support Vector Machine
[0017] The thumbnail images are presented to a support vector machines (SVM) 150. SVMs are actually not machines but methods for pattern recognition, regression and other learning problems. SVMs have been known for decades. SVMs are primarily designed for two-class classification problems, see Vapnik et al. in “Pattern Recognition using Generalized Portrait,” Automation and Remote Control, 24:6, 1963.
[0018] As shown in FIG. 2 , the SVM locates an optimal manifold or hyperplane 200 that leaves the largest possible fraction of data points of the same class (male 210 or female 220) on the same side of the hyperplane 200. FIG. 2 shows example data in two-dimensions for the purpose of illustrating the invention. It should be understood that the actual data can have much higher dimensions making the manifold 200 a hyperplane. The SVM 150 also maximizes the distance or “margins” 201–202 of either class from the hyperplane such that the expected recognition error for the actual samples to be classified is minimized. Maximizing the margins of the hyperplane is equivalent to minimizing the Vapnik-Chervonenkis (VC) dimension of the SVM while constructing the optimal hyperplane. According to the structural risk minimization inductive principle, a function that describes the training data well and belongs to a set of functions with lowest VC dimension will generalize well regardless of the dimensionality of the input space. The hyperplane 200 can be represented by a small number of support vectors 203, e.g., circled training samples 203. As a result the SVM provides a compact classifier.
[0019] Based on this principle, the SVM adopts a systematic approach to find a linear function that belongs to a set of functions with the lowest VC dimension. The SVM also provides non-linear function approximations by mapping the input vectors into a high dimensional feature space where the hyperplane 200 is constructed. Although there is no guarantee that a linear hyperplane will always exist in the high dimensional feature space, in practice it is possible to construct a linear SVM in a projected space. The following sections describe the vector support machine in greater detail.
[0020] Given a set of samples (x1, y1), (x2, y2), . . . , (xl, yl) where xi(xiεRN) is the input vector of N dimension, i.e., the thumbnail images 141, and yi is its label (yiε{−1,1}) for the present recognition problem, the SVM 150 finds the hyperplane 200 as described above. The problem of finding the optimal hyperplane is a constrained optimization problem. We solve the problem by using quadratic techniques. The optimal hyperplane is in the form:
where a and b are constants, k(.,;) is a kernel function, and the sign of f(x) determines the label of x. Constructing the optimal hyperplane is equivalent to determining a nonzero ai. Any vector xi that corresponds to the nonzero ai is a supported vector (SV) 203 of the optimal hyperplane 200. As stated above, the number of supported vectors for defining the hyperplane 200 is small, making the SVM a compact classifier.
where a and b are constants, k(.,;) is a kernel function, and the sign of f(x) determines the label of x. Constructing the optimal hyperplane is equivalent to determining a nonzero ai. Any vector xi that corresponds to the nonzero ai is a supported vector (SV) 203 of the optimal hyperplane 200. As stated above, the number of supported vectors for defining the hyperplane 200 is small, making the SVM a compact classifier.
[0021] For a linear SVM, the kernel function is the simple dot product of vectors in the N dimensional input space, while the kernel function in a nonlinear SVM projects the samples to an Euclidean feature space of higher dimensions M via a nonlinear mapping function ψ:
ψ: RN→FM, M>>N
and construct a hyperplane in F. We do this because it is more likely to find a linear function in the high dimensional feature space F.
ψ: RN→FM, M>>N
and construct a hyperplane in F. We do this because it is more likely to find a linear function in the high dimensional feature space F.
[0022] Using the well known Mercer's theorem, the lengthy calculations for projecting samples into the high dimensional feature space can be significantly reduce by using a suitable function k such that
k(x, xi)=ψ(x)·ψ(xi)
where ψ is the nonlinear projection function. Several kernel functions, such as polynomial functions and radial basis functions, see below, are known to satisfy Mercer's theorem.
k(x, xi)=ψ(x)·ψ(xi)
where ψ is the nonlinear projection function. Several kernel functions, such as polynomial functions and radial basis functions, see below, are known to satisfy Mercer's theorem.
[0023] By using different kernel functions, the SVM can construct a variety of learning machines, some of which coincide with classical architectures.
[0024] However, this also results in a drawback since we need to find the “right” kernel function while using nonlinear SVMs.
Classification Techniques
Radial Basis Function Networks
[0025] A radial basis function network is a method for improved generalization to new data based on regularization theory, please see Poggio et al. in “Networks for approximation and learning,” Proceedings of the IEEE, volume 78, pp. 1481–1497, 1990. One powerful network is the RBF with K Gaussian basis functions of the form:
where the G is the ith Gaussian basis function with center ci and variance σ2. The weight coefficients wi combines the basis functions into a single output value and b is a bias term.
where the G is the ith Gaussian basis function with center ci and variance σ2. The weight coefficients wi combines the basis functions into a single output value and b is a bias term.
[0026] Training a Gaussian RBF network for a given learning task involves determining the total number of Gaussian basis functions, locating the Gaussian basis function centers, computing the cluster variance for each basis function, and solving for the weight coefficients and bias. With careful design on K, ci, and ai, Gaussian RBF networks are very powerful in many recognition problems.
[0027] In a conventional RBF network, the number of neurons is predetermined before training. On the other hand, the number of Gaussian basis for a large ensemble-RBF network is iteratively increased until the mean square error is below a threshold. The centers of the Gaussian basis for both networks are determined by k-means clustering.
[0028] In contrast, SVM with RBF kernel automatically determines the number and locations of the Gaussian centers, weights and threshold that minimize an upper bound on the expected test error.
Fisher Linear Discriminant
[0029] Fisher Linear Discriminant (FLD) is an example of a class specific method that finds the optimal projection for gender classification. Rather than finding a projection that maximizes the projected variance, FLD determines a projection, y=WTFLDx, that maximizes the ratio between the between-class (variance) and the within-class scatter (variance). Consequently, classification is simplified in the projected space.
[0030] Consider a c-class problem where the between-class scatter matrix is defined as:
and the within-class scatter matrix be defined as:
where μ is the mean of all samples, μi is the mean of class Xi, and Ni is the number of samples in class Xi. We chose an optimal projection WFLD as the matrix with orthonormal columns which maximizes the ratio of the determinant of the between-class scatter matrix of the projected samples to the determinant of the within-class scatter matrix of the projected samples, i.e.,
where {ωi|=1, 2, . . . , m} is the set of generalized eigenvectors of SB and SW, corresponding to the m largest generalized eigenvalues {λi|i=1, 2, . . . , m}. However, the rank of SB is c−1 or less. Thus, the upper bound on m is c1. To avoid the singularity problem of SW, we apply principal component analysis (PCA) to first reduce the dimension of the feature space to Nc, and then apply the standard FLD to reduce the dimension to N−C, and then we apply the standard FLD to reduce the dimensions of c−1.
and the within-class scatter matrix be defined as:
where μ is the mean of all samples, μi is the mean of class Xi, and Ni is the number of samples in class Xi. We chose an optimal projection WFLD as the matrix with orthonormal columns which maximizes the ratio of the determinant of the between-class scatter matrix of the projected samples to the determinant of the within-class scatter matrix of the projected samples, i.e.,
where {ωi|=1, 2, . . . , m} is the set of generalized eigenvectors of SB and SW, corresponding to the m largest generalized eigenvalues {λi|i=1, 2, . . . , m}. However, the rank of SB is c−1 or less. Thus, the upper bound on m is c1. To avoid the singularity problem of SW, we apply principal component analysis (PCA) to first reduce the dimension of the feature space to Nc, and then apply the standard FLD to reduce the dimension to N−C, and then we apply the standard FLD to reduce the dimensions of c−1.
[0031] We use a Gaussian to model the distribution of projected samples of each male and female class in c−1 dimensional space. We then determine the class label of a sample on the Mahalanobis distances to the centers of these two classes. The Mahalanobis distance is a very useful way of determining the “similarity” of a set of values from an “unknown” sample to a set of values measured from a collection of “known” samples. We use the Mahalanobis distance method because it is very sensitive to inter-variable changes in the training data. In addition, because the Mahalanobis distance is measured in terms of standard deviations from the mean of the training samples, the reported matching values give a statistical measure of how well the unknown sample matches (or does not match) the original training data.
Linear and Quadratic Classifiers
[0032] If x is a sample in our two gender class problem, then the decision boundary of a quadratic classifiers is given by a quadratic form in x using the Bayes decision rule for minimum error. If the distribution of each class is a Gaussian, then the decision rule for a quadratic classifier is:
where μi and Σi(i=1,2) are the mean and covariance matrix in a Gaussian distribution, respectively.
where μi and Σi(i=1,2) are the mean and covariance matrix in a Gaussian distribution, respectively.
[0033] A linear classifier finds a linear discriminator with the assumption that Σ1=Σ2=Σ, which simplifies the above equation into:
For both classifiers, the sign of f(x) determines the class to which the sample x belongs. e.g., positive female, and negative male.
Experimental Results
For both classifiers, the sign of f(x) determines the class to which the sample x belongs. e.g., positive female, and negative male.
Experimental Results
[0034] In order to determine the effectiveness of the present invention, we trained with 1496 images, and tested with 259 images. We used high and low resolution images, 80×40 and 21×12 pixels, respectively. Each image was represented by a raster scan eight-bit vector. With the RBF kernel, the overall error rate for classifying low resolution thumbnail images was about 3.4%. This is significantly better than error rates obtained with prior art gender classifiers. We have also applied our classifier to high resolution images. It is worth noting that our method works equally well in both low and high resolution images with only 1% error rate difference. This demonstrates a degree of robustness and relative scale invariance of our method.
[0035] Table A below compares error rate percentages of prior art classifiers with the SVM classifiers (bold) of the present invention. The error rates percentages are for overall results, and male and female error rates in rank order.
[0036]
| TABLE A | |||
|---|---|---|---|
| Classifier | Overall | Male | Female |
| SVM with RBF kernel | 3.38 | 2.05 | 4.79 |
| SVM with cubic polynomial kernel | 4.88 | 4.21 | 5.59 |
| Large ensemble of RBF | 5.54 | 4.59 | 6.55 |
| Classical RBF | 7.79 | 6.89 | 8.75 |
| Quadratic classifier | 10.63 | 9.44 | 11.88 |
| Fisher linear discriminant | 13.03 | 9.44 | 13.78 |
| Nearest neighbor | 27.16 | 26.53 | 28.04 |
| Linear classifier | 58.95 | 58.47 | 59.45 |
[0037] In order to calibrate the performance of our classifiers, we used human subject to classify the high and low resolution images. All subjects were asked to classify the gender from the face images, presented in random order, as best as they can without time constraints. For high resolution images, the overall error was about 6.5%, and for low resolution images the error rate was about 30%. These experiments clearly show that our method performs significantly better than humans in both high resolution and low resolution gender classification. It is not surprising that human subjects perform far better with high resolution images than with low resolution images. Surprisingly, our method perform almost equally well in both cases.
[0038] Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.
Claims
We claim:
1. A method for classifying images of faces according to gender, comprising the steps of:
supplying a vector support machine with a plurality of training images, the training images including images of male and female faces;
determining a plurality of support vectors from the training images for identifying a hyperplane;
supplying the support vector machine with a test image;
classifying the gender of the test image with respect to the hyperplane; and
reducing the resolution of the training images and the test image by sub-sampling before supplying the images to the support vector machine;
wherein the test image contains less than 260 pixels.
Patent Citations (1)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US5710833(A) | 1998-01-01 | Moghaddam et al. | Applicant |
Non-Patent Literature (12)
- Osuna et al. Training Support Vector Machines: An Application to Face Detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Jun., 1997. 130-136.Examiner
- Gutta et al. Gender Classification of Human Faces Using Hybrid Classifier Systems. IEEE International Conference on Neural Networks. Jun., 1997. vol. 3, 1353-1358.Examiner
- Brunelli et al.; “HyberBF Networks for Gender Classification”; In Proceedings of the Image Understanding Workshop, DARPA, San Diego, pp. 311-314, 1992.Applicant
- Cottrell et al., “EMPATH: Face, Emotion and Gender Recognition Using Holons”; In Advances in Neural Information Processing Systems, pp. 564-571, 1993.Applicant
- Edelman et al.; “Sex Classification of Face Areas: How Well Can A Linear Neural Network Predict Human Performance?”; Journal of Bilogical Systems, 6(3): 241-264, 1998.Applicant
- Golomb et al., “Sexnet: A Neural Network Identifies Sex from Human Faces”; In Advances in Neural Information Processing Systems (NIPS), vol. 3, Lippmann et al. (Eds.), Morgan Kaufmann, 1990, pp. 572-577.Applicant
- Gutta et al., “Gender and Ethnic Classification of Face Images”; In Proceedings of the IEEE International Automatic Face and Gesture Recognition, pp. 194-199, 1998.Applicant
- Hill et al.; “Perceiving the Sex & Race of Faces: The Role of Shape and Colour”; Proc. R. Soc. Lond. B (1995) 261, pp. 367-373.Applicant
- O'Toole et al.; “The Perception of Face Gender: The Role of Stimulus Structure in Recognition and Classification”; Memory and Recognition, vol. 25, 1997.Applicant
- Sim et al.; “High-Performance Memory-based Face Recognition for Visitor Identification”; 1999.Applicant
- Tamura et al.; “Male/Female Identification from 8x6 Very Low Resolution Face Images by Neural Network”; Pattern Recognition 29(2): pp. 331-335, 1996.Applicant
- Wiskott et al.; “Face Recognition and Gender Determination”; Proc. Intl. Workshop on Automatic Face and Gender Recognition, pp. 92-97, 1995.Applicant