US 8,300,900 B2Grant

Face recognition by fusing similarity probability

Issue Date:2012-10-30

•12 Claims

•5 Drawing Sheets

Abstract

The invention discloses a face recognition method that reconstructs a 3D face model from a single face image, synthesizes a set of face images under different conditions (such as pose, light . . . ) via the 3D face model, feeds the set of face images under different conditions to the face recognition classifier for training, and making intermediate decisions whether to-be identified individual from a series of video frames is a legal system user by the face recognition classifier. Moreover, the method not only recognizes legal system users, but also rejects imposters, a function inspired by the idea of LLE. Finally, better reliability can be achieved by fusing temporal intermediate decisions.

Metadata

Assignee

National Tsing Hua University

Inventors

Shang-Hong Lai
Chia-Te Liao

Application Information

Application Number:US 12/349,260

Filing Date:2009-01-06

Priority Date:2008-12-02

Art Unit:2624

Classifications

IPC:

G06K9/00G06K9/62G06T15/00

Patent Drawings (5 sheets)

Description

Background of the Invention

[0001] 1. Field of the Invention

[0002] The present invention relates to a face recognition method, particularly reconstructing a 3D morphable face model from a single face image, and recognizing a series of face video frames from a video receiving apparatus by fusing similarity probability of each video frame.

[0003] 2. Description of the Related Art

[0004] Face recognition is one of the most important research areas in computer vision and biometric recognition. For 2D face recognition, the main challenge is the variations of face images for the same individual under different capturing conditions such as angles, light source, etc. To overcome the problem, most developed algorithms rely on a large number of training examples under different conditions for each individual. However, it is difficult to collect 2D face images under a precise condition with the variation in real-world applications.

[0005] Recently, 3D face modeling from images is very popular for many applications, such as facial animation and face recognition. Model-based statistical techniques have been widely used for robust face modeling. Most of the previous 3D face reconstruction techniques require more than one 2D face image to achieve satisfactory 3D face modeling. Although another approach for 3D face reconstruction requires only a single image, the problem is restricted by requiring a statistical head model known as a priori. Besides, most of the previous face recognition method cannot differentiate imposters who have not been present before. For the increasingly popular face recognition system, simplifying sampling complexity and training difficulties, and increasing recognition accuracy are problems need to be solved.

Summary of the Invention

[0006] To solve aforementioned problem, one objective of the present invention is to propose a face recognition method, which constructs a 3D face model using a single 2D face image, and simulate face image variations under different conditions. The synthesized face images are used to train a face recognition classifier. Besides, the present invention employs a locally linear embedding concept, which not only can recognize face images of legal system users, but also can reject imposters. Furthermore, the present invention temporally integrates single-frame decisions by a probabilistic approach to increase the reliability of face recognition.

[0007] To achieve aforementioned objective, one embodiment of the present invention proposes a face recognition method including obtaining an original face image from a database, reconstructing a 3D face model for the original face image using a 3D morphable face model, providing synthesized face images from the 3D face model for training a face recognition classifier, receiving a plurality of video frames of a face, obtaining a similarity probability for each video frame from the face recognition classifier, and fusing the temporal similarity probability for each video frame by accumulation to obtain a face recognition confidence value, and if the face recognition confidence value is larger than a threshold, the face is recognized to be the same as the face in the original face image.

Brief Description of the Drawings

[0008] FIG. 1 is a flowchart of the face recognition method of one embodiment of the present invention;

[0009] FIG. 2a-FIG. 2d are diagrams showing the 3D morphable model reconstructed from an original face image according to one embodiment of the present invention;

[0010] FIG. 3 shows the synthesized images according to one embodiment of the present invention; and

[0011] FIG. 4 is a diagram showing the conceptual framework of fusing temporal information for face recognition of one embodiment of the present invention.

Detailed Description of the Invention

[0012] The face recognition method of the present invention reconstructs a 3D face model from a single 2D face image using a 3D morphable model, and synthesizes a large set of 2D face images under different head poses for training the face recognition model. Also, this method temporally integrates single-frame recognition decisions via a probabilistic approach to increase recognition accuracy, and rejects non-system allowed imposters based on a LLE (locally linear embedding) method.

[0013] The flowchart of one embodiment of the present invention is shown in FIG. 1. This embodiment uses face recognition as an example, but the method can also be applied to recognizing diagrams or other images. In step S01 of this embodiment, an original face image is obtained from a database, and a 3D face model is reconstructed from the original face image using a 3D morphable face model. The 3D morphable face model provides prior knowledge of geometry and texture information of a normal face. A 3D morphable face model can be represented by a 3D geometry data set, S=(x₁, y₁, z₁, x₂, y₂, z₂, . . . , x_N, y_N, z_N)ε custom character

, and 3D texture data set, T=(r₁, g₁, b₁, r₂, g₂, b₂, . . . , r_N, g_N, b_N)ε custom character

, where N is the number of sampling points for the curved surface of a 3D face. Then the 3D geometry data set and 3D texture data set of the 3D face is approximated by a linear combination of a plurality of eigenhead basis vectors, shown as follows:

[0014]

\begin{matrix} S (α) = \overline{S} + \sum_{i = 1}^{m} α_{i} s_{i} & (1) \\ T (β) = \overline{T} + \sum_{i = 1}^{m} β_{i} t_{i} & (2) \end{matrix}

wherein S is the mean shape vector, T is the mean texture vector, s_iis the i_theigen-shape basis vector, t_iis the i_theigen-texture basis vector, α=[α₁, . . . , α_m] and β=[β₁, . . . , β_m] are the shape and texture coefficients, respectively, to represent the 3D face.

[0015] The embodiment uses 3D scans and image database of BU-3DFE (3D facial expression database) as the data source for producing eigen head basis vectors. Please referring to FIG. 2a to FIG. 2d, they are conceptual diagrams of a 3D morphable face model in one embodiment of the present invention. FIG. 2a shows the generic face model labeled with a plurality of feature points; FIG. 2b is the original face image from a database; FIG. 2c shows the reconstructed 3D face model of FIG. 2b after registration, re-sampling and smoothing; and FIG. 2d shows the triangulation detail of FIG. 2c.

[0016] The abovementioned 3D face reconstruction is initialized by the feature points in FIG. 2a, and then a shape data set, α, a texture data set, β, and a plurality of 3D characteristic parameters are obtained respectively by photometric error minimization to achieve a 3D morphable face model best represent the input face. The 3D characteristic parameters are found from minimizing a cost function of the following form:
∥I_input−I_model(α,β,P_ortho,f,R,t_2d)∥ (3)
wherein I_inputis the original face image, P_orthois an orthographic projection matrix, f is the scaling factor, R is the 3D rotation matrix, and t_2dis the translation vector. The minimization of function (3) can be solved by using the Levenberg-Marquart (LM) optimization.

[0017] In step S02, possible variations of the image are simulated by the reconstructed 3D face model, and used to train a face recognition classifier for the original face image. FIG. 3 shows the 2D face images synthesized with pre-defined characteristic parameters, based on the original 2D face image, by using the reconstructed 3D face model. The image in the middle is the original face image, and others are the synthesized face images under different poses.

[0018] In step S03, a plurality of video frames of a face are received, and a similarity probability is obtained for each video frame from the face recognition classifier. In step S04 the probability of each video frame is fused temporally to obtain a confidence value. If the confidence value is greater than a threshold, the received face video frames are recognized to have the same identity as the individual in the original face image. The method for fusing previous recognition results is shown in expression (4):

[0019]

\begin{matrix} P_{i}^{t} = \frac{(ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}{\sum_{i} (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}, i = 1, 2, \dots, n & (4) \end{matrix}

wherein ω is a forgetting factor, P_i^t,SVCis a posterior probability denoting the probability that this user belongs to class i decided by a single video frame at time t. In one embodiment, we merge P_i^t,SVCwith the previously accumulated probability P_i^t−1and obtain a final similarity probability P_i^taccumulated up to time t.

[0020] On the other hand, when it comes to an applicable face recognition system, it must have the ability to reject imposters who have not been present before and only accept legal system users. Borrowing the idea from LLE that nonlinear data is locally linear on a manifold, if a to-be recognized image cannot be well-reconstructed linearly via the training images of the same person, the image is likely to be from an imposter. Therefore, a recognized face I^tat frame t is supposed to be reconstructed well by its K nearest neighbors in the predicted class c. The similarity error ε^tof an input face I^tis defined to be:

[0021]

\begin{matrix} ɛ^{t} (w) =  I^{t} - \sum_{j = 1}^{K} w_{j}^{c, t} I_{j}^{c}  & (5) \end{matrix}

wherein w_j^c,tdenotes the weights of K nearest neighbors in class c that best approximate said video frame I^t, and

[0022]

\sum_{j = 1}^{K} w_{j}^{c, t} = 1; {I_{j}^{c}}_{j = 1}^{K}

is the set of K nearest neighbors of the same class. Then a sigmoid function is applied to transform ε^tto the corresponding rejection probability P_rej^twith a recursive definition:

[0023]

\begin{matrix} P_{rej}^{t} = 1 - \exp (\frac{- ɛ^{t}}{ɛ_{threshold}^{c}}), {\hat{P}}_{rej}^{t} = ω \times {\hat{P}}_{rej}^{t - 1} + (1 - ω) \times P_{rej}^{t} & (6) \end{matrix}

wherein ε_threshold^cis the parameter controlling the rejection rate. It turns out that face I^tcannot belong to a system user if it has a large rejection probability {circumflex over (P)}_rej^t. Please referring to FIG. 4, it is a diagram showing the conceptual framework of fusing temporal information for face recognition. Finally, the similarity probability P_i^tis summed up with the rejection probability {circumflex over (P)}_rej^t, and normalized to obtain the posterior probability {circumflex over (P)}_i^tof the video frame belonging to class c up to time t, which can be expressed by expression (7):

[0024]

\begin{matrix} {\hat{P}}_{i}^{t} = \frac{β^{t} \times (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}{\begin{matrix} (ω \times {\hat{P}}_{rej}^{t - 1} + (1 - ω) \times P_{rej}^{t}) + β^{t} \times \\ \sum_{i} (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC}) \end{matrix}}, & (7) \end{matrix}

wherein i=1, 2, . . . , n, and β^t=1−{circumflex over (P)}_rej^tis the acceptance rate at time t, and {circumflex over (P)}^t=[{circumflex over (P)}₁^t, {circumflex over (P)}₂^t, . . . , {circumflex over (P)}_rej^t]^Tis the posterior probability distribution. If {circumflex over (P)}_i^texceeds a threshold P_th, the present invention can decide the identity of the face in video frames. If

[0025]

\arg \max_{i} {\hat{P}}_{i}^{t} = n + 1

(class n+1 for the imposter class), then the to-be recognized face is rejected; otherwise, the face is determined to be of a legal system user and his/her identity is further recognized, i.e.

[0026]

c = \arg \max_{i} {\hat{P}}_{i}^{t}, c = 1, 2, \dots, n .

[0027] In conclusion, the present invention reconstructs a face model via a 3D morphable model, and synthesizes variations of a face image as the training database for face recognition classifier. In the process of constructing 3D face model and training face recognition classifier, only a 2D image is needed for the user of each class. This improves from the statistical approach which requires collecting a large amount of face training samples.

[0028] The embodiments described above are to demonstrate the technical contents and characteristics of the preset invention to enable the persons skilled in the art to understand, make, and use the present invention. However, it is not intended to limit the scope of the present invention. Therefore, any equivalent modification or variation according to the spirit of the present invention is to be also included within the scope of the present invention.

Claims

What is claimed is:

1. A face recognition method comprising:

obtaining an original face image from a database and reconstructing a 3D face model for said original face image using a 3D morphable face model;

providing synthesized face images from said 3D face model for training a face recognition classifier;

receiving a plurality of video frames of a face, and obtaining a similarity probability for each said video frame from said face recognition classifier; and

fusing said similarity probability for each said video frame by accumulation to obtain a face recognition confidence value, and if said face recognition confidence value is larger than a threshold, said face is recognized to be the same as the face from said original face image.

2. The face recognition method according to claim 1, wherein said original face image is a 2D image.

3. The face recognition method according to claim 1, wherein steps for reconstructing said 3D face model for said original face image include:

constructing said 3D morphable face model, wherein said 3D morphable face model comprises a 3D geometry data set and a 3D texture data set;

expressing said 3D morphable face model by a linear combination of a plurality of eigen head basis vectors; and

obtaining a geometry data set, a texture data set and 3D characteristic parameters by photometric error minimization for reconstructing said 3D face model corresponding to said original face image.

4. The face recognition method according to claim 3, wherein said 3D geometry data set can be expressed as: S=(x₁, y₁, z₁, x₂, y₂, z₂, . . . , x_N, y_N, z_N)ε custom character

wherein N is the number of facial sampling points.

5. The face recognition method according to claim 3, wherein said 3D texture data can be expressed as:

T=(r₁, g₁, b₁, r₂, g₂, b₂, . . . , r_N, g_N, b_N)ε custom character

wherein N is the number of facial sampling points.

6. The face recognition method according to claim 3, wherein said geometry data set is:

S (α) = \overline{S} + \underset{i = 1}{\sum^{m}} α_{i} s_{i}; and

said texture data set is:

T (β) = \overline{T} + \underset{i = 1}{\sum^{m}} β_{i} t_{i},

wherein S is the mean shape vector, T is the mean texture vector, s_iis the i_theigen-shape basis vector, t_iis the i_theigen-texture basis vector, α=[α₁, . . . , α_m] are the corresponding coefficients of said eigen-shape basis vectors and β=[β₁, . . . , β_m] are the corresponding coefficients of said eigen-texture basis vectors.

7. The face recognition method according to claim 3, wherein said 3D characteristic parameters are produced by minimizing a cost function: ∥I_input−I_model(α,β,P_ortho,f,R,t_2d)∥, wherein I_inputis said input original face image, P_orthois an orthographic projection matrix, f is the scaling factor, R is the 3D rotation matrix, and t_2dis the translation vector.

8. The face recognition method according to claim 1, wherein the step for obtaining said face recognition confidence value by fusing said temporal similarity probability of each said video frame by accumulation is:

P_{i}^{t} = \frac{(ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}{\sum_{i} (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}, i = 1, 2, \dots, n,

wherein ω is a forgetting factor, P_i^t,SVCis a posterior probability, denoting the probability that a single video frame belongs to the i_thclass at time t, and P_i^t,SVCis merged with previously accumulated probability P_i^t−1to contribute to the final similarity probability P_i^taccumulated up to time t.

9. The face recognition method according to claim 1 including:

calculating a rejection probability P_rej^tfor each said video frame; and

summing up said similarity probability P_i^talong with said rejection probability {circumflex over (P)}_rej^tup to time t for normalization and obtaining a posterior probability {circumflex over (P)}_i^tof said video frames up to time t belonging to class c.

10. The face recognition method according to claim 9, wherein steps for calculating said rejection probability of each said video frame comprises:

ɛ^{t} (w) =  I^{t} - \sum_{j = 1}^{K} w_{j}^{c, t} I_{j}^{c} ; and

P_{rej}^{t} = 1 - \exp (\frac{- ɛ^{t}}{ɛ_{threshold}^{c}}), {\hat{P}}_{rej}^{t} = ω \times {\hat{P}}_{rej}^{t - 1} + (1 - ω) \times P_{rej}^{t},

wherein w_j^c,tdenotes the weights of K nearest neighbors in class c that best approximate said video frame I^t, and

\underset{j = 1}{\sum^{K}} w_{j}^{c, t} = 1; {I_{j}^{c}}_{j = 1}^{K}

is the set of K nearest neighbors of the same class; and ε_threshold^cthreshold is the parameter for controlling rejection rate.

11. The face recognition method according to claim 9, wherein the step for summing up said similarity probability along with said rejection probability up to time t for normalization can be expressed as:

{\hat{P}}_{i}^{t} = \frac{β^{t} \times (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}{\begin{matrix} (ω \times {\hat{P}}_{rej}^{t - 1} + (1 - ω) \times P_{rej}^{t}) + β^{t} \times \\ \sum_{i} (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC}) \end{matrix}},

wherein i=1, 2, . . . , n, β^t=1−{circumflex over (P)}_rej^tis the acceptance rate at time t, and {circumflex over (P)}^t=[{circumflex over (P)}₁^t, {circumflex over (P)}₂^t, . . . , {circumflex over (P)}_rej^t]^Tis the posterior probability distribution.

12. The face recognition method according to claim 9, wherein if {circumflex over (P)}_i^tis greater than said threshold and

\arg \max_{i} {\hat{P}}_{i}^{t} = n + 1,

then said face image is determined to be belonging to an imposter; otherwise, said face image is recognized as a legal system user and his/her identity is further recognized, i.e.

c = \arg \max_{i} {\hat{P}}_{i}^{t},

and c=1, 2, . . . , n.

Patent Citations (13)

Patent	Date	Inventor	Cited By
US6556196(B1)	2003-04-01	Blanz et al.	Examiner
US7499574(B1)	2009-03-01	Yang et al.	Examiner
US7536030(B2)	2009-05-01	Wang et al.	Examiner
US7596247(B2)	2009-09-01	Ioffe	Examiner
US7664962(B2)	2010-02-01	Kuhlman	Examiner
US7728839(B2)	2010-06-01	Yang et al.	Examiner
US7783082(B2)	2010-08-01	Koshizen et al.	Examiner
US8050465(B2)	2011-11-01	Ianculescu et al.	Examiner
US2002/0106114(A1)	2002-08-01	Yan et al.	Examiner
US2003/0149803(A1)	2003-08-01	Wilson	Examiner
US2007/0031028(A1)	2007-02-01	Vetter et al.	Examiner
US2007/0122001(A1)	2007-05-01	Wang et al.	Examiner
US2010/0134487(A1)	2010-06-01	Lai et al.	Examiner

Non-Patent Literature (5)

“Video-Based Face Recognition Based on View Synthesis From 3D Face Model Reconstructed From a Single Image,” Chia-Te Liao, et al. 2008 IEEE International Conference on Multimedia and Expo, Jun. 23 2008-Apr. 26, 2008, pp. 1589-1592.Examiner
“Face Identification by Fitting a 3D Morphable Model using Linear Shape and Texture Error Functions,” Sami Romdhani, et al, the European Conference on Computer Vision, 2002, pp. 1-15.Examiner
“Face Recognition with Weighted Locally Linear Embedding,” Nathan Mekuz, et al. Proceedings of the Second Canadian Conference on Computer and Robot Vision (CRV'05), 7 pages.Examiner
“Separating Style and Content on a Nonlinear Manifold,” Ahmed Elgammal, et al., Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'04), 8 pages.Examiner
“Fitting a Morphable Model to 3D Scans of Faces,” Volker Blanz, et al., Proc. Of Int. Conf. on Computer Vision ICCV 2007, 8 pages.Examiner

Abstract

Description

Background of the Invention

[0001] 1. Field of the Invention

[0003] 2. Description of the Related Art

Summary of the Invention

Brief Description of the Drawings

[0008] FIG. 1 is a flowchart of the face recognition method of one embodiment of the present invention;

[0009] FIG. 2a-FIG. 2d are diagrams showing the 3D morphable model reconstructed from an original face image according to one embodiment of the present invention;

[0010] FIG. 3 shows the synthesized images according to one embodiment of the present invention; and

[0011] FIG. 4 is a diagram showing the conceptual framework of fusing temporal information for face recognition of one embodiment of the present invention.

Detailed Description of the Invention

, and 3D texture data set, T=(r₁, g₁, b₁, r₂, g₂, b₂, . . . , r_N, g_N, b_N)ε custom character

[0014]

\begin{matrix} S (α) = \overline{S} + \sum_{i = 1}^{m} α_{i} s_{i} & (1) \\ T (β) = \overline{T} + \sum_{i = 1}^{m} β_{i} t_{i} & (2) \end{matrix}

[0019]

\begin{matrix} P_{i}^{t} = \frac{(ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}{\sum_{i} (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}, i = 1, 2, \dots, n & (4) \end{matrix}

[0021]

\begin{matrix} ɛ^{t} (w) =  I^{t} - \sum_{j = 1}^{K} w_{j}^{c, t} I_{j}^{c}  & (5) \end{matrix}

wherein w_j^c,tdenotes the weights of K nearest neighbors in class c that best approximate said video frame I^t, and

[0022]

\sum_{j = 1}^{K} w_{j}^{c, t} = 1; {I_{j}^{c}}_{j = 1}^{K}

is the set of K nearest neighbors of the same class. Then a sigmoid function is applied to transform ε^tto the corresponding rejection probability P_rej^twith a recursive definition:

[0023]

\begin{matrix} P_{rej}^{t} = 1 - \exp (\frac{- ɛ^{t}}{ɛ_{threshold}^{c}}), {\hat{P}}_{rej}^{t} = ω \times {\hat{P}}_{rej}^{t - 1} + (1 - ω) \times P_{rej}^{t} & (6) \end{matrix}

[0024]

\begin{matrix} {\hat{P}}_{i}^{t} = \frac{β^{t} \times (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC})}{\begin{matrix} (ω \times {\hat{P}}_{rej}^{t - 1} + (1 - ω) \times P_{rej}^{t}) + β^{t} \times \\ \sum_{i} (ω \times P_{i}^{t - 1} + (1 - ω) \times P_{i}^{t, SVC}) \end{matrix}}, & (7) \end{matrix}

[0025]

\arg \max_{i} {\hat{P}}_{i}^{t} = n + 1

(class n+1 for the imposter class), then the to-be recognized face is rejected; otherwise, the face is determined to be of a legal system user and his/her identity is further recognized, i.e.

[0026]

c = \arg \max_{i} {\hat{P}}_{i}^{t}, c = 1, 2, \dots, n .