Person recognizing apparatus which can reduce time and year changing effects and method therefor

[0002] 1. Field of the Invention

[0003] The present invention relates to an apparatus and method for recognizing persons.

[0004] 2. Description of the Prior Art

[0005] Recently there has been a growing concern in security. A person recognition method utilizing human biometrical information involves identification of a person by discriminating in feature patterns between faces, voiceprints, fingerprints, irises and so on.

[0006] The face recognition method, among these, is characterized in that data is readily obtainable through a camera or the like without imposing mental or physical burden on the user.

[0007] Although the face recognition method have offered various study reports and literatures, there is a document (“Recent Research on Face Recognition by Computer”: Shigeru Akamatsu, The Journal of the Institute of electronics, information and communication engineers, vol. 80 No. 3 pp. 257-266 (1997)) disclosed as a survey document. This document reports that a recognition rate drops to 40-60% in the case that a picture is taken long time after entering an face image.

[0008] Japanese Patent Laying-open No. H9-138851 discloses a method of writing face information on a card in order to identify whether it is a card owner or not.

[0009] Face feature information on the card has to be rewritten in order to avoid card appropriation. In this method, the data is completely substituted by the recognition data from time to time, resulting in automatic decrease in recognition time interval. However, there is a problem that recognition accuracy will be deteriorated if this method is utilized as it is.

[0010] In identification of persons, it is generally preferred to utilize unchangeable fingerprint or the like. However, there involves a problem that fingerprint might inflict a mental displeased feeling upon a user.

[0011] Non-contact recognition methods includes the use of voiceprint or face, which is effective in obtaining information to distinguish between persons without burdening on the user.

[0012] In the case of face, however, there are aging changes due to lapse of time, such as in hair style, makeup, mustache and physical condition or due to lapse of years, such as in growth and maturity. Further, there exist other variation factors such as change in expression and face direction.

[0013] The aging change is also true for voiceprint.

[0014] FIG. 2 typically represents a similarity degree to entered data on a some person with respect to time. The horizontal axis denotes days from the entry, while the vertical axis represents a similarity degree expressed in 0.0-1.0, wherein 1.0 means identical. As in the figure, it is general that the similarity degree lowers as the number of days are elapsed. The data entry only once is not sufficient for conducting accurate authentication.

[0015] Therefore the present invention provides an apparatus and method for recognizing a person which can reduce the effect of time or year aging change.

[0016] The first aspect of the invention is a person recognizing apparatus, comprising: a feature pattern extracting means for extracting a feature pattern obtainable from a person to be recognized; a recognition information calculating means for calculating recognition information containing at least a subspace from the feature pattern extracted by the feature pattern extracting means; a registered information holding means for holding recognition information in multiple manner as registered recognition information; a recognizing means for recognizing the person by using recognition information calculated by the recognition information calculating means or a feature pattern extracted by the feature pattern extracting means and the registered recognition information held in the registered information holding means; a registered information updating means for updating the registered recognition information concerning the person held in the registered information holding means by using the recognition information calculated by the recognition information calculating means or the feature pattern extracted by the feature pattern extracting means.

[0017] In the person recognizing apparatus according to the first aspect, the registered information updating means Includes an update determining means for determining a predetermined method as to update of the registered recognition information by the registered information holding means, and an update information creating means for creating new registered recognition information based on the update method determined by the update determining means by using the recognition information calculated by the recognition information calculating means or the feature pattern extracted by aid feature pattern extracting means.

[0018] In the person recognizing apparatus according to the first aspect, the registered information holding means holds the registered recognition information in plurality of number concerning one objective person or one group, and the recognizing means recognizes with using registered recognition information in plurality of number concerning the one objective person or the one group in the registered information holding means.

[0019] In the person recognizing apparatus according to the first aspect, the update determining means selects recognition information to be updated among the recognition information calculated by the recognition information calculating means based on a predetermined rule, and the update information creating means including an update information creating means for creating new registered recognition information by using the selected recognition information by the update determining means.

[0020] In the person recognizing apparatus according to the first aspect, the update determining means determines an update condition based on a time of registering or updating the registered recognition information used by the recognizing means, and the update information creating means including an update information creating means for creating new registered recognition information based on the update condition determined by the update determining means.

[0021] In the person recognizing apparatus according to the first aspect, the recognizing means makes recognition using a method of calculating a similarity value, and the update determining means using another method of calculating a similarity value different from the method of calculating a similarity value by the recognizing means.

[0022] The second aspect of the invention is a person recognizing method comprising: a feature pattern extracting step to extract a feature pattern obtainable from a person to be recognized; a recognition information calculating step to calculate recognition information containing at least a subspace from the calculation amount extracted by the feature pattern extracting step; a registered information holding step to hold recognition information in plurality of number as registered recognition information; a recognizing step to recognize the person by using recognition information calculated by the recognition information calculating step or a feature pattern extracted by the feature pattern extracting step and the registered recognition information held in the registered information holding step; a registered information updating step to update the registered recognition information concerning the person held in the registered information holding step by using the recognition information calculated by the recognition information calculating step or the feature pattern extracted by the feature pattern extracting step.

[0023] In the person recognizing method according to the second aspect, the registered information updating step includes an update determining step to determine a predetermined method as to update of the registered recognition information by the registered information holding step, and an update information creating step to create new registered recognition information based on the update method determined by the update determining step by using the recognition information calculated by the recognition information calculating step or the feature pattern extracted by aid feature pattern extracting step.

[0024] In the person recognizing method according to the second aspect, the registered information holding step holds the registered recognition information in plurality of number concerning, one objective person or one group, and the recognizing step recognizes with using registered recognition information in plurality of number concerning the one objective person or the one group in the registered information holding step.

[0025] In the person recognizing method according to the second aspect, the update determining step selects recognition information to be updated among the recognition information calculated by the recognition information calculating step based on a predetermined rule, and the update information creating step including an update information creating step to create new registered recognition information by using the selected recognition information by the update determining step.

[0026] In the person recognizing method according to the second aspect, the update determining step determines an update condition based on a time of registering or updating the registered recognition information used by the recognizing step, and the update information creating step including an update information creating step for creating new registered recognition information based on the update condition determined by the update determining step.

[0027] In the person recognizing method according to the second aspect, the recognizing step makes recognition using a method of calculating a similarity value, and the update determining step using another method of calculating a similarity value different from the method of calculating a similarity degree by the recognizing step.

[0028] The third aspect of the invention is a person recognizing program recording means, comprising: a feature pattern extracting function for extracting a feature pattern obtainable from a person to be recognized; a recognition information calculating function for calculating recognition information containing at least a subspace from the calculation amount extracted by the feature pattern extracting function; a registered information holding function for holding recognition information in plurality of number as registered recognition information; a recognizing function for recognizing the person by using recognition information calculated by the recognition information calculating function or a feature pattern extracted by the feature pattern extracting function and the registered recognition information held in the registered information holding function; a registered information updating function for updating the registered recognition information concerning the person held in the registered information holding function by using the recognition information calculated by the recognition information calculating function or the feature pattern extracted by the feature pattern extracting function.

[0029] The invention of the first, second or third aspect can update the registration information and suppress against aging change.

[0030] The invention of the first or second aspect may perform automatic updating during authentication, thereby eliminating the necessity of troublesome re-registration and preventing the recognition rate due to aging change from lowering.

[0031] The invention of the first or second aspect makes it possible to recognize certain one person according to the registered recognition information in plurality of number. There is no requirement to adjust at all times a person condition in a same state, making possible to cope with large variation in the feature pattern of an objective person. Also, it is possible to image recognition with using any of the registered recognition information in plurality of number, further improving convenience.

[0032] FIG. 1 is a configuration diagram of a person recognizing apparatus 10;

[0033] FIG. 2 is a typical graph on aging change;

[0034] FIG. 3 is a configuration diagram of a feature pattern extracting section 12;

[0035] FIG. 4 is a view as an example of a feature pattern;

[0036] FIG. 5 is a flowchart for a recognition section 16;

[0037] FIGS. 6(a) and 6(b) are explanatory diagrams of recognition data multiplexing;

[0038] FIG. 7 is an explanatory view of recognition data combining;

[0039] FIG. 8 is an explanatory graph of weight coefficient;

[0040] FIG. 9 is a flowchart for an update method;

[0041] FIG. 10 is a graph wherein similarity values are determined on data being inputted in time series to represent their changes in time in a graph.

[0042] The present invention will be explained hereinbelow on one embodiment thereof.

[0043] Here, a person recognizing apparatus 10 is explained as an example, which authenticates on a human face as an object to be recognized by inputting an image through a TV camera in order to detect the face. The person recognizing apparatus 10 comprises, as shown in a configuration diagram of FIG. 1, a feature pattern extracting section 12, a subspace calculating section 14, a recognition section 16, a register information holding section 18 and a register-information updating section 20.

[0044] Such a structure is feasible by using a personal computer with a frame grabber connected with a TV camera. In order to give functioning to the structure, a program for functioning this is memorized on a hard disc, CD-ROM, MD, FD, MO, or DVD.

[0045] In the below explanation, information will appear which involves image input data, an extracted feature pattern, a subspace, eigenvectors for forming the subspace, a correlation matrix, situational information such as a time, day and hour and a place upon registration, and personal information such as a password number and identification code. Recognition data, when referred to, includes a subspace or eigenvectors for forming the subspace. Registered information, when referred to, includes image input data, an extracted feature pattern, a subspace, eigenvectors for forming the subspace, a correlation matrix, situational information, and personal information. Accordingly, recognition data is included in registered information.

[0046] (Feature Pattern Extracting Section 12)

[0047] A feature pattern extracting section 12 is to take a feature pattern from an object, which herein performs facial image analysis through image recognition as shown in FIG. 4.

[0048] FIG. 3 shows a configuration of the feature pattern extracting section 12 comprising an image input section 121, a face area detecting section 122, a facial parts detecting section 123 and a feature pattern creating section 124.

[0049] The image input section 121 inputs therein a facial image through an TV camera to perform A/D conversion and then send it to the face area detecting section 122

[0050] The face area detecting section 122 detects a face area out of the image.

[0051] The face area detecting section 122 uses a detecting method to move a previously prepared template in an image to determine a correlation value thereby obtaining a highest correlation value point as a face area.

[0052] Alternatively, it may be a face detecting means such as a face extracting method utilizing an eigenspace method or subspace method.

[0053] The facial parts detecting section 123 then detects facial parts such as an eye, nose and mouth.

[0054] Next the facial parts detecting section 123 detects positions of an eye and nose in detected face area region. The detecting method may use a method such as disclosed in a document (“Facial Feature Point Extraction Method Based on Combination of Shape Extraction and Pattern Matching”: Kazuhiro Fukui and Osamu Yamaguchi, The transactions of the institute of electronics, information and communication engineers, D-II, vol. J80-D-II, No. 8, pp 2170-2177 (1997)).

[0055] Next the feature pattern creating section 124 crop a face area into given size and shape based on positions of a detected part, and use its light-and-shade information as a feature pattern. The image intensity value of an area with m×n pixels is herein used as information, and m×n dimensional information is used as feature vector.

[0056] Instead of using image intensity information, another information such as edge intensity information may be used.

[0057] An upper figure in FIG. 4 is face image data obtained through processing an input image by the feature pattern extracting section 12, which is obtained from temporal image sequences.

[0058] (Subspace Calculating Section 14)

[0059] The subspace calculating section 14 determines a correlation matrix (or covariance matrix) for feature vectors and, by its K-L expansion, determine an orthonormal vector (eigenvector), thereby calculating for a subspace.

[0060] The subspace is expressed by using a set of eigenvectors in number k corresponding to eigenvalues and singled out in the order of greater eigenvalue.

[0061] In the present embodiment, a correlation matrix Cd is determined from feature vectors and diagonized as correlation matrix Cd=ΦdΛdΦd^T, thereby determining an eigenvector matrix Φd.

[0062] FIG. 4 is face image patterns obtained through processing the input image by the feature pattern extracting section 12. The subspace is computed from the correlation matrix of the feature vectors, and the eigenvectors are calculated by K-L expansion of the correlation matrix.

[0063] The lower figure of FIG. 4 is represented by imaging the eigenvectors.

[0064] This subspace is utilized as an identification dictionary for identifying a person.

[0065] Registration may be previously made to register it as a dictionary.

[0066] Otherwise, the subspace itself may be used as input data for identification, as will be stated later. Accordingly, a result of calculation by the subspace calculating section 14 is sent to a recognition section 16 or registered information holding section 18.

[0067] (Recognition Section 16)

[0068] Now the recognition section 16 is explained.

[0069] The recognition section 16 compares recognition data (subspace) stored in the registered information holding section 18 with a feature pattern obtained in the feature pattern extracting section 12 or a subspace obtained in the subspace calculating section 14, thereby identifying as to who is a person taken by the camera or whether it is the relevant person or not.

[0070] In order to identifying a person, it is satisfactory to determine what person data is most resembled to. A result of identification may be obtained as a person who corresponds to the data with the maximum similarity value.

[0071] Meanwhile, where performing a facial authentication using a card, registration number, password number, key, etc., a similarity value is calculated to the identification data corresponding to personal information, such as each person's card or number. A threshold is set to make identification as the person when the similarity value exceeds the threshold.

[0072] The recognition method may employ the subspace method or composite similarity method where using the information by the feature pattern extracting section 12.

[0073] The recognition method in the present embodiment uses a mutual subspace method described in a document (“A Pattern Matching Method with Local Structure”: by Ken-ichi Maeda and Sadakazu Watanabe, The transaction of the institute of electronis, information and communition engineers (D), vol. J68-D, No. 3, pp 345-352 (1985)).

[0074] In this method, the recognition data in pre-registered information and the data to be inputted are expressed as subspaces, and the “angle” given by the two subspace is defined as a similarity value. The subspace inputted is referred to as an input subspace.

[0075] A correlation matrix Cin is calculated from the input data sequence in the similar manner and diagonized as Cin=ΦinΛinΦin^T, thereby determining an eigenvector Φin.

[0076] The inter-subspace similarity (0.0-1.0) is determined on the two subspaces expressed by Φin and Φd, which are taken as a similarity value for recognition.

[0077] The recognition section 16 operates as in FIG. 5.

[0078] First, the recognition section 16 operates different work depending on whether to perform distinguishment or identification (step 1).

[0079] When performing identification, a target identification code for a target person is first read in (step 2).

[0080] Then registered information (subspace) corresponding to the target identification code is read from the registered information holding section 18 (step 3).

[0081] Then similarity value is calculated between the subspace of the registered information and the input vector (feature vector from the feature pattern extracting section 12) or the input subspace (step 4).

[0082] Then a threshold used in setting the similarity value is compared (steps 5, 6) and a result of recognition is outputted (step 7).

[0083] When performing distinguishment, all the data to be considered for distinguishment is read out from the registered information holding section 18 (step 8).

[0084] They are then calculated for similarity values to respective ones of registered information (step 9).

[0085] Then a maximum one is singled out of the similarity values calculated (step 10) and outputted as a recognition result (step 12).

[0086] As shown by a step 11 enclosed by broken lines in FIG. 5, it is also possible to verify as to whether the recognition result is correct or not by comparing the maximum similarity with the threshold (step 13). For example, where the similarity value is excessively low, it can also be determined that there are no recognized targets.

[0087] The similarity value and determination result calculated there is also sent to the registered information updating section 20, and utilized as information for updating.

[0088] (Registered information Holding Section 18)

[0089] The registered information holding section 18 can accumulate subspaces (or correlation matrix, etc.) utilized for person identification or situational information, such as registration time, day and hour, site, etc. They may be image input data or extracted feature patterns.

[0090] The registered information holding section 18 holds not only subspaces but also a correlation matrix at a prior stage for calculating a subspaces.

[0091] The registered information holding section 18 holds one or a plurality of recognition data corresponding to one person or a certain identification code. The subspace is memorized together with its accompanying information such as its acquired time. Also, these of the registered information are newly created, modified, deleted or the like by instructions from the registered information updating section 20.

[0092] The reason of holding multiple recognition data includes to make recognition on a plurality of subspaces for one person which are simultaneously sent to the recognition section 16.

[0093] In FIG. 6(a) shows an example wherein multiple recognition data are prepared. For example, where change frequently occurs such as wearing on eye glasses and taking off the eye glasses, two of recognition data are prepared. When identification with any of the prepared recognition data is made, it is determined that person-identification is made.

[0094] Besides eye glasses, it is possible to utilize to determine on the presence or absence of a women makeup.

[0095] This relieves a user of a burden of taking off eye glasses with troublesomeness for authentication.

[0096] Also, one subspace may be newly calculated from different ones of data which are obtained different in day and hour, as shown in FIG. 7. The multiple data are held in order to achieve recognition using the subspace. This will be explained in the registered information updating section 20.

[0097] Also, FIG. 6(b) shows an example wherein multiple recognition data are prepared for one code. For example, where facial authentication is utilized in security with a a “card usable by family members”, there is a requirement to deal with multiple face information.

[0098] In this case, recognition data in the number of family members are prepared for the numbers given for the card. When identified with any one, it is determined that identification through the recognition data is made.

[0099] This provides for improvement in convenience. There is of course no limitation to a family case.

[0100] (Registered information Updating Section 20)

[0101] The registered information updating section 20 performs an update on the registered information in the registered information holding section 18, as required. The registered information updating section 20 comprises an update determining section 201 for determining an update method and condition, and a update information creating section 202 for updating the registered information according to the determined method and condition.

[0102] <Update Determining Section 201>

[0103] The update determining section 201 in one example operates on a flowchart as shown in FIG. 9.

[0104] [Update Determination Based on a Time Lapsing]

[0105] The update determining section 201 receives a result of recognition (input data and similarity value) performed by the recognition section 16 (step 91).

[0106] Then the information of recognition data having been recognized is obtained from the registered information holding section 18 (step 92). This makes it possible to know on which data has been used and to obtain a lapse time from a time of recognition.

[0107] Whether an update should be made or not is determined from the data recognized, a length of a lapse time for updating a dictionary and a similarity value. It is first determined whether an elapsed time is greater than Ts (step 93). If the elapsed time is comparatively small, an update parameter is set for short term (step 94). These parameters, although explained later, refers to α, β and γi in the later explanation.

[0108] Similarly, whether the elapsed time is greater than T1 is determined (step 95). If no recognition has been made for a long time period, a long term parameter is set (step 96). Further, where no recognition is made for a long time period, it may be considered that a great variation has occurred and multiplexing to render recognition data a multiple manner may be made (step 97). In this manner, an update method for the recognition data is decided. Incidentally, Ts and T1 are empirically determined.

[0109] [Data Level Update Determination]

[0110] As for the input information used in recognition which received at the step 91, a similarity value between its input data and the dictionary has been determined. The data is selected not to be preferably used for updating which includes, among the input data having been used in recognition, data with a similarity value sufficiently obtained, input data with excessive variation and the data that is mistaken in image processing (step 93).

[0111] First, each of the input data is roughly divided into the following three groups depending on the magnitude in similarity value.

[0112] A. Input data exceeding by far a threshold for recognition (this input data is listed here because of high similarity value and no necessity to update using new data).

[0113] B. Input data having a possibility of mistaking in recognition and not included in the dictionary.

[0114] C. Input data having been recognized with clear mistake.

[0115] FIG. 10 is a graph wherein similarity values are calculated on input data sequence (black dots in the figure) to represent their changes in time.

[0116] There are various ones of data with high and low similarity values as shown in FIG. 10. The data is selected into three groups, e.g., one belonging to A and B or one belonging only to B.

[0117] For example, it is possible, in the above update determination with time lapse, to configure such that both A and B of data are used for updating when lapse of time is long, while only the data belonging to B is used when lapse of time is short. Correlation matrices or subspaces are calculated on them, and sent to the update information creating section 202.

[0118] It is of course possible to previously determine similarity values using another evaluation criteria (recognition method) in order to conduct test independent of the recognition section 16.

[0119] For example, in the recognition section 16 similarity values may be calculated using different algorithms to select respective feature patterns such that a mutual subspace method is used to calculate a similarity value to perform person identification and recognition while a subspace method is used in test for updating.

[0120] Such algorithms may be installed irrespective of their forms, that is, the recognition section 16 may possess a means to calculate a plurality of algorithms or the recognition section 16 and the update determining section 201 may be provided with respective algorithm calculating means. This test for updating may be an authentication in a level of such feature patterns (input data) or an authentication in a level expressed by subspaces.

[0121] <Update Information Creating Section 202>

[0122] The update information creating section 202 creates new registered information using the data already registered and the data used in authentication.

[0123] The update method includes

[0124] 1) a method to update a certain subspace

[0125] 2) a method to combine a new subspace

[0126] 3) a method to add a subspace having been used in recognition

[0127] These are explained in order hereinbelow.

[0128] 1) Method to update a certain subspace

[0129] The present embodiment performs update in the following manner because correlation matrix is held in the recognition data.

[0130] It is assumed that a correlation matrix for the recognition data is Cd and a correlation matrix for the input data is Cin.

[0131] New recognition data (correlation matrix) is calculated based on the following equation from the above

[0132] Cin=αCd+βCin.

[0133] Here, the update method can be adjusted by varying setting of parameters α and β.

[0134] Assuming that a 1.0 and β=1.0, addition is sequentially made and recognition data is made up using all the data collected so far.

[0135] Assuming that α=1.0 and β=1.0, the update ratio with current data decreases but the increases in number of times decreases a contribution of time-elapsed data. For example, if updates are made n number of times, the contribution ratio of the first recognition data is 1/2ⁿ.

[0136] The parameters α and β may be adjusted by taking into account lapse of time, and so on.

[0137] 2) Method to Combining a New Subspace

[0138] Another embodiment is provided wherein a plurality of correlation matrices are held each time recognition is performed, and a subspace in a dictionary used for recognition is newly calculated from the correlation matrices.

[0139] For example, using correlation matrices Ci and weight coefficients γi accumulated by each recognition $D=\sum _i^n\gamma \mathrm{iCi}$

[0140] Using the matrix D, subspaces are calculated to obtain a dictionary (eigenvectors).

[0141] The weight coefficients γi are determined using a sigmoid function taking lapse of time into account, as shown in FIG. 8. It is of course possible to select another function for determining the weight coefficients. An empirical weight coefficient table may be prepared. Also, the number of correlation matrices Ci is not fixed.

[0142] The method of combining subspaces may, of course, utilize a method to calculate on common portions between subspaces, besides the above-stated additional calculation.

[0143] 3) Method to Add Subspaces Used in Recognition

[0144] It is also possible to automatically multiplex in multiple recognition data.

[0145] For example, where exceeding a threshold but the similarity value is excessively low, Cin may be added as recognition data as they are to increase the number, instead of performing update as stated before.

[0146] Incidentally, although the correlation matrix was also held in order to calculate subspaces, a matrix Φ^TΦ may be calculated, without having a correlation matrix, in place of a correlation matrix by using an eigenvector matrix Φ representative of subspaces.

[0147] In such a case, the memory capacity can be reduced as compared with holding a correlation matrix.

[0148] The subspaces are updated by appropriately using these method jointly or in combination, thereby maintaining accuracy of the dictionary and performing aging-suppressed recognition.

[0149] (Modification)

[0150] Although the present embodiment was explained on the recognition example using faces, voice prints may be used as feature patterns. Alternatively, other feature pattern may be utilized.

[0151] The present invention can be practiced in various modified forms within a range of not departing from its gist.

[0152] According to the present invention, it is possible to automatically update the registered information by using recognition information containing at least a subspace. Further, it is possible to increase resistance to variation, cope with aging change and maintain the recognition rate. That is, the effect of time and year aging changes are relieved by updating the registered information from time to time.

[0153] Besides being possible to eliminate the effect of time and year aging changes, it is also possible to eliminate the effect of change caused in a brief time, such as change in expression or face direction. In other words, a subspace calculated from a feature pattern is used as registered information in recognition procedure so as to absorb the change caused in a brief time and, in addition, to flexibly cope with the time and year aging changes.

[0154] If the registered data is updated by providing an automatic update mechanism using information upon recognition instead of explicit implementing, the recognition rate can be avoided from lowering without burdening on a person.

[0155] Furthermore, if multiple registered data is held in order to cope with large variation in the presence or absence of spectacles or makeup, the convenience is enhanced and the recognition rate is improved by collation with the multiple data.