Method for verifying the authenticity of an image recorded in a person identifying process

[0001] The present invention relates to a method for verifying the authenticity of an image recorded in a person identifying process and belonging to a person to be identified.

[0002] Known person-identifying methods which are e.g. based on face recognition as described in German patent 44 06 020 only compare single photographs of the person to be identified with reference pictures. Thus only static information is processed in said methods. Such methods have the drawback that an access control operating with such a method can easily be deceived by showing a photo or a mask-like picture of the person.

[0003] Therefore, various methods have been developed for reducing the risk of such deceptions.

[0004] For instance, it is known from German Utility Model 2 950 660 that the spectral range of the light used for identifying a person is limited by an IR filter disc to a range in which a photo or a mask-like picture has reflectance characteristics which differ from those of an actually recorded face. Such an IR filter disc has the drawback that the image quality is reduced thereby and that the demands made on the quality of the identifying process increase. On the other hand, there is the risk that an image medium is found with reflectance characteristics in the IR range that are similar to the reflectance characteristics of a person's face in the IR range, whereby a deception is again possible with such an image medium.

[0005] Furthermore, the prior art discloses methods in which the picture of the person to be identified is taken with a stereo imaging system. For instance, such a method is used in the face recognizing system described in the article “In-Your-Face Security” in PCWeek of Mar. 26, 1997. Other methods for detecting the three-dimensional structure of a face either fully or in part are laser triangulation or the so-called shape-from-shading methods disclosed by Attick, Griffin and Redlich in Neural Computation 8, 1321-1340, 1996; these make it possible to draw conclusions from the gray-level curve at a given direction of illumination with respect to the depth information of the object viewed. Although deceptions in the case of which an image is held in planar fashion in front of the recording system can be detected with such methods, deceptions with a cylindrically bent image or a three-dimensional mask cannot be recognized as such.

[0006] Moreover, the prior art discloses methods in which the pictures of the person to be identified are taken by means of thermographic methods; see e.g. German patent 4 009 051. In this instance, the radiant heat emitted by the person to be identified is detected and compared with a given reference value for identifying the person. This method, however, has the drawback that the devices for carrying out the methods are very expensive, in particular the thermographic camera used for image-recording the person to be identified. Thus thermographic methods cannot be used in access controls which are to be used in many places, e.g. in cash dispensers for withdrawing money and/or in banking transactions.

[0007] In view of the above-discussed drawbacks of the devices known from the prior art, it is the object of the present invention to provide a method for verifying the authenticity of an image recorded in a person identifying process and belonging to a person to be identified, in the case of which the reliability during verification of the authenticity is improved over the prior art on the one hand and thus the security to be achieved with an access control, and which can be implemented at low costs on the other hand.

[0008] This object is achieved by a method for verifying the authenticity of an image recorded in a person identifying process and belonging to a person to be identified, the method comprising the steps of recording a sequence of consecutive individual images of the person and of determining the authenticity of the recorded image if in at least two consecutive individual images of the sequence intrinsic movements of the person are detected.

[0009] Since in the method of the invention intrinsic movements of a person to be identified, i.e. movements carried out by every person, i.e. also a person to be identified, consciously or unconsciously and without changes in his or her center of gravity are used for verifying the authenticity of the recorded images, the authenticity will only be detected if the consecutively recorded individual images actually originate from a person performing intrinsic movements, i.e. from a living person. Thus a deception of an access control using the method of the invention is no longer possible by showing a cylindrically bent photo or a mask-like picture of the person to be identified, which considerably enhances the security guaranteed by the access control.

[0010] Furthermore, the method according to the invention can be implemented at relative low costs because only an image recording means (called recording means in the following) and an image evaluating means are required for carrying out the method and because said two means for carrying out the image recognition proper are already provided for. Consequently, the method of the invention for verifying the authenticity of a recorded image is also suited for use in access controls which are to be used in many places, e.g. in cash dispensers for withdrawing money and/or banking transactions.

[0011] According to a preferred development of the inventive method at least one differential image of two consecutive individual images of the sequence can be evaluated for determining the intrinsic movements.

[0012] The amount of data to be processed in the method of the invention can considerably be reduced by the formation of such differential images, whereby the method can be carried out at a faster pace. This has, inter alia, the effect that the demands made on the hardware for performing the method can be reduced, whereby the total costs of the system are immediately decreased.

[0013] Furthermore, according to a preferred variant a differential image which results from an AND operation of two consecutive differential images can be evaluated for determining the intrinsic movements.

[0014] Such a development yields, in particular, a differential image in which changes are considerably reduced that follow e.g. from a release of a dominating background in two consecutive individual images and are thus reflected in the differential image of two consecutive individual images. In other words, the differential image to be assessed in the final analysis is freed by this variant from changes that do not result from intrinsic movements.

[0015] According to another preferred development each differential image can be binarized prior to evaluation.

[0016] This measure, in turn, greatly reduces the amount of data to be processed, resulting in an increase in the speed of the method and thus in smaller demands on the hardware having the same efficiency.

[0017] Binarization may e.g. be carried out by means of a threshold value which is determined by evaluating the background of one of the differential images, e.g. by averaging the intensities of those pixels in the differential image that are in the area of the static background. This has the advantage that binarization is always determined in dependence upon the instantaneously existing image-recording situation. Thus the method can be adapted to environment-induced changes, for instance to changing light and/or contrast conditions, thanks to such a development.

[0018] Furthermore, at least one section of enhanced movement can be selected in each differential image, and it is possible to detect intrinsic movements if said at least one section in the differential image is consistent with a predetermined region in which intrinsic movements are to be expected.

[0019] Thanks to such a development of the method of the invention, the amount of data to be processed can be reduced again, which results in the already above-discussed advantages.

[0020] To this end, according to a first alternative, the section of enhanced movement can be defined by a vertical and a horizontal extent, the vertical extent being substantially determined by a peak of the function of the products from the horizontal projections with the horizontal variances of the differential image, and the horizontal extent being substantially determined by a peak of the function of the products from the vertical projections with the vertical variances of the differential image.

[0021] Advantageously, each function can be smoothed with a low-pass filter prior to the determination of the corresponding peak. This measure helps to avoid high-frequency noise in the differential images, which might lead to an impaired determination of the peak and thus of the section of enhanced movement. On the whole, this yields a more robust determination of the section of enhanced movement.

[0022] According to a further development of the first alternative, sections can be chosen from the differential image that are supposed to be in symmetrical relationship with one another, and intrinsic movements can be detected when the symmetrical relationship is verified by a correlation analysis of the corresponding sections. Since intrinsic movements, e.g. in the face of a person to be identified, are symmetrical as a rule, such a design helps to check whether the intrinsic movement detected in a specific region is actually an intrinsic one or just a different movement, e.g. noise or the like.

[0023] Moreover, independently of or together with said check, a further check can be carried out as to whether the enhanced movement in the region of enhanced movement is irregularly distributed over said region.

[0024] It can thereby be ruled out that small changes which might add up under adverse conditions are wrongly detected in their sum as an intrinsic movement.

[0025] According to a second alternative, a function which depends on the vertical position and which is determined for a given vertical position through the product of the horizontal projection with the horizontal variance of the differential image, and a function which depends on the horizontal position and which is determined for a given horizontal position through the product from the vertical projection with the vertical variance of the differential image can also be evaluated for detecting intrinsic movements.

[0026] Preferably, in the second alternative, it is also possible to smooth each function with a low-pass filter prior to the determination of the corresponding peak so as to filter high-frequency noise out of the differential images.

[0027] According to an advantageous development, it is possible for the evaluation of the above-mentioned functions to determine at least one peak in the function depending on the vertical position, and at least one peak in the function depending on the horizontal position, and finally intrinsic movements can be detected if said peaks are within predetermined limits in the vertical and horizontal direction, said limits being defined by the vertical and horizontal extent of at least one area in which intrinsic movements are to be expected.

[0028] By analogy with the first alternative, checking methods which are based on the fact that intrinsic movements are normally exhibiting symmetries can also be carried out for intrinsic movements detected with the second alternative.

[0029] For instance, sections which are supposed to be in a symmetrical relationship with one another can be evaluated on the basis of the function depending on the vertical position and/or the function depending on the horizontal position, and intrinsic movements can finally be detected if the symmetrical relationship is verified by a correlation analysis of the corresponding sections.

[0030] To make said checking method more robust with respect to relative deviations of the sections of the function which are supposed to be in a symmetrical relationship with one another, the sections may be standardized with respect to one another prior to the correlation analysis.

[0031] In addition or as an alternative to said checking method, the function depending on the vertical position and/or the function depending on the horizontal position can be evaluated, and intrinsic movements can be detected if it is determined by way of a correlation analysis that the values of the function depending on the vertical position and/or of the function depending on the horizontal position are irregular over a predetermined range.

[0032] Such an additional check will rule out that small changes that may add up in adverse cases are wrongly recognized in their sum as an intrinsic movement.

[0033] According to a preferred development of the above-discussed development of the inventive method intrinsic movements can be determined in the head portion of the person to be identified for verifying the authenticity of the recorded images.

[0034] This development has the advantage that the head portion as compared to the whole body of a person to be identified shows most of the intrinsic movements, which are also the most dominating ones. Thus, the method of the invention can be carried out in the most efficient way in the head region of a person to be identified.

[0035] Regions with particularly pronounced intrinsic movements in the head portion are e.g. the mouth region and/or the cheek region and/or the nose region and/or the eye region.

[0036] In a preferred development the intrinsic movements to be determined are intrinsic movements in at least two regions that are symmetrical to each other in the head portion of the person to be identified, and said at least two symmetrical regions are used for determining the axis of symmetry of the head portion.

[0037] With the symmetry axis found in this way it is possible to produce an image which can be used for identifying a person and in which a frontoparallel-rotated head region is transformed into a head region with a rotated axis of symmetry.

[0038] In addition or alternatively, an image of the person to be identified which can be used for identifying the person and which is composed of the left head region and the mirrored left head region or of the right head region and the mirrored right head region, respectively, can be produced by way of the symmetry axis found.

[0039] Both alternatives have the advantage that improved images can be provided for a later person identification. For instance, in the case of the first alternative, recorded images in which the head region of the person to be identified is inclined relative to the vertical can be used for image recognition. With the second alternative, it is also possible to use those images for person identification in which an image half is glared and which would thus be useless for the identification of a person.

[0040] According to another preferred development of all of the above-discussed embodiments the regions in the individual images or frames in which intrinsic movements are expected can be extracted from the individual images prior to the detection of the authenticity.

[0041] This development has the advantage that the images required for detecting the authenticity can be extracted from any desired video sequences.

[0042] Thus, the person to be identified has just to step into the visual field of the recording means. This has the advantage that optimum individual images can be selected from a plurality of individual images for respectively detecting the authenticity. Furthermore, it is not necessary that the person to be identified assumes a fixedly predetermined position so that the image recordings required for identifying the person and for verifying the authenticity can be made of said person.

[0043] Like in the detection of the authenticity, advantageously at least one differential image of two consecutive individual images of the sequence can be evaluated for reducing the data to be processed during extraction of the regions.

[0044] Furthermore, during extraction of the regions it is also possible to evaluate a differential image resulting from an AND operation of two consecutive differential images if the differential image to be finally evaluated is to be freed from changes resulting e.g. from the release of a dominating part of the background.

[0045] Likewise, each differential image can be binarized prior to evaluation and the binarizing operation can e.g. be carried out by means of a threshold value which is determined by evaluating the background of one of the differential images, e.g. by averaging the intensities of those pixels in the differential image that are in the region of the static background. As already discussed above in connection with the detection of intrinsic movements, the amount of the data to be processed can considerably be reduced by this development.

[0046] Analogously to the detection of the intrinsic movements the head region of the person to be identified can be extracted in an advantageous manner.

[0047] According to a preferred development the extracted head region can be transformed to a predetermined standard size for identifying the person. Variations in size which might lead to problems, in particular with respect to the computing time, in the complicated algorithms for face recognition can thereby be taken into account.

[0048] For extracting the head portion at least two head boundaries can be determined in the corresponding individual images or differential images, and the head region can be extracted on the basis of said head boundaries from the corresponding individual images or differential images.

[0049] According to a first alternative the determination of the head boundary in the corresponding individual images or differential images may include the determination of the upper and the left head boundary.

[0050] According to a second alternative the right head boundary can additionally be determined.

[0051] Expediently, but not limited thereto, each head boundary may be defined by a head boundary line which extends such that the contour of the head is substantially entirely located within the head boundary lines.

[0052] According to a preferred development the function of the vertical projections of the binarized differential image can be determined for determining the upper head boundary, and the upper head boundary can be defined by the first maximum of the absolute value of the first derivative of said function that is above a predetermined threshold value.

[0053] If high-frequency noise is to be eliminated, the function of the vertical projections could be smoothed with a low-pass filter prior to the definition of the head boundary.

[0054] For the determination of the left head boundary the function of the horizontal projections of the binarized differential image is determined, and the left head boundary can be defined by the first maximum of the absolute value of the first derivative of said function that is above a predetermined threshold value.

[0055] Alternatively, for the determination of the left head boundary the differential image can be divided into a plurality of vertically successive strips, the first strip being vertically downwardly adjacent to the upper head boundary determined; in each strip the function of the horizontal projections of the binarized differential image can then be determined, the absolute values of the first derivatives of the resulting plurality of functions of the horizontal projections can subsequently be formed, the sum of the resulting plurality of absolute values can be added, and the left head boundary can finally be defined as the first maximum of said sum that is above a predetermined threshold value.

[0056] The advantage of said second alternative is that the head region is only detected through the strips below the upper head boundary, and the shoulder portion positioned thereunder, which might lead to inaccurate results in the determination of the left head boundary, is masked.

[0057] In this case, too, the function(s) of the horizontal projections can optionally be smoothed with a low-pass filter prior to the definition of the head boundary.

[0058] By analogy with the first alternative regarding the determination of the left head boundary, it is possible to determine the function of the horizontal projections of the binarized differential image according to the preferred development in which the right head boundary is also determined, and the right head boundary can be defined by the last maximum of the absolute value of the first derivative of said function that is above a predetermined threshold value.

[0059] By analogy with the second alternative in the determination of the left head boundary, it is possible according to the preferred development in which the right head boundary is also determined, to divide the differential image into a plurality of vertically successive strips for the determination of the right head boundary; in this case the first strip is vertically downwardly adjacent to the upper head boundary determined, the function of the horizontal projections of the binarized differential image is determined in each strip, the absolute values of the first derivatives of the resulting plurality of functions of the horizontal projections are formed, the sum of the resulting plurality of absolute values is added, and the right head boundary is defined as the last maximum of said sum that is above a predetermined threshold value.

[0060] It is here also possible to smooth the function or functions of the horizontal projections with a low-pass filter prior to defining the head boundary.

[0061] According to a preferred development it might be intended in the determination of the head boundaries that two successive individual images or the differential images obtained therefrom are only used for determining the head boundaries if a change between the two consecutive individual images is within a predetermined range.

[0062] It is thereby easily possible to make sure even before the determination of the head boundaries that only those differential images are used for a determination in the case of which it is ensured that sufficiently reliable results for the head boundaries can be obtained therefrom. Since the change between two consecutive individual images must be greater than a predetermined value (so that head boundaries can be calculated according to said preferred development), it is ensured that in both images a change going beyond normal noise and thus an actual movement of the person to be identified have taken place. Since the change between two consecutive individual images must be smaller than a predetermined value (for calculating head boundaries), differential images in which an excessively strong movement is reflected, as is e.g. produced by shaking the recording means or an extremely fast movement of the person to be identified, are left out of consideration in determining the head boundaries.

[0063] According to a particularly advantageous development with respect to the necessary computing time, a motional intensity can be calculated for determining the change between two consecutive individual images. In the case of a gray-level image the motional intensity can substantially be calculated through the sum of the gray level of the differential image. In a binarized differential image the motional intensity can expediently be calculated as the sum of the 1 pixel or 0 pixel of the binarized differential image.

[0064] According to a preferred development predetermined head boundaries can be used for extracting the head region in the case where no head boundaries can be determined.

[0065] According to a further advantageous development predetermined head boundaries can be used for extracting the head region if one of the head boundaries determined is not within predetermined limits. Alternatively, the step for determining the head boundaries can be carried out with successive differential images.

[0066] It is thereby possible to perform an additional plausibility check of the boundaries determined in the method. In other words, it is checked whether in the region defined by said head boundaries the head of a person would actually have to be expected with respect to the arrangement of the device. If the head boundaries determined are not plausible, predetermined head boundaries are assumed in the end. Thanks to this additional check the quality of the determination of the head boundary and, in the final analysis, the quality of the verification of the authenticity can additionally be enhanced.

[0067] Expediently, in the above-described developments for determining the head boundaries, the lower or the right and lower head boundaries can be defined such that a square head region is extracted from the corresponding individual images or differential images.

[0068] According to a preferred development of all of the previously described variants of the method of the invention a stabilized differential image which is evaluated for determining the intrinsic movements and/or for extracting the regions in which intrinsic movements are to be determined can be evaluated on the basis of two consecutive images.

[0069] This development can be used independently of and alternatively to the extraction methods for regions in which intrinsic movements are expected. Thus all of the advantages mentioned in connection with the determination of the head boundaries could here be mentioned as an advantage.

[0070] On the other hand, said development can also be used for increasing the accuracy of the regions with intrinsic movements, as determined in the extracting process.

[0071] A stabilization can here be carried out by means of a correlation correction.

[0072] According to a predetermined development the correlation correction can be carried out by template matching which comprises calculating the differential image. in the case of which in calculating the differential image the first individual image used for forming the differential image is shifted relative to the second individual image used for forming the differential image in such a way that a correction function between the two individual images is maximum.

[0073] According to another advantageous development, two consecutive individual images or differential images obtained therefrom are only used in the above-described methods for determining the intrinsic movements if the change between the two consecutive individual images is within a predetermined range.

[0074] It is thereby possible to select consecutive individual images already at the beginning of the inventive method for obtaining sufficiently reliable results of the verification.

[0075] In particular, it is easily possible to make sure already before the determination of the intrinsic movements that for the determining process use is only made of differential images by which it is ensured that sufficiently reliable results can be obtained therefrom. Since the change between two consecutive individual images must be greater than a predetermined value (so that according to said preferred development the attempt is made to determine an intrinsic movement), it is ensured that in the two images a change going beyond normal noise and thus an actual movement or an intrinsic movement of the person to be identified has taken place. Since the change between two consecutive individual images has to be smaller than a predetermined value (for calculating head boundaries), differential images which reflect excessive movements, as are e.g. caused by shaking the recording means or by an extremely fast movement of the person to be identified, are not used for determining the intrinsic movements.

[0076] Preferably, the change between two consecutive individual images can be calculated through a motional intensity. Like in the determination of the head boundaries, the motional intensity can be calculated for a gray-level image through the sum of the gray levels. By analogy, the motional intensity can expediently be calculated for a binary differential image through the sum of the 1 pixel or 0 pixel of the binarized differential image.

[0087] FIG. 1 shows a sequence diagram of an inventive method for verifying the authenticity of an image recorded in a person identifying process and belonging to a person to be identified.

[0088] In a first step S110 a sequence of consecutive individual images or frames K(t), t=1, . . . , n of a person is recorded. To this end, use is expediently made of prior-art video cameras which record the sequence in digitized form. Individual images recorded with such a video camera are normally present in the form of pixels, each individual image comprising c×r pixels if c designates the number of the columns and r the number of the lines of the corresponding individual image.

[0089] In the following K(t) therefore stands for the whole individual image at time t and K_cr (t) for the pixel in the c-th column and the r-th row.

[0090] The individual images or frames are recorded in the inventive method at a speed of 8-12 images/per second.

[0091] In step S120, two consecutive individual images K(t−1) and K(t) are chosen from the recorded sequence of individual images.

[0092] In the subsequent step S130, the individual images (K(t−1) and K(t) are compared with each other to find out whether the person subjected to an identification has performed intrinsic movements between two individual images K(t−1) and K(t).

[0093] To this end the two individual images K(t−1) and K(t) can be compared with each other pixel by pixel, and the change resulting from said comparison of pixels can be compared with a predetermined threshold value. Thus if the amount of change is above the predetermined threshold value, the person has performed an intrinsic movement between two individual images K(t−1) and K(t), whereas if the amount of change is below the threshold value, no intrinsic movements of the person could be detected. Said pixel-by-pixel comparison yields satisfactory results if no additional movements are superimposed on the intrinsic movements of the person, i.e. if the person does not move or only moves minimally with respect to the image recording means.

[0094] How the comparison has to be made in detail depends on the type of the consecutive individual images. If the individual images are e.g. present as gray-level images, the amount of the difference of the gray levels of two corresponding pixels in the individual image K(t−1) and K(t) can be regarded as a measure of the change in a pixel, and the sum of the amounts of said differences as a measure of the change between the two individual images K(t−1) and K(t) that are finally compared with the predetermined threshold value.

[0095] If the individual images are e.g. present in binarized form (i.e. in the form of white or black pixels), the change between two individual images follows from the sum of the differing pixels corresponding to each other in the individual images K(t−1) and K(t). Thus, in the case of binarized individual images said change has to be compared with a predetermined threshold value. The binarization of gray-level images will be described in detail in connection with the second embodiment (step S240).

[0096] Finally, if the comparison in step S130 shows that intrinsic movements of the person have taken place between the two individual images, the authenticity of the recorded individual images K(t−1) and K(t) is verified in step S170.

[0097] By contrast, if the comparison shows that the person has not performed intrinsic movements between the individual images K(t−1) and K(t), the authenticity of the recorded individual images K(t−1) and K(t) cannot be verified in step S180.

[0098] In this case it is possible to interrupt the method and if said method is used within the scope of an access control, access can be denied. Alternatively, as shown by step S190, it is possible to jump back to step S120, and the method for verifying the authenticity can be carried out again with two further individual images, e.g. K(t) and K(t+1).

[0099] The loop shown by step S190 can be passed through j times, j being a predetermined number which is n−1 at the most.

[0100] Alternatively to the comparison of the individual images K(t−1) and K(t), a differential image D(t)=K(t)−K(t−1) can be formed for reducing the amount of data to be processed, and finally the differential image D(t) can directly be analyzed to find out whether the person has performed intrinsic movements between the two individual images K(t−1) and K(t).

[0101] According to a further alternative of the first embodiment, instead of the differential image D(t) an AND operation of two consecutive differential images D(t) and D(t−1) can be evaluated for detecting the intrinsic movements. Said alternative is particularly useful when there are great changes in two consecutive individual images due to the release of a dominating background, for instance a source of illumination, in two consecutive individual images, and when these are reflected in the differential image. Thus, the differential image to be assessed in the final analysis can be freed by said alternative from changes that do not result from intrinsic movements.

[0102] Since in said alternative two differential images D(t) and D(t+1) are used, three consecutive individual images K(t−1), K(t) and K(t+1) are needed for producing said two differential images.

[0103] FIG. 2 shows a sequence diagram of a second embodiment of the method according to the invention.

[0104] Said second embodiment is predominantly an alternative to step 130 in the first embodiment.

[0105] Thus, according to the second embodiment, steps S110 and S120 are also passed through first, i.e. a sequence of consecutive individual images is recorded and two consecutive individual images are chosen from said sequence.

[0106] Following step S230 the differential image D(t) is determined in step S230 from the individual images K(t) and K(t−1).

[0107] In the optional step S240 said differential image can be binarized according to the following equation:
B_δ(t)=Θ(D(t)−δ).  (1)
δ is here a predetermined threshold value, and θ(l) designates the step function, i.e., θ(l)=0 for l<0, and θ(l)=1 for l≧0.

[0108] The threshold value δ may be fixedly predetermined or determined by evaluation of the background.

[0109] In the last-mentioned case, the threshold value δ should be defined as a function of the upper m lines of the differential image because said lines only show static background, and values different from 0 in the differential image are thus due to noise caused, for instance, by the recording device. Consequently, noise is suppressed if only gray-level values above said threshold value are considered. Thus the maximum value found in the uppermost m lines of the differential image should be used as the threshold value, i.e. $\begin{array}{cc}\delta =\underset{c,r<m}{\max }{D}_{\mathrm{cr}}\left(t\right),& \left(2\right)\end{array}$

[0110] In the next step S250, at least one region of enhanced movement is determined in the differential image D(t) or in the binarized differential image B(t).

[0111] To this end a masking frame of any desired size may be predetermined and guided over the corresponding differential image, and upon each shift the movement is calculated, i.e. the number of the sum of the pixels with the value 1 or, alternatively, of the pixels with the value 0. The masking frame of the shift with the largest sum finally defines the region of enhanced movement.

[0112] After the region of enhanced movement has been determined in the differential image or the binarized differential image, it is checked whether said region corresponds to a predetermined region in which intrinsic movements are to be expected.

[0113] If this is the case the authenticity of the recorded individual images K(t−1) and K(t) underlying the differential image D(t) or the binarized differential image B(t) are verified in step S170.

[0114] If the region of enhanced movement is not identical with the predetermined region, the authenticity of the recorded individual images cannot be verified in step S180, and the method may be interrupted or repeated according to step S190 (see FIG. 1).

[0115] The region in which most of the intrinsic movements of a person have to be expected is the head region. In the head region as such, most of the intrinsic movements are to be expected in the mouth region, the cheek region, the nose region or the eye region. Preferably, according to the second embodiment in FIG. 2, the at least one determined region of enhanced movement is therefore compared with the mouth region, the cheek region, the nose region and/or the eye region.

[0116] As already implicitly pointed out, it is of course also possible to determine several regions of enhanced movement in the differential image and to check whether said determined regions of enhanced movement correspond to the mouth region, the cheek region, the nose region and/or the eye region.

[0117] FIG. 3 shows a sequence diagram of a third embodiment of the method of the invention, which illustrates an alternative to step S250 in the second embodiment, i.e. the determination of the region of enhanced movement, and, moreover, corresponds to the second embodiment.

[0118] Hence, in the third embodiment a differential image D(t) or, alternatively, a binarized differential image B(t) is started from. The differential image D(t) or the binarized differential image B(t) will be abbreviated in the following text as differential image K.

[0119] In step S350, the function Y_V of the products from the horizontal projections P_H and the horizontal variances V_H is determined on the basis of the differential image K, in formulae: $\begin{array}{cc}{Y}_V=P_H·V_H=\left(\sum _c{K}_{\mathrm{cr}}\right)·\left(\frac{1}{\mathrm{nc}}\sum _c{\left(K_{\mathrm{cr}}-{\stackrel{\_}{K}}_r\right)}^2\right),\mathrm{with}{\stackrel{\_}{K}}_r=\frac{1}{\mathrm{nc}}\sum _c{K}_{\mathrm{cr}},\mathrm{and}\mathrm{nc}=\mathrm{number}\mathrm{of}\mathrm{columns}\mathrm{of}K.& \left(3\right)\end{array}$

[0120] Furthermore, in step S351, the function Y_H of the products from the vertical projections P_V and the vertical variances V_V is determined from the differential image K according to the following formula: $\begin{array}{cc}{Y}_H=P_V·V_V=\left(\sum _r{K}_{\mathrm{cr}}\right)·\left(\frac{1}{\mathrm{nr}}\sum _r{\left(K_{\mathrm{cr}}-{\stackrel{\_}{K}}_c\right)}^2\right),\mathrm{with}{\stackrel{\_}{K}}_c=\frac{1}{\mathrm{nr}}\sum _r{K}_{\mathrm{cr}},\mathrm{and}\mathrm{nr}=\mathrm{number}\mathrm{of}\mathrm{columns}\mathrm{of}K.& \left(4\right)\end{array}$

[0121] In step S352, the functions Y_V and Y_H obtained according to the above formulae can be smoothed with a suitable low-pass filter TPF{ }.

[0122] In functions Y_V and Y_H or the smoothed functions TPF{Y_V} and TPF{Y_H}, peaks are identified in step S353. The peaks of Y_V and Y_H, respectively, define the vertical and horizontal extent and position of the region of enhanced movement.

[0123] Various definitions which are known in the prior art can be given as a relation between the peak of one of the functions and the corresponding region of enhanced movement. For instance, the position of the corresponding region can be indicated by the maximum value of a peak and the extent of said region by the FWHM (full width of half maximum) of the peak. Alternatively, the extent can also be indicated by the turning points of the peak. As a further alternative, the extent can also be indicated by the boundaries which lead to a predetermined area below the peak.

[0124] Standard prior-art methods can be used for finding the individual peaks. These methods substantially consist in looking for the sections of a function that exceed a predetermined threshold value.

[0125] The definition to be used for the peak and the best method for finding the peak strongly depend on the specific intrinsic movement to be detected; the most advantageous definition for the peak and the most advantageous method for finding the peak can be determined by way of tests for each individual case. It should be noted that a detailed description of a specific definition for the peak and of the method used for finding the peak will be given in connection with the embodiment described in FIG. 5.

[0126] FIG. 4 is a sequence diagram showing a fourth embodiment of the method of the invention. Said fourth embodiment is an extension of the second and third embodiment.

[0127] It is presupposed for the performance of the method according to the fourth embodiment that a first region of enhanced movement has already been determined and that it has been checked whether said region corresponds to a predetermined region in which intrinsic movements are to be expected.

[0128] If such a region has been found, a further region of enhanced movement which is supposed to be in a symmetrical relationship with the first region of enhanced movement is determined according to the fourth embodiment in step S461.

[0129] For determining said second region, step S250 (second embodiment) may be passed through, or steps S350 to S353 (third embodiment).

[0130] It should here be noted that a symmetrical relationship between two regions in a differential image, i.e. a symmetrical relationship in the movements, must be expected when the regions resulting in the intrinsic movements are symmetrical with each other. For instance, when the head region of a person to be identified is considered, the mouth region in the left and right face halves, the cheek region in the left and right face halves, the nose region in the left and right face halves as well as the left and right eye, are suited as the first and second region within the scope of the fourth embodiment.

[0131] When two of said regions that are supposed to have a symmetrical relationship have been found in the differential image, it can be verified by means of a correlation analysis whether or not the symmetrical relationship exists.

[0132] On account of the fact that the intrinsic movements, e.g. in the head region of a person, are symmetrical, a plausibility check of the determined regions of enhanced movement. is thus obtained through the fourth embodiment.

[0133] Hence, if it is verified in step S462 by way of a correlation analysis that the symmetrical relationship exists, the authenticity of the recorded individual images can be verified in step S170.

[0134] In the other case the authenticity cannot be verified (see step S180) and the method is thus interrupted with such a result or, as already discussed above, repeated.

[0135] The various possibilities of performing a correlation analysis according to S462 will be explained in detail below in connection with FIG. 7.

[0136] FIG. 5 shows the sequence diagram of a fifth embodiment of the method according to the invention. Steps S530 to S565, which are shown in FIG. 5, illustrate a preferred embodiment of step S130 according to the first embodiment.

[0137] Hence, a sequence of consecutive individual images of a person is first recorded in the fifth embodiment as well, and two consecutive individual images are then selected from said sequence. This is illustrated in FIG. 5 by step S120.

[0138] On the basis of said individual images, a differential image is determined in step S530 and optionally binarized in step S540 (steps S530 and S540 correspond to steps S230 and S240 in the second embodiment, so that a description is here not needed and reference is made to the corresponding description of steps S230 and S240).

[0139] According to the fifth embodiment the vertical position and the extent as well as the horizontal position and the extent of the mouth region of a person to be identified are determined on the basis of the differential image in step S550.

[0140] In contrast to step S250 in the second embodiment, in which regions of enhanced movements are determined in the differential image and a check is carried out whether said regions can be assigned to specific regions, a specific region in which intrinsic movements are expected, namely the mouth region, is looked for in step S550.

[0141] It is then checked in step S560 whether or not the determined mouth region is within predetermined boundaries. If the mouth region is within predetermined boundaries, the authenticity of the recorded individual images can be verified in step S170; if the determined mouth region is not within predetermined boundaries, the verification of the authenticity of the individual images must be denied.

[0142] Moreover, as an alternative to step S560, a correlation analysis can be carried out in step S565 with the determined mouth region for detecting intrinsic movements.

[0143] Various possibilities of carrying out a correlation analysis will be explained in detail in connection with FIG. 7.

[0144] In a modification of the fifth embodiment, it is also possible to carry out steps S560 and S565 not alternatively, but cumulatively. Thus, according to step S170 the authenticity of the individual images will only be detected if the determined mouth region is within predetermined boundaries and if the presence of intrinsic movements follows from a correlation analysis of the mouth region.

[0145] Although the fifth embodiment specifically relates to the mouth region, it is of course possible to carry out the method with other regions in which intrinsic movements are expected, or with combinations of such regions.

[0146] FIG. 6 is a sequence diagram showing a sixth embodiment of the inventive method. Said sixth embodiment is a preferred development of step S550 in the fifth embodiment for determining the vertical position and extent and the horizontal position and extent of the mouth region on the basis of the differential image. Thus, it is presupposed that according to step S530 a differential image D(t) or according to S540 a binarized differential image B(t) is present.

[0147] On the basis of said differential image, the function Y_V of the products from the horizontal projections P_H and the horizontal variances V_H is calculated in step 650 according to equation (3).

[0148] The function Y_V can optionally be smoothed with a low-pass filter.

[0149] The vertical position and the vertical extent of the mouth region are determined on the basis of said function or the smoothed function in step S651.

[0150] The vertical mouth position is here defined as the position of the last (lowermost) maximum of Y_V that is greater than a predetermined threshold value. In the case of the mouth region twice the mean of Y_V has turned out to be a suitable threshold value.

[0151] The width of the peak in which the peak has dropped to a predetermined percentage of its maximum value can be indicated as the vertical extent of the mouth region. 50% has here turned out to be an advantageous value.

[0152] As an alternative, the vertical extent of the mouth region can also be indicated as the width of the peak which follows from the distance of the two closest turning points at the left and right side from the peak position.

[0153] Moreover, the values obtained for the vertical position and the vertical extent can be checked as to their plausibility. To this end a minimum mouth position and a maximum mouth position can be predetermined, and it can be checked whether the determined mouth position is within said two extreme values. Said procedure can be carried out alternatively or cumulatively for the vertical extent of the mouth region.

[0154] The function Y_H;t,b of the products from the vertical projections P_V;t,b and the vertical variances V_V;t,b in a horizontal strip extending from the upper mouth boundary t determined in step 651 to the lower mouth boundary b, namely $\begin{array}{cc}{Y}_{H;t,b}=P_{V;t,b}·V_{H;t,b}=\left(\sum _{r=1}^b{K}_{\mathrm{cr}}\right)·\left(\frac{1}{n}\sum _{r=1}^b{\left(K_{\mathrm{cr}}-{\stackrel{\_}{K}}_c\right)}^2\right),\mathrm{with}{\stackrel{\_}{K}}_c\left(t\right)=\frac{1}{n}\sum _{r=t}^b{K}_{\mathrm{cr}},n=b-t+1,\mathrm{and}\mathrm{nr}=\mathrm{number}\mathrm{of}\mathrm{columns}\mathrm{of}K,& \left(5\right)\end{array}$
is then calculated in the next step 652.

[0155] The function Y_H;tb calculated in this way can also be smoothed in an optimum manner.

[0156] In the next step S653, the horizontal position and extent of the mouth region are determined.

[0157] To this end the absolute value of the derivative of the function Y_H;tb or of the smoothed function TPF{Y_H;t,b} is determined, i.e.
b_H;t,b=|y_H+1;t,b−y_H+1;t,b|,  (6)

[0158] The left horizontal boundary of the mouth region is then defined as the first maximum of b_H that is greater than a predetermined threshold value. The right horizontal boundary of the mouth region is defined accordingly as the last maximum of b_H that is greater than said threshold value. In the instant case the mean of b_H has been used as the threshold value.

[0159] In the case of the horizontal extent of the mouth a plausibility check can also be carried out such that the determined mouth region has to lie within a predetermined maximum region so that the region of enhanced movement is regarded as the mouth region.

[0160] FIG. 7 shows a sequence diagram of a correlation analysis, as can e.g. be carried out in step S462 according to the fourth embodiment or in step S565 according to the fifth embodiment.

[0161] Although the procedure according to the sequence diagram in FIG. 7 is explained with reference to the functions Y_V and Y_H, it can readily be applied to the regions as determined in the fourth embodiment.

[0162] In step S766, function Y_V and/or Y_H is split up into sections which are supposed to be in a symmetrical relationship with one another.

[0163] In step S767, it is checked by means of a correlation analysis whether there is the symmetrical relationship assumed in step S766.

[0164] Steps S766 and S767 can of course also be carried out for several symmetrical relationships.

[0165] If it can be verified with the correlation analysis that there is/are the symmetrical relationship(s), the authenticity of the recorded individual images can be verified according to S170.

[0166] If the symmetrical relationship cannot be verified with the correlation analysis, the authenticity of the individual images cannot be verified either, and the method may be interrupted or, alternatively, be repeated for a new pair of individual images.

[0167] The following correlations can be checked in step S767 for the already discussed case regarding the determination of intrinsic movements in the mouth region.

[0168] On the one hand, an intrinsic mouth movement is symmetrical in itself. Therefore, the correlation of curve Y_H and of the smoothed curve TPF{Y_H}, respectively, with the mirrored curve Y_H and the mirrored smoothed curve TPF{Y_H}, respectively, must be greater than a predetermined value.

[0169] Expediently, prior to the correlation analysis curve Y_H is transformed to standardized variables so that the function Y_H has the mean value 0 and the standard deviation 1. When said standardized variables are designated as n(Y_H/V), the correlation is calculated according to the formula $\begin{array}{cc}{g}_m=\sum _{c=-\mathrm{nc}/2}^{\mathrm{nc}/2-1}{n}_{m+c}\left(Y_H\right)·n_{m-c}\left(Y_H\right),& \left(7\right)\end{array}$
where n_c(Y_H) is continued for indices c<0 and c≧n_c with 0.

[0170] It has been found in the case of a mouth movement that a value of g_m>0,9 corresponds to a sufficiently symmetrical mouth movement. Thus, the expected symmetrical relationship would be confirmed if g_m>0,9 followed from equation (7).

[0171] As an alternative or in addition to the above-described correlation analysis, it is possible to check the correlation of the image section from the differential image of the mirrored mouth with the mouth itself. When said correlation is smaller than a predetermined value, it can be assumed that an actual intrinsic movement has not taken place, but the determined movement is due to movements, noise, or the like.

[0172] The correlation of the differential image of the mirrored mouth with the mouth itself can be determined by the expression $\begin{array}{cc}{h}_m=\sum _{c=-e/2}^{e/2}{n}_{\mathrm{cnt}+c}\left(\sum _{r^{\prime }=t}^b{K}_{{\mathrm{cr}}^{\prime }}\right)·n_{\mathrm{cnt}-c}\left(\sum _{r^{\prime }=1}^b{K}_{{\mathrm{cr}}^{\prime }}\right),& \left(8\right)\end{array}$
where cnt is the horizontal position and e the width of the mouth and can be determined by $\begin{array}{cc}\mathrm{cnt}=\arg \underset{m}{\max }{g}_m& \left(9\right)\end{array}$

[0173] It has been found that a value of h_M=0.5 is suited for discriminating between intrinsic and non-intrinsic movements.

[0174] Together with the above-described correlation analyses or alternatively, a further correlation analysis is provided for in steps S768 and S769.

[0175] For carrying out said correlation analysis function Y_V and/or Y_H can be split up into predetermined sections in step S768. Said step, however, is only provided for the case that specific regions of the corresponding function are to be analyzed. If the function Y_V and/or Y_H is to be analyzed in its entirety, step S768 can consequently be dispensed with.

[0176] In step S769, it is checked by means of a correlation analysis whether the values of the corresponding function within their range of definition (or in the predetermined sections if step S768 is carried out) are substantially irregular.

[0177] If the values are irregular, the conclusion can be drawn that an intrinsic movement is actually concerned. By contrast, in the case of regular values of the function the conclusion can be drawn that the movements determined are not movements but rather noise, or the like.

[0178] For carrying out the correlation analysis according to step S769, correlations between the vertical projections and vertical variances for the right and left side of the mouth can be checked in the case of the mouth region.

[0179] The correlation between the vertical projection and the vertical variance for the right and left side of the mouth can be calculated by the expressions $\begin{array}{cc}{i}_m=\sum _{c=0}^{e/2}{n}_{\mathrm{cnt}+c}\left(\sum _{r^{\prime }=t}^b{K}_{{\mathrm{cr}}^{\prime }}\right)·n_{\mathrm{cnt}+c}\left(\sum _{r^{\prime }=t}^b{\left(K_{{\mathrm{cr}}^{\prime }}-{\stackrel{\_}{K}}_c\right)}^2\right),i_m=\sum _{c=-e/2}^0{n}_{\mathrm{cnt}+c}\left(\sum _{t^{\prime }=t}^b{K}_{{\mathrm{cr}}^{\prime }}\right)·n_{\mathrm{cnt}+c}\left(\sum _{r^{\prime }=t}^b{\left(K_{{\mathrm{cr}}^{\prime }}-{\stackrel{\_}{K}}_c\right)}^2\right).& \left(10\right)\end{array}$

[0180] In the case of the mouth region it has been found that the correlations i_m and j_m between the vertical projection and the vertical variance must be ≦0 to ensure that the detected movement is an intrinsic movement.

[0181] In the methods according to FIGS. 5 to 7 the mouth region is shown as an exemplary embodiment of the method according to the invention. However, as already explained in connection with the second embodiment, the method of the invention is not limited to the mouth region.

[0182] Rather any regions in which intrinsic movements have to be expected, e.g. the cheek region, the nose region and the eye region, or any combinations of said regions, are possible.

[0183] If a region which differs from the mouth region is to be analyzed as to intrinsic movements, only the regions in which the functions are formed, the correlations which are analyzed within the correlation analysis, as well as the threshold values with which a correlation is verified, have to be adapted accordingly to the regions considered.

[0184] Since the principles according to which such an adaptation has to be carried out are obvious by virtue of the above-explained example of the mouth region and the fundamentals found in standard textbooks on correlation analysis, a detailed description of the adaptation to the other above-mentioned regions is dispensed with.

[0185] FIG. 8 shows a sequence diagram of an eighth embodiment of the method according to the invention. The steps of the eighth embodiment serve to optimize the individual images after recording for determining the intrinsic movements.

[0186] Thus, as shown in FIG. 8, two consecutive individual images are chosen in step S120 after a sequence of consecutive individual images has been recorded in step S110.

[0187] In step S825, it is first of all checked whether or not the change between the two consecutive individual images is within a predetermined range.

[0188] Since the change between two consecutive individual images must be greater than a predetermined value, it is ensured that in both individual images a change going beyond normal noise has taken place and thus an actual movement or an intrinsic movement of the person to be identified.

[0189] On the other hand, since the change between two consecutive individual images must be smaller than a predetermined value, it is ensured that differential images in which an excessively strong movement is reflected, as is e.g. caused by shaking the recording means or by an extremely fast movement of the person to be identified, are not used for determining the intrinsic movement because in such a case a distinction between an actual movement and an intrinsic movement can no longer be made in a sufficiently reliable manner.

[0190] Step S825 can be carried out in that a motional intensity is calculated for determining the change between two consecutive individual images.

[0191] In case the differential images are gray-level images, the motional intensity can substantially be calculated through the sum of the gray level of the differential image. In case the differential image is binarized, the motional intensity is expediently calculated through the sum of the 1 pixel or 0 pixel of the binarized differential image.

[0192] In step S826, which can be carried out alternatively or cumulatively with respect to step S825, the regions in the individual images in which intrinsic movements are expected are extracted from the corresponding individual images before the individual images are compared with each other. The amount of data to be processed can thus be reduced considerably.

[0193] A region which can expediently be extracted because a multitude of intrinsic movements take place therein is the head region.

[0194] Thus for extracting purposes a predetermined section in the individual images where according to experience the head of the person to be identified comes to rest can be extracted.

[0195] With reference to FIG. 9. an alternative to said method shall be described below. According to said alternative the head region of the person to be identified is determined by determining the head boundaries from a differential image. With said determined head boundaries the head region can then finally be extracted from the respective individual images.

[0196] Again alternatively or cumulatively with respect to steps S826 and S825, respectively, the individual images can be stabilized in step S827.

[0197] Again alternatively or cumulatively with respect to steps S826 and/or step 825, two or more individual images can be stabilized with respect to each other in step 827 or, in other words, aligned with respect to each other.

[0198] To this end a residual shift (l,m) between K(t) and K(t−1) can be determined by template matching. For this purpose K(t) and K(t−1) are first superimposed in centered fashion (this is equal to a residual shift (l,m)=(0,0)). The standardized correlation c(0,0) is then determined. In the next step the standardized correlations c(l,m) are calculated in dependence upon a variation of the residual shift in the vicinity of (l,m)=(0,0).

[0199] The correlation correction (x,y) finally follows from $\begin{array}{cc}\left(x,y\right)=\arg \underset{\left(x,y\right)}{\max }c\left(x,y\right).& \left(11\right)\end{array}$

[0200] With this residual correction a stabilized differential image D(t) can finally be determined according to
D_cr(t)=K_c-x,r-y(t−1)−K_cr(t)  (12)

[0201] FIG. 9 is a sequence diagram of the already above-mentioned alternative for determining the head boundaries with the help of which the head region in which most of the intrinsic movements are expected can be extracted from individual images K(t−1) and K(t).

[0202] Starting from two selected consecutive individual images K(t−1) and K(t), the differential image D(t) is first determined in step S921.

[0203] In step S922, the upper head boundary is then determined from the differential image D(t). To this end the horizontal projection s_H(t) is first determined according to $\begin{array}{cc}{s}_H\left(t\right)=\sum _c{D}_{\mathrm{cr}}\left(t\right)& \left(13\right)\end{array}$

[0204] In this context it should be remembered that c designates the columns and r the lines of the differential image.

[0205] Optionally, function s_H(t) can be smoothed by filtering with a suitable low-pass filter TPF.

[0206] The absolute value of the first derivative is then formed from the function s_H(t) or the smoothed function TPF{s_H(t)}, respectively
b_H(t)=|s_r+1(t)−s_r−1(t)|  (14)

[0207] The position of the first maximum of b_H that is greater than a predetermined threshold value, here half the mean of b_H, is now determined as the upper head boundary.

[0208] If such a value is not found, the calculation of the upper head boundary can be repeated or set to a predetermined value which has e.g. been determined for the individual images K(t−2) and K(t−1).

[0209] Finally, in step S293 the left and right head boundaries are determined. To this end the differential image can be divided into q successive strips which extend in vertical direction downwards, starting from the upper head boundary t.

[0210] In each strip the vertical projection of the differential image is then calculated according to $\begin{array}{cc}{s}_V^{\left(q\right)}\left(t\right)=\sum _{r=t+\mathrm{qz}}^{1+\left(q+t\right)z-1}{D}_{\mathrm{cr}}\left(t\right)& \left(15\right)\end{array}$

[0211] As in the case where the upper head boundary has been calculated, the function s_V^(q)(t) can be smoothed by filtering with a suitable low-pass filter TPF.

[0212] In the next step the absolute value of the first derivative of the function s_V^(q)(t) and the smoothed function TPF{s_V^(q)(t)} is formed and subsequently summed over q: $\begin{array}{cc}{b}_V\left(t\right)=\sum _qs_{c+1}^{\left(q\right)}\left(t\right)-s_{c-1}^{\left(q\right)}\left(t\right)& \left(16\right)\end{array}$

[0213] Of this function b_V all maxima are determined that are greater than a predetermined threshold value, here half the mean b_V.

[0214] The first maximum (the maximum farthest to the left) is defined as the position of the left head boundary. The last maximum (the maximum farthest to the right) is defined as the right head boundary.

[0215] If function b_V has less than two of such maxima, the determination of the left and right head boundaries for the next following differential image can be carried out. Alternatively, the left and right head boundaries can be replaced by predetermined values which have e.g. followed from the determination of the head boundary with the help of the differential image D(t−1).

[0216] Finally, in step S924, the lower head boundary is determined with the help of the upper, left and right head boundaries. To this end the width of the extracted region is determined from the difference between the right head boundary and the left one. The lower head boundary can be adjusted such that a square region is obtained on the whole.

[0217] Apart from the variant described in FIG. 9 for determining the head boundaries, alternative embodiments are possible.

[0218] For instance, it is possible to determine only the upper and left head boundaries according to the above-described procedures. The lower and right head boundaries can then be calculated by way of a predetermined width and a predetermined height, which typically correspond to the head region in the individual images.

[0219] Moreover, the head boundaries can also be determined from a differential image which has been obtained as an AND operation from two consecutive differential images.

[0220] Furthermore, it can optionally be checked in the embodiment shown in FIG. 9 whether the change between the individual images K(t−1) and K(t) is within a predetermined range.

[0221] Since the same procedures as explained in connection with step S825 can be used for this purpose, a description is here not needed, and reference is just made to the description regarding step S825.