Object identification

[0001] This application is the U.S. national phase of international application PCT/GB02/02228, filed May 24, 2002, which designate the U.S.

[0002] 1. Technical Field

[0003] Statistical models of appearance are widely used in computer vision and have many applications, including interpreting medical images and interpreting images containing faces.

[0004] 2. Related Art

[0005] Conventionally, a statistical model is built which represents intensity (greyscale or colour) variation across an image or part of an image. In the case of a facial appearance model, images of many faces are used to generate the model, these images being known as training images. The intensity variation that is likely to be seen in any given face will tend to include similar patterns, and the statistical model represents these patterns. Once the statistical model has been built it may be used to determine whether a new image includes a face, by determining whether the new image contains the facial intensity patterns. The identity of the face in the image may be determined by comparing the intensity patterns found in the image with identity patterns of known faces. A positive identification is given if a suitable error measure, such as a sum of squared error metric, is below a predetermined threshold value.

[0006] There are several known methods of generating statistical appearance models, and using the models to identify and/or recognize faces or other objects in images. Two known models, the Active Shape Model (ASM) [3,4] and the Active Appearance Model (AAM) [2] were developed at the Victoria University of Marichester, United Kingdom and have been widely used. Both of these models are based upon the use of normalized intensity values. The ASM and the AAM are both generalizations of eigen-face models [6]. Eigen-face models are also based upon the use of intensity values and have also been widely used.

[0007] A disadvantage of statistical appearance models that are based upon intensity variation information is that these models are prone to function incorrectly in the presence of changing lighting effects. For example, a face illuminated from an angle different to the illumination which was used when generating the model may cause an error.

[0008] It is Known to generate an ASM using images to which a non-linear filter has been applied. One such filter is arranged to locate edges in a given image, and then set to zero the intensity of everything that is not an edge (this is known as the Canny Edge Operator). The same filter is applied to a new image when the ASM is used to identify and/or recognise an object in the image the output of the filter is a pair of images, one of which represents the directions of edges of the image, and the other represents the magnitude of the edges. This method suffers from two disadvantages. The first disadvantage is that the resulting images are binary images, each edge being represented by ‘on’ values, with the result that a significant amount of information relating to the image is lost. In particular, information relating to structure close to the edges is lost. The second disadvantage of the method is that the filter parameters remain the same for every area of each image, and remain the same for every image. Although it is possible to adjust the filter parameters, tile method does not provide any measurement upon which to base any adjustment.

[0009] It is an object of the present invention to provide an appearance model which overcomes or mitigates at least one of the above disadvantages.

[0010] According to an exemplary embodiment of the invention there is provided a method of object identification comprising, for an image in which an object is to be identified, determining for each of a set of locations in the image the direction in which the gradient of intensity change to an adjacent location is greatest together with the magnitude of that gradient, modifying the gradient magnitude using a nonlinear function, the modified gradient and the associated direction for the set of locations providing a vector representative of the image, and comparing the vector with a previously generated statistical model which provides identification of the object.

[0011] The term ‘image’ is intended to mean an image or a region of the image which is of interest. The locations of interest in the image may be selected regions which are expected to provide inflation useful for identification.

[0012] Suitably, the nonlinear function maps the magnitudes of the gradients to a fixed range of values.

[0013] Suitably, the parameters of the nonlinear function are set for a given location or region in the image, depending upon properties of that location or region.

[0014] Suitably, one of the properties is the scale of that location or region of the image. Suitably, one of the properties is the orientation of that location or region of the image.

[0015] Suitably, one of the properties is the statistics of that location or region of the image.

[0016] Suitably, the nonlinear function is the modulus of the gradient magnitude divided by the sum of the gradient magnitude and the mean or median of the expected values of the gradient magnitude.

[0017] Suitably, the nonlinear function is a monotonic function representative of the cumulative probability distribution of gradient magnitudes due to effective noise in the image.

[0018] Suitably, the effective noise, as represented by noise variance, is obtained from the distribution of the gradient over the image or a region of the image.

[0019] Suitably, the effective noise, as represented by noise variance, is pre-computed for the image as a whole.

[0020] Suitably, the nonlinear function acts to normalise the gradient magnitude.

[0021] Suitably, the direction of the gradient is represented by modulo 2π.

[0022] Suitably, the direction of the gradient is represented by modulo π.

[0023] Suitably, the location and size of an object of interest is estimated prior to determining the gradient directions and magnitudes.

[0024] Suitably, the method provides identification of the class of an object.

[0025] Suitably, the method provides recognition of a particular object within a class of objects.

[0026] Suitably, the class of objects is faces.

[0027] Suitably, the previously generated appearance model is an Active Shape Model.

[0028] Suitably, the Active Shape Model uses profile models.

[0029] Suitably, the profile models are one-dimensional models.

[0030] Suitably, the previously generated appearance model is Combined Appearance Model.

[0031] Suitably, the previously generated appearance model is an Active Appearance Model.

[0032] The exemplary embodiment of the invention also provides an appearance model comprising a set of data representative of images of objects of interest, the model being constructed by determining for each of a set of locations in the image the direction in which the gradient of intensity change to an adjacent location is greatest together with the magnitude of that gradiet, modifying t&e gradient using a nonhnear function, and combining die resulting set of modified gradients and associated directions with sets of modified gradients and associated directions determined for other images, to form the set of data.

[0035] A first embodiment of the invention is based upon the Active Shape Model (ASM) [4 ], which is described in detail in Appendix 1. The known prior art ASM comprises a statistical model of the intensity of the image at each model point.

[0036] The first embodiment of the invention, instead of simply using intensity measurements, represents the image structure as a gradient, which is specified as a magnitude and an orientation. For any given pixel, the orientation of the gradient indicates the direction of greatest intensity change when moving from that pixel to an adjacent pixel. Where an edge of a feature passes through or adjacent the pixel, the direction of the gradient will be transverse to the edge. The magnitude of the gradient provides a measure of belief in the accuracy of the measurement (in other words a measure of belief that an edge is present). Essentially the measure is strong when on a well defined edge, since the orientation of the gradient is well defined, but is weak or zero in essentially flat noisy regions of the image. Use of the gradient instead of intensity information is advantageous because it provides inherent identification of edges in images.

[0037] The gradient for a given pixel is operated on by a nonlinear filter, and is then added to the ASM (during ASM generation). This is in marked contrast to the known prior art in which a nonlinear filter is applied across an image and the gradient for pixels is subsequently determined.

[0038] A first advantage of the invention is that it retains more information than is retained by the prior art ASM. In particular, the filter is not used to locate edges, which by necessity means discarding information relating to areas which are found not to be edges, but instead is used to provide gradient information relating to an entire image (information does not need to be discarded). The nonlinear filter acts to suppress the importance of information that is less likely to relate to an area of interest of the image.

[0039] A second advantage of the approach used by the invention is that the parameters of the filter are easily adjusted for the particular region of the image being explored. In particular, when searching a cluttered image for an example of the object of interest, the parameters of the normalisation function are chosen based on the statistics of the image in the region being explored launder the current estimate of model position, rather than on the statistics of the image as a whole. This provides a substantial improvement over the prior art, where a single set of filter parameters is used for the whole image.

[0040] When generating an ASM, be it the prior art ASM or the ASM according to the invention, locations of importance are manually marked on each image which is added to the ASM (this is described in Appendix 1). Where the ASM models faces, the locations of interest include the comers of the mouth, the comers of the eyes, etc. Since the location of the features of interest is known, this allows the parameters of the filter to be adjusted for different areas of the image. For example, for a face model the filter parameters may be set to suppress the effect of pixels located immediately outside the comers of the mouth since these pixels are unlikely to include significant useful information (the pixels represent the cheeks). Similarly, the filter parameters may be set to enhance horizontal edges located between the comers of the mouth since these pixels are likely to include very useful information (the pixels represent the mouth). Since the orientation of the face is known, from the orientation of the marked up points, any rotation of the face may be corrected before it is filtered.

[0041] When using the ASM to locate an object of interest in a new image, the filter parameters are again adjusted in the same way that they were adjusted during ASM generation. Once the initial location, scale and orientation of an object of interest have been determined (a method of doing this is described in Appendix 2), the filter parameters are set depending on the scale, orientation and statistics of the image in the region around the current estimate of position.

[0042] The appearance model represents image structure in regions around the object of interest. In the case of the ASM this is a set of small regions around each model point, either a profile normal to the boundary at the point or a small 2D region around the point.

[0043] During both model generation and searching of a new image, the image of interest is sampled over each region of interest, the sample values being placed in a vector representing the image structure in the region. In prior art ASM's this was commonly simply a case of taking the intensity values at each pixel in the region and placing them in the vector. The first embodiment of the invention uses the following approach to represent each region with a vector:

• [0044] 1. Estimate the local gradients at a point x in x and y. The gradient is determined by first applying an edge filter in x (essentially convolving with a (−101) filter), then in y. This provides gradient values g_x, g_y for each point x. The direction in which the gradient is a maximum is given by the vector (g_x, g_y).
• [0045] 2. Compute the magnitude of the gradient, g=√{square root over (g_x²+g_y²)}
• [0046] 3. Apply a non-linear function, f(g, x,) to obtain a representation of gradient direction and strength, (g′_x, g′_y)=f(g, x)(g_x/g, g_y/g).
• [0047] 4. Add the values g′_x, g′_y to a feature vector containing values representative of the region.
• [0048] 5. Repeat for all points n the region.

[0049] All other steps of model building remain unchanged from the prior art, as described in Appendix 1, and the listed references. The steps are:

• [0050] 6. Add the feature vector to the ASM.
• [0051] 7. Repeat for a large number of regions, the number of regions required being determined by the quality of the statistics of the model.

[0052] The orientation of the gradient is represented as modulo 2π (i.e. an angle between zero and 2π). The directions along which the gradients are measured are defined in the model frame, to ensure that the method is independent of the rotation and scale of the model instance in the image.

[0053] When adding values to the new Active Shape Model (ASM), the normalised intensity feature vector g, which in the prior art ASM is a concatenation of all the intensity values for a given image, is replaced with a feature vector twice as long, of the form g^T=(. . . g′_xi, g′_yi, . . . ), a concatenation of all the gradient values for the image. All other calculations for the ASM remain the same.

[0054] The non-linear function ƒ(g, x) is chosen so as to accentuate likely edges and suppress structure which is likely to be noise. If the function were not used then the values input to the ASM would simply be gradients. Tests have shown that the use of unmodified gradients will provide an ASM which has an unimproved performance compared to the known intensity value based ASM.

[0055] Without loss of generality, 0≦ƒ(g, x)≦1 for all g, i.e. the nion-linear function provides normalisation to values between 0 and 1 of the gradients. This allows the generation of a statistical model with easily manipulated values. It will be appreciated that normalisation of the gradients between any two values flay be used.

[0056] The non-linear function may vary across the image. This may be done for example to suppress non-maximal edges or anticipate different noise properties in different parts of an image. Alternatively, the non-linear function may be independent of position in the image, in which case ƒ(g, x)=ƒ(g).

[0057] An effective non-linear function has been found to be ƒ(g)=|g|/(|g|+g₀), where g₀ is the mean or median of the expected values of g. In the case of the Active Shape Model, this can be the mean or median of g measured over a region of interest of an image. Different values of g₀ may be used for different regions of the image (for example by separating the image into regions, and determining the mean or median separately for each region). The function has the desired property that values of g less than the mean noise value tend to get mapped towards zero and those above the mean (those more likely to be edges of interest) get mapped towards 1. The useful properties of the function stem from the fact that the average gradient will be quite small compared to the maximum gradient; this is because there are likely to be many regions in any given image within which there is little variation of intensity between adjacent points.

[0058] An alternative non-linear function is ƒ(g)=P_n(g), where P_n(a) is the cumulative probability distribution of the edge magnitudes due to effective noise in the image. P_n(a) is a monotonic function with a value in [0,1] such that the probability of a random value, b, drawn from a distribution, being less than a given a is P_n(a) (i.e. Prob[b<a]=P_n(a). Again, values of g likely to be due to noise effects are mapped toward zero, those less likely to be noise are treated as interesting structure and are effectively enhanced. Note that if the noise on each pixel is Gaussian. the noise on the derivatives, g_x, g_y, is also Gaussian and thus g²=g_x²+g_y² will be distributed as X²-allowing an analytic estimate of P_n(g) in terms of gamma functions. An estimate of the noise variance can be obtained from the distribution of g over the sampled region, or can be pre-computed for the image as a whole as appropriate.

[0059] The new ASM, once generated, can be used to identify and/or recognise faces or other objects in images. This is done by determining an initial estimated position of the object of interest (described in Appendix 2 below), then performing steps 1 to 5 as set out above to generate a feature vector which is compared to the ASM. The nonlinear filters used, and their associated parameters, are based upon the filters and parameters used to generate the model. Noisy, approximately flat regions in the new image tend to be represented as near zero values and thus have little effect on the optimisation. Experiments have demonstrated that the invention provides more accurate and reliable location of the position of the boundaries of structures of interest in an image, particularly when the target image has significantly different lighting conditions compared to the original training set.

[0060] A new image, or region of an image, can be verified as belonging to a particular class by checking that the feature vector is a valid example for that class (i.e. it is ‘close enough’ to a particular fixed vector, or is sufficiently likely to be generated by a probability density function representing the class).

[0061] In some cases it is advantageous to represent the orientation of the edge structure only up to modulo π, essentially identifying the direction of the edge gradient but not its polarity. This is useful in cases where a background of an image is not known in advance. An object of interest in an image may be darker than the background or lighter than the background, so that the position and direction of edges are known, but not their polarity (dark to light or light to dark). This situation can be dealt with by first representing the edge direction in polar co-ordinates, (g_x, g_y)→ (g, θ). The angle is then doubled and mapped back: (h_x, h_y,)=(gcos2θ, gsin2θ)=(g_x²−g_y², 2g_xg_y)/g.

[0062] A suitable noin-linear normalisation is then applied as described above:
(g′_x, g′_y)=ƒ(g)(h_x/g, h_y/g)=ƒ(g)(g_x²−g_y²,2g_xg_y)/g²

[0063] The modulo π data is used to Generate a new Active Shape Model as described above (modulo π and miodulo 2π data are not combined in a single model).

[0064] The ASM according to the first embodiment of the invention models the structure of the region about each point using a statistical model of the gradient measured along a profile through the point, orthogonal to the boundary at this position. Where profile models are to be used, they can either represent the structure as described above, with each pixel contributing both an x and y component to the vector, or the model may be used in just one dimension.

[0065] Where just one dimension is used, the gradient at a point along the profile is simply measured along the direction of the profile (for instance g(i)=I(p+u(i+1))−I(p+u(i−1)) where p is the centre of the profile and u is the step along the profile).

[0066] The embodiment of the invention applies a non-linear normalisation to this, g′(i)=ƒ(g′(i), X), where ƒ(g, x) is a suitable non-linear function identical in properties to those described above. This does not contain quite as much information as representing both g′_x and g′_y, but is usually faster to compute.

[0067] A second embodiment of the invention is based upon the Active Appearance Model (AAM), which is described in detail in Appendix 2. The known prior art AAM uses statistical models of variations of intensity across an object in an image, after first warping the image to correct for shape variability. In the second embodiment of the invention, an image is warped to correct for shape variability, and then instead of using variations of intensity, the non-linearly normalised gradient is used both to generate the new DM and to identify and/or recognise objects using the new AAM.

[0068] The appearance model represents image structure in regions around the object of interest. Typically in the case of the AAM this is a 2D region covering all or part of the object.

[0069] During both model building and image search, the image of interest is sampled over each region of interest, the sample values being placed in a vector representing, the image structure in the region. In prior art AAM's this was commonly simply a case of taking the intensity values at each pixel in the region and placing them in the vector. The second embodiment of the invention uses the following approach to represent each region within a vector:

[0070] For a 2D image, the following steps are performed at each pixel:

• [0071] 1. Estimate the local gradients at a point x in x and v. The gradient is determined by first applying an edge filter in x (essentially convolving with a (−101) filter), then in y. This provides gradient values g_x, g_y for each point x. The direction in which the gradient is a maximum is given by the vector (g_x, g_y)
• [0072] 2. Compute the magnitude of the gradient, g=√{square root over (g_x²+g_y²)}
• [0073] 3. Apply a non-linear function, ƒ(g, x) to obtain a representation of gradient direction and strength, (g′_x, g′_y)=ƒ(g, x)(g_x/g,g_y/g).
• [0074] 4. Add the values g′_x, g′_y to a feature vector containing values representative of the region.
• [0075] 5. Repeat for all points in the region.

[0076] All other steps of model building remain unchanged from the prior art, as described in

[0077] Appendix 1, and the listed references. The steps are:

• [0078] 6. Add the feature vector to the AAM.
• [0079] 7. Repeat for a large number of regions, the number of regions required being determined by the quality of the statistics of the model.

[0080] It will be appreciated that features of the first embodiment of the invention which have been described above may be applied to the second embodiment of the invention. For example, the manner in which the filter parameters are adjusted for different areas of an image may be applied to the second embodiment of the invention.

[0081] The invention is advantageous because the model generated using, the nonlinear gradient representation is more robust to lighting changes than a straightforward model of raw intensity values. Thus, when used in a classification/recognition framework the invention provides more reliable results.

[0082] The first aspect of the invention is based upon the Active Shape Model ([3,4]). To ensure that the manner in which the invention is applied to the Active Shape Model is fully understood, the prior art Active Shape Model is described below:

[0083] Active Shape Models are statistical models of the shapes of objects which iteratively deform to fit to an example of the object in a new image. The shapes are constrained by a Statistical Shape Model to vary only in ways seen in a training set of labelled examples.

[0084] In addition to the shape model, we require models of the image appearance around each model point. The simplest is to assume the points lie on strong edges. More complex models can be built to represent the statistical variation of the gradient along profiles through the points, normal to the boundary curve at that point.

[0085] We assume we have an initial estimate for the pose and shape parameters (eg the mean shape). This is iteratively updated as follows:

• • [0086] Look along normals through each model point to find the best local match for the model of the image appearance at that point (eg strongest nearby edge)
  • [0087] Update the pose and shape parameters to best fit the model instance to the found points
  • [0088] Repeat until convergence

[0089] The performance can be significantly improved using a multi-resolution implementation, in which we start searching on a coarse level of a gaussian image pyramid, and progressively refine. This leads to much faster, more accurate and more robust search.

[0090] The Active Shape Model uses the Statistical Shape Model described below:

[0091] Given a set of examples of a shape, we can build a statistical shape model. Each shape in the training set is represented by a set of n labelled landmark points, which must be consistent from one shape to the next. For instance, on a hand example, the 7th point may always correspond to the tip of the thumb.

[0092] Given a set of such labelled training examples, we align them into a common co-ordinate frame using Procrustes Analysis. This translates, rotates and scales each training shape so as to minimise the sum of squared distances to the mean oft the set. Each shape can then be represented by a 2n element vector:
x=(x_—1, . . . , x_—n, y_—1, . . . , y_—n).

[0093] The aligned training set forms a cloud in the 2n dimensional space, and can be considered to be a sample from a probability density function.

[0094] In the simplest formulation, we approximate the cloud with a gaussian. We use Principal Component Analysis (PCA) to pick out the main axes of the cloud, and model only the first few, which account for the majority of the variation.

[0095] The shape model is then
x=x_mean+Pb
where x_mean is the mean of the aligned training examples, P is a 2n ×t matrix whose columns are unit vectors along the principal axes of the cloud, and b is a t element vector of shape parameters.

[0096] (This model has been dubbed a “Point Distribution Model” (PDM), but has little to do with the Point Distribution in statistics)

[0097] By varying the shape parameters within limits learnt from the training set, we can generate new examples.

[0098] Such models are used in the Active Shape Model framework to locate new examples in new images, as described above.

[0099] The second aspect of the invention is based upon the Active Appearance Model (G. Edwards, C. Taylor, and T. Cootes. Interpreting face images using active appearance models. In 3^rd International Conference on Automatic Face and Gesture Recognition 1998, pages 300-305, Nara, Japan, Apr. 1998. IEEE Computer Society Press) and described further by Cootes et al (T. Cootes, G. J. Edwards, and C. J. Taylor. Active appearance models. In 5^th European Conference on Computer Vision, pages 484-498. Springer, June 1998). To ensure that the manner in which the invention is applied to the Active Appearance Model is fully understood, the prior art Active Appearance Model is described below:

[0100] The Active Appearance Model uses the difference between a reconstructed image generated by the model and the underlying target image, to drive the model parameters towards better values. In a prior learning stage, known displacements, δv, are applied to known model instances and the resulting difference between model and image, δv, is measured. Multivariate linear regression is applied to a large set of such training displacements and an approximate linear relationship is established:
δc=Aδv

[0101] When searching an image, the current difference between model and image, δv, is used to predict an adjustment, −δc, to the model parameters which improves model fit. For simplicity of notation, the vector δc is assumed to include displacements in scale, rotation, and translation.

[0102] The Active Appearance Model is constructed using sets of face images. To do this, Facial appearance models are generated following the approach described by Edwards et al (G. Edwards, A. Laintis, C Taylor and T. Cootes, Statistical model of face images—improving specificity. Image and Vision Computering, 16:203-211, 1998). The models are generated by combining a model of face shape variation with a model of the appearance variations of a shape-normalised face. The models are trained on 400 face images, each labelled with 122 landmark points representing the positions of key features. The shape model is generated by representing each set of landmarks as a vector, x and applying a principal component analysis (PCA) to the data. Any example can then be approximated using:
x= x+P_sb_s  (1)
where x is the mean shape, P_s is a set of orthogonal modes of variation and b_s is a set of shape parameters. Each example image is warped so that its control points match the mean shape (using a triangulation algorithm), and the grey level information g is sampled from this shape-normalised face patch. By applying PCA to this data a similar model is obtained:
g= g+P_gb_g  (2)
The shape and appearance of any example can thus be summarised by the vectors b_s and b_g. Since there are correlations between the shape and grey-level variations, a further PCA is applied to the concatenated vectors, to obtain a combined appearance model of the form:
x= x+{tilde under (O)}_sc  (3)
g= g+{tilde under (O)}_gc  (4)
where c is a vector of appearance parameters controlling both the shape and grey-levels of the model, and Q_s and Q_g map the value of c to changes in the shape and shape-normalised grey-level data. A face can be synthesised for a given c by generating the shape-free grey-level image from the vector g and warping it using the control points described by x (this process is described in detail in [3]). Equations 3 and 4 define a statistical model of appearance known as the Combined Appearance Model.

[0103] The 400 examples lead to 23 shape parameters, b_s, and 114 grey-level parameters, b_g. However, only 80 combined appearance model parameters, c are required to explain 98% of the observed variation.

[0104] A two stage strategy is adopted for matching the appearance model to face images. The first step is to find an approximate match using a simple and rapid approach. No initial knowledge is assumed of where the face may lie in the image, or of its scale and orientation. A simple eigen-face model (M. Turk and A. Pentland. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1)31-86, 1991) is used for this stage of the location. A correlation score, S, between the eigen-face representation of the image data, M and the image itself, I can be calculated at various scales, positions and orientations:
S=|I—M² |

[0105] Once a reasonable starting approximation of the position of a face has been determined, the parameters of the appearance model are then adjusted, such that a synthetic face is generated which matches the image as closely as possible. The basic idea is outlined below, followed by details of the alorithm.

[0106] Interpretation is treated as an optimisation problem in which the difference between a real face image and one synithesised by the appearance model is mimimised. A difference vector δI can be defined:
δI=I_i−I_m  (6)
where I_iis the vector of grey-level values in the image, and I_m, is the vector of grey-level values for the current model parameters.

[0107] To locate a best match between model and image, the magnitude of the difference vector, Δ=|δI²|, is mimimised by varying the model parameters, c.

[0108] Since the model has around 80 parameters, this appears at first to be a very difficult optimisation problem involving search in a very high-dimensional space. However, it is noted that each attempt to match the model to a new face image, is actually a similar optimisation problem. Therefore, the model learns something about how to solve this class of problems in advance. By providing a-prior knowledge of how to adjust the model parameters during image search, it arrives at an efficient run-time algorithm. In particular, it might be expected that the spatial pattern in I, to encode information about how the model parameters should be changed in order to achieve a better fit. For example, if the largest differences between the model and the image occurred at the sides of the face, that would imply that a parameter that adjusted the width of the model face should be adjusted.

[0109] In adopting this approach there are two parts to the problem: learning the relationship between δI and the error in the model parameters, δc and using this Knowledge in an iterative algorithm for minimising Δ.

[0110] The simplest model that could be chosen for the relationship between δI and the error in the model parameters (and thus the correction which needs to be made) is lineal:
δc=AδI  (7)

[0111] This is a good enough approximation to provide good results. To find A, a multiple multivariate linear regression is performed on a large sample of known model displacements, δc, and the corresponding difference images, δI. These large sets of random displacements are generated, by perturbing the true model parameters for the images in the training set by a known amount. As well as perturbations in the model parameters, small displacements in 2D position scale, and orientation are also modeled. These extra 4 parameters are included in the regression; for simplicity of notation, they can, however, be regarded simply as extra elements of the vector δc. In order to obtain a well-behaved relationship it is important to choose carefully the frame of reference in which the image difference is calculated. The most suitable frame of reference is the shape-normalised face patch described above. A difference is calculated thus: for the current location of the model, calculate the image grey-level sample vector, g_i, by warping the image data at the current location into the shape-normalised face patch. This is compared with the model grey-level sample vector, g_m, calculated using equation 4:
δg=g_i−g_m  (8)

[0112] Thus, equation 7 can be modified:
δ_c=Aδg  (9)

[0113] The best range of values of δ c to use during training is detennined experimentally. Ideally it is desired to model a relationship that holds over as large a range errors, δ g as possible. However, the real relationship is found to be linear only over a limited range of values. In experiments, the model used 80 parameters. The optimum perturbation level was found to be around 0.5 standard deviations (over the training set) for each model parameter. Each parameter was perturbed from the mean by a value between 0 and 1 standard deviation. The scale, angle and position were perturbed by values ranging from 0 to +/-10% (positional displacements are relative to the face width). After performing linear regression, an R² statistic is calculated for each parameter perturbatton, δ c₁ to measure how well the displacement is ‘predicted’ by the error vector δ g. The average R² value for the 80 parameters was 0.82, with a maximum of 0.98 (the 1st parameter) and a minimum of 0.48.

[0114] Given a method for predicting the correction which needs to made in the model parameters, an iterative method may be constricted for solving the optimisation problem. For a given model projection into the image, c, the grey-level sample error vector, δg , is calculated, and the model estimate is updated thus:
c′=c−Aδg  (10)

[0115] If the initial approximation is far from the correct solution the predicted model parameters at the first iteration will generally not be very accurate but should reduce the energy in the difference image. This can be ensured by scaling A so that the prediction reduces the magnitude of the difference vector, |δg²|, for all the examples in the training set. Given the improved value of the model parameters, the prediction made in the next iteration should be better. The procedure is iterated to convergence. Typically the algorithm converges in around 5-10 iterations from fairly poor starting approximations. More quantitative data are given below.

[0116] The method was tested on a set of 80 previously unseen face images.

[0117] The reconstruction error of AAM search was tested over a test set of 80 unseen images. The reconstruction error for each image was calculated as the magnitude of the shape-normalised grey-level sample vector, |δ g |².

• [0118] [1] H. Boscli, S. C. Mitchell, P. F. Boudewijn, P. F. Leieveldt, F. Nijland, O. Kamp, M Sonka, and J. Reiber. Active appearance-motion models for endocardial contour detection in time sequences of echocardiograms. In SPIE Medical Imaging, Feb. 2001.
• [0119] [2] T. F. Cootes, G. J. Edwards, and C. J. Taylor. Active appearance models. In H. Burkhardt and B. Neumann, editors, 5^th European Conference in Computer Vision, volume 2, pages 484-498. Springer, Berlin, 1998
• [0120] [3] T. F. Cootes, A. Hill, C. J. Taylor, and J. Haslam. The use of active shape models for locating structures in medical images. Image and Vision Computing, 12(6):276-285, July 1994.
• [0121] [4] T. F. Cootes, C. J. Taylor, D. Cooper, and J. Graham. Active shape models—their training and application. Computer Vision and Image Understanding, 61 (1):38-59, Jan. 1995.
• [0122] [5] D. Hond and L. Spacek. Distinctive descriptions for face processing. In 8^th British Machine Vision Conference, volume 1, pages 320-329, Colchester, UK, 1997.