Approximated invariant method for pattern detection

[0002] The invention relates generally to systems and methods for assigning input patterns into one of two classes and more particularly to a system and a method for classifying input patterns based on a linear classification.

[0003] Pattern classification has drawn increasing attention in recent years, due to its application in a number of technological areas, such as acoustic signal processing and image processing. The objective in pattern classification is to separate input patterns into two classes, a target class and a non-target class. For example, for face detection in images, vectors (blocks of pixels) are extracted from an image and each vector is examined to determine whether it contains an image of a human face (target class) or not (non-target class). Another example is the detection of a specific sound, e.g., a piano sound, in an arbitrary sound signal stream. In this case, the components of the sound signal stream are classified into the specific sound class (target class) or background class (non-target class).

[0004] Various types of pattern classifiers are known in the art. Most pattern classifiers are configured to discriminate between the two classes by “learning” from two training sets, {circumflex over (X)}={x_i}_{i=1 . . . Lx}(target samples) and Ŷ={y_i}_{i=1 . . . Ly}(non-target samples). Such classifiers are known as example-based classifiers. The simplest and computationally fastest type of pattern classifier is a linear classifier C(z), where the classifying operation is based on a mathematical projection followed by a thresholding. For a given input pattern, i.e., a vector z, the projection of z is performed onto a projection vector u and, thus, C(z)=f(u^Tz), where f(*) is a thresholding operation or some other decision rule.

[0005] Various types of pattern classifiers are known in the art. Most pattern classifiers are configured to discriminate between the two classes by “learning” from two training sets, {circumflex over (X)}={x_i}_{i=1 . . . Lx}(target samples) and Ŷ={y_i}_{i=1 . . . Ly}(non-target samples). Such classifiers are known as example-based classifiers. The simplest and computationally fastest type of pattern classifier is a linear classifier C(z), where the classifying operation is based on a mathematical projection followed by a thresholding. For a given input pattern, i.e., a vector z, the projection of z is performed onto a projection vector u and, thus, C(z)=f(u^tz), where f(*) is a thresholding operation or some other decision rule and where U^Tis the related transpose.

[0006] Therefore, what is needed is a system and a method for efficiently and accurately classifying input patterns into two classes, even in cases where the configuration of the two classes is complex and cannot be separated by a linear hyper-plane.

[0007] A system and a method for classifying input vectors into one of two classes, a target class and a non-target class, utilize iterative rejection stages to first label the input vectors that belong in the non-target class in order to identify the remaining non-labeled input vectors as belonging in the target class. Each iterative rejection stage involves a linear projection of an input vector that is followed by a thresholding to determine whether that input vector belongs in the non-target class. The system and method may be used in a number of applications, such as face detection, where the members of the two classes can be represented in a vector form. In a face detecting application, the system operates to determine whether an input vector, representing a block of image pixels, contains a picture of a human face.

[0008] The operation of the system can be divided into an off-line (training) procedure and an on-line (actual classification) procedure. During the off-line procedure, a training module of the system computes the projection vectors and their corresponding threshold values that will be used during the on-line procedure. During the on-line procedure, the projection vectors and the threshold values are utilized by a pattern classifier of the system to execute the iterative rejection stages to determine if an input vector should be labeled as a non-target. After all the iterative rejection stages are executed, the input vector is identified as a target, if it had not been previously labeled as a non-target.

[0009] An off-line method of computing the projection vectors and their corresponding threshold values in accordance with the invention begins when a training set of vectors, including sample targets and sample non-targets, is received by the training module of the system. Mean-variance pairs for the sample targets and the sample non-targets are then computed by the mean-variance pair calculator of the training module. Using a proximity measure approach, a first projection vector is calculated by the projection vector calculator of the training module. Preferably, the first projection vector facilitates a positive identification of a large portion of the sample non-targets. In particular, the first projection vector is chosen such that projection values of many sample non-targets fall outside a target-containing interval, i.e., an interval of projection values that contains the projection values of all the sample targets when projected onto the first projection vector.

[0010] Next, a pair of threshold values is chosen for the first projection vector by the threshold setter of the training module. The pair of threshold values defines the upper and lower limits of the target-containing interval. The threshold values are chosen to include the smallest and the largest projection values of the sample targets. The separation between the two threshold values may include the separation between the smallest and the largest projection values of the sample targets, as well as a safety margin. The safety margin may be some percentage of the separation between the smallest and the largest projection values of the sample targets, e.g., five percent, to reduce the risk of erroneous non-target classification. After the pair of threshold values has been set, the sample non-targets having projection values that fall outside of the target-containing interval defined by the two threshold values are removed from the training set. The result is a reduced training set that contains fewer sample non-targets.

[0011] Using the reduced training set, a new mean-variance pair for the remaining sample non-targets is then computed by the mean-variance pair calculator. Next, a second projection vector and its corresponding pair of threshold values are calculated by the projection vector calculator and the threshold setter, respectively, in the same manner in which the first projection vector and its corresponding pair of threshold values were derived. This process is repeated until all the sample non-targets have been removed from the training set. The end product of the off-line procedure is a number of projection vectors and their corresponding threshold values that can be utilized during an on-line procedure to classify unknown input vectors into one of the two classes.

[0012] An on-line method of classifying input vectors into one of the two classes is primarily executed by the pattern classifier of the system. The method begins at a step in which the input vectors, as well as the computed projection vectors and their corresponding threshold values, are received by the pattern classifier. For each input vector, successive rejection stages are performed by the linear projection unit and the threshold module of the pattern classifier. During each stage, an input vector is linearly projected using one of the projection vectors computed during the off-line procedure to derive a projection value. The projection value is then compared to corresponding pair of threshold values. If the projection value falls outside the target-containing interval defined by the threshold values, the input vector is labeled as a non-target. After the successive rejection stages, the input vector is identified as a target, if the input vector has not been previously labeled as a non-target. In this manner, all of the input vectors are classified as either non-targets or targets.

[0013] FIG. 1 is a block diagram of an approximated invariant method (AIM) system, including its components, in accordance with the present invention.

[0014] FIG. 2 is an exemplary illustration for the construction of the intervals C_t, C_cand C_uin accordance with the invention.

[0015] FIG. 3 illustrates the first rejection stage at which many Clutters are rejected using a first projection vector in accordance with the invention.

[0016] FIG. 4 illustrates the second rejection stage at which additional Clutters are rejected using a second projection vector in accordance with the invention.

[0017] FIG. 5 is a block diagram of a training module in the AIM system of FIG. 1, including its components, in accordance with the invention.

[0018] FIG. 6 is a flow diagram of an off-line method of computing projection vectors and their corresponding threshold values in accordance with the invention.

[0019] FIG. 7 is a block diagram of a pattern classifier in the AIM system of FIG. 1, including its components, in accordance with the invention.

[0020] FIG. 8 is a flow diagram of an on-line method of classifying input vectors into one of two classes, a Target class and a Clutter class, in accordance with the invention.

[0021] With reference to FIG. 1, an approximated invariant method (AIM) system having a pattern classifier 12, a training module 14, a processing unit 16, memory 18, a user interface device 20, and an output device 22 is shown. The AIM system operates to classify input vectors within a signal stream into one of two classes, a Target class and a non-Target (“Clutter”) class. The AIM system may be used in a number of applications, such as face detection, where the members of the two classes can be represented in a vector form. In a face detecting application, the input vectors represent blocks of pixels from an image that may or may not contain a picture of a human face. In this application, the AIM system operates to discriminate between the two possibilities.

[0022] A mathematical description for the overall operation of the AIM system will be first described. The classification problem can be defined as follows: Given two classes X (Target class) and Y (Clutter class) in , a point z drawn from one of these classes needs to be labeled correctly as either a Target or a Clutter. An important observation is that it is known a priori that for a typical input stream, the vast majority of the inputs are in Clutters, i.e.:

[0023] P{X}<<P{Y}.  (1)

[0024] Based on this knowledge, the Clutter labeling should be made fast, even at the expense of a slow Target labeling, to yield a fast output, i.e., the determination of whether the input point z is a Target or a Clutter, with as few operations as possible.

[0025] Similar to other linear classifiers, the fundamental approach utilized by the AIM system 10 to label the point z as either a Target or a Clutter is to project the point z onto a projection vector u and then label the point z based on the projected value α=u^Tz. Projecting the Target class onto the projection vector u results in a Probability Density Function (PDF) P{α|X}, or the “Target PDF”, defined as:

[0026] P{α|X}=∫_aP{z|X}δ(u^Tz−α)dz,  (2)

[0027] where δ(x) is the Dirac's function. The term P{α|X} defines the probability that a given input drawn from the Target class will have the value α after it has been projected onto the projection vector u. Similarly, projecting the Clutter class onto the projection vector u results in a PDF P{α|Y}, or the “Clutter PDF”, defined as:

[0028] P{α|Y}=∫_aP{z|Y}δ(u^Tz−α)dz.  (3)

[0029] The following intervals are based on P{α|X} and P{α|Y}:

[0030] C_t={α|P{α|X}>0, P{α|Y}=0}

[0031] C_c={α|P{α|X}=0, P{α|Y}>0}

[0032] C_u={α|P{α|X}>0, P{α|Y}>0},  (4)

[0033] where t=Target, c=Clutter and u=Unknown. After the projection, z is labeled either as a Target, Clutter or Unknown based on the following decision rule: $\begin{array}{cc}\mathrm{Classifier}\left\{z\right\}=\{\begin{array}{cc}\mathrm{Target}& u^{T}z\in C_t\\ \mathrm{Clutter}& u^{T}z\in C_c\\ \mathrm{Unknown}& u^{T}z\in C_u\end{array}& \left(5\right)\end{array}$

[0034] The Unknown classifications are obtained only in the C_uinterval, where a definite decision cannot be made. FIG. 2 illustrates an example for the construction of the intervals C_t, C_cand C_uand their appropriate decisions. In FIG. 2, a Target PDF 24 and two Clutter PDFs 26 and 28 are shown, super-imposed. The overlapping zones between the two types of PDFs define the Unknown regions C_u, in which definite decisions cannot be made.

[0035] The probability of the Unknown decision is given by:

[0036] P{Unknown}=∫_αεC[P{Y}P{α|Y}+P{X}P{α|X}]dα.  (6)

[0037] The equation (6) is a function of the projection vector u. It is desirable to find the vector u that minimizes the Unknown probability. However, it turns out that this problem is difficult to solve directly, even for simplified cases. Thus, some heuristics need to be applied. An alternative approach is to use a proximity measure.

[0038] If P{α|X} and P{α|Y} are far apart and separated from each other, P{Unknown} will be small. Therefore, the alternative approach is to minimize the overlap between these two types of PDFs. This approach will be defined using the following expected distance between a point α₀and a distribution P{α}: $\begin{array}{cc}D\left({\alpha }_0\left|\right|P\left\{\alpha \right\}\right)={\int }_{\alpha }\frac{{\left({\alpha }_0-\alpha \right)}^{2}P\left\{\alpha \right\}}{{\sigma }^2}d\alpha ,& \left(7\right)\end{array}$

[0039] where τ is the variance of P{α} and the division by τ is performed to make the distance scale-invariant (or unit-invariant). Calculating the integral in the equation (7), it is easy to verify that: $\begin{array}{cc}D\left({\alpha }_0\left|\right|P\left\{\alpha \right\}\right)=\frac{{\left({\alpha }_0-\mu \right)}^2+{\sigma }^2}{{\sigma }^2}& \left(8\right)\end{array}$

[0040] where μ is the mean of P{α}. Using this distance definition, the distance of P{α|X} from P{α|Y} can be defined as the expected distance of (α|Y} from P{α|X} and expressed as: $\begin{array}{cc}\begin{array}{c}D\left(P\left\{\alpha \left|Y\right\}\right|\left|P\left\{\alpha \right|X\right\}\right)={\int }_{\alpha }D\left(\alpha \left|\right|P\left\{\alpha |X\right\}\right)P\left\{\alpha |Y\right\}d\alpha \\ ={\int }_{\alpha }\frac{{\left(\alpha -{\mu }^x\right)}^2+{\sigma }_x^2}{{\sigma }_x^2}P\left\{\alpha |Y\right\}d\alpha \\ =\frac{{\left({\mu }_y-{\mu }_x\right)}^2+{\sigma }_x^2+{\sigma }_y^2}{{\sigma }_x^2},\end{array}& \left(9\right)\end{array}$

[0041] where [μ_x, τ_x] and [μ_y, τ_y] are the mean-variance pairs of P{α|X} and P{α|Y}, respectively. Since the two distributions should have as small an overlap as possible, it is desirable to maximize this distance or minimize the proximity between P{α|X} and P{α|Y}, which can be defined as the inverse of their mutual distance. Note that this measure is asymmetric with respect to the two distributions. The proximity between P{α|X} and P{α|Y} defines the closeness of P{α|Y} to P{α|X}, but not vice versa. Therefore, the overall proximity between the two distribution is defined as follows: $\begin{array}{cc}\mathrm{Prox}\left(P\left\{\alpha |Y\right\},P\left\{\alpha |X\right\}\right)=P\left\{X\right\}\frac{{\sigma }_y^2}{{\sigma }_x^2+{\sigma }_y^2+{\left({\mu }_y-{\mu }_x\right)}^2}+P\left\{Y\right\}\frac{{\sigma }_x^2}{{\sigma }_x^2+{\sigma }_y^2+{\left({\mu }_y-{\mu }_x\right)}^2}.& \left(10\right)\end{array}$

[0042] Compared to the original expression in Equation (6), the minimization of this term with respect to μ is easier. If P{X}=P{Y}, i.e., if there is an even chance to get Target or Clutter inputs, the proximity becomes: $\begin{array}{cc}\mathrm{Prox}\left(P\left\{\alpha |Y\right\},P\left\{\alpha |X\right\}\right)=\frac{{\sigma }_x^2+{\sigma }_y^2}{{\sigma }_x^2+{\sigma }_y^2+{\left({\mu }_y-{\mu }_x\right)}^2}.& \left(11\right)\end{array}$

[0043] This is the cost function minimized by the Fisher Linear Discriminant (FLD). Specifically, the inverse of the equation (11) is minimized by the FLD. An alternative approach would have been to use an overall distance instead, resulting with the expression $\begin{array}{cc}\frac{{\left({\mu }_y-{\mu }_x\right)}^2+{\sigma }_x^2+{\sigma }_y^2}{{\sigma }_x^2}+\frac{{\left({\mu }_y-{\mu }_x\right)}^2+{\sigma }_x^2+{\sigma }_y^2}{{\sigma }_y^2},& \left(12\right)\end{array}$

[0044] for P{X}=P{Y}. However, this expression is more difficult to minimize, as compared to the FLD.

[0045] However, when P{X}<<P{Y} (Eq. 1), the first term of equation (10) is negligible and can be omitted. Therefore, the preferred u should minimize the resulting term: $\begin{array}{cc}d\left(u\right)=\frac{{\sigma }_x^2}{{\sigma }_x^2+{\sigma }_y^2+{\left({\mu }_y-{\mu }_x\right)}^2)},& \left(13\right)\end{array}$

[0046] where τ²_y, τ²_x, μ_yand μ_xare all function of u.

[0047] Note again that d(u) is asymmetric, and its minimization is equivalent to the maximization of equation (9). This means that most Clutter samples must be distant from the Target class, but the opposite is not necessarily required. A given Target sample can be close to some of the Clutters. This implies that most of the Clutters will be projected onto C_c, while C_tmight even be an empty set. After projection, many of the Clutter inputs are usually classified, whereas Target labeling may not be immediately possible. In fact, this serves the intended objective because for a Clutter input, there is a high probability that a decision will be made. Since Clutter inputs are more frequent, this means a faster decision for the vast majority of the inputs.

[0048] Minimization of equation (13) usually results with a small τ_xand a large τ_y. This means that projection of Target inputs tends to concentrate near a constant value, whereas the Clutter inputs will spread with a large variance. Therefore, the projection onto the projection vector u stands as an approximated invariant for the Target class.

[0049] In order to find the preferred projection vector u, the τ²_y, τ²_x, μ_yand μ_xmust be expressed as function of u. $\begin{array}{cc}\begin{array}{c}{\mu }_x={\int }_{\alpha }\alpha P\left\{\alpha |X\right\}d\alpha ={\int }_{\alpha }{\int }_z\alpha P\left\{z|X\right\}\delta \left(u^{T}z-\alpha \right)dzd\alpha \\ ={\int }_{z}P\left\{z|X\right\}\left[{\int }_{\alpha }\mathrm{\alpha \delta }\left(u^{T}z-\alpha \right)d\alpha \right]dz\\ =u^T{\int }_z\mathrm{zP}\left\{z|X\right\}dz=u^T{M}_x\end{array}\mathrm{and}\begin{array}{c}{\sigma }_x^2={\int }_{\alpha }{\left(\alpha -{\mu }_x\right)}^{2}P\left\{\alpha |X\right\}d\alpha ={\int }_{\alpha }\left({\alpha }^2-2{\mathrm{\alpha \mu }}_x+{\mu }_x^2\right)P\left\{\alpha |X\right\}d\alpha \\ ={\int }_{\alpha }{\alpha }^{2}P\left\{\alpha |X\right\}d\alpha -2{\mu }_x{\int }_{\alpha }\alpha P\left\{\alpha |X\right\}d\alpha +{\mu }_x^2\\ ={\int }_{\alpha }{\alpha }^{2}P\left\{\alpha |X\right\}d\alpha -u^T{M}_x{M}_x^{T}u\\ =u^T\left[{\int }_z{\mathrm{zz}}^{T}P\left\{z|X\right\}dz\right]u-u^T{M}_x{M}_x^{T}u\\ =u^T\left[{\int }_z\left(z-M_x\right){\left(z-M_x\right)}^{T}dz\right]u\\ =u^T{R}_{\mathrm{xx}}u\end{array}& \left(14\right)\end{array}$

[0050] and similarly,

[0051] μ_y=u^TM_y

[0052] τ²_y=u^TR_yyu.

[0053] Therefore,

[0054] μ_x=u^TM_xand τ²_x=u^TR_xxu  (15)

[0055] and

[0056] μ_y=u^TM_yand τ²_y=u^TR_yyu,  (16)

[0057] where it is defined that

[0058] M_x=∫_azP{z|X}dz R_xx=∫_a(z−M_x)(z−M_x)^TP{z|X}dz

[0059] M_y=∫_azP{z|Y}dz R_yy=∫_a(z−M_y)(z−M_y)^TP{z|Y}dz.  (17)

[0060] As can be seen from the above equations (15) and (16), only the first and second moments of the classes play a role in the choice of the projection vector u. The function that is to be minimized is therefore: $\begin{array}{cc}\begin{array}{c}d\left(u\right)=\frac{{\sigma }_x^2}{{\sigma }_x^2+{\sigma }_y^2+{\left({\mu }_y-M_x\right)}^2}\\ =\frac{u^T{R}_{\mathrm{xx}}u}{u^T\left[R_{\mathrm{xx}}+R_{\mathrm{yy}}+\left(M_y-M_x\right){\left(M_y-M_x\right)}^T\right]u}.\end{array}& \left(18\right)\end{array}$

[0061] The u that minimizes the equation (18) satisfies the equation

[0062] R_xxu=λ[R_xx+R_yy+(M_y−M_x)(M_y−M_x)^T]u  (19)

[0063] and should correspond to the smallest possible λ. A problem of the form Au=λBu, as in equation (19), is known as the generalized eigen-value problem and can be easily solved. Notice that given any solution u for this equation, bu is also a solution with the same λ, where b is any scalar. Therefore, without loss of generality, normalized solution ||u||=1 is used.

[0064] After finding the preferred projection vector u, the intervals C_t, C_cand C_uare determined according to equation (4). An input z is labeled as a Target or Clutter, if its projected value u^Tz is in the interval C_tor the interval C_c, respectively. FIG. 3 presents this first rejection stage for the case where the interval C_tis empty, i.e., there are no inputs which can be classified as a Target. In FIG. 3, the Clutters and Targets 32 are shown in two-dimensional space. For illustrative purposes, Target PDF 34 and Clutter PDF 36 for the Clutters and Targets are also shown. Note that the preferred projection vector 38 was chosen such that many of the Clutters are rejected, which are identified in FIG. 3 as rejected Clutters 40. The thresholds are chosen such that erroneous rejection of Targets is not possible for the given Targets.

[0065] Input vectors having projected values within in the interval C_uare not labeled. For these inputs another step of classification is applied, where the design of the preferred projection vector in this step is performed according to the following new distributions:

[0066] P{α|Y & α₁εC_u} and P{α|X & α₁εC_u},  (20)

[0067] where u₁is the first projection vector and α₁=u^T₁z. The next projection vector u₂is defined as the vector which minimizes the following proximity measure:

[0068] Prox(P{α|Y & α₁εC_u}, P{α|X & α₁εC_u}).  (21)

[0069] This minimization is performed in the same manner as explained for the first rejection stage. FIG. 4 illustrates the second rejection stage, after the application of the first stage, shown in FIG. 3. During the second rejection stage, the next projection vector 42 is chosen such that large portions 46 of the remaining Clutters 44 are rejected. In FIG. 3, the Clutters that were rejected during the first rejection stage are shown in phantom. Again, corresponding thresholds are chosen such that erroneous rejection of Targets 32 is not possible for the given Targets. In FIG. 4, new Clutter PDF 48 and new Target PDF 50 for the remaining Clutters and Targets are also shown.

[0070] After the second rejection stage, the process can continue with projection vectors u₃, u₄, . . . , etc. Due to the high rejection rate of the projection vectors for a given set of Clutters, it is expected that a large portion of the Clutter input vectors will be labeled as Clutters at each rejection stage, such that the remaining Clutter input vectors are labeled by subsequent projection vectors. Such an iterative process ensures a good performance of the classification with respect to an accurate labeling and a fast classification rate.

[0071] In practice, the probabilities P{α|X} and P{α|Y} are not known and the inference on the Target or Clutter class is achieved through examples. Two example sets are collected and denoted as {circumflex over (X)}={x_k}_k=1^Lxand Ŷ={y_k}_k=1^Ly. It is assumed that these two collected sets are sufficiently large to be considered as good representations of the actual classes X and Y. First, an alternative definition of the function to be minimized is found, expressed in terms of the training sets. Replacing the mean-covariance pairs, M_x, R_xx, M_yand R_yy, with empirical approximations, the mean-covariance pairs can be expressed as: $\begin{array}{cc}{\hat{M}}_x=\frac{1}{L_x}\sum _{k=1}^{L_x}{x}_k{\hat{R}}_{\mathrm{xx}}=\frac{1}{L_x}\sum _{k=1}^{L_x}\left(x_k-{\hat{M}}_x\right){\left(x_k-{\hat{M}}_x\right)}^T& \left(22\right)\\ \mathrm{and}& \\ {\hat{M}}_y=\frac{1}{L_y}\sum _{k=1}^{L_y}{y}_k{\hat{R}}_{\mathrm{yy}}=\frac{1}{L_y}\sum _{k=1}^{L_y}\left(y_k-{\hat{M}}_y\right){\left(y_k-{\hat{M}}_y\right)}^T.& \left(23\right)\end{array}$

[0072] Substituting the empirical approximations into equation (18) results in the empirical proximity: $\begin{array}{cc}\hat{d}\left(u\right)=\frac{u{}^T\hat{R}_{\mathrm{xx}}u}{u^T\left[{\hat{R}}_{\mathrm{xx}}+{\hat{R}}_{\mathrm{yy}}+\left({\hat{M}}_y-{\hat{M}}_x\right){\left({\hat{M}}_y-{\hat{M}}_x\right)}^T\right]u}.& \left(24\right)\end{array}$

[0073] As previously described, the desired projection vector u is computed by a generalized eigen-value routine.

[0074] Since the class probabilities have been exchanged with sets of points, it is impractical to define the intervals C_t, C_cand C_uusing equation (4). This is because the intervals will be composed of many fragments, each of which results from a particular example. Moreover, the domain of a cannot be covered by a finite set of examples. Thus, it is more natural to define for each set, two thresholds bounding its projection values. Due to the function that is being minimized, in most cases, the Target thresholds define a small interval located inside the Clutter interval (see FIG. 3). Therefore, for simplicity, a single interval, ┌=[T₁, T₂], which is bounding the Target set, is defined. Using the defined interval ┌, points that are projected outside the interval ┌ are classified as Clutter and points that are projected inside the interval ┌ are classified as Unknown.

[0075] Turning back to FIG. 1, the components of the AIM system 10 operate to compute the preferred projection vectors and their corresponding pairs of threshold values, i.e., two threshold values for each computed projection vector, and then to utilize the computed vectors and the threshold values to determine whether each vector within an input signal belongs to the Target class or the Clutter class. The operation of the AIM system can be divided into an off-line (training) procedure and an on-line (actual classification) procedure. During the off-line procedure, the projection vectors and their corresponding pairs of threshold values that will be used during the on-line procedure are computed. During the on-line procedure, an input vector is projected onto each projection vector and the derived projection value is thresholded using its corresponding pair of threshold values to determine whether the input vector belongs in the Clutter class in an iterative manner. If the input vector is not labeled as belonging in the Clutter class for every projection vector, the input vector is identified as belonging in the Target class. Although the AIM system is described as utilizing only two thresholds for each computed projection vector, the AIM system may utilize more than two thresholds for each computed projection vector.

[0076] If more than two thresholds are utilized by the AIM system 10, the target and non-target classes are separated, during the training procedure, into several sub-intervals in accordance with their projection values, and training proceeds for each of these sub-intervals separately. The on-line procedure using more than two thresholds simply requires that after each projection, the projected value is compared to the existing thresholds, and further treatment will depend on the sub-interval into which the projected value fell.

[0077] An off-line method of computing the projection vectors and their corresponding intervals in accordance with the invention is illustrated by a flow diagram, shown in FIG. 6. The off-line method is primarily executed by the training module 14 of the AIM system 10. Therefore, the off-line method will be described with reference to the components of the training module, which is shown in FIG. 5. The training module includes a mean-variance pair calculator 52, a projection vector calculator 54 and a threshold setter 56.

[0078] With reference to FIG. 6, the off-line method begins at step 58, in which a training set of vectors, including a Target sample subset {circumflex over (X)} and a Clutter sample subset Ŷ, is received by the mean-variance pair calculator 52 of the training module 14. The vectors may be digitized representations of images, such as blocks of pixels, or of sounds. Using equation (22), {circumflex over (M)}_xand {circumflex over (R)}_xxfor {circumflex over (X)} are then computed by the mean-variance pair calculator, at step 60. Next, at step 62, the variable k is set to one. At step 64, {circumflex over (M)}^k_yand {circumflex over (R)}^k_yyfor Ŷ^kare computed by the mean-variance pair calculator, using equation (23). The projection vector u_kis then calculated by the projection vector calculator 54, at step 66. The projection vector u_kis the vector that minimizes the equation (24). After the calculation of the projection vector U_k, threshold values T^k₁and T^k₂that define the corresponding interval ┌^kfor the current projection vector u^kare determined by the threshold setter 56, at step 68. During this step, T^k₁is set to min_{xε{circumflex over (X)}}(u^T_kx_i)−Δ/2, while the T^k₂is set to max_{xε{circumflex over (X)}}(u^T_kx_i)+Δ/2. The term Δ/2 in the T^k₁and T^k₂includes a safety margin in the interval ┌^k. The Δ can be 5% of the actual interval for the Target set, i.e., [min_{xε{circumflex over (X)}}(u^T_kx_i), max_{xε{circumflex over (X)}}(u^T_kx_i)], for the current projection vector u_k, or any other reasonable value.

[0079] After the projection vector u_kand its corresponding threshold values T^k₁and T^k₂have been determined, the Clutter sample vectors having projection values outside the interval ┌^kare discarded, while the Clutter sample vectors having projection values within the interval ┌_kare kept for further processing. Therefore, at step 70, the Clutter set Ŷ_k+1is defined as including Clutter sample values that are within the interval ┌^k. Next, at step 72, the variable k is set to k+1. At step 74, a determination is made whether |Ŷ_k| is greater than zero, where |Ŷ_k| is the number of Clutter sample vectors remaining in Ŷ_k. If |Ŷ_k| is greater than zero, the method proceeds back to step 64. If |Ŷ_k| is equal to zero, the off-line method comes to an end.

[0080] In an alternative embodiment, the values {circumflex over (M)}_x, {circumflex over (R)}_xx, {circumflex over (M)}_yand {circumflex over (R)}_yyare modified in accordance to a regularization scheme. The modification of these values solves two potential problems that may be encountered when attempting to solve the generalized eigen-value problem, as shown in equation (19). The first problem is purely numerical. The matrix {circumflex over (R)}_yymay become singular after several rejection stages, when few Clutters remain. The second problem involves an over-fitting problem associated with insufficient samples in the training set of Clutter samples and Target samples. An over-fitting problem results in a classifier that operates well on the example sets, but not on the actual unknown input vectors.

[0081] The regularization scheme involves taking two given example-sets, {circumflex over (X)}={x_k}_k=1^Land Y={y_k}_k=1^L, and enriching these two sets in the following manner, thereby solving the over-fitting problem, as well as the numerical stability problem. For any Target point x_k, it is assumed that all the points z within its neighborhood are also Targets. More formally, defining the neighborhood by a sphere, it is assumed that for a predefined radius δ, all z satisfying ||z−x_k||≦δ are Targets as well. In this fashion, the finite set {circumflex over (X)}={x_k}_k=1^Lare replaced with a new richer set of points, which contains infinite number of Targets, {circumflex over (X)}_reg(the subscript reg stands for regularization). In a similar manner, the Clutter set is also enriched, and Ŷ_regis obtained. The quality of this solution depends on the validity of the assumption that δ neighborhood contains a similar class points, and consequently, the predefined radius δ should be carefully chosen to ensure that δ is not too large. Adopting a different point of view for the above scheme, an assumption is made that when adding zero mean white Gaussian noise with moderate variance δ²to the points {circumflex over (X)}={x_k}_k=1^L, their classification does not change. Thus, if x_kis a Target, so are all the realizations of x_k+v_k, where v_kis a multi-variate Gaussian noise with zero mean and δ²l covariance matrix. (V_k˜N(0, δ²l)). Similarly, such noise can be used to enrich the Clutter set as well.

[0082] The first and second moments for the regularized sets, {circumflex over (X)}_regand Ŷ_reg, can be computed by exploiting the statistical representation of the regularization process, where the neighborhood of each given point is regarded as additive noise with a known variance. Looking at the regularized Pattern set, it can be referred to as an finite mixture of multi-dimensional (dimension n) Gaussians. These Gaussians are the following: $\begin{array}{cc}\mathrm{Prob}\left\{z|X_{\mathrm{reg}}\right\}=\frac{1}{\mathrm{Const}}\sum _{k=1}^{k=L_x}{G}_{\left\{x_k,{\delta }^{2}l\right\}}\left(z\right)& \left(25\right)\end{array}$

[0083] where: $G_{\left\{M,{\delta }^{2}l\right\}}\left(z\right)=\frac{1}{{\left(2{\mathrm{\pi \delta }}^2\right)}^{n/2}}\exp \left\{-\frac{\left|\right|z-M{\left|\right|}^2}{2{\delta }^2}\right\}.$

[0084] The constant is a normalization factor that ensures that the above probability function sums into 1. Its value is given by: $\begin{array}{cc}\mathrm{Const}={\int }_{z\in {\Re }^n}\sum _{k=1}^{L_x}\frac{1}{{\left(2{\mathrm{\pi \delta }}^2\right)}^{n/2}}\exp \left\{\frac{-\left|\right|z-x_k{\left|\right|}^2}{2{\delta }^2}\right\}dz=L_x& \left(26\right)\end{array}$

[0085] and similarly: $\begin{array}{cc}\mathrm{Prob}\left\{z|Y_{\mathrm{reg}}\right\}=\frac{1}{L_y}\sum _{k=1}^{k=L_y}{G}_{\left\{y_k,{\delta }^{2}l\right\}}\left(z\right).& \left(27\right)\end{array}$

[0086] The first moment (the mean) of the regularized Pattern set is given by: $\begin{array}{cc}\begin{array}{c}{\hat{M}}_x^{\mathrm{reg}}=E\left\{z|X_{\mathrm{reg}}\right\}\\ ={\int }_{z\in {\Re }^n}z·\mathrm{Prob}\left\{z|X_{\mathrm{reg}}\right\}dz\\ ={\int }_{z\in {\Re }^n}z·\left\{\frac{1}{L_x}\sum _{k=1}^{L_x}\frac{1}{{\left(2{\mathrm{\pi \delta }}^2\right)}^{n/2}}\exp \left\{\frac{-\left|\right|z-x_k{\left|\right|}^2}{2{\delta }^2}\right\}\right\}dz\\ =\frac{1}{L_x}\sum _{k=1}^{L_x}{x}_k\\ =\frac{1}{L_x}\sum _{k=1}^{L_x}{x}_k\\ ={\hat{M}}_x\end{array}& \left(28\right)\end{array}$

[0087] where {circumflex over (M)}_xis the first moment of {circumflex over (X)}, the original set before regularization. Similarly for the regularized Non-Pattern set, {circumflex over (M)}_y^reg={circumflex over (M)}_y.

[0088] The second moment for the regularized Pattern set is computed through: $\begin{array}{cc}\begin{array}{c}{\hat{R}}_{\mathrm{xx}}^{\mathrm{reg}}=E\left\{{\left[z-{\hat{M}}_x^{\mathrm{reg}}\right]\left[z-{\hat{M}}_x^{\mathrm{reg}}\right]}^T|X_{\mathrm{reg}}\right\}\\ ={\int }_{z\in {\Re }^n}{\left[z-{\hat{M}}_x^{\mathrm{reg}}\right]\left[z-{\hat{M}}_x^{\mathrm{reg}}\right]}^T·\mathrm{Prob}\left\{z|X_{\mathrm{reg}}\right\}dz\\ =\frac{1}{L_x}\sum _{k=1}^{L_x}{\int }_{z\in {\Re }^n}{\left[z-{\hat{M}}_x^{\mathrm{reg}}\right]\left[z-{\hat{M}}_x^{\mathrm{reg}}\right]}^T\frac{1}{{\left(2{\mathrm{\pi \delta }}^2\right)}^{n/2}}\exp \left\{\frac{-\left|\right|z-x_k{\left|\right|}^2}{2{\delta }^2}\right\}dz\\ =\frac{1}{L_x}\sum _{k=1}^{L_x}\left[{\delta }^{2}l+x_k{x}_k^T-{x_k\left({\hat{M}}_x^{\mathrm{reg}}\right)}^T-{\hat{M}}_x^{\mathrm{reg}}{x}_k^T+{{\hat{M}}_x^{\mathrm{reg}}\left({\hat{M}}_x^{\mathrm{reg}}\right)}^T\right]\\ ={\delta }^{2}l+\frac{1}{L_x}\sum _{k=1}^{L_x}{\left[x_k-{\hat{M}}_x^{\mathrm{reg}}\right]\left[x_k-{\hat{M}}_x^{\mathrm{reg}}\right]}^T\\ ={\delta }^{2}l+{\hat{R}}_{\mathrm{xx}}\end{array}& \left(29\right)\end{array}$

[0089] and therefore, {circumflex over (R)}_xx^regδ²l+{circumflex over (R)}_xx. Similarly, the Non-Pattern second moment is give by {circumflex over (R)}_yy^reg=δ²l+{circumflex over (R)}_yy.

[0090] Thus, the following regularized first and second moments are derived.

[0091] {circumflex over (M)}_y^reg={circumflex over (M)}_y{circumflex over (R)}_xx^regδ²l+{circumflex over (R)}_xx

[0092] {circumflex over (M)}_y^reg={circumflex over (M)}_y. {circumflex over (R)}_yy^reg=δ²l+{circumflex over (R)}_yy  (30)

[0093] where {circumflex over (M)}_x, {circumflex over (R)}_xx, {circumflex over (M)}_y, and {circumflex over (R)}_yyare the mean-covariance pairs of the original sets before modification. Note that the means do not change, but the covariance matrices become well-posed because of the addition of the identity matrix that is multiplied by δ². Also note that the two sets need not be treated similarly with regularization. In cases where the Target set contains enough examples, only the Clutter set can be regularized. In such cases, the parameter δ can be determined, for example, by finding the nearest neighbor Of Y_kin the Target set, for each y_k, and choosing δ to be half of the smallest distance found for all kε[1, L_y].

[0094] The extracted projection vectors and their corresponding threshold values are stored in the memory 18 of the system 10, until needed for the on-line procedure. An on-line method of classifying input vectors into one of the two classes in accordance with the invention is illustrated by a flow diagram, shown in FIG. 8. The on-line method is primarily executed by the pattern classifier 12 of the AIM system 10. Therefore, the on-line method will be described with reference to the components of the pattern classifier, which is shown in FIG. 7. The pattern classifier includes a linear projection unit 76, a thresholding module 78, and a decision module 80.

[0095] With reference to FIG. 8, the on-line method begins at step 82, in which a set of input vectors Z₁={z_i}ⁿ_i=1are received by the linear projection unit 76 of the pattern classifier 12. The integer n represents the number of input vectors included in the set. In addition, the stored projection vectors {u_k}^K_k=1and their corresponding thresholds {[T^k₁, T^k₂]}^K_k=1are transmitted to the pattern classifier from the memory 18 by the processing unit 16 of the AIM system 10. The integer K represents the number of projection vectors that will be utilized to determine the classification of the input vectors. Again, depending on the application, each input vector can be a digitized, representation of images, such as blocks of pixels, or of sounds. At steps 84 and 86, the variables i and k are set to one. Next, at step 88, a projected value α is computed by the projection vector calculator 54, using the equation α=u^T_kz_i. At step 90, a determination is made whether the projected value α is outside the interval ┌^kdefined by the threshold values T^k₁and T^k₂that correspond to the projection vector u_k. This determination is executed by the thresholding module 78. If step 90 is satisfied, the method proceeds to step 92, where the input vector z_iis labeled as Clutter. However, if step 90 is not satisfied and the projected value α is within the interval ┌^k, the method proceeds to step 94. At step 94, the variable k is set to k+1.

[0096] After the input vector z_ihas been analyzed using the projection vector u_kand its associated threshold values T^k₁and T^k₂, a determination is made whether the variable k is greater than K, at step 96. If step 96 is not satisfied, the method proceeds back to step 88, in order to further analyze the input vector z_i. If step 96 is satisfied, the method proceeds to step 98. During this step, the input vector z_iis labeled as Target by the decision module 80, if it has not been already labeled as Clutter at step 92. Next, at step 100, the variable i is set to i+1, so that the next input vector can be processed. At step 102, a determination is made whether i is greater than n. If step 102 is not satisfied, the method proceeds back to step 86, in order to process the next input vector. However, if step 102 is satisfied, the on-line method comes to an end. The result of the on-line method is that each input vector has been labeled as either a Target or Clutter.

[0097] The final determination of the input vectors may be displayed on the output device 22 of the AIM system in a desired format, depending on the parameter prescribed by an operator via the user interface device 20. As an example, in a face detecting application, the output displayed on the output device may be an image with highlighted blocks of pixels that correspond to the input vectors that have been labeled as Targets.