Automatic target recognition system with Elliptical Laplacian Pyramid filter

[0001] 1. Field of the Invention

[0002] The present invention relates to target recognition. More particularly, the present invention relates to automatic target recognition systems which employ a filter to pre-screen image data and nominate candidate targets.

[0003] 2. Background Information

[0004] Weapon system performance, and in particular, military weapon system performance is generally measured in terms of system accuracy. In weapon delivery systems, accuracy may depend on the system's ability to deliver a weapon on target. In weapon systems that track targets, accuracy typically depends on the ability to periodically or continuously establish the correct position of a target. To improve accuracy, these systems often employ automatic target recognition systems and/or techniques.

[0005] Automatic target recognition generally involves processing large quantities of image data (e.g., thermal image data) in order to identify and locate candidate targets. Automatic target recognition system performance, in turn, is generally measured in terms of probability of detection (P_d) and probability of false detection (P_fd), where P_d represents the likelihood that the system correctly detected and classified a target as a true target, and where P_fd represents the likelihood that the system incorrectly classified a non-target as a true target. Curve A in FIG. 1 illustrates the relationship between P_d and P_fd for a typical automatic target recognition system. Clearly, it is more desirable to have a system which maximizes P_d while minimizing P_fd, as indicated by the dotted line to the left of curve A, as compared to the dashed line to the right of curve A.

[0006] Conventional target recognition systems employ what is known as pipeline processing to maximize performance. Pipeline processing is a well-known term which refers to a data processing technique that involves multiple, concurrently operating processing stages, where each stage performs a separate processing function.

[0007] FIG. 2 illustrates an exemplary pipeline processing sequence 200 which might be employed in a conventional target recognition system. As shown, the exemplary pipeline processing sequence includes a pre-screen filter 205, a target delineator 210, a feature extractor 215 and a target classifier 220. The pre-screen filter 205 processes the image data and, based on this processing, identifies candidate targets. Typically, the pre-screen filter 205 also provides a location of each candidate target identified within the image, e.g., by providing an x-y coordinate of a pixel that corresponds with the spatial center of each candidate target. The target delineator 210 then uses the image data and the location information to determine the extent or boundary of any candidate target. For instance, the target delineator 210 may determine which pixels in proximity to the spatial center of a given candidate target also reflect or correspond to a portion of the candidate target. The feature extractor 215, as the name suggests, measures certain predefined target features, such as length, width, perimeter and texture. The feature extractor 215 uses the measurements to generate a vector for the candidate target. The vector is then passed to the target classifier 220. The target classifier 220, which has been previously trained using measurements of features from known true targets and known false targets, classifies the candidate target as a true target or a non-true target based on the vector it received from the feature extractor 215.

[0008] The pre-screen filter stage 205 illustrated in FIG. 2 is of particular importance. As stated, it identifies and locates candidate targets. As such, it establishes the processing load for the remaining stages. If the filter nominates a large number of candidate targets, the processing load is relatively large. If the filter nominates a small number of candidate targets, the processing load is relatively small. Therefore, it is important to employ a filter that effectively nominates targets, that is, with a relatively high P_d and a relatively low P_fd. Else, a significant amount of time is wasted processing data associated with false targets. It is also important to employ a filter that efficiently processes image data so that candidate targets are nominated in a timely manner.

[0009] The present invention is directed to an automatic target recognition system and/or method, where the pre-screen filter stage of the pipeline processing sequence is implemented using an Elliptical Laplacian Pyramid (ELP) based algorithm. A Laplacian is a mathematical function which may be generated using a technique known as the Difference of Gaussians, as explained in greater detail below. As such, a Laplacian is sometimes referred to as a Difference of Gaussians (DOG). If the Gaussians are circular, so to is the resulting Laplacian. An elliptical Laplacian, as the name suggests, may be generated by a difference of two elliptical Gaussians. Thus, an elliptical Laplacian may be referred to as the Elliptical Difference of Gaussians (EDOG).

[0010] The ELP based algorithm is a resolution sequence matched filter that capitalizes on the spatial nature of targets, which tend to exhibit a rectangular and/or elliptical shape at most aspect angles. Consequently, the algorithm efficiently and effectively identifies and locates candidate targets. This, in turn, helps to insure a relatively high P_d:P_fd ratio.

[0011] Accordingly, it is an object of the present invention to provide an automatic target recognition system that is capable of efficiently and effectively identifying candidate targets based on image data.

[0012] Another object of the present invention is to provide an automatic target recognition system that is capable of efficiently and effectively nominating candidate targets based on image data regardless of target range.

[0013] Still another object of the present invention is to provide an automatic target recognition system that is capable of efficiently and effectively nominating candidate targets based on image data regardless of aspect angle.

[0014] In accordance with a first embodiment of the present invention, the aforementioned and other objectives are achieved by a method for identifying an object in an image. The method involves processing the image with an elliptical Laplacian operator and locating a peak value in the processed image. The object is then identified as a function of the peak value.

[0015] In accordance with another embodiment of the present invention, the aforementioned and other objectives are achieved by a method for locating a target in an image. The method involves processing pixel data associated with the image through the use of an elliptical Laplacian operator. A vector is then generated, where the vector comprises a set of resultant values based on the previous step of processing the pixel data. From the set of resultant values, a peak value is identified, and therefrom, the target is located as a function of a pixel location corresponding with the identified peak value.

[0016] In accordance with yet another embodiment of the present invention, the aforementioned and other objectives are achieved through a target recognition system. The system includes means for receiving data associated with an image. The system also includes an elliptical Laplacian based pre-screen filter, which processes the image data and identifies therein a candidate target.

[0027] To facilitate a better understanding of the present invention, the present invention is described herein below with references to military type weapon systems. However, it will be understood that the present invention may apply to any system, military or otherwise, that requires efficient and effective recognition of targets, or more generally, objects embedded in image data.

[0028] FIG. 3 depicts a pipeline processing sequence 300. Like the pipeline processing sequence depicted in FIG. 2, the pipeline processing sequence 300 includes a pre-screen filter 305, a target delineator 310, a feature extractor 315 and a target classifier 320. Unlike the pipeline processing sequence 200, the pre-screen filter 305 is, according to exemplary embodiments of the present invention, implemented using an ELP based algorithm. The ELP based algorithm employs elliptical Laplacian operators which closely reflect the spatial characteristics of prospective targets, such as trucks, tanks, missile sites and the like, to identify and locate these targets in the image data.

[0029] FIG. 4 is a block diagram which illustrates the processing functions that are associated with the ELP based pre-screen filter algorithm 305. As shown, the processing functions include an elliptical Laplacian kernel (ELK) 405, a peak detector 410 and a peak pixel location recording function 415. In accordance with a preferred embodiment of the present invention, each of these functions are implemented in software and/or firmware using well-known development techniques. However, as one skilled in the art will readily appreciate, some or all of these functions may be implemented in hardware.

[0030] Generally, the ELK 405 receives image data from an imaging sensor (e.g., thermal image data from a forward-looking infrared sensor). Typically, image data involves a matrix of picture elements known as pixels, where each pixel represents a value (e.g., image intensity value) at a corresponding position (e.g., an x-y coordinate position) in the image. The ELK 405 then down-samples the image one or more times, as explained in greater detail below. The ELK 405 also generates an appropriate elliptical Laplacian operator, which it then uses to process (i.e., filter) the original and one or more down-sampled images. This processing results in a vector (i.e., a set of values) for each of the original and one or more down-sampled images, where each value in a given vector reflects a level of correlation between the elliptical Laplacian operator and a corresponding pixel in a corresponding one of the original or down-sampled images, as well as a number of neighboring pixels surrounding the corresponding pixel. The peak detector 410 then processes each vector. This processing results in the identification of one or more peak values (assuming there are one or more candidate targets embedded in the image data) and the position (e.g., the x-y position) of each pixel associated with each of the one or more peak values. Assuming the peak value exceeds a predefined level, the location of the corresponding pixel is stored as the location of a corresponding candidate target by the peak pixel location recording function 415.

[0031] FIG. 5 illustrates, in greater detail, the ELP based pre-screen filter 305 and, more particularly, the processing tasks associated with the ELK 405. As stated, the ELK 405 receives the image data. The image data is then down-sampled as shown by task 505. The data associated with the original image and each down-sampled image are then stored in memory. Down-sampling may involve a low-pass filtering process, whereby high frequency components are progressively removed from the image data. This may be achieved through decimation, whereby groups of adjacent pixels in the original image are averaged such that the resolution of the first down-sampled image is, for example, one-half that of the original image. The first down-sampled image may then be down-sampled in the same manner. The second down-sampled image would then have a resolution that is one-half that of the first down-sampled image. In a preferred embodiment, the down-sampling process continues until a pre-specified number of down-sampled images are obtained, where each subsequently generated down-sampled image has a progressively lower resolution than the image from which it was generated. If one were to picture the original image and all of the down-sampled images stacked in order, one on top of the other, with the highest resolution image on the bottom and the lowest resolution image on the top, one may think of the image stack in terms of a pyramid.

[0032] In accordance with one embodiment of the present invention, the ELK 405, after generating and storing the down-sampled images, processes (i.e., filters) the original image data and the data associated with each down-sampled image using an elliptical Laplacian operator (ELO), as shown by task 510. In an alternative embodiment, the ELK 405 may select, based on target range information (i.e., the distance between a prospective target and the image sensor), a number of images from amongst the original image and each of the down-sampled images, such that the size of any candidate target as it appears in the image data closely approximates the estimated size of the target given the aforementioned target range information. The ELK 405 then processes (i.e., filters) only the data associated with the selected images using the elliptical Laplacian operator. Target range information may be obtained through a variety of sources depending upon the nature of the weapon system. For example, target range information may be available through navigation equipment integral to or in communication with the weapon system, radar information or advanced intelligence. Prior to describing how the original image and/or one or more of the down-sampled images are processed (i.e., filtered) using an elliptical Laplacian operator, the manner in which the ELK 405 may generate the elliptical Laplacian operator is described.

[0033] FIG. 6A illustrates an exemplary three-dimensional elliptical Laplacian operator. For purposes of illustration only, this elliptical Laplacian operator covers a 50×50 pixel area in the xy plane, where the values contained in each pixel represent the amplitude (i.e., the z direction values) of the operator. FIGS. 6B and 6C illustrate the amplitude of the operator in the xz and yz planes, along the major and minor axes of the elliptical operator, respectively. Again, the elliptical Laplacian operator shown in FIG. 6A is exemplary, and it will be understood that the amplitude and distribution will approximate the size and overall spatial characteristics of the expected target(s), as well as the level of contrast between the target(s) and the background imagery.

[0034] In a preferred embodiment of the present invention, the ELK 405 generates the elliptical Laplacian operator using a Difference of Gaussians (DOG) technique. A Gaussian is mathematical function that has a normal distribution. FIG. 7A illustrates two exemplary Gaussian functions, where Gaussian A has a greater amplitude than Gaussian B, but a smaller standard deviation (i.e., data spread about the mean value). In accordance with the DOG approach, subtracting Gaussian B from Gaussian A results in a Laplacian, for example, the Laplacian illustrated in FIG. 7B. If, as shown, Gaussian A and B are three-dimensional circular Gaussians, the resulting three-dimensional Laplacian will be circular as shown, for example, in FIG. 7C.

[0035] In accordance with one embodiment of the present invention, a Laplacian operator may be generated by subtracting one circular Gaussian from another, as explained above. The resulting circular Laplacian is then manipulated to achieve an elliptical Laplacian operator which closely matches the size, spatial characteristics and image intensity of prospective targets. However, in a preferred embodiment of the present invention, the Laplacian operator is generated, not by subtracting one circular Gaussian from another, but instead subtracting one elliptical Gaussian from another elliptical Gaussian. The resulting Laplacian will be elliptical.

[0036] In order to generate an elliptical Laplacian operator which matches the size, spatial characteristics and image intensity of prospective candidate targets, it is necessary to select appropriate Gaussians, such that the subtraction of one from the other results in the desired elliptical Laplacian operator.

[0037] As one skilled in the art will understand, a Gaussian may be represented by the following equation:

[0038] $G\left(X\right)=\frac{1}{{\left(2\pi \right)}^{n/2}{\Sigma }^{1/2}}\exp \left[-1/2{\left(X-M\right)}^T{\Sigma }^{-1}\left(X-M\right)\right]$
where X is a vector comprising, for example, xy coordinate values that define the Gaussian distribution in three-dimensional space; where M is a vector reflecting the mean value of vector X; where T is the symbol for transpose matrix; and where Σ represents the standard deviation of Gaussian G(X). Further, standard deviation Σ may be represented by the following covariance matrix:

[0039] $\Sigma =\left[\begin{array}{cc}{\sigma }_{11}& {\sigma }_{12}\\ {\sigma }_{21}& {\sigma }_{22}\end{array}\right]$
where σ₁₁ represents the distribution of the corresponding Gaussian along the x axis, σ₂₂ represents the distribution of the Gaussian along the y axis, and σ₁₂ and σ₂₁ represent the distribution of the Gaussian where the Gaussian has been rotated about the x and y axes respectively. The terms σ₁₁ and σ₂₂ are referred to as the auto-correlation terms, and if they are of equal value, the corresponding Gaussian is circular. If σ₁₁ and σ₂₂ are not of equal value, the corresponding Gaussian is elliptical. Thus, in order to generate an appropriate elliptical Laplacian operator, the values of σ₁₁ and σ₂₂ must be set accordingly.

[0040] The terms σ₁₂ the σ₂₁ are referred to as the cross-correlation terms. In addition to setting σ₁₁ and σ₂₂ to appropriate values, σ₁₂ and σ₂₁ must also be set to appropriate values. If σ₁₂ and σ₂₁ are both equal to zero, the corresponding Gaussian will be symmetric about the x and y axes. However, in order to accommodate an image sensor that is tilted with respect to the horizon, or more specifically, tilted with respect to the major axis of the prospective candidate target(s), the values for σ₁₂ and σ₂₁ should be selected so that the major axis of the resulting elliptical Laplacian operator is parallel to the horizon, where the major axis of the prospective candidate target(s) are parallel to the horizon, or more specifically, parallel to the major axis of the prospective targets themselves, where the major axis of the prospective candidate target(s) are parallel to neither the horizon or the image sensor.

[0041] The DOG is but one approach that may be employed for generating an elliptical Laplacian operator. Certainly, functions other than Gaussians may be used in a similar manner to generate an elliptical Laplacian operator, such as quadratic functions or sine waves. In fact, it is also possible to generate the desired elliptical Laplacian operator by brute force. That is, constructing the operator by defining all of its desired characteristics (e.g., major axis, minor axis, amplitude, standard deviation).

[0042] Regardless how the elliptical Laplacian operator is generated, the ELK 405 uses the operator to process (i.e., filter) each of the original and down-sampled images or, alternatively, a select number of these images based on target range. In a preferred embodiment, processing each image involves a convolution process. This may be achieved by overlaying the elliptical Laplacian operator atop a given pixel in a given image. For those images having higher resolutions, the size of the elliptical Laplacian operator will in all likelihood be larger than a single pixel. Thus, the elliptical Laplacian operator will overlay not only the given pixel but a number of neighboring pixels as well. The value of the given pixel, according to a typical convolution process, would be multiplied by the corresponding elliptical Laplacian operator value. Additionally, the value of each neighboring pixel covered by the operator are likewise multiplied by a corresponding value associated with the elliptical Laplacian operator. The multiplication values are then summed and averaged. The average value is then attributed to the given pixel. In general, this process is repeated for each pixel in each image (i.e., the original image and the down-sampled images) or, alternatively, each pixel in each of the images selected from amongst the original and down-sampled images based on target range. However, pixels located on the edge of an image or within a certain distance from the edge of an image, depending on the size of the elliptical Laplacian operator, may be excluded from the convolution process. When the one or more images have been processed using the elliptical Laplacian operator, the resulting vector (i.e., set of values) for each processed image is passed on to the peak detector 410, as shown in FIGS. 4 and 5.

[0043] For each resulting vector, the peak detector 410 scans the values and identifies a peak value. In a preferred embodiment, the peak value is compared to a threshold, where the threshold level may be a function of the image background, and more specifically, a predefined number of standard deviations above the mean image background value. If the peak value is greater than the threshold value, the peak value is attributed to a candidate target. If the peak value is not greater than the threshold value, it is assumed there is no distinguishable candidate target in that particular image. The position (e.g., the x-y position) of the pixel that corresponds with the peak value is stored by the record peak pixel location function 415. It should be noted that more than one peak value may be identified in a given image (i.e., in a given vector), particularly where more than one candidate target is embedded in the image data.

[0044] Referring back to FIG. 3, the image data and candidate target location information are passed to the Target delineator 310, which, as stated, establishes the extent of each candidate target. The feature extractor 315 then measures the various features associated with each candidate target and, based thereon, the target classifier 320 classifies each candidate target as a true target or a non-true target.

[0045] FIG. 8 illustrates the processing tasks associated with an alternative ELP based pre-screen filter algorithm. In FIG. 5, the ELK 405 receives the image data. Per task 505, the image data is down-sampled. Then, per task 510, the ELO is used to process (i.e., filter) the input image and/or one or more down-sampled images. Alternatively, the input image may not be down-sampled. Instead, per task 810, the ELO may be enlarged. In FIG. 8, the ELO is progressively enlarged by a 2:1 ratio. It will be readily apparent, however, that the ELO may be enlarged using a ratio other than a 2:1 ratio. It will also be readily apparent that by processing (i.e., filtering) the input image with progressively larger ELOs, the P_d associated with relatively large targets appearing in the input image increases.

[0046] FIGS. 9A-C illustrate the concept of progressively enlarging the ELO, per task 810. If, for example, FIG. 9A reflects the ELO, FIG. 9B reflects the larger 2×ELO. Likewise, FIG. 9C reflects the even larger 4×ELO.

[0047] Since numerous additional modifications and alternative embodiments of the invention will be apparent to those skilled in the art in view of the foregoing description, the above description is to be construed as illustrative only, and is for the purpose of teaching those skilled in the art the best mode of carrying out the invention. The details of the present invention described above may be varied substantially without departing from the spirit of the invention, and the exclusive use of any modification which comes within the scope of the appended claims is reserved.