System and method for object detection from a moving platform

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 61/107,543, filed Oct. 22, 2008, titled “Framework for Object Detection from a Moving Platform,” which is herein incorporated by reference.

[0002] This invention was made with U.S. Government support under contract number DAAB07-01-D-G601. The U.S. Government has certain rights in this invention.

[0003] The present invention relates generally to the detection of objects in a camera's field of view. More specifically, the framework described herein relates to the detection of objects or targets from a moving platform in a 3D-rich environment.

[0004] Significant interest exists in systems integrated into or onto a mobile platform (e.g., a vehicle) that can detect objects and utilize the object-detection information to properly classify and identify the detected objects. Prior art systems are configured to detect such potential objects; however, said systems are not able to classify the detected objects without returning a large number of false positives, leading to inefficiencies and an increased computational burden. False positives occur when a system improperly classifies a detected object as belonging to a particular category of objects the system is intended to identify.

[0005] One way these systems have attempted to lower the false positive rate is by utilizing filters. Some examples of filters include long wave infrared (LWIR) filtering and filters based on classifiers, e.g., AdaBoost or Support Vector Machine (SVM) classifiers with scale and/or rotation invariant feature descriptors (like histogram of oriented gradients (HOG) or Scale Invariant Feature Transform).

[0006] Each type of filtering has its advantages and disadvantages. LWIR filtering may be easy to compute, however it has a high rate of false positives. Conventional classifiers that use HOG feature descriptors have a lower rate of false positives compared to LWIR filtering; however, it requires significant computational resources and can produce false negatives (i.e., a false negative occurs when a filter removes a candidate region, identified by a detector, that contains an object) when objects appear against complex and highly textured backgrounds because image gradient-based features become fragile in the presence of multiple gradient directions in a local image patch.

[0007] Furthermore, in general object detection and filtering, it is often required to search for an optimal region of interest (ROI) size and position to obtain valid classification scores. This is due to the sensitivity of the classifiers to ROI alignment as rigid placement of the local feature sampling windows inside the ROI become susceptible to different object configuration changes, such as different body poses in the case of pedestrian detection. This results in the need for an exhaustive search over multiple positions and scales for each input ROI.

[0008] Therefore, there is need in the art for a method and system that provides for efficient object detection and classification with a low false positive rate.

[0009] According to an embodiment of the present invention, systems and methods are provided for detecting moving and stationary objects of a certain class (e.g., people) from a moving or mobile platform in the presence of strong parallax, such as, for example, in the case of a ground vehicle or robot moving through an urban environment.

[0010] According to an embodiment of the present invention, the systems and methods use input from sensors of different modalities with a common field of view (e.g., visible, image intensifier, thermal, radar and time-of-flight sensors such as LADAR).

[0011] According to an embodiment of the present invention, multiple sensor sets may be distributed around a moving platform to provide 360 degree situational awareness.

[0012] According to an embodiment of the present invention, a plurality of detectors and filters are utilized to reduce the number of false positive detections (i.e., reduce the false positive rate).

[0013] According to an embodiment of the present invention, the combination of detectors and filters includes using an AdaBoost or SVM classifier that uses features derived from a combination of contour-based matching and HoG feature descriptor.

[0014] According to an embodiment of the present invention, the system and methods include first applying a LWIR filter in order to weed out or eliminate obvious false positives; then applying a filter of a second type to further identify and remove false positives from the detected region(s).

[0015] According to an embodiment of the present invention, systems and methods are provided for inferring behavior or type of activity of a detected object by tracking the object for a short time interval. This inferred activity can be used both to confirm object detection and to identify interesting, suspicious or threatening behaviors.

[0027] The following description relates to a framework (system and method) for detecting and classifying moving and stationary objects of a certain class (e.g., people, vehicles) from a moving platform in the presence of strong parallax, such as in the case of a ground vehicle moving through an urban environment, herein referred to as an object detection system.

[0028] The object detection system uses input from sensors of different modalities (e.g., visible, image intensifier, thermal) with a common field of view. According to an embodiment of the present invention, a plurality of sensors of different modality can be combined to form a sensor set (e.g., a sensor set comprising a visible sensor, thermal sensor and image intensifier sensor). Multiple sensor sets may be distributed around a vehicle to provide 360 degrees situational awareness for the vehicle crew.

[0029] FIG. 1 illustrates an exemplary object detection system and process flow including one sensor set. Input data from different sensors (i.e., sensor 1 through sensor x) is aligned both spatially and temporally to produce a synchronized sequence, according to methods known by those having ordinary skill in the art. A visual odometry module 10 recovers the camera pose for every frame of the synchronized sequence, according to techniques known by those having ordinary skill in the art. According to an embodiment of the present invention, the visual odometry module 10 calculates the position and orientation of the moving platform The object detection system according to embodiments of the present invention may comprise any suitable number of detectors 20 (e.g., detector 1 through detector n in FIG. 1) and filters 30 (e.g., filter 1 through filter m in FIG. 1), a subset of them being “active” at any particular time.

[0030] According to an embodiment of the present invention, detectors 20 and filters 30 can be activated or deactivated based on several criteria. Typically, the “active” subset of detectors 20 and filters 30 is based on determinations made by the object detection system regarding environmental properties (e.g., distance to a target, latency requirements, speed of the target, speed of the mobile platform). It should be understood that the active subset of detectors 20 and filters 30 may consist of every detector 20 and every filter 30 included in the object detection system.

[0031] The active detectors 20 are applied to the input data and their responses are OR-ed together to form a set of “candidate detections”. As used herein, the term “candidate detections” includes but is not limited to regions of interest in which a detector has potentially identified an object. The candidate detections are then passed through a series of active filters 30 of differing types (i.e., a filter of a first type, a filter of a second type) to reduce the number of false positives (i.e., produced when a detector 20 or filter 30 falsely identifies an object in a region of interest) and produce a set of “confirmed detections.”

[0032] It should be understood that the filters 30 may either remove a candidate detection from a set of candidate detections, classify a candidate detection in a set of candidate detections (e.g., candidate detection comprises an object, candidate detection does not comprise an object, etc.), or both. A filter 30 can also be configured to classify an object. In this manner a filter 30 may be understood to be a classifier. Exemplary filter types are described in detail below. Each time a set of candidate detections is passed through a filter, a refined set of candidate detections is produced. After the set of candidate detections has been through the series of active filters, a final set of candidate detections is returned (i.e., a set of confirmed detections). The set of confirmed detections is a subset of a set of candidate detections that were not previously discarded by the filtering process. The set of confirmed detections is reported to the user through perceptible means.

[0033] According to an embodiment of the present invention, an exemplary detector 20 suitable for use in the object detection system is a moving target detector, known by those of ordinary skill in the art. An exemplary moving target detector may utilize the epipolar constraint method and shape consistency method, as described in United States Patent Application Publication No. 2007/0273696, which is hereby incorporated by reference herein.

[0034] According to an embodiment of the present invention, an exemplary detector 20 suitable for use in the object detection system is a slow moving and stationary targets detector, known by those having ordinary skill in the art. An exemplary slow moving and stationary targets detector suitable for use in connection with embodiments of the present invention is a 3D template detector that searches the scene ahead of each sensor (e.g., camera) for regions of depth that match a template that describes the expected size and shape of an object. Examples of slow moving and stationary targets detectors are described in U.S. Pat. Nos. 6,956,469 and 7,263,209, and United States Patent Application Publication No. 2008/0159620, which are hereby incorporated by reference herein.

[0035] According to an embodiment of the present invention, the object detection system can use either monocular or stereo input. Exemplary embodiments of the present invention may use range information recovered from motion stereo for the 3D template detector or use range recovered from stereo for the 3D template detector.

[0036] According to an embodiment of the present invention, the detectors 20 may be of different modality types (e.g., visible, image intensifier, thermal). For example, an IR hot-spot detector may be used for locating people or other objects capable of generating a heat signature.

[0037] Although FIG. 1 depicts the filters in series, it should be understood that the filters 30 may be operated in parallel, in series, or a combination of both.

[0038] FIG. 6 depicts exemplary results achieved by the methods and systems of the present invention. The graph shows the fraction of the targets that were detected (y axis), with “1” meaning that all targets were detected, versus the number of false positives generated for an entire test sequence (x axis). In this example, each test sequence was approximately 2 minutes long. Each point in FIG. 6 represents the output of a module (e.g., a detector, a detector and a infrared filter, a detector plus a infrared filter and a activity classification filter) in the processing pipeline. After the detection stage 601, all targets are detected, but with a large number of false positives (e.g., approximately 120). Applying an infrared filter 602 after the detection stage 601 reduces the number of false positives (e.g., approximately 100). Applying an exemplary filter 603 (i.e., “C1s” in the graph) will also result in the reduction of false positives (e.g., approximately 80). By applying multiple filters 604 (e.g., an IR based segmentation filter and a contour-based filter) the number of false positives is reduced further than either of the filters by themselves.

[0039] Furthermore, according to embodiments of the present invention, an object detection system can be implemented in a manner that is optimized for efficiency. For example, cheap to compute filtering methods may be utilized before sophisticated and computationally intensive filters in order to efficiently remove obvious false positives with the least amount of computing power. For example, a Long Wave Infrared (LWIR) filter may be utilized before using a classifier based on Contour matching and Histogram of Oriented Gradient (HOG) feature descriptor. Since the LWIR filter can reject obvious false positives with relatively low computing power, an exemplary system does not get bogged down in a target rich 3D environment.

[0040] According to embodiments of the present invention, a wide variety of filter types may be used in order to detect false positives in a set of candidate detections. Some of the exemplary filter types that may be utilized with the object detection system and methods of the present invention are described in further detail below.

[0041] Filter/Classifier Types

[0042] I. Infrared (IR) Based Segmentation and IR Classifier Filters

[0043] According to an embodiment of the present invention, LWIR filters are used to classify candidate detections based on several statistical properties (e.g., mean, variance). FIG. 2 shows an example where 5 of the 7 candidate detections are rejected based on their mean and variance values. In FIG. 2, the boxes labeled 201 and 202 are rejected based on their mean and variance. The two boxes 203 are confirmed detections.

[0044] FIG. 3 depicts an exemplary method for a LWIR filter. First, the LWIR filter computes a derivative of LWIR data (i.e., thermal intensity) across image columns (i.e., accentuate vertical edges). Second, for each detection region (R), the LWIR filter computes a threshold (T) (i.e., a first threshold score) such that a percentage (e.g., 80%) of the derivative pixels within region (R) are below the threshold (T) (i.e., a set of pixels a with low thermal intensity derivative) and the remaining pixels (e.g., 20%) are above the threshold (T) (i.e., a set of pixels with a high thermal intensity derivative). Third, the LWIR filter binarizes the region (R) (i.e., changes each pixel in R to either black or white) using the threshold (T). Pixels with the highest thermal intensity derivative are chosen first, leaving the pixels with lower thermal intensity derivative to be below the threshold (T). For example, if the threshold (T) was such that 80% of the pixels are below the threshold (T) (i.e., the first set of pixels), only the 20% of the pixels with the highest thermal intensity derivative would be above the threshold (T) (i.e., the second set of pixels). In this example, 80% of pixels below the threshold (T) are binarized to black and the 20% of pixels with the highest thermal intensity derivative are binarized to white.

[0045] After the LWIR filter binarizes the pixels, a connected component analysis is performed. A connected component is a grouping of pixels that are above the threshold. The connected component analysis consists of identifying the number of connected components within the region (R). The LWIR filter then stores the number of connected components (N).

[0046] The LWIR filter then determines if the computed threshold (T) is less than a certain minimum level of thermal intensity derivative (L). If the computed threshold (T) is less than level (L), then region (R) is rejected as a false positive.

[0047] As an alternative to, or in combination with the previous step, the LWIR filter computes the area of pixels (A) and a number of connected components (N). If the number of connected components (N) divided by area (A) is greater than or equal to a second threshold (C), the candidate detection is identified as a false positive. For an example, take a candidate region comprising a pedestrian. After making the candidate region binary, only a few connected components would remain (e.g., a connected component representing the boundaries of the head, a connected component representing the boundaries of the torso, etc). However, if applied to a candidate region comprising a road surface, after making the candidate region binary, it is likely many small connected components would remain (e.g., different parts of the road surface that are of the highest thermal intensity derivative). Therefore, if a candidate region has a few large connected components (i.e., the pedestrian), it is likely to be an object of interest. Inversely, if the candidate region has several small connected components (i.e., the road surface), it is likely to be a false positive.

[0048] II. Consistency of Detection Over Multiple Frames Filters

[0049] A Consistency of Detection over Multiple Frames Filter is configured to associate instantaneous detections over time, and only those that persist for a certain number of frames are confirmed as being an object of interest. Since true targets are present in the field of view for longer periods of time and some of the false positives are due to spurious detections, the consistency filter reduces the number of false positives.

[0050] A Consistency of Detection over Multiple Frames Filter may be part of an active subset of filters based on an operating scenario. The number of frames over which the consistency is checked can be modified depending on the acceptable detection latency. For applications where a very low latency is required, the number of frames will be small, or the Consistency of Detection over Multiple Frames Filter can be deactivated. When the number of false alarms is more critical than the latency, the number of frames can be increased.

[0051] III. Implausible Depth, Size and 3D Constraints for Detected Object Filters

[0052] An Implausible Depth, Size and 3D Constraints for Detected Object Filter, the range to an object of interest is determined from motion stereo or from the target image location and the ground plane equation. Given the target image size, range to target and nominal size for an object, detections for which these measurements are inconsistent are eliminated.

[0053] IV. Scene Classification Filters

[0054] A Scene Classification Filter is configured to obtain 3D range information from motion-stereo detectors, and analyze the range information to extract local scene structures (e.g., vertical wall like regions, trees). Knowledge of the scene structure is then used to reject some of the candidate detections that overlap with the background.

[0055] V. Activity Classification Filters

[0056] An Activity Classification Filter can be used to confirm detections that contain objects performing some specific activity (e.g., a person walking, a person running, a bird flying). In this case the spatio-temporal volume for a particular target is analyzed to determine the type of activity it represents. The main challenges are the great variability due to change in viewing direction (same type of actions may look completely different) and the huge intra-class variations due to high spatial-temporal dimensionality. To address these challenges, the embedded perceptual distance to a number of typical action exemplars as the underlying representation rather than modeling the high dimensional action volume in space-time. Changes in viewing direction may be handled by having a number of exemplars for the same action, but from different viewpoints. The perceptual distance computation does not require subtraction of complex backgrounds.

[0057] FIG. 4 depicts an exemplary Activity Classification Filtering process, suitable for use in connection with embodiments of the present invention. The Activity Classification Filter might be part of an active subset of filters only for targets that are comprised of enough pixels to enable discrimination of a certain activity type.

[0058] VI. Contour+Histogram of Oriented Gradients-based Classifier Filters (C-HOG Filters)

[0059] A C-HOG Filter, according to an embodiment of the present invention, is configured to compute a two-dimensional spatial-orientation histogram where the weights in the histogram are weighted by their gradient magnitude. Furthermore, an exemplary C-HOG Filter utilizes a set of refined sampling windows as well as a Global Foreground Mask 530 calculated by a Contour Matching Filter, as described below, to further improve the efficacy of the C-HOG Filter.

[0060] According to an embodiment of the present invention, an exemplary Contour Matching Filter is described in detail herein. First, a Contour Vocabulary is constructed in an offline procedure by sampling binary contours from a database of object silhouettes. In the Contour Vocabulary matching step, first, an edge map and edge gradients are computed from a given candidate detection. Second, a local part contour set is matched to the extracted edge map by chamfer matching, which refines each local feature sampling window. Specifically, the chamfer score for each part template is computed in the local region and the associated sampling window is anchored at the position of the maximum chamfer score.

[0061] With reference to FIG. 5, image 510 shows how a typical prior art approach refines sampling window positions. In image 510, the sampling windows are rigid. The application of an exemplary Contour Matching Filter, according to embodiments of the present invention, to the same candidate region as shown in image 510 is illustrated in images 520, 540. In image 520, the Contour Matching Filter refines the sampling window positions on top of underlying parts of the object through the chamfer matching process described above.

[0062] According to an embodiment of the present invention, the Contour Matching Filter additionally creates a Global Foreground Mask 530 by overlapping the set of matched local templates, each of which is weighted by matching scores. In this mariner, the Contour Matching Filter explicitly handles local feature alignment with body contour segments and suppresses background noise from foreground mask composition and filtering. The set of matched local templates represents a collection (e.g., a database) of stored contours frequently associated with a given object (e.g., the contours of a person's head, the contours of a vehicle's wheel, etc.). FIG. 9 depicts an exemplary set of matched local templates.

[0063] The Global Foreground Mask 530 can then be used in conjunction with an exemplary C-HOG Filter to provide better results due to the suppression of background noise. In accordance to embodiments of the present invention, a C-HOG Filter utilizes the set of refined sampling windows as well as the Global Foreground Mask 530 provided by the Contour Matching Filter, for object classification. The C-HOG utilizes the refined sampling feature window and the composed Global Foreground Mask 530 for computing HOG feature 530.

[0064] FIG. 7 depicts an exemplary C-HOG filtering scheme, in accordance with embodiments of the present invention, which applies a Contour Based Matching Filter in combination with a HOG classification to determine whether a candidate detection (ROI) belongs to a particular class. First, a candidate detection (ROI) is sent to a Contour-Based Matching Filter. Second, the Contour-Based Matching Filter extracts the features from the candidate detection and computes an edge map and edge gradients. Third, binary contours from the set of matched local templates are matched to the edge map. Fourth, the Contour-Based Matching Filter computes a set of sampling window localization information and a Global Foreground Mask 530, as described above. Finally, the Contour-Based Matching Filter transmits the set of sampling window localization information and the Global Foreground Mask 530 to a HOG classifier for further processing in which the area processed is the subset of the candidate region present in the overlapping sections of the Global Foreground Mask 530 and the set of refined sampling windows. Image 540 depicts a visual representation of the results of a combination of a Contour-Based Matching and a HOG classification that excluded a majority of background noise during the HOG feature computation process.

[0065] FIG. 8 depicts exemplary results achieved by the previously described C-HOG Filter, in accordance with embodiments of the present invention. The graph shows the quantitative evaluation between typical HOG-based Filter and an exemplary C-HOG Filter by showing fractions of targets that were detected (y axis) in relation to the fraction of detections that were false positives (x axis). In FIG. 8, both Filters are trained and evaluated under the same condition of training set of 2000 positives and 4000 negatives and test on 4841 positives and 15494 negatives. The figure shows the distinctive advantage of the C-HOG Filter described herein, illustrating the improved detection rate and false positive rejection rate achieved according to embodiments of the present invention.

[0066] It is to be understood that the above-described embodiments are merely illustrative of the present invention and that many variations of the above-described embodiments can be devised by one skilled in the art without departing from the scope of the invention. It is therefore intended that such variations be included within the scope of the following claims and their equivalents.