Correlation based in frame video tracker

[0001] The present invention relates to video correlation trackers, and in particular, video correlation trackers that efficiently track, identify and register objects of interest.

[0002] There are numerous applications in the video processing arts where it is necessary to either track objects of interest from one image frame to the next, or to record an image and compare the saved image with another saved image or a current image. In general, such applications include unmanned air vehicle (UAV) navigation, mapping, object identification, and scene reconstruction.

[0003] For example, when identifying an object on the ground from a camera on a UAV, the system tracks the object from image frame to image frame until a level of certainty is reached that the object is identified. Objects are identified by passing the information from each successive image frame to a classification algorithm. In identifying the object of interest from image frame to image frame, each image frame or a portion thereof must be processed and reprocessed until the associated classification algorithm's output level has exceeded a predetermined threshold at which point an identification is declared. Such processing and reprocessing requires a tremendous amount of computer resources.

[0004] Consequently, video correlation tracking would benefit from an improved system that tracks objects and registers images with a minimal expenditure of processing time and other

[0005] The present invention uses a video correlation system to track objects from image frame to image frame. The invention identifies and stores reference patches from a first image frame. A reference patch is an area of the image delineated by a boundary, and is either placed along the leading edge of the image in search of a contrast anomaly, or it is placed around an object of interest (that has been identified via a classification algorithm). The invention then estimates, through the use of a dead reckoning algorithm, where the reference patch should appear in a successive image frame, and places a trial patch at that location in the image frame.

[0006] Thereafter, the invention cross-correlates the positions of the trial patches and reference patches. The cross-correlation produces two benefits. First, the displacement of a correlation peak generated by the cross correlation is used to estimate corrections to the dead reckoning algorithm. These corrections are fed into the dead reckoning algorithm and used to correct future predictions. Second, the cross-correlation produces tie points that are used to register the image. The tie points are also used to identify where objects of interest are located, and in turn where processing resources should or should not be concentrated.

[0007] It is consequently an object of the present invention to estimate corrections to a dead reckoning algorithm.

[0008] It is another object of the present invention to identify reference patches whose associated object of interest has been identified by a classification algorithm, and release said reference patch to free up processing resources.

[0009] FIG. 1 illustrates a camera mounted on an air vehicle and its corresponding field of view.

[0010] FIG. 2 illustrates the steps of the present invention.

[0011] FIG. 1 illustrates an environment in which the present invention functions. A camera 20, or other video sensing or recording device, is positioned such that its field of vision encompasses an area of interest. Such a set up includes a camera mounted on a manned or unmanned air vehicle, a security camera mounted on a building, or a camera positioned in a manufacturing facility to track one or more manufacturing lines. In these environments, the present invention may perform one or more functionsthe identification and tracking of images through the use of contrast anomalies and classification algorithms, the prediction of future positions of objects of interest, the identification of areas of interest in the image frame where processing power is to be concentrated, and/or the identification of areas of little or no interest on which processing resources should not be expended.

[0012] When implemented on a UAV with a camera that scans the terrain below, for example to identify objects of interest, the present invention functions as illustrated in FIG. 2. The system first captures an image frame (110). At least one, and preferably three or more, reference patches are selected for that image frame (120). Reference patches are two-dimensional pixel arrays, or a memory segment containing data about the two-dimensional reference patch, that outline a portion of the entire image. The boundaries of a reference patch are large enough to account for uncertainties in camera motion.

[0013] The system selects reference patches in one of two ways. First, it can select a portion of the image frame along the leading portion of the image frame. Such a selection along the leading edge of the image depends upon the presence of contrast anomalies along that leading edge of the image (e.g. the contrast between the water in a lake and the shoreline) (115). Contrast anomalies are identifiable because in a natural scene, it is unusual to see high contrasts. When high contrasts appear, it is normally the result of an object of interest, e.g., a body of water. Alternatively, the reference patch may be selected to have strong texture as this will tend to produce a narrower, more pronounced correlation peak. In addition to contrast anomalies, the system can place a patch around an object of interest that has been identified by a classification algorithm.

[0014] After selection of reference patches, the data regarding these patches are stored in some manner, preferably in a processor-based medium. At this point, the system captures and begins processing a successive image. Specifically, the system estimates where each reference patch from the previous frame should appear in the new image frame. The system then places a trial patch at this estimated location. The trial patch marks the location corresponding to its associated reference patch (according to variables such as the speed of the camera) in this new image frame. The position of a trial patch is estimated and placed in the image frame for each reference patch.

[0015] The placement of a trial patch in the new image is accomplished through the use of well-known dead reckoning techniques (130). In general, dead reckoning takes into account the speed, acceleration, rotation and course of travel of an object of interest, and uses that information to predict where the object of interest, and hence the trial patch, should be located. There are several tools that can be used to implement dead reckoning, such as Kalman filters, alpha/beta trackers, or even simple integration techniques. All of these tools depend on state vectors that have multiple inputs for parameters describing the dynamics of the system. While each of these systems involve complex computations and state vector analysis, they all use the fact that the speed of a stationary object of interest is the speed of the UAV on which a camera is mounted. This fact, coupled with the known elapsed time between image frames, and the known width of the area covered by the image frame, is used to locate the position of the trial patch.

[0016] The system cross-correlates (using two dimensional fast Fourier transforms) each trial patch with its respective reference patch in the current frame (140). The cross correlation identifies the difference in the position of a trial patch and its associated reference patch. If the position of the trial patch correlates exactly with the position of the reference patch, no change need be made to the dead reckoning algorithm. If the coordinates of the trial patch do not correlate with the coordinates of the reference patch, then depending on whether the mismatch is a result of speed, direction, or a combination of the two, corrections are made to the dead reckoning algorithm's coefficients.

[0017] The cross-correlation also produces tie points, which are a new estimate of the center of the reference patch (of the next image frame). The tie points also provide coefficients for an interpolation (e.g. a bilinear interpolation) so that points from a previous image can be mapped onto points for a successive image. It is in this manner that the images are registered in a common coordinate system (150, 160).

[0018] As the UAV and camera travel over the area of interest capturing successive images, the dead reckoning algorithm's coefficients will tend to converge and become more accurate due to the normal accumulation of statistics. In addition, since each new image adds information for the classification algorithm to process, the confidence level of the classification algorithm will also increase with each successive frame. One way of determining the threshold level required to achieve a given confidence level for classification is met is through the use of a receiving operating curve (ROC). When the threshold is met, and the object is identified and registered, the patch dedicated to that object of interest can be released to be used in connection with new objects of interest coming into the field of vision at the leading edge of the image frame. Patches are also released when they move off the trailing edge of the image. After the system releases the patch of a particular object of interest, processing resources are no longer expended on that object of interest, and can be dedicated to processing new objects of interest, or concentrating on an area of interest.

[0019] While certain objects of interest are readily identifiable by the system and will require few image frames before a level of confidence is obtained, other objects may be more difficult to identify. Such objects are prime candidates for the concentration of processing resources. For example, if the system is trying to identify telephone poles in an area of interest, and the UAV is slightly offset above a pole, the contrast produced by the pole indicates a short longitudinal object, then indicates a circular object as the UAV moves over the pole, then once again indicates a short longitudinal object as the UAV begins to move away from the pole, and then finally indicates the appearance of a telephone pole as the UAV moves away from the pole and the camera and classification algorithm see the outline of the telephone pole. During this time period, numerous image frames are captured and processed. The present invention is able to relinquish processing resources from objects that have been identified to a certain level of confidence, and use those resources to identify comparatively difficult to identify objects such as telephone poles.

[0020] If the reference patch identifies an object of interest that is in motion, that reference patch is rejected and recycled since a moving object introduces error into the system (170). One way that the system determines that an object is moving is that the object moves in the image frame in a different direction than all the other objects of interest. The increased error caused by moving objects is due to the fact that the estimation of trial patch location is based on a dead reckoning algorithm, which takes into account the known speed of the camera, not the unknown speed of the object of interest.

[0021] While the invention has been described in its preferred embodiment, the words used are words of description rather than limitation, and changes may be made within the purview of the appended claims without departing from the true scope and spirit of the invention in its broader aspects.