Feature based high resolution motion estimation from low resolution images captured using an array source

[0001] The current application is a continuation of U.S. patent application Ser. No. 13/975,159, entitled “Feature Based High Resolution Motion Estimation from Low Resolution Images Captured Using an Array Source” filed Aug. 23, 2013, which application claims priority to U.S. Provisional Patent Application Ser. No. 61/692,547, entitled “Feature Based High Resolution Motion Estimation From Low Resolution Images Captured Using an Array Source” filed Aug. 23, 2012, the disclosures of which are incorporated by reference herein in their entirety.

[0002] The present invention relates generally to feature detection in digital images and more specifically to the use of array cameras and super resolution to improve the performance and efficiency of feature detection.

[0003] In digital imaging and computer vision, feature detection is a fundamental operation that is typically a preliminary step to feature-based algorithms such as motion estimation, stabilization, image registration, object tracking, and depth estimation. The performance of these algorithms depends sensitively on the quality of the feature point estimates.

[0004] Various types of image features include edges, corners or interest points, and blobs or regions of interest. Edges are points where there is a boundary between two image regions, and are usually defined as sets of points in the image which have a strong gradient magnitude. Corners or interest points can refer to point-like features in an image that have a local two dimensional structure. A corner can be the intersection of two edges, or a point for which there are two dominant and different edge directions in a local neighborhood of the point. An interest point can be a point which has a well-defined position and can be robustly detected, such as a corner or an isolated point of local maximum or minimum intensity. Blobs or regions of interest can describe a type of image structure in terms of regions, which often contain a preferred point. In that sense, many blob detectors may also be regarded as interest point operators.

[0005] A simple but computationally intensive approach to corner detection is using correlation. Other methods include the Harris & Stephens corner detection algorithm that considers the differential of the corner score with respect to direction using the sum of squared differences.

[0006] Achieving effective feature detection depends in part on providing high quality data, i.e., high resolution image(s), to the feature detector.

[0007] Systems and methods in accordance with embodiments of the invention enable feature based high resolution motion estimation from low resolution images captured using an array camera. One embodiment includes performing feature detection with respect to a sequence of low resolution images using a processor configured by software to identify initial locations for a plurality of detected features in the sequence of low resolution images, where the at least one sequence of low resolution images is part of a set of sequences of low resolution images captured from different perspectives, synthesizing high resolution image portions from the set of sequences of low resolution images captured from different perspectives using the processor configured by software to perform a super-resolution process, where the synthesized high resolution image portions contain the identified plurality of detected features from the sequence of low resolution images, performing feature detection within the high resolution image portions to identify high precision locations for said plurality of detected features using the processor configured by software, and estimating camera motion using the high precision locations for said plurality of detected features using the processor configured by software.

[0008] In a further embodiment, wherein the detected features are selected from the group consisting of: edges, corners, and blobs.

[0009] In another embodiment, performing feature detection with respect to a sequence of low resolution images further includes detecting the location of features in a first frame from the low resolution sequence of images, and detecting the location of features in a second frame from the low resolution sequence of images.

[0010] In a still further embodiment, detecting the location of features in a second frame from the sequence of low resolution images further includes searching the second frame from the sequence of low resolution images to locate features detected in the first frame from the sequence of low resolution images.

[0011] In still another embodiment, searching the second frame from the sequence of low resolution images to locate features detected in the first frame from the sequence of low resolution images further includes identifying an image patch surrounding the location of the given feature in the first frame in the sequence of low resolution images, and searching the second frame in the sequence of low resolution images for a corresponding image patch using a matching criterion.

[0012] In a yet further embodiment, the matching criterion involves minimizing an error distance metric.

[0013] In yet another embodiment, performing feature detection within the high resolution image portions to identify high precision locations for said plurality of detected features further comprises searching the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images using the high resolution image regions containing the features from the first frame in the low resolution sequence of images.

[0014] In a further embodiment again, searching the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images further comprises comparing high resolution image regions containing features from the second frame in the sequence of low resolution images to the high resolution image portions containing the features from the first frame in the sequence of low resolution images using a matching criterion.

[0015] In another embodiment again, the matching criterion involves minimizing an error distance metric.

[0016] In a further additional embodiment, the processor is part of an array camera that further comprises an imager array, the method further comprising capturing at least a plurality of the sequences of low resolution images in the set of sequences of low resolution images from different perspectives using the imager array.

[0017] In another additional embodiment, the high precision locations for said plurality of detected features estimate feature location at a subpixel precision relative to the size of the pixels of the frames in the sequence of low resolution images.

[0018] In a still yet further embodiment, performing feature detection with respect to a sequence of low resolution images further comprises performing feature detection with respect to a plurality of sequences of low resolution images, where each sequence is from a different perspective.

[0019] In sill yet another embodiment, the set of sequences of low resolution images comprises sequences of low resolution images captured in a plurality of different color channels, and performing feature detection with respect to a sequence of low resolution images further comprises performing feature detection with respect to a at least one sequence of low resolution images in each color channel.

[0020] Another embodiment includes an imager array, a processor configured by software to control various operating parameters of the imager array. In addition, the software further configures the processor to: capture a set of sequences of low resolution images captured from different perspectives using the imager array; perform feature detection with respect to one of the set of sequences of low resolution images to identify initial locations for a plurality of detected features in the sequence of low resolution images; synthesize high resolution image portions from the set of sequences of low resolution images captured from different perspectives, where the high resolution image portions contain the identified plurality of detected features from the sequence of low resolution images; perform feature detection within the high resolution image portions to identify high precision locations for said plurality of detected features; and estimate camera motion using the high precision locations for said plurality of detected features.

[0021] In a further embodiment, the detected features are selected from the group consisting of: edges, corners, and blobs.

[0022] In a still further embodiment, the processor is further configured to perform feature detection with respect to a sequence of low resolution images by detecting the location of features in a first frame from the sequence of low resolution images, and detecting the location of features in a second frame from the sequence of low resolution images.

[0023] In still another embodiment, the processor is further configured by software to detect the location of features in a second frame from the sequence of low resolution images by searching the second frame from the sequence of low resolution images to locate features detected in the first frame from the sequence of low resolution images.

[0024] In a yet further embodiment, the processor is further configured by software to search a second frame from the sequence of low resolution images to locate a given feature detected in the first frame from the sequence of low resolution images by: identifying an image patch surrounding the location of the given feature in the first frame in the sequence of low resolution images; and searching the second frame in the sequence of low resolution images for a corresponding image patch using a matching criterion.

[0025] In yet another embodiment, the matching criterion involves minimizing an error distance metric.

[0026] In a further embodiment again, the processor is further configured by software to perform feature detection within the high resolution image portions to identify high precision locations for said plurality of detected features by searching the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images using the high resolution image regions containing the features from the first frame in the low resolution sequence of images.

[0027] In another embodiment again, the processor is further configured by software to search the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images by comparing high resolution image regions containing features from the second frame in the sequence of low resolution images to the high resolution image portions containing the features from the first frame in the sequence of low resolution images using a matching criterion.

[0028] In a further additional embodiment, the matching criterion involves minimizing an error distance metric.

[0029] In another additional embodiment, the high precision locations for said plurality of detected features estimate feature location at a subpixel precision relative to the size of the pixels of the frames in the sequence of low resolution images.

[0030] In a still yet further embodiment, a plurality of the imagers in the imager array sense different wavelengths of light and the set of sequences of low resolution images comprises sequences of low resolution images captured in a plurality of different color channels.

[0031] In still yet another embodiment, the processor is further configured by software to perform feature detection with respect to a sequence of low resolution images by performing feature detection with respect to a at least one sequence of low resolution images in each color channel.

[0032] In a still further embodiment again, the processor is further configured by software to perform feature detection with respect to a sequence of low resolution images by performing feature detection with respect to a plurality of sequences of low resolution images, where each sequence is from a different perspective.

[0039] Turning now to the drawings, systems and methods for feature based high resolution motion estimation from low resolution images captured using an array camera in accordance with embodiments of the invention are illustrated. Two images sequentially captured using a legacy camera can reflect a relative displacement due to motion of the camera. This camera motion, or equivalently, the 3D structure of the scene, can be recovered from the images. The first step toward these goals is to perform feature matching between the two images, the initial step of which is to detect features in one or both images independently. Initial correspondences are then formed between image features by selecting a patch around each feature and minimizing an error distance metric, such as normalized cross correlation, between the patch and candidate patches in the other image. The set of initial correspondences between features can then be refined using a validation procedure such as Random Sample Consensus (RANSAC) for a given motion model.

[0040] Array cameras offer a number of advantages and features over legacy cameras. An array camera typically contains two or more imagers, each of which receives light through a separate lens system. The imagers operate to capture images of a scene from slightly different perspectives. The images captured by these imagers are typically referred to as low resolution images and super resolution processing can be used to synthesize a high resolution or super resolution image from a subset of the low resolution images. A comparison of a minimum of two low resolution images can provide parallax information used in super resolution processing. The terms low resolution and high resolution are used relatively and not to indicate any specific image resolution. Imagers in the array may sense different wavelengths of light (e.g., red, green, blue, infrared), which can improve performance under different lighting conditions and the performance of super resolution processing on images captured using the array. Super resolution processes that can generate higher resolution images using low resolution images captured by an array camera include those disclosed in U.S. patent application Ser. No. 12/967,807 entitled “Systems and Methods for Synthesizing High Resolution Images Using Super-Resolution Processes,” the disclosure of which is hereby incorporated by reference in its entirety.

[0041] A sequence of low resolution images captured by the imagers of an array camera typically contain temporal displacement between the frames due to camera motion, as in a legacy camera, but also intra-frame displacement between the constituent images of the array (i.e. the low resolution images captured by each imager in the array) for each frame due to parallax. Because the offset distances of each imager in the array are known, the parallax displacement can be calculated and used to register the images to perform super resolution processing.

[0042] In several embodiments, feature detection can be performed on a sequence of super resolution images generated using low resolution images captured by the array camera. Performing feature detection in this way can yield a subpixel estimate of feature positions (i.e. an estimate has a precision smaller than the size of the pixels of the sensors in the array camera used to capture the low resolution images). Referring to FIG. 1a, a low resolution image is shown with pixels forming edges 101 and 102. As discussed above, an edge is a feature that can be defined as a boundary between two image regions and a corner can be defined as the intersection of two edges. Point 103 is identified as a corner at the intersection of edges 101 and 102. In a subsequent image at time t+1 shown in FIG. 1b, point 104 is identified as a corner corresponding to point 103.

[0043] By applying a super resolution process, such as one of the processes described in U.S. patent application Ser. No. 12/967,807, the accuracy of the feature detection can be increased. Higher resolution images obtained by applying super resolution processing to low resolution images including the low resolution images shown in FIGS. 1a and 1b respectively are shown in FIGS. 2a and 2b. Featured-based algorithms such as motion estimation, stabilization, image registration, object tracking, and depth estimation can be performed on the higher resolution images in the same manner that they are performed on lower resolution images with the benefit of subpixel accuracy. Performing super resolution processing on the entire image, however, can require considerable computation that uses power and computing resources, and may not perform optimally on devices with limited processing capabilities such as mobile platforms.

[0044] In many embodiments of the invention, accurate feature detection can be achieved in a computationally efficient manner by initially identifying the location of features in low resolution images and then selectively performing super resolution processing to obtain in higher resolution the portions of the low resolution images containing the identified features. By only performing super resolution processing to obtain the portions of the super resolution images utilized in feature detection, feature detection can be performed at a higher speed (i.e. with fewer computations) while preserving the benefits of increased accuracy. In this way advanced functionality relying upon feature recognition such as (but not limited to) real time image stabilization during video capture can be performed in a computationally efficient manner. Array cameras and the use of super resolution processes to obtain high resolution image portions for performing feature detection in accordance with embodiments of the invention are discussed further below.

Array Camera Architecture

[0045] An array camera architecture that can be used in a variety of array camera configurations in accordance with embodiments of the invention is illustrated in FIG. 3. The array camera 100 includes an imager array 106, which is connected to a processor 108. Imagers 110 in the array 106 are evenly spaced in a 5×5 square. In other embodiments, imagers may have different spacing or can be arranged in other orientations in the array. The processor 108 is hardware, software, firmware, or a combination thereof that controls various operating parameters of the imager array 106. The processor 108 can also function to process the images received from imager array 106 to produce a synthesized higher resolution image using super resolution processes, or transfer the images to other hardware, software, firmware or a combination thereof to process the images. In several embodiments, the array camera includes memory containing an image processing application that can be utilized to perform feature based high resolution motion estimation using low resolution images captured by the array camera utilizing any of the techniques described below.

[0046] Although a specific architecture is illustrated in FIG. 3, any of a variety of architectures that enable the capture of low resolution images and application of super resolution processes to produce a synthesized high resolution image can be utilized in accordance with embodiments of the invention.

Obtaining High Resolution Image Portions

[0047] In many embodiments of the invention, super resolution is performed to obtain a portion of a high resolution image corresponding to the portion of a low resolution image containing an identified feature. Once the high resolution image portion is obtained, feature correspondences that were initially determined using the low resolution images can be refined at the higher resolution. A flow chart illustrating a process 120 for refining feature correspondences using high resolution image portions obtained using super resolution processing in accordance with an embodiment of the invention is shown in FIG. 4. Through the following discussion, reference is made to sample images illustrated in FIGS. 5a and 5b.

[0048] A feature detection algorithm is run on a first low resolution image captured by an imager in an array camera to identify (122) features in the image. An image from any of the imagers in the array camera can be chosen, so long as the second low resolution image used to perform feature detection is captured from the same imager. In many embodiments, feature detection can be performed with respect to sequences of images captured by multiple cameras to obtain additional information concerning the location of features. In a number of embodiments, the array camera includes cameras that capture images in different color channels and the array camera performs feature detection with respect to a sequences of images captured by cameras in multiple cameras. In certain embodiments, feature detection is performed with respect to a sequence of images captured by at least one camera in each color channel.

[0049] As discussed above, the types of features that can be detected in the low resolution image can include (but are not limited to) edges, corners, and blobs. Typically, a feature detection algorithm identifies one type of feature based upon the definition of the feature. A corner detector such as the Harris and Stephens detection algorithm can be used to identify corners. In the Harris and Stephens algorithm, an image patch is considered over a specified area and shifted. A corner is characterized by a large variation in the weighted sum of squared differences between the two patches in all directions.

[0050] Referring to FIG. 5a, a low resolution image captured at time t is illustrated. Point 140 in the image is identified as a corner using a corner detection algorithm. Similarly, the feature detection algorithm is run on the second low resolution image to identify (124) features in the image. A second image at some later time t+1 is depicted in FIG. 5b. Point 142 in the second image is identified as a corner.

[0051] In some embodiments of the invention, each feature in the first frame is matched (i.e. determined to correspond) to a feature in the second frame where possible. This initial correspondence may not be possible if the feature has moved out of the second frame or has moved a significant distance. In other embodiments, the features are not matched in the low resolution images, but are matched after performing super resolution on portions of the low resolution images (frames).

[0052] A neighborhood of pixels around each feature is selected (126) in each frame. Suitable dimensions for such a neighborhood can be 20 pixels by 20 pixels (20×20) to 60 pixels×60 pixels (60×60), although smaller or larger neighborhoods are possible and may be determined by the limitations of the computing platform carrying out calculations on the image. Moreover, the neighborhood can be of any shape and need not be square. The feature typically can fall within the boundaries of the neighborhood, but need not be centered in the neighborhood.

[0053] Referring to FIGS. 5a and 5b, a neighborhood of 20×20 pixels 144 is selected around point 140 in the first frame. Similarly, a 20×20 neighborhood of pixels 146 is selected around point 142 in the second frame. For each neighborhood, super resolution processing is performed (128) using parallax information to apply any necessary pixel shifts in the low resolution images captured by the other imagers in the camera array. Super resolution processing can be applied using a subset of the low resolution images generated by the array camera. As discussed above, an array camera captures images with multiple imagers simultaneously. A subset (i.e., minimum of two) of low resolution images obtained from different perspectives provides parallax information that can be used in super resolution processing. Suitable super resolution processes can include (but are not limited to) those disclosed in U.S. patent application Ser. No. 12/967,807 (incorporated by reference above).

[0054] As discussed above, differences exist in the low resolution images captured by the imagers of a camera array due to the effects of parallax. In order to synthesize a high resolution image portion containing a designated neighborhood, the effects of parallax are accounted for by determining the parallax between the images and applying appropriate pixel shifts to the pixels of the low resolution images. The pixel shifts may involve moving pixels into the designated neighborhood and shifting pixels out of the designated neighborhood. Accordingly, although a specific neighborhood of pixels in the synthesized high resolution image is identified, the super resolution algorithm may utilize pixels from the low resolution images that are outside the neighborhood and exclude pixels from the low resolution images within the neighborhood following correcting for parallax. Therefore, the input pixels from the low resolution images utilized to obtain a designated neighborhood of a high resolution image using super resolution processing are not limited to pixels within the designated neighborhood identified by performing feature detection within the initial low resolution image pair. The designated neighborhood simply guides the super resolution process with respect to the low resolution pixels to utilize to synthesize the portion of the high resolution image corresponding to the designated neighborhood. Methods for obtaining distance and other information using parallax calculations via an array camera that can be used in super resolution processing include those disclosed in U.S. Patent Application Ser. No. 61/691,666 entitled “Systems and Methods for Parallax Detection and Correction in Images Captured Using Array Cameras,” the disclosure of which is incorporated by reference herein in its entirety.

[0055] The resulting frames are illustrated in FIGS. 6a and 6b. Super resolution of the designated neighborhood 144 gives the high resolution 40×40 neighborhood 144′ and point 140′ shown in FIG. 6a. Super resolution of the low resolution neighborhood 146 gives the high resolution 40×40 neighborhood 146′ and point 142′ shown in FIG. 6b.

[0056] In high resolution neighborhood 146′, the position of point 142′ is slightly to the right of where it appears in low resolution neighborhood 146. Because super resolution restores the actual high frequency content of the image, the higher resolution neighborhood provides a “truer” representation of the point's actual position. In many embodiments of the invention, the newly calculated positions of points 140′ and 142′ within high resolution neighborhoods 144′ and 146′ can be used in matching (i.e. determining a correspondence between) points 140′ and 142′.

[0057] Referring to FIGS. 6a and 6b, an initial correspondence is formed between the first point 140′ and the second point 142′. Correspondence can be established using a variety of methods. A common method is to select a patch around each point and minimize an error distance metric, such as (but not limited to) normalized cross correlation, between the patch and candidate patches in the other image. Point 142′ is thus determined to correspond to point 140′ in the previous frame. Other methods for finding correspondence are known in the art.

[0058] Using the initial correspondences, any of a variety of feature-based algorithms, including (but not limited to), motion estimation, stabilization, image registration, object tracking, or depth estimation, can be performed on the images. The model that is developed using the features and correspondences (for example, a motion model) can be further refined using high resolution neighborhoods of pixels that encompass the relevant features.

[0059] The initial correspondences between points 140′ and 142′ are refined (130) using the high resolution neighborhoods (i.e. the high resolution image portions). Refinement may be accomplished using a variety of methods, including (but not limited to) recomputing a matching metric (e.g., normalized cross-correlation) between a pair of corresponding high resolution neighborhoods. Recomputing a matching metric can involve finding the normalized cross-correlation between high resolution neighborhoods 144′ and 146′, and using the metric to compute an estimated position of point 142, i.e., future position of point 140′ in the subsequent frame. In other embodiments, any of a variety of methods can be utilized appropriate to the requirements of a specific application.

[0060] A variety of validation procedures can be used such as the RANdom SAmple Consensus (RANSAC) method for a given model that was formed using the initial features and correspondences (such as a motion model for motion estimation). The RANSAC method utilizes a set of observed data values, a parameterized model which can be fitted to the observations, and confidence parameters. A random subset of the original data is iteratively selected as hypothetical inliers and tested by: fitting the parameters of a model to the hypothetical inliers, testing all other data against the fitted model, including a point as a hypothetical inlier if it fits well to the estimated model, keeping the estimated model if sufficiently many points have been classified as hypothetical inliers, re-estimating the model from the updated set of all hypothetical inliers, and estimating the error of the inliers relative to the model. Other suitable validation procedures appropriate to a specific application can also be utilized in accordance with embodiments of the invention.

[0061] Although a specific process is illustrated in FIG. 4, any of a variety of processes for detecting features in low resolution array frames and refining feature correspondences using super resolved regions over the features can be utilized in accordance with embodiments of the invention. While the figures and processes discussed herein depict a single corner in an image, embodiments of the invention can operate on images that include multiple features of various types.

[0062] Although the description above contains many specificities, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the presently preferred embodiments of the invention. Various other embodiments are possible within its scope.