Method for real time correlation of stereo images

[0002] The invention relates to a method of comparing two stereo images to determine a location of an object or point in a field of view.

[0003] It is well-known that the position of an object in a volume can be determined using two spaced apart cameras. Both cameras take an image of the object at the same time or nearly the same time. Then the images are compared to determine the location in each image of a point or series of points on the object. From that information one can calculate the location of the object in the volume such that each point on the object has a known and different x, y, z coordinate.

[0004] Today there are algorithms which allow computers to perform image matching of two stereo images. Typical images from a video camera contain a 640×480 array of pixels. In a “black and white” image each pixel will have a gray scale value of from 0 to 255. Current algorithms use the gray scale values to perform pixel comparisons to identify the position of an object in one image with respect to the other image. Although this method is quite accurate, substantial computer capacity is needed to perform the image matching and the process is relatively slow. As a result more expensive computer hardware is needed to do stereo image processing. Thus, one must either use expensive image processing hardware to achieve near real time processing or be satisfied with the slow processing speeds that occur with off the shelf computing components such as a personal computer (PC). Consequently, there is a need for a method of determining the position of an object from stereo images which is fast and can be performed on a low cost computer.

[0005] For many years the art has used the Laplacian pyramid to process and compress images as part of stereo processing. Compressed images are easier to store and transmit. When an image is subjected to a series of Laplacian transforms via pyramid processing the image becomes successively smaller dimensionally; however, the gray scale information remains at 8 bits. Each higher level array is half the dimensions of its predecessor. Prior to the present invention the art used these full gray scale Laplacian images for stereo image correlation which requires much computational complexity. Yet, I have found that by reducing the gray scale dimensionality of the Laplacian images I can correlate stereo images significantly faster using a simple processor.

[0006] I provide a method for correlating two stereo images in which the images are subjected to a Laplacian operator to produce reduced grayscale Laplacian images in which the pixels have a value of +1, 0 or −1. Then, I overlap the two images to produce pairs of overlapping pixels. The values of the two overlapping pixels are summed in a manner so that if both pixels are +1 or both are −1 the summed value is +1, if one pixel is +1 and the other pixel is −1, the resulting sum is −1 and if one or both pixels are zero the resulting sum is zero. All of the correlation values for the regions about the two overlapping pixels are combined to get a correlation value for the pair of pixels that overlap generating a correlation image. Then, the two Laplacian images are shifted relative to one another and correlation values are computed for each pair of pixels for this particular overlap. This process is repeated several times resulting in correlation images for each overlap. The overlap which has the highest correlation value is the best match for that pixel. Having determined the best match for each pixel, one can then determine the location of an object or point in the field of view using standard stereo processing techniques. Other objects and advantages of the method will become apparent from a description of certain present preferred embodiments illustrated in the drawings.

[0007] FIG. 1 is a diagram illustrating the images used for stereo processing.

[0008] FIG. 2 is a diagram illustrating creation of correlation images by shifting Laplacian images in a vertical direction.

[0009] FIG. 3 is a diagram illustrating creation of correlation images by shifting Laplacian images in a horizontal direction.

[0010] To correlate stereo images I begin with two images, each from one or the other of two spaced apart cameras. The first image will be called the reference image and the second image will be called the shift image. In the present method the original image 1 and 11 from each camera 10 is subjected to a Gaussian pyramid to produce transformed images 2, 3, 12 and 13. The original gray scale images are dimensionally reduced and scaled by using the Gaussian pyramid. The original 640×480 image is reduced to a 320×240 image (level 1 Gaussian image) and further reduced to a 160×120 image (level 2 Gaussian image). The images are shown in the drawings to contain a person's head. For purposes of illustration only a simple outline is shown. However, in actual images there would be much greater detail, particularly in the original image. The amount of detail becomes progressively less for each level from the original image at level 0 to the level 2 Gaussian image 3 and 13. This process reduces the physical size of the image as well as making subsequent processing more robust with respect to sensor or other noise that may be present. It should be noted that the image reduction is not a necessary step for this invention and the original image 1 and 11, could be used in subsequent steps.

[0011] Next, the Laplacian of image 3 and 13 is taken giving images 4 and 14 each a 160×120 gray scale image. A Laplacian image shows the high frequency content of the original image and is generally a signed 8 bit number. This Laplacian may be part of the Gaussian pyramid used to generate images 2, 3, 12 and 13 or the Laplacian may be performed using standard formulae.

[0012] The next step of the process is to reduce the gray scale of image 4 and 14, producing images 5 and 15, called a reduced gray scale Laplacian image. A reduced gray scale Laplacian image is one in which each pixel will have one of three values {−1, 0, +1 }. Values greater than or equal to 1 are assigned a value of 1, those less than or equal to −1 are assigned a value of −1 and those with a value of 0 are assigned the value of 0. This gray scale mapping accentuates the edges of the image. This reduced gray scale Laplacian is used to perform the correlation needed for stereo processing.

[0013] The reduced gray scale Laplacian images 5 and 15 from each camera are overlapped as indicated in FIGS. 2 and 3 and summed to produce a correlation image 20 or 22. Although one could sum all pairs of overlapping pixels to create correlation images I have found that summing pixels along objects' edges of one image with corresponding pixels in the second image is sufficient. When only edge pixels are summed correlations can be made much faster. Consequently, the correlation image may be derived for simpler reduced grayscale Laplacian images. This is the reason for generating reduced grayscale Laplacian images. In summing the pixel values of overlapping pixels if both pixels are +1 or both pixels are −1, the summed value is +1. If one pixel is +1 and the other pixel is −1, the summed value is −1. If one or both pixels have a zero value, the sum is zero. The following matrix shows correlation values for various pairs of pixels. $\begin{array}{cc}& \mathrm{Image}1\\ \mathrm{Image}2& \begin{array}{cccc}& -1& 0& +1\\ -1& +1& 0& -1\\ 0& 0& 0& 0\\ +1& -1& 0& +1\end{array}\end{array}$

[0014] The correlation image 20 or 22 is also a reduced grayscale image containing only the values {−1, 0, 1}. For stereo processing it is desired to find a maximum correlation value corresponding to each pixel location in the reference image. Thus, images 20 or 22 must be processed further to generate a region combination gray scale image 7. This image 7 is a gray scale image which has continuous values through each pixel. Image 7 is created by combining all pixel pair values for a surrounding region such as by using a weighted kernel. Thus, in the case of a uniformly weighted kernel, all the signed numbers resulting from the correlation surrounding a given pixel would be added together. One example of kernel size would be a 3×3 kernel, the center of the kernel would be placed on coordinates of the pixel of interest of image 20 or 22 and the correlation values would be multiplied by the corresponding kernel values to form the corresponding value in image 7.

[0015] In making the correlation the Laplacian images are overlapped several times to create a set of correlation images. The various overlaps are created by shifting one image relative to another in a vertical direction as indicated in FIG. 2 or by shifting one image relative to the other image in a horizontal direction as indicated in FIG. 3. The use of a vertical or horizontal shift is dependent on the orientation of the cameras, and how they are spaced apart. In general stereo cameras 10 are only displaced in one direction with their optical axes being separated by some distance x as indicated in FIG. 1. After correlation images have been found for several overlaps I select for each pixel in the non-shifted reference image the overlap image (in the format of image 7) having the highest correlation value as the best match. Then I can use each overlap to determine the position of the object in the field of view using standard stereo processing algorithms and techniques.

[0016] This technique can be used to correlate all types of stereo images. Depending on the subject and the background greater or fewer combinations of pairs of overlapping pixels can be used to obtain a correlation value.

[0017] Although I have described certain present preferred embodiments of my method, the invention is not limited thereto, but may be variously embodied within scope of the following claims.