Method and apparatus for aligning a pair of digital cameras forming a three dimensional image to compensate for a physical misalignment of cameras

[0001] The present invention relates to digital camera systems, and more particularly to a method and apparatus for electronically altering the digital image being produced by one of a pair of digital cameras cooperatively forming a three dimensional image, to thereby account for misalignment between the two cameras.

[0002] Present three dimensional (3D) imaging systems used in remote vision applications require very accurate physical alignment of the pair of cameras being used to form the 3D image. More particularly, very fine pixel-to-pixel coincidence is needed to accurately produce a three dimensional image from the two cameras. The ability to provide an accurate, three dimensional image is especially important in certain applications, such as with military refueling tanker aircraft. With refueling tanker aircraft, a 3D vision system must be capable of providing an accurate and clear 3D image to an operator monitoring the positioning of a boom relative to an aircraft being refueled. This implementation has previously required the two digital cameras to be mounted on a common, highly complex platform and for prealignment of the cameras to be performed in a laboratory environment while the cameras were positioned on the platform before installing the platform on the aircraft. The need to perform this prealignment procedure in a laboratory environment contributes to the time and expense associated with the use of a 3D vision system.

[0003] In view of the foregoing, it will be appreciated then that there is a need for a method for aligning a pair of digital cameras to produce an accurate 3D image. More particularly, there is a need for a method to perform this alignment electronically, particularly aboard an aircraft so that same can be accomplished in real time and without the necessity of removing the digital cameras, and any supporting platform that they may be mounted on, to a laboratory or test facility.

[0004] The present invention is directed to a system and method for aligning a pair of digital cameras to produce an accurate three dimensional (3D) image of a target being viewed by the two cameras. The method generally involves aiming a first digital camera at a target to produce a first image which is comprised of a first plurality of pixels. A second digital camera, spaced apart from the first digital camera, is also aimed at the target to produce a second image comprised of a second plurality of pixels. Using known distances of the target from each of the digital cameras, the positions of the pixels forming each of the first and second images can be analyzed. Appropriate corrections can then be determined electronically which effectively alter one or the other, or both, of the images as needed to produce an accurate, composite, three dimensional image of the target.

[0005] In one preferred method, the analysis of the two images comprises determining an image offset value for each pixel forming at least one of the two images. The image offset value is comprised of a lateral offset value, an elevation offset value, and a roll offset value. These three offset values cooperatively alter one of the first and second images along X, Y and θ (roll) axes to reposition and/or resize the image being electronically altered such that the composite three dimensional image is accurately produced. It will be appreciated that in applications where both cameras can be mechanically mounted in the same platform to eliminate roll shift, the value of θ becomes zero and does not have to be included in computing the image offset value.

[0006] In one preferred implementation, the method is able to alter one or both of the digital images electronically to provide pixel coincidence at the operational distance (boom tip, e.g., in a refueling aircraft application). In this implementation a pixel offset of a dimensioned target, located at a distance less than the operational distance, can be generated. The offset can be adjusted and applied to the operational distance using trigonometric functions.

[0007] A principal advantage of the present invention is that since the digital images produced by each of the two digital cameras are electronically aligned, such alignment can be performed very quickly. It also becomes unnecessary to remove the cameras from their common mounting platform or from an aircraft or other vehicle to which they may be directly mounted on. This eliminates the time and expense associated with previous approaches for aligning digital images that involve the requirement of removing the entire mounting platform on which the cameras are mounted and transporting the platform to a laboratory environment where the alignment can be effected.

[0008] Further areas of applicability of the present invention will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples are intended for purposes of illustration only and are not intended to limited the scope of the invention.

[0013] The following description of the preferred embodiment(s) is merely exemplary in nature and is in no way intended to limit the invention, its application, or uses.

[0014] Referring to FIG. 1, there is shown a digital imaging system 10 in accordance with a preferred embodiment of the present invention for forming a composite, three dimensional digital image. System 10 includes a pair of digital cameras 12 and 14 which each are directed at a common target 16. The cameras 12 and 14 are also mounted on a platform 18 which is in turn supported on a different structure or a platform 20, such as possibly an aircraft, bus or other motor vehicle, ship, or even stationary support surface. Accordingly, while it will be appreciated that the present invention 10 will have particular utility with regard to use on mobile platforms, the invention is not so limited and may be used in connection with fixed (i.e., ground-based) support structures.

[0015] The imaging system 10 further includes an alignment correction circuit 22 which is in bidirectional communication with each of the digital cameras 12 and 14. The alignment correction circuit 22 may be mounted on the mounting platform 18 along with the cameras 12 and 14, or it may be mounted on the platform 20 apart from the platform 18, or even remotely from the platform 20.

[0016] In operation, each of the digital cameras 12 and 14 produces an image, 16a and 16b, respectively of the target 16. The images 16a and 16b are painted in the pixel arrays 12a and 14a of the cameras 12 and 14, respectively, in such a manner that there is “pixel-to-pixel coincidence”. By “pixel-to-pixel coincidence” it is meant that the images must be produced within the two pixel arrays 12a and 14a such that they essentially occupy the same number of pixels. Thus, if one of the images 16a or 16b is shifted in an X direction or a Y direction within its respective pixel array 12a or 14a, the resulting composite three dimensional image will not be clear and unambiguous. It will be appreciated that even very minor positional misalignments of one or the other of the cameras 12 and 14 can produce misalignments that will negatively effect the resulting composite 3D image generated by the system 10.

[0017] Referring briefly to FIG. 2, it will be appreciated that the target 16, when viewed as a digital image, will typically be represented by hundreds, or perhaps thousands, of pixels. The target can be thought of as having a center portion 24 and an outer bound or periphery 26. The center 24 of the target 16 has a distance D2, while the outer boundary or periphery of the image of the target is defined by distance D1. The dimensions D1 and D2 can be used to more accurately derive the distance from the target to the platform 18.

[0018] Referring to FIG. 3, a preferred method implemented by the system 10 of the present invention is illustrated in simplified flow chart form. The method involves capturing a first digital image, as indicated at step 28, from the first digital camera 12. A second digital image frame is captured at step 30 from the second digital camera 14. A pixel set is then selected from each frame which indicates the identical pixel content of the target 16. Then, the X-Y values are determined in step 34. The distance from the lens 12b of the first digital camera 12 to the target 16, as indicated by line 32, is determined at step 38. The distance, represented by line 36, separating the target 16 from the lens 14b of the second digital camera 14 is the same as line 32.

[0019] The alignment correction system 22 uses the known distances represented by lines 32 and 36 in analyzing the images 16a and 16b to determine the correction that needs to be applied to one or the other of the images 16a and 16b. The next step is to compute the offset necessary to be applied to the operational distance (camera to boom) using trigonometric functions in step 44. As described above, the correction is that alteration of the pixels used to form one or the other of the images 16a and 16b (or possibly both images) that is sufficient to produce pixel-to-pixel coincidence of the two images. This step is indicated by reference numeral 40 in FIG. 3.

[0020] At step 42, the alignment correction system 22 determines the appropriate correction offset values in X, Y and θ (roll) (i.e., lateral, elevation and roll) directions that are needed to alter the position of one of the images 16a or 16b to produce pixel-to-pixel coincidence of the two images. It will be appreciated that while the pixel arrays 12a and 14a are two dimensional displays, that reference to a roll offset correction value represents modifications to one of the images 16a or 16b, in terms of the pixels being used to form the image, that effectively rotates the image 16a or 16b in its associated pixel array 12a or 14a, respectively. Thus, one of the images 16a or 16b may not only need to be magnified or reduced in size, but it may also need to be shifted in the X direction or the Y direction, as indicated in FIG. 2. By analyzing the pixels representing the images 16a and 16b, together with the distances represented by lines 32 and 36, alignment correction system 22 can determine the needed corrections to produce accurate pixel-to-pixel coincidence for the two images 16a and 16b, as corrected for the operational target. The determined corrections are then applied to at least one of the images 16a or 16b, at step 46, and the process is repeated if necessary.

[0021] The analysis and manipulation of one or the other of the images 16a and 16b by the alignment correction system 22 can be performed by a suitable software application. The software application essentially mathematically compares the pixels making up the images 16a and 16b and applies suitable modifications to one or the other of the images as needed to produce the pixel-to-pixel coincidence of the displays 12a and 14a.

[0022] The system 10 and method of the present invention provides the important benefit that the offset correction values are calculated and applied electronically in real time to one or the other (or both) of the images 16a or 16b on the ground, when the target is a ground based target. The method can also be used in flight, e.g., with an aircraft where the actual boom tip forms the target 16. Thus, any sudden vibration or movement that would cause misalignment of one or the other of cameras 12 or 14 can be immediately detected and corrected for by the alignment correction system 22. Also importantly, there is no need to remove the platform 18 supporting the cameras 12 and 14 for calibration in the event a misalignment is detected. This saves significant time and money in the maintenance of a 3D vision system.

[0023] While it will be appreciated that the present invention will find particularly utility in connection with military tanker aircraft employing a refueling boom, the system 10 can be used with a wide variety of other mobile platforms such as busses, ships and other land, sea, air and space vehicles where use of a 3D vision system is needed. The present invention is further not limited to mobile platforms, but may also be used in connection with a ground-based structure where a 3D vision system is needed.

[0024] Those skilled in the art can now appreciate from the foregoing description that the broad teachings of the present invention can be implemented in a variety of forms. Therefore, while this invention has been described in connection with particular examples thereof, the true scope of the invention should not be so limited since other modifications will become apparent to the skilled practitioner upon a study of the drawings, specification and following claims.