Alignment of images for stitching

[0001] The present invention relates to enhanced image capturing methods and apparatus and more particularly relates to the use of fiducial marks in image capturing devices to place fiducial marks in related images to improve and automate the stitching of related images together into a panoramic image.

[0002] Photography is a well known and widely used technique for capturing images for later appreciation and use. Digital photographydigital image capturingis a recent development made possible by lower cost optical to electronic conversion devices such as relatively high resolution charge coupled device (CCD) arrays and similar mechanisms, more powerful and energy efficient microprocessors, and small and low cost memory devices. Advantages to the user of a digital camera, as it is commonly known, are the ability to capture, store, and immediately view a picture, the ability to correct defects in the stored picture either in situ or later using a computer system, the ability to erase and retake a similar picture, and the ease and ability to digitally transmit the picture to others. The immediacy and the correction ability has stimulated others to produce and offer application programs (for example Stitcher 3.0 offered by RealViz S.A.) that merge overlapping images together into a panoramic scene while correcting for lens distortion, color, and brightness across the panoramic sceneso long as the overlap is about 30% or more. Of course, panoramic scenes have also been created using digital movie techniques, but these scenes are viewed over time rather than as an instantaneous presentation.

[0003] Image stitching application programs rely upon the ability to locate the same objects appearing in overlapping scenes in order to provide alignment targets. Scenes that do not contain easily locatable objects in the overlap area are problematic for the stitching algorithms. Thus, a technology that simplifies the relationship between scenes without leaving undesirable vestiges in the final image would greatly improve the speed and quality of stitched images.

[0004] An apparatus for and a method of enabling the stitching of digital images together includes the generation of a fiducial and the projection of the fiducial into a subject scene. The first portion of the subject scene is captured as a first image and includes the projected fiducial. A second portion of the subject scene is captured as a second image which also includes the projected fiducial and overlaps at least that part of the first portion that contains the fiducial.

[0005] FIG. 1 is a representation of an environment which would accommodate the present invention.

[0006] FIG. 2 is a block diagram of a digital camera, fiducial generator, and computer which may employ the present invention.

[0007] FIG. 3A is a diagram of a fiducial generator which may employ the present invention.

[0008] FIG. 3B is a diagram of a power and gating function for the fiducial generator of FIG. 3A.

[0009] FIG. 4 is a flowchart of an image capture process which may be employed in the present invention.

[0010] FIG. 5 is a flowchart of an image stitching process which may be employed in the present invention.

[0011] FIG. 6 is a flowchart of a fiducial removal process which may be employed as part of the process of FIG. 5 in the present invention.

[0012] An enhanced image capturing device, such as a digital camera, coupled to a positionable reference uses fiducial marks placed into a scene by a fiducial mark generator to improve and automate the stitching of related images together into a panoramic image. Such an arrangement is shown in the representation of FIG. 1. A scene 101 is visible to a digital camera 103 in at least two parts, part A and part B as framed by the camera's optics and two user orientations of the camera. The user ultimately wishes to create a panoramic picture of scene part A and scene part B. In order to avoid difficulties experienced by earlier stitching application programs like requiring a minimum scene overlap and the necessity of clearly recognizable objects in the overlapped scenes, a reference mark, a fiducial, is placed into the overlap region of the scene being photographed.

[0013] In a preferred embodiment of the present invention, the user is provided an external fiducial generating device 105, mounted on a stable support, such as a tripod, external to the camera. The fiducial generating device receives its power and control information via a control cable 107 connected to a control port 109 located on the camera body. The fiducial generating device projects one or more marks of predetermined shape and color onto one or more objects in the scene selected by the user. As shown in FIG. 1, one of the fiducials is an X and another of the fiducials is an O. In advanced systems, the fiducials are selected by the user to be a meaningful character or set of characters such as 1, 2, etc., which may provide meaning to its presence in the stored digital representation of the captured scene. In any event, the X fiducial is placed by the user in a position of scene part A which will overlap with scene part B, thus providing a common reference mark that can be used later to stitch the scenes together. The O fiducial is placed in a non-overlapping part of scene B, but can be expected to be used if a third scene part, part C (not shown), is to be included in the panoramic composite. One or more adjustment devices 111 is placed on the fiducial generating device to allow moving a fiducial mark into another scene without moving the fiducial generating device itself.

[0014] A block diagram of a digital camera 103 coupled to the fiducial generator 105 and a separate user computer system 201 is shown in FIG. 2. The selected scene is timed and focused in conventional digital camera fashion by the lenses and shutters of the optical portion 203 upon an electromagnetic to digital converter 205, which in the preferred embodiment is a 16001200 pixel, nominally 100 mm square area CCD substrate. Electronic shutter speed, white balance, focus, pixel scan speed, and other image capture parameters as well as user interface and memory control are performed by an image processor and control 207. Additionally, the image processor and control interfaces to and controls the fiducial generator 105 via the control port 109 and cable 107. The user interfaces include a 5 cm (2 inch) TFT color LCD viewer 209 for scene selection and user controls 211 for activating the shutter and for interacting with the image control and processor. A conventional USB output port 213 is provided to enable interfacing with the user computer system to, inter alia, download stored images from the memory 215. Additional outputs, such as IEEE 1394, Fast IrDA, and composite video may also be provided as enhancements to the output port 213. The memory 215, in the preferred embodiment, is a 32 Mbit CompactFlash removable memory element.

[0015] A diagrammatic cut-away view of the preferred embodiment fiducial generator 105 is depicted in FIG. 3. Contained within a housing 301, which has a type 20 UNC screw thread mounting (not shown) for attachment to a conventional tripod or equivalent stand, are two solid-state class IIIa lasers, 303 and 305, having 1-5 mW of output power at a wavelength of between 640 and 650 nm as preferred illumination sources. (While these lasers provide low power visible red light for ease of alignment and are relatively safe if not pointed directly into the eye, an alternative embodiment can employ a class II (less than 1 mW output power) laser). The lasers are mounted on pivots 307 and 309 respectively and are mechanically coupled to the adjustment device 111 so that the lasers can be aimed at selected targets within the scenes that are to be stitched together. Character masks 311 and 313 are placed at the output of each laser to provide the characteristic fiducial projected by each laser. Power and gating circuitry 315 is coupled to the cable 107 and digital camera 103 to receive electric power and control signals useable to turn the lasers on and off. Alternatively, power is supplied from a source separate from the digital camera. In a simplified version of a fiducial generator, only a single laser is disposed in the housing 301 and it is aimed in a fixed direction.

[0016] In operation and after the digital camera 103 is turned on, a process shown in the flowchart of FIG. 4 is followed in the preferred embodiment. The image focused on the camera CCD is converted to a digital electrical signal at 401. This electrical signal representation is presented to the user on the viewer 209, at 403, for the user to aim and prepare the scene. The image processor and control 207 of the digital camera polls the control port 109 to detect the electrical presence of the fiducial generator 105, at 405. In a preferred embodiment, the illumination sources of the fiducial generator are not maintained in a constant on condition but are turned on for a period of time, for example 10 seconds, to enable the user to direct the fiducial to a desired position within the selected scene. The user, however, may elect to have the fiducial generator remain on constantly, or for advanced systems, may want to assign particular meaning to unique characters generated and projected by the fiducial generator. In any event, the user enters instructions regarding the fiducial generator at 407, and enters instructions for image adjustments such as focus, shutter speed, etc. at 409. When the user is ready and activates the shutter control button, the digital camera accepts the activation, at 411, energizes the shutter, activates the fiducial generator to project the fiducials into the scenes to be captured (if they are not already activated), and stores the electrical signal representative of the image, at 413. The shutter is not the traditional mechanical apparatus normally found in photographic film cameras, but is an electronic switch that, in simple terms, electronically energizes the CCD to accept and convert the light photons from the scene to be captured into electronic pulses. Some digital cameras also employ a mechanical optical blocking shutter for improved performance. The user provides an instruction to the camera that there is to be another image taken that is related to (and stitched with) the last image stored. The digital camera accepts the instruction and decides based upon the instruction, at 415, whether there is to be another related image. If there is to be another related image, an indicator of the relationship is stored, at 417, with the last image stored to link the next image with the last. The user may then proceed to reposition the camera to capture the next image without moving the fiducial generator's positioning (thereby maintaining the position of the projected fiducial between the images) and capture the next image. Multiple fiducial generation and projection enable several images to be captured and stitched together with relative ease. If the next image is not to be related to the last, the user may proceed with normal image capture.

[0017] Once the user desires to stitch the related images together, the user energizes the digital camera, couples the camera to a computer, and activates the recall of the first set of related images, illustrated in the flowchart of FIG. 5. In a preferred embodiment of the present invention, a stitching process is activated in a user's computer, into which the suite of images have been uploaded from the digital camera. Of course, for those cameras having sufficient processing power, the stitching can occur without the mediation of a separate computer. Referring, now, to the flowchart of FIG. 5, it can be seen that the image suite is to be uploaded into the computer, at 501, from the digital camera. Information and characteristics relating to the one or more fiducials used in each captured image is also uploaded, at 503. The images which are to be stitched into a panoramic image are identified by image identifier and a count of the number of images in each potential panoramic image are uploaded and determined, at 505. It is a feature of the present invention that a substantial amount of information is already known about the fiducial(s) (the shape, the color, the fact that a fiducial is found in each of the images identified into a panoramic image, for example) and such information makes the search for fiducial(s) in the images a great deal simpler. The stitching process continues with a search for the first fiducial in the first identified image, at 507. The search can be accomplished by several means, such as that described in U.S. Pat. No. 5,611,033 or similar feature-searching processes, but since the details regarding the shape and color of the fiducial are already known, the search process is basically one of looking for the correct combination of pixels in the stored image. Once this search is concluded successfully, the found fiducial pixel locations are stored with an association to the first image. Assuming the fiducial has been found, at 509, the process moves to the next identified image for the panorama. (An error message and options are presented to the user if the fiducial is not found). The next image is searched in a manner similar to the search for the first image for the substantially identical fiducial that was found in the first image, at 511. The found fiducial pixel locations are stored for this second image. If another image is to be stitched to the second image, determined at 513 from the uploaded fiducial information, the second image is searched for the next fiducial shape/color and that found fiducial pixel locations are also stored in association with the second image. Once all the fiducials are successfully found, at 515, the next image is searched and the process continues, at 517, until all of the images for the given panorama have been searched.

[0018] The computation required for the stitching can continue, at 519, as the images are processed into the panoramic image. Several techniques are known and can be employed in the preferred embodiment to provide a panoramic image having a corrected image perspective, initial images color correction, and the like, once the images are correlated when the fiducials are matched image to image, at 518. Having known fiducial shapes, sizes, and colors greatly simplifies the overlaying of images since the fiducials not only provide an overlaying reference mark, but also image size comparisons (so that the images can be enlarged or reduced for best match), and image rotation information. For example, that technique described in the aforementioned U.S. Pat. No. 5,611,033 or that described in U.S. Pat. No. 5,987,164 or a similar technique can be used to orient the images and modify the size of the images for a satisfactory stitching into a panorama. Once the images are processed and stitched into a panoramic image, the panoramic image is stored, at 521, and a second set of images for another panoramic image may be processed.

[0019] A user would, understandably, not want to have the fiducial(s) remain a part of the panoramic image. Accordingly, the fiducial(s) are removed, in the preferred embodiment, as part of the image processing for stitching 519. A subroutine, shown as a flowchart in FIG. 6, is called from step 519 and the location(s) of the fiducials are recalled, at 601. An aforementioned feature of the present invention is that information about the fiducial(s) is already known. In the present fiducial removal subroutine, a process similar to red-eye removal in digital photographs is practiced. Such processes are described in U.S. Pat. Nos. 6,016,354 and 6,009,209, for example. The difficult task of identifying the pixels for replacement has already been accomplished in the preferred embodiment. The luminance and color of the pixels adjacent to the fiducial pixels are identified and are used to replace the fiducial pixels, at 603, in a manner that causes the fiducial to disappear into the background.

[0020] Thus, an apparatus and method that enables the capture of several scenes and the stitching together of the scenes into a panorama has been shown and described. This apparatus and method simplifies the establishment of a relationship between scenes without leaving undesirable vestiges in the final image.