Method for providing image alignment feedback for panorama (composite) images in digital cameras using edge detection

[0001] The present invention relates to digital cameras and methods.

[0002] Newer model digital cameras have included a “panorama” feature, which allows a user of the digital camera to capture multiple adjacent images that are “stitched” together into a single image at a later time. In order for a stitching algorithm to succeed, there needs to be a substantial overlap between adjacent images so that the stitching algorithm can find features common between the two adjacent images.

[0003] The most common existing solutions for providing image alignment feedback to the user consist of one of two techniques. The first technique displays a portion of the previously captured image (usually 33% or 50% of the previous image) adjacent to a reduced-size liveview image. This technique requires the liveview image to be reduced in size, meaning that fewer visual features will be available to the user. This technique is used in Canon digital camera models S30, S40, and S45.

[0004] A second technique is to overlay a portion of the previous image opaquely over a full-screen liveview image. While this approach does not reduce the liveview image, it conceals up to half of the liveview by obstructing a substantial portion of the previous image. The disadvantage of this approach is that the user is unable to see any changes in the image composition that takes place in the portion of the liveview image that is obstructed by the overlaid previous image.

[0005] The present invention provides for digital cameras and methods that improves on existing methods of image alignment for panorama image capture. The present invention identifies those features that likely will be used by a stitching algorithm, and it highlights those features for a user by superimposing them on top of a “liveview” image displayed on a display of the digital camera. An edge detection algorithm (filter) is applied to the previously captured image to identify desired edge features, and the camera selects and displays (overlays) those features of the previous image on top of the subsequent “liveview” image that are relevant to aligning the camera for the subsequent image.

[0006] The various features and advantages of embodiments of the present invention may be more readily understood with reference to the following detailed description taken in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which:

[0007] FIGS. 1a and 1b are rear and front views, respectively, of an exemplary digital camera in accordance with the principles of the present invention; and

[0008] FIG. 2 is a flow diagram that illustrates an exemplary method in accordance with the principles of the present invention.

[0009] Referring to the drawing figures, FIGS. 1a and 1b are rear and front views, respectively, of an exemplary digital camera 10 in accordance with the principles of the present invention. As is shown in FIGS. 1a and 1b, the exemplary digital camera 10 comprises a handgrip section 20 and a body section 30. The handgrip section 20 includes a power button 21 or switch 21 having a lock latch 22, a record button 23, a strap connection 24, and a battery compartment 26 for housing batteries 27. The batteries may be inserted into the battery compartment 26 through an opening adjacent a bottom surface 47 of the digital camera 10.

[0010] As is shown in FIG. 1a, a rear surface 31 of the body section 30 comprises a liquid crystal display (LCD) 32 or viewfinder 45, a rear microphone 33, a joystick pad 34, a zoom control dial 35, a plurality of buttons 36 for setting functions of the camera 10 and an output port 37 for downloading images to a computer, for example. As is shown in FIG. 1b, a zoom lens 41 extends from a front surface 42 of the digital camera 10. A metering element 43 and front microphone 44 are disposed on the front surface 42 of the digital camera 10. A pop-up flash unit 45 is disposed adjacent a top surface 46 of the digital camera 10.

[0011] An image sensor 11 is coupled to processing circuitry 12 (illustrated using dashed lines) are housed within the body section 30, for example. An exemplary embodiment of the processing circuitry 12 comprises a microcontroller (μC) 12 or central processing unit (CPU) 12. The CPU 12 is coupled to a nonvolatile (NV) storage device 14, and a high speed (volatile) storage device 15, such as synchronous dynamic random access memory (SDRAM) 15.

[0012] In the digital camera 10, the processing circuitry 12 (microcontroller (μC) 12 or CPU 12) embodies a processing algorithm 13 that comprises an edge detection algorithm 13 in accordance with the principles of the present invention to provide image alignment feedback for panorama (composite) images. This will be discussed in more detail with reference to FIG. 2.

[0013] An exemplary edge detection algorithm 13 is disclosed in “A Simplified Approach to Image Processing” by Randy Crane, ISBN: 0-13-226416-1. This Laplacian convolution provides a fast and efficient method for identifying edges within an image.

[0014] FIG. 2 is a flow diagram that illustrates an exemplary method 60 in accordance with the principles of the present invention. The exemplary method 60 comprises the following steps.

[0015] A user captures 61 (stores 61) an image that is to be used as part of a panorama image sequence. The captured image is processed 62 using the edge detection filter 13.

[0016] The camera applies 63 the edge detection filter to that portion of the captured image that will overlap with the next image of the panorama sequence. For instance, if the user is taking a sequence of pictures for a panorama composite, traversing from left to right, the camera can apply the edge detection filter to the right half of each image, since the right half of an image will overlap with the left half of the next image.

[0017] Once the camera has applied the edge detection filter, it computes 64 a histogram of the resultant values in order to identify a threshold value. The threshold value is selected 65 such that a small portion of the total number of image pixels are considered “edges”. For instance, 10-15% of the total pixels may be designated edges. By using the histogramming function, the camera 10 can accommodate either very busy images (those with many edges) or very bland images (those with few edges) without cluttering the liveview image.

[0018] Once the edges are identified and the threshold has been applied, the camera extracts 66 the image information (pixels) near the edges from the previously taken image, and superimposes 67 only those pixels over the liveview image. The definition of “near” in this case is dependent on display size, but a radius of 2-3 pixels is considered reasonable.

[0019] By applying the method of identifying edge features and superimposing only those elements of the previous image that are near those edges over the current liveview image, the user is provided a full-screen display that is minimally obscured with prior image information. Only those elements of the previous image that are important for alignment obscure the liveview image.

[0020] This approach eliminates the disadvantage of using a reduced-size image as found in Canon digital cameras, for example. Furthermore, it eliminates the disadvantage of overlaying a large portion of the previous image (as much as 50% of it) over a full-screen liveview, interfering with the composition of the current image.

[0021] This approach also allows a number of adjustments to be made, by either changing 68 the percentage of pixels that will be identified as “edges”, or by changing 69 the radius (or extent) around the edges that are sampled to display on the liveview image. By altering the threshold value 65, the user of this technique can alter the number of edges that the user of the camera will see superimposed on the liveview image. A higher threshold value increases the likelihood of the user seeing false edges, while a lower threshold value decreases the number of edges displayed. The false edges that can be detected with a higher threshold may provide more alignment features, but it also obscures more of the current liveview image. Fewer edges decrease the clutter obscuring the current liveview image, allowing for better composition of the current image, but provides fewer features for alignment. Altering the radius of pixels that are sampled around edges likewise alters the level of obscuration of the liveview image. A larger radius includes more features around the edges to use for alignment, but it also obscures more of the liveview image. Conversely, a smaller radius reduces the visibility of alignment features.

[0022] This method is easily identifiable in application, since it essentially overlays only edge features from the previous image over the current liveview image. By obscuring the current liveview as little as possible, the camera 10 provides the user the best possible solution to the problem of aligning adjacent images during a panorama sequence.

[0023] Thus, digital cameras and methods that use edge detection for providing image alignment feedback for panorama (composite) images have been disclosed. It is to be understood that the above-described embodiments are merely illustrative of some of the many specific embodiments that represent applications of the principles of the present invention. Clearly, numerous and other arrangements can be readily devised by those skilled in the art without departing from the scope of the invention.