Method and apparatus to provide improved image quality in a camera

[0001] This patent application claims the benefit under 35 U.S.C. §119(e) of U.S. provisional application No. 60/843,312, filed on Sep. 8, 2006.

[0002] The present invention relates to improved image quality of pictures taken with a camera, and more particularly to using motion information to provide improved image quality.

[0003] One of the historical problems with photographs taken by small handheld cameras is hand motion. Most people do not hold their camera perfectly steady, and often the depression of the activator button itself is sufficient to cause a jiggle. An image taken when the camera is not stable will generally exhibit unwanted characteristics such as blurriness, fuzziness and streaks.

[0004] Methods of measuring the motion of an object, such as accelerometers, are becoming more common. Accelerometers are being integrated into cameras and video recorders to provide motion data. In particular, mobile telephones often include cameras. Due to their light weight and unusual form factor, such mobile phone cameras are particularly likely to have blurry pictures due to the user's motion.

[0005] One prior art method of using motion data to improve image quality includes providing a floating lens or similar apparatus to compensate for the motion. Alternative methods include gyroscopes or other systems providing counter-movement to balance any hand motions accidentally made by a user. However, both of these solutions require large hardware elements, and specialized cameras.

[0006] Another prior art method attempts to correct the image taken after the fact, based on the accelerometer data. However, this is computing intensive, and can be impossible if there is not sufficient data to correct for the problems caused by the motion.

[0007] An improved camera system including an accelerometer to detect motion, and stabilization logic to utilize the accelerometer data to select an image having a minimal motion, when a user provides an indication.

[0016] The method and apparatus described is a camera stabilization system which does not require additional corrective hardware or post-processing. In one embodiment, the system uses motion data, in one embodiment as detected by a three dimensional accelerometer, to take an image which has the minimal motion artifacts such as blurring, fuzziness, or streaks. Instead of counter balancing, the system takes one or more images, and selects an image having the minimum amount of motion. In one embodiment, the system starts taking images prior to the user clicking the shutter release.

[0017] FIG. 1 is a system diagram of one embodiment of a camera system which may be used with the present invention. The camera apparatus 105 includes a camera sensor 140 and a shutter release button 160. In one embodiment, the camera apparatus 105 may be any existing camera apparatus, and does not require hardware modification.

[0018] Camera sensor 140 in one embodiment is used to detect light levels and power levels for the flash. This data, in one embodiment, is communicated as light measurement 150 and flash and power state 150 to the stabilization logic 130. The flash state indicates whether the flash is on or off, and the flash power indicates how much power is available to the flash to illuminate the image. Generally, cameras integrated into other devices such as camera phones or pen cameras have low powered flashes. This means that in low light conditions, the exposure is longer (i.e. the shutter is open for a longer time.) When the shutter is open for longer, ensuring that there is no motion during the entire exposure is even more important.

[0019] In one embodiment, the shutter release button's indication 165 is also communicated to the stabilization logic 130. The stabilization logic provides a level of indirection between the user physically pushing the shutter button and the shutter release control signal 155 sent to the camera sensor 140 to trigger the taking of an actual picture.

[0020] The stabilization logic 130 is coupled to an accelerometer 110, or alternative motion detection system. In one embodiment, the accelerometer 110 is a three dimensional accelerometer, which measures motion along the X, Y, and Z axes. In another embodiment, a two dimensional accelerometer may be used as acceleration along the Z-axis (towards and away from the object) is often negligible compared to the overall distance, and can be compensated for by the focus length. In another embodiment, two two-dimensional accelerometers may be used.

[0021] In one embodiment, the accelerometer 110 is not directly coupled to the stabilization logic 130. Rather, the stabilization logic 130 is coupled to a virtual accelerometer 120. The virtual accelerometer 120 is used in circumstances where more than one application may utilize accelerometer data. For example, in a system which includes motion controls, the accelerometer data may be used by the motion controller as well as the stabilization logic 130. For another example, the accelerometer data may be used to provide step information—i.e. pedometer data. In one embodiment, the accelerometer 110 may be coupled to the device including the camera. For example, the accelerometer 110 may be on a PCMCIA card, or other insertible device which is capable of communicating with the camera logics.

[0022] In one embodiment, the stabilization logic 130 receives all accelerometer data when it is activated. Thus, the virtual accelerometer 120 output to the stabilization logic 130 reflects all of the actual accelerometer 110 data. In one embodiment, the stabilization logic 130 is only activated when the camera application is active. In a standard camera this may be when the camera is on. In a multi-function device such as a camera phone, this may be when the camera application is active.

[0023] FIG. 2 is a block diagram of one embodiment of the stabilization logic. The system receives as its input accelerometer data. As noted above, this data may be received directly, via a virtual accelerometer, or from an external device.

[0024] Stability detection logic 210 is designed to calculate an overall relevant motion of the device. The accelerometer data, for example, may show motions toward and away from the image target. However, if the focus is far away compared to the motions,—for example, focus is on a mountain far away—this motion may be discounted, or removed from the overall motion calculated by stability detection logic. In one embodiment, the goal of stability detection logic 210 is to identify a point when the camera is not moving. However, accelerometers only detect acceleration, not velocity. In one embodiment, stability detection logic 210 determines whether the accelerations are static or tending towards static. In one embodiment, tending towards static means that the current acceleration on each axis is close to zero or is zero. In one embodiment, tending towards static means that the acceleration average for each axis is close to zero or is zero. In one embodiment, the stability detection logic 210 also checks if the jerk or change in acceleration is getting smaller or is small, so tending towards steady state. Generally speaking, this does not handle constant velocity. However, constant velocity is very rare in human motion.

[0025] In one embodiment, once stability detection logic 210 detects that the camera is stable, it triggers buffering logic 220 to take sequential images and store them in buffer 225. The buffer 225 stores the images and the associated accelerometer data. In one embodiment, the accelerometer data is attached to the images as metadata. Thus, when an image is later evaluated, or corrected through other means, the accelerometer data is available. In one embodiment, depending on the time it takes to completely record the image, multiple accelerometer readings may be taken. Thus, the accelerometer data may include multiple readings from the accelerometer. In one embodiment, once the indicator signal is received by buffering logic 220 a preset number of additional images are taken.

[0026] Selection logic 230 takes the images from buffer 225 and selects the best image. The best image, ideally, is near the timing of the indicator and has no or minimal motion, which has no impact on the quality of the image. Thus, each frame taken has a quality rating which is, in one embodiment, a function of proximity to the indicator and motion data. In one embodiment, a lighting rating is also included as a parameter in the quality function.

[0027] In one embodiment, the system further includes a “mode indicator” 260. The mode indicator 260 enables the user to select “still image” or “action image” mode for the picture. In one embodiment, this mode indication is used by the selection logic 230, to determine the relative weight of the proximity to the indicator. For still images, stability is more important than proximity to the indicator (since still images generally do not change rapidly). Still images include images of scenery, portraits, images of nature, etc. For action images, the proximity of the frame to the indicator is important, since the scene is likely to be changing rapidly. Action images include images of people performing in sports activities, animals in motion, thunder storms, and other rapidly changing scenes.

[0028] The selection is then stored in memory 250. In an alternative embodiment, the selection logic 230 enables a user to make a selection from among the buffered images.

[0029] In another embodiment, instead of the stability detection logic triggering the taking of pictures, the indicator triggers image acquisition system 240. The system then takes a series of images, and stores them in the buffer 225, along with the accelerometer data.

[0030] In another embodiment, stability detection logic 210 receives the indication, and waits until the camera is stable before taking a single image, and storing it in memory 250.

[0031] Thus, in one embodiment, not all of these elements are present in any single implementation of the system. One implementation may simply include stability detection logic 210 which receives accelerometer data and indicator data, and memory 250.

[0032] Another implementation may include stability detection logic 210, buffering logic 220, buffer 225, selection logic 230, and memory 250. While yet another implementation may include image acquisition system 240, buffer 225, selection logic 230, and memory 250. Yet another implementation may include all of these elements, and enable the user to select between the various possible configurations. Thus, the user may activate a subset of these elements, based on the user's preferred method of providing better quality images.

[0033] FIG. 3 is an overview flowchart of one embodiment of using the stabilization logic. The process starts at block 310. At block 315, the process determines whether the camera function has been activated. If the camera function is not active, the process ends at block 350. In one embodiment, the process only starts when the camera function is activated. Note that in a stand-alone camera, this occurs when the camera is turned on. In other applications, the user may manually or otherwise select the camera functionality.

[0034] At block 320, the overall accelerometer data is received.

[0035] At block 330, the process utilizes the accelerometer data to select an image, and store it. In one embodiment, accelerometer data associated with the instant an image is taken is stored along with the image. In one embodiment, the accelerometer data may become part of the metadata associated with the image. In one embodiment, the system continues to store accelerometer data while the image is being taken, thus multiple accelerometer readings may be part of the accelerometer data. The accelerometer data may be used to time taking a singular image, or to select an optimum one of a plurality of images. In one embodiment, the accelerometer data (which may include multiple accelerometer readings) is stored as metadata associated with the selected image. The process then ends at block 350.

[0036] FIG. 4 is a more detailed flowchart of using one embodiment of the stabilization logic. The process starts at block 410. In one embodiment, FIG. 4 corresponds to block 330 of FIG. 3.

[0037] At block 425, the process determines whether the camera is being held stably. In general, when users are preparing to take a picture, they frame the picture, and start holding the camera steady. Block 425 is tested until the camera becomes stable. As noted above, in one embodiment this is indicated by the accelerations being static or tending towards static. In one embodiment, if the camera is not stable when the indicator is received (i.e. the user snaps a picture) the picture is taken in the conventional way.

[0038] Once steadiness is detected, at block 430 a picture is taken, and stored in a buffer. While the picture is being taken, the accelerometer readings are being stored as well. In one embodiment, the accelerometer readings are stored as a single batched accelerometer metadata. In one embodiment, the accelerometer data is attached to the sub-portion of the image data which is it associated with. That is, for example, each line of an image may have an accelerometer reading associated with it. This may enable post-processing to correct for image blur based on the accelerometer data, within the image itself. For example, at a very simple level, if one line of the image was scanned while the camera was moving to the right, the image line may be corrected by moving the line to the left, to correct.

[0039] The process then tests whether the indicator has been received. If no indicator has been received—i.e. the user has not clicked to take a picture—the process continues to block 440.

[0040] At block 440, the process determines if the number of pictures in the buffer is larger than a preset maximum number, M. In one embodiment, this maximum may be set by the user. In another embodiment, the maximum may be based on a buffer size. In one embodiment, the maximum may be based on an available RAM memory in the camera system. In one embodiment, the system may instead determine whether the total size of the images has reached a maximum size. Thus, in one embodiment, if the user is taking smaller pictures, more images may be buffered. In one embodiment, the maximum number M is set to two.

[0041] If the number of images in the buffer has not reached M, the process continues to block 430 to take a next picture. If the number of images has reached M, the process continues to block 445. At block 445, the oldest picture in the buffer is deleted. The process then continues to block 430 to take a next picture.

[0042] This system is designed to take multiple pictures in a sub-second timeframe. In one embodiment, the value of M is set such that the next picture can be taken while the oldest picture in the buffer is being deleted (i.e. that the two actions can be performed in parallel), to ensure that the deletion does not slow down the process. In one embodiment, the buffer is a ring buffer, so that older data is overwritten as newer data is added.

[0043] If, at block 435, an indicator was detected, the process continues to block 450.

[0044] At block 450, N more pictures are taken and stored in the buffer. In one embodiment, N is set by the user. In one embodiment, the default value for N is 3. In one embodiment, the values of M and N are chosen to ensure that N+M pictures can be stored in the buffer without difficulty.

[0045] At this point, the picture-taking aspect of the process is complete. There are now N+M (or fewer) pictures in the buffer. Note that each of the pictures has associated accelerometer data, which as noted above may include multiple accelerometer readings.

[0046] At block 455, the process determines whether the picture taken closest to the time when the shutter release was clicked by the user is stable. In one embodiment, this image may have been actually stored in the buffer due to the stability, just before the indicator, as the indicator was received, or shortly thereafter. If so, at block 460 it is selected, since it is likely to be showing the image the user wished to preserve. The remaining images are discarded from the buffer. In another embodiment, the user may be presented with all of the images, and permitted to select from them. In another embodiment, a subset of the images may be selected, and the presented to the user. The process then ends at block 470.

[0047] If the picture immediately after the indicator, or immediately at the indicator, was not stable or does not exist, the process continues to block 465. At block 465, the best picture is selected from the buffer, and the rest are discarded. In one embodiment, the best picture is the most stable picture. In another embodiment, the best picture is calculated based on a combination of stability, image artifacts, and proximity to the receipt of the indicator. In another embodiment, other quality indicators such as lighting may be included to evaluate the quality of the image.

[0048] The selection of the best image may be based on a scoring system. At its simplest, each image is scored using a sequential set of unique scores. For example, for five images, each image is assigned a score from 1 to 5, with 5 being best, i.e. least motion/closest to the indicator. The image with the highest total score is then selected. Alternative mechanisms of scoring may be used. In one embodiment, the picture mode (action v. still) is also utilized in this calculation. In one embodiment, for action shots, the scoring emphasizes proximity to the indicator, while for still images, the scoring focuses on motion. As noted above, in another embodiment, two or more images may be presented to the user and the user may be able to select the preferred image. The process then ends at block 470.

[0049] In one embodiment, prior to the scoring the evaluation first discards all images having a level of motion above a threshold. Then, all images which do not have any in-focus portions, indicating artifacts from motion, are discarded. The remaining images are evaluated as described above.

[0050] In one embodiment, blocks 455 and 460 may be skipped. In one embodiment, instead, the scoring method is used, and the image immediately proximate to the indicator is given an additional point.

[0051] FIG. 5 is a more detailed flowchart of using another embodiment of the stabilization logic. The process of FIG. 5 is similar to that described in FIG. 4. However, instead of starting to buffer images upon receiving a stable signal, the image buffering is initiated in response to receiving the indicator, as shown in block 535.

[0052] Then, N pictures are taken in rapid sequence. In one embodiment, the pictures are taken as fast as the camera and buffer can handle.

[0053] The system then evaluates the timing (proximity to indicator) and motion (accelerometer data) to select the best picture, at block 560. The various techniques described above with respect to FIG. 4 may be used. Alternative methods of evaluating the best picture, including simply selecting the picture with the least motion, may be utilized. The process then ends at block 570.

[0054] FIG. 6 is a more detailed flowchart of using another embodiment of the stabilization logic. The process starts when the camera application is activated. The process monitors to see whether the indicator is received, at block 620.

[0055] When the indicator is received, the process continues to block 630. At block 630, the process determines whether the camera is stable. This is determined based on the motion data. In one embodiment, the motion data may be accelerometer data.

[0056] If the camera is stable, a picture is taken at block 650. The process then ends.

[0057] If the camera is not stable, the process at block 640 determines whether time T has elapsed. Time T, in one embodiment, is the maximum delay permitted by the system. In one embodiment, time T is set to be less than 2 seconds. In one embodiment, time T is adjustable by the user. In one embodiment, time T is set to be less than the maximum shutter delay projected for the system. If time T has not yet elapsed, the process returns to block 630 to attempt to take an image when the camera is stable. If time T has elapsed, the process continues to block 650, to take a picture. The process then ends.

[0058] This process simply attempts to time the taking of the picture until the camera is properly stable. In general, motion may be caused by clicking. Thus, even a minimal delay of a tenth of a second may be sufficient to produce a much better quality image.

[0059] FIG. 7 is a flowchart of one embodiment of the process. The process is similar to that described above with respect to FIG. 4. However, the process of storing images is different. At block 715, the process waits until the camera is stable. At block 720, the image is taken, and the image and its accelerometer data are stored in a buffer. At block 730, the prior image in the buffer, if any, is erased. In one embodiment, “erased” can simply mean overwritten. In one embodiment, “erased” can simply mean that the write pointer is directed at the prior image, such that the next image stored in the buffer will overwrite the prior image.

[0060] At block 740, the process determines whether the stability of the camera is better than the stability associated with the prior stored image. If so, the process returns to block 720, to take an image.

[0061] If the stability is not better than the stability of the prior image, the process continues to block 745. At block 745, the process determines whether the shutter release signal has been received. If not, the process returns to block 740, to continue evaluating the stability of the camera, compared to the stability associated with the already stored image.

[0062] If the shutter release signal has been received, the process continues to block 750. At block 750, the process waits until the camera is stable. At block 755, the image is taken, and the image and its accelerometer data are stored in a buffer.

[0063] At block 760, the process determines whether the current time from shutter release is closer than the time associated with the stored image. That is, the process determines whether it is possible to take a new image that is closer in time to the shutter release than the already stored pre-shutter release image. If not, the process continues to block 780. Otherwise, the process continues to block 770.

[0064] At block 770, the process evaluates the stability of the camera. If the camera is stable, then another image is taken. In one embodiment, this image is stored in the same buffer as the images taken prior to the shutter release. Therefore, the 2nd prior image is replaced. The process continues to block 760, to determine the current time from shutter release.

[0065] At block 780, the images are evaluated, and the best image is selected. The image evaluation may take into account the stability of each image (i.e. the accelerometer readings, and the integral of the accelerometer reading, or second integral of accelerometer), and the proximity of each image to the time of the shutter release. In one embodiment, the remaining images are deleted.

[0066] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.