Lightweight image manipulation for maximizing the capacity of digital photo storage media

[0002] 1. Technical Field

[0003] This invention relates generally to digital photo storage media such as exists in a digital camera and, in particular, to methods for more efficient storage of digital content in such media.

[0004] 2. Description of the Related Art

[0005] Recently, the computer industry has sought to add computer processing and communications capabilities to devices other than what would normally be considered a traditional computer. Such devices are quite varied and include, for example, personal digital assistants (PDAs), smartphones, cellular phones, desktop screen phones, in-vehicle devices, business organizers (e.g., IBM WorkPad™, PalmPilot™, and the like), peripherals (such as printers, fax machines, and the like), handheld or palmtop computing devices, and the like. For convenience, these devices, as a class, are referred to herein as “pervasive computing” clients as they are devices that are designed to be connected to servers in a computer network and used for computing purposes regardless of their location.

[0006] Digital cameras are now being proposed as pervasive computing devices. The recent increase in the popularity of digital photography has created a need for improved storage schemes in such devices. Because digital cameras must store images in either RAM or removable media, each with a finite capacity, it is desirable to maximize the amount of photos that can be stored in the camera. To this end, most digital cameras use a compression scheme, such as JPEG, to reduce the size of the photos.

[0007] While such compression schemes are quite useful, there remains a need to provide enhanced methods for storing digital photos in digital photo storage media. Preferably, such methods would not require special image processing functions or new compression techniques.

[0008] The present invention addresses this problem.

[0009] It is a primary object of this invention to maximize the capacity of digital photo storage media such as exists in a digital camera.

[0010] It is another primary object of this invention to provide a mechanism that uses lightweight image manipulations, such as clipping, inversion, rotation and reflection, to minimize the total space required to store digital pictures on digital storage media, such as disk or RAM cards.

[0011] It is a further object of this invention to maximize digital photo storage media capacity without special image processing functions or new compression techniques.

[0012] Another object of this invention is to augment existing compression techniques (e.g., JPEG) in digital cameras to provide for improved storage of digital photos.

[0013] Still another more general object of this invention is to provide a novel approach to storing a collection of images in a digital camera in a given manner to maximize storage savings with very little overhead.

[0014] Another general object is to provide a digital camera having enhanced photo storage capability.

[0015] These and other objects are achieved in a digital camera that includes a lightweight image processing mechanism according to the present invention The mechanism divides up each picture stored in the camera, preferably into equal size subcomponents. The subcomponents from a given set of the photos are then “quilted” together, preferably into one large composite image. Preferably, the subcomponents are stitched together in such a way that maximizes the camera's existing compression technique. In particular, given subcomponents are selectively transformed (e.g., clipped, inverted, rotated, reflected, and the like), to position the subcomponents as desired. For example, if the camera uses JPEG compression, two predominately blue subcomponents are placed next to each other as JPEG is a compression technique that exploits color redundancies. After the subcomponents are processed, the coordinates of the sectors are indexed and stored so that an individual image can be reconstructed by clipping out its component subcomponents and reassembling the image. With this processing, the more similar the photos in the set, the more compression benefit is achieved.

[0016] Thus, preferably the present invention is implemented in a digital camera having a digital storage media and a compression routine for storing digital photos in a compressed formal in the digital storage media. According to the invention, a set of digital photos are processed into subcomponents, which are then collected into a composite image and then applied to the compression routine. The coordinates of the subcomponents within the composite image are then indexed and stored so that the individual photos can be reconstructed.

[0017] The foregoing has outlined some of the more pertinent objects and features of the present invention. These objects should be construed to be merely illustrative of some of the more prominent features and applications of the invention. Many other beneficial results can be attained by applying the disclosed invention in a different manner or modifying the invention as will be described. Accordingly, other objects and a fuller understanding of the invention may be had by referring to the following Detailed Description of the Preferred Embodiment.

[0018] For a more complete understanding of the present invention and the advantages thereof, reference should be made to the following Detailed Description taken in connection with the accompanying drawings in which:

[0019] FIG. 1 is a simplified block diagram of a digital photo capture and storage device;

[0020] FIG. 2 is a simplified illustration of how a set of individual images are arranged into a composite image according to the present invention;

[0021] FIG. 2A is an illustration showing how an individual image is subdivided into sectors to facilitate processing of the composite image;

[0022] FIG. 3 is a simplified representation of how the composite image is stored and accessed using a file system of a digital photo capture and storage device;

[0023] FIG. 4 is a simplified flowchart of a composite image creation and storage routine; and

[0024] FIG. 5 is a simplified flowchart of a routine for extracting an individual image from the stored composite image.

[0025] FIG. 1 illustrates a simplified block diagram of a digital photo capture and storage device 10, e.g., a digital camera. The device comprises a lens 12, digitizer 14 and charge coupled device (CCD) 16 for capturing image data in a digital format. The image data is typically compressed before storage. To this end, the device 10 includes a compression device 18. In particular, it is well-known that digital cameras include special hardware to handle native compression techniques, for example, an ASIC having JPEG-based compression. JPEG, as is well-known, is a compression technique that takes an image and creates a frequency map of a given color characteristic (e.g., hue, saturation or luminosity). Pixels sharing the characteristic are then represented by a smaller piece of data to reduce storage requirements. Thus, according to known techniques, following compression, the image data is stored in digital storage medium 20, which may be nonvolatile RAM, a disk drive, a floppy drive, a flash RAM card, or the like. The digital storage medium 20 also includes a directory. As can be seen, each compressed image is stored as a separate file. As a result, there is significant overhead required to store a set of such files because each file has file format constraints, e.g., headers and other common structures, that are repeated across each of the individual files.

[0026] Typically, many of the separate images in the digital storage medium have features in common. In particular, most photographers tend to frame subjects in the same way, and often many images in a set of photographs have similar backgrounds (e.g., a blue sky). The present invention is based on the recognition that, in the usual case, there is a high probability of redundancy between the various individual photos of the stored set. Accordingly, the present invention provides a mechanism for using such redundancy to maximize storage of such photos, preferably without altering the device's existing compression scheme. This goal is achieved by arranging a number of images into a single composite image or “mosaic” that takes advantage of the existing compression algorithm's strength.

[0027] FIG. 2 illustrates the inventive approach. In this example, there are four (4) discrete images 22, 24, 26, and 28, each of which show a rugged mountain landscape and a blue sky. According to the invention, the discrete images are stitched or “quilted” together into a single composite image in such a way as to maximize the value of a given compression routine. Thus, in this example, photo 22 and photo 24 have been placed side-by-side given the redundancies in the mountain images and the commonality in the blue sky. Photos 26 and 28 have each been flipped and then arranged on top of the photos 22 and 24 to increase the shared common areas (namely, the blue sky). Photos 26 and 28 also share the somewhat common mountain range images. As a result of the transformation of individual images and the aggregation as shown, the overall composite image (comprising the set of four photos) has more overall redundancy and commonality. Thus, when the composite image is then compressed, a lesser amount of memory is required (to store the compressed composite image) as compared to the amount of memory that would otherwise be required to store separate compressed versions of each of the four photos.

[0028] As is well-known, compression routines such as JPEG exploit commonality, namely, slight variations in color (as opposed to abrupt or drastic changes) or some other image characteristic. JPEG, for example, is based on the fact that humans are much more aware of small changes in brightness (luminance) than small changes in color or large changes in color or brightness. Typically, a JPEG compression starts by converting color data from RGB to a system that identifies the brightness of each pixel. Once the conversion is made, a first data reduction takes place in a process called subsampling. The brightness scale is left alone, while half of the other two scales are eliminated by replacing two neighboring pixels with a single value representing their average. This reduces the entire image to two-thirds of its original size with no noticeable loss in quality. The next steps work on 8 by 8 blocks in the image. Using a direct cosine transform (DCT) and quantization, the changes in brightness and color within each block are identified and rounded off. The amount of rounding increases with the size of the change, thus giving more weight to the smaller changes. This produces a set of values that still describes the image data, although not as accurately as before, and contains many identical values because of the rounding process. All identical values are then compressed via a standard lossless compression technique.

[0029] The present invention exploits the known property that compression routines exploit commonality by arranging a set of individual images into a composite image that takes better advantage of the compression routine than can the individual photos themselves. The present invention thus provides “lightweight” image processing, because it preferably just augments the device's existing compression functionality. The digital camera does not require any significant new chips or added processing capability to achieve the storage-savings provided by the technique. Stated differently, the invention preferably leverages an underlying type of compression rather than creating a new compression type.

[0030] While the present invention has been illustrated in the context of a digital camera, this is not a limitation. The inventive technique is useful in any system or device (preferably at the file system level, as will be seen) in which it is desired to reduce the storage requirements of digital media such as photos. Thus, the invention is useful in any type of pervasive computing client, especially where there are storage constraints and there may not be significant computing power available for new or additional CPU-intensive compression algorithms. More generally, the invention is useful in any photo capture and storage system or device including, without limitation, a digital archive, a laser disk system, a scanner, a digital photo album storage device, and the like.

[0031] Referring now to FIG. 3, the invention is preferably implemented as an abstraction at the file system level of the digital storage medium. Thus, the fact that individual photos are compiled into a “mosaic” before compression and storage need not be readily surfaced to the user. If the inventive technique is built into the device, individual photos are aggregated into the composite as they are taken. For an existing device, the invention may be applied as pictures are extracted from the camera and placed in storage. Thus, in the preferred embodiment, as illustrated in FIG. 3, the device's file system 30 stores the composite image 32 in the digital storage medium 34 together with a lookup table 36. Table 36 contains mapping information necessary to extract or reconstruct the individual images.

[0032] FIG. 4 is a flowchart describing the preferred process for creating and storing the composite image. FIG. 5 is a flowchart described the preferred routine for reconstructing an individual image. In each case, it is preferred that the control functions are carried out through the existing file system as described above.

[0033] Referring now to FIG. 4, the routine begins at step 40 with the processing of an data representing a discrete image to be aggregated into the composite image. If desired, the inventive technique may be performed on an entire image (as illustrated above) or on given components or “sectors” of an image. Subdividing each image into subcomponents or sectors often maximizes redundancies, thereby enhancing the degree of compression achieved by the underlying compression routine. To this end, at step 42, the image is broken into subcomponents (as illustrated in FIG. 2A). The routine then continues at step 44 to “rate” each component of the image being processed according to some given criteria (e.g., hue, saturation, luminosity, content or the like). Preferably, each sector is assigned a number or “rank” based on some given characteristic (e.g., its color and/or content) that is normally exploited by the underlying compression routine. Other factors that may be taken into consideration in generating a ranking may include the picture content, the capability of the processor, the ease or difficulty in reconstructing the sectors, and the like.

[0034] The routine continues at step 46 with the sector being added to a list of components generated from the other digital images being aggregated into the composite image. As step 48, the routine composites all component images together, applying transforms to individual components if necessary as has been previously described. At step 50, the composite image is stored. The routine then continues at step 52 by recording extraction information in the lookup table. This completes the composite image creation and storage routine.

[0035] The actual algorithm for arranging and ranking the subcomponents of an image are implementation-specific. Thus, an implementation for a camera's images using JPEG preferably is different than a digital archive's images using PNG, for example. In general, however, one illustrative method is as follows. The first step is to identify the key components of the underlying compression algorithm that determine the effectiveness of the compression. A subcomponent may be anything from the whole image to some smaller portion of the image, again depending on the specific characteristics of the compression algorithm, i.e. if JPEG works best on 8 by 8 pixel squares, that would be the size of a segment. Next, the segments of the image are analyzed in terms of hue, luminosity, saturation, or whatever other parameters the specific compression algorithm acts upon. With a composite value representing the general character of the subcomponent, a determination is then made as to the appropriate position of the subcomponent in a larger composite grid. As previously described, subcomponents are preferably located in an area of like segments, as described by calculated values. Then, the size and position of the subcomponent within the grid is recorded so that the subcomponent can be retrieved and viewed at a later time.

[0036] As described above, the present invention also provides a means to extract a given image from the composite. Referring now to FIG. 5, a simplified flowchart of an extraction routine is illustrated. The routine begins at step 60 with a request for a given image. At step 62, the image location for the image is found in the lookup table. At step 64, the image data is extracted from the composite image. Prior to step 64, the composite image is decompressed and the routine may crop the composite image to a given area to facilitate processing. The routine then continues at step 66 to apply any necessary transforms. This step may be necessary if a given component of an image was transformed prior to being aggregated into the composite image. At step 68, the image components are reassembled and returned as a single image. This completes the extraction routine processing.

[0037] According to the invention, image or image component transforms may be of several different types. As noted above, the particular transforms used are designed to maximize the operation of the underlying compression routine. In an illustrative embodiment, the transforms include: mirroring, cutting, cropping, inverting, flipping, reversing, rotating and color shifting. Inversion renders the negative of the image. One of ordinary skill in the art will appreciate that other transforms, or combinations of such transforms, may be used as well.

[0038] As noted above, the present invention may be implemented in any pervasive computing client device. A representative pervasive client is x86-, PowerPC®- or RISC-based, includes a runtime operating system such as WindRiver VXWorks™, QSSL WNX4™, or Microsoft Windows CE, and includes a Web browser.

[0039] While the invention has been described in the context of a JPEG-based compression routine, one of ordinary skill will appreciate that this is not a limitation of the invention. Similar advantages are also obtained by applying the invention with other types of compression (e.g., .gif, .png or the like). In particular, other applications use different file formats that contribute to storage overhead when individual files are stored separately. As has been previously described, the present invention substantially reduces this overhead by obviating storage of redundant headers and other data structures as the individual files are aggregated into the composite file.

[0040] The inventive functionality of the lightweight image processing routine preferably is implemented in software executable in a processor, namely, as a set of instructions (program code) in a code module resident in the random access memory of the computer. Until required by the computer, the set of instructions may be stored in another computer memory, for example, in a hard disk drive, or in a removable memory, or downloaded via the Internet or other computer network.

[0041] In addition, although the various methods described are conveniently implemented in a general purpose computer selectively activated or reconfigured by software, one of ordinary skill in the art would also recognize that such methods may be carried out in hardware, in firmware, or in more specialized apparatus constructed to perform the required method steps.

[0042] Having thus described our invention, what we claim as new and desire to secure by letters patent is set forth in the following claims.