Method and apparatus for organizing digital images with embedded metadata

[0001] 1. Field of the Invention

[0002] The present invention relates to computer systems that manipulate digital images. More specifically, the present invention relates to a method and an apparatus for automatically organizing digital images using metadata which is embedded in the digital images.

[0003] 2. Related Art

[0004] Digital imaging technology is rapidly improving, and the cost of digital cameras is continuing to decrease. Furthermore, digital cameras are now being embedded in many different devices, such as cell phones, Personal Digital Assistants (PDAs), and portable MP3 players.

[0005] In addition to the improvements in digital imaging technology, online photo sharing networks and online galleries are more prevalent than ever before. With these modern technologies, it is possible to take hundreds of pictures, view them, touch them up, and annotate them before making them available on a public server.

[0006] A significant problem with non-digital photography is not being able to see what the images look like until after the film is developed. With digital imaging, you can typically view the image that has been captured immediately after the photograph is taken. In addition, memory cards let users take many more pictures than traditional film based cameras, and each additional picture incurs little or no additional cost.

[0007] The biggest burden with digital photography is to organize and annotate the digital images. A photographer typically uploads the images and has to review each one to identify attributes, such as locations, events, and people or objects associated with the images.

[0008] Hence, what is needed is a method and an apparatus for organizing digital images without the problems described above.

[0009] One embodiment of the present invention provides a system that facilitates automatically organizing digital images using metadata which is embedded in the digital images. The system operates by receiving a digital image with embedded metadata, wherein the embedded metadata includes Global Positioning System (GPS) data. The system uses the embedded metadata, including the GPS data, to identify additional metadata associated with the digital image. The system then saves the digital image on a storage medium. In addition, the system saves the embedded metadata and the additional metadata on the storage medium in a searchable format.

[0010] In a variation of this embodiment, the additional metadata includes an identifier for a location associated with the GPS data.

[0011] In a variation of this embodiment, the metadata includes a time at which the digital image was acquired, and the additional metadata includes a weather condition for a location associated with the GPS data at the time the digital image was acquired.

[0012] In a variation of this embodiment, using the embedded metadata to identify additional metadata involves querying a web service.

[0013] In a variation of this embodiment, using the embedded metadata to identify additional metadata involves performing a lookup in a Personal Information Manager (PIM) application.

[0014] In a further variation, the system queries a user's calendar information to identify additional metadata including an event associated with the digital image.

[0015] In a further variation, the system queries a user's contact information to identify additional metadata including a location associated with the digital image.

[0016] In a variation of this embodiment, using the embedded metadata to identify additional metadata involves performing a lookup in a custom dictionary.

[0017] In a variation of this embodiment, the system annotates the image with the additional metadata so that the additional metadata can be displayed to a user.

[0018] In a variation of this embodiment, the system uses the metadata to identify digital images that are similar to the digital image. Once similar digital images are identified, the system presents a user with annotations from the similar digital images as possible options to facilitate annotating the digital image.

[0019] In a further variation, presenting the user with annotations from the digital images involves presenting the user with annotations that have not been automatically generated.

[0020] In a variation of this embodiment, the system additionally facilitates browsing a collection of digital images organized by metadata.

[0021] In a variation of this embodiment, the system adds the digital image to a gallery of images that have metadata similar to the metadata from the digital image.

[0022] In a variation of this embodiment, the system uses the metadata to perform a dynamic lookup query to present a user with a collection of digital images.

[0023] In a further variation, performing the dynamic lookup query involves performing a join operation with the metadata.

[0027] TABLE 1 illustrates various possible browsing options in accordance with an embodiment of the present invention.

[0028] TABLE 2 illustrates various possible search queries in accordance with an embodiment of the present invention.

[0029] The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0030] The data structures and code described in this detailed description are typically stored on a computer readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. This includes, but is not limited to, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs) and DVDs (digital versatile discs or digital video discs), and computer instruction signals embodied in a transmission medium (with or without a carrier wave upon which the signals are modulated). For example, the transmission medium may include a communications network, such as the Internet.

Background

[0031] FIG. 1 illustrates the digital imaging process in accordance with an embodiment of the present invention. During this process, digital images, such as digital image 108, are typically associated with metadata, such as the date the image was taken, the time the image was taken, the exposure settings, etc. For example, the metadata can be Exchangeable Image File Format (EXIF) data which is stored in Joint Photographic Experts Group (JPEG) images.

[0032] The addition of digital imaging technology and Global Positioning System (GPS) capabilities to cell phones 106 and Personal Digital Assistants (PDAs) 104 has enabled some additional metadata to be captured when the digital image 108 is taken. One embodiment of the present invention uses this GPS location information, along with other metadata 110 imbedded in the digital image 108 to organize, share, browse and search a collection of digital photographs on behalf of a user 100 or a group of users.

[0033] Currently, when an image is captured by a digital camera 102, the digital camera 102 typically stores metadata 110 in the image header, such as date the image was captured, time of the day the image was captured, camera make and model, exposure settings, ISO settings, etc. In addition to this metadata 110, in one embodiment of the present invention, location-specific information, such as the coordinates of the place where the photograph was taken and the altitude from which the image was taken is also stored in the embedded image metadata 110. Note that this is possible with current technology since it simply requires a GPS device to be embedded in a digital camera 102. Furthermore, cellular phones of today are equipped with both digital cameras and GPS technologies, and they are increasingly used as a primary camera device for many users. The newest generation of cellular phones has better imaging technology than ever before, and consequently, the number of users who use their phone as their primary camera continues to grow.

Organization

[0034] In one embodiment of the present invention, embedded metadata 110 can be used to auto annotate digital image 108 and organize digital image 108 with other digital images in a collection or albums. Note that in one embodiment of the present invention, digital images which are stored inside a database, such as an Oracle™ database, can be organized into collections through the use of database VIEWS over the image data.

[0035] The title or annotation for an individual image, such as digital image 108, can be generated by obtaining the name of the place where the photograph was taken either through a dictionary or by connecting to a service that maps coordinates to place names. For example, a service could map coordinates to terms such as “my office”, “my backyard”, or the Shoreline Amphitheater. Coupled with this time and date information, the system could generate a collection of images that have related times and dates, such as spring 2003 in Niagara Falls or winter 2002 in Alaska.

[0036] Yet another example provides a collection of all Alaska photos, or all photographs in 2003. In this example, the embedded date information is used to generate season/month and the location information.

[0037] Furthermore, the system can tag the photographs with information such as the weather at the time and location that the photograph was taken by. This can be accomplished by connecting to a weather service and obtaining the conditions at a particular time and place. Note that the present invention is not meant to be limited to the examples listed above. Virtually any combination of data from metadata 110 can be used to associate digital image 108 with other metadata that would be useful in annotating and organizing digital image 108.

Browsing

[0038] Digital images with embedded metadata coupled with the powerful indexing and integration features of a database can be used to provide a multi-dimensional access to the digital images. A system that organizes images into a simple file systems allows folder-based viewing of the images. In contrast, using a database to store images allows flexible menu-based viewing. Possible examples of menus are:

[0039]

	TABLE 1
	All Images -> Year -> 2004 -> Months -> October;
	All Images -> Year -> 2004 -> Seasons -> Fall;
	All Images -> Location -> North America -> USA -> Niagara
	Falls;
	All Images -> Time of the Day -> Sunset;
	All Images -> Occasions -> Birthday (Note that this would
	require integration with a personal Information database);
	All Aerial Images -> Location -> North America -> USA ->
	Death Valley.

Searching

[0040] The metadata stored with the images, along with the annotations generated while organizing the images, can be used to search over the image collections on behalf of a given user or a group of users. These searches can be based on keywords, or can be performed through a structured column based form, or a combination of the two. These searches can also be based on information from other sources, such as weather information, location information, occasion information, etc. Examples of some queries are:

[0041]

	TABLE 2
	Niagara Falls;
	All photographs within x miles of Eiffel Tower that have
	count greater than y (Can be used to find preferred tourist
	spots);
	Fall photographs on highway 108 in Stowe, Vermont;
	Sunset photographs on highway 1;
	Britney Spears at Bali;
	All photographs taken in 2001 (from personal collection);
	Photographs taken in rain last year (links weather info to
	date).

[0042] Searching can also be performed on user-provided annotations that are stored along with the images in an object-relational database. For example, a query can find all photographs of Scooby Doo, if the photographs that contain the image of Scooby Doo are annotated with tag “Scooby Doo”.

Sharing and Collaboration

[0043] Embedding location metadata into images facilitates a natural link between photographs taken by different users, which enables users to share annotations photographs taken at same location. These annotations can be additionally ranked by the number of other image attributes that match, such as year, time of day, weather conditions, etc. Users can enforce an additional constraint to obtain only those annotations that were not automatically generated. The system can then rank annotators based on number of people who select their annotations. This can increase the quality of the automatic suggestions that the system provides. Note that in one embodiment of the present invention, users do not need to upload their images to the service if they do not want to share the image. However, they can still share the annotations by simply uploading the image metadata.

Server Technology

[0044] In one embodiment of the present invention, the system can be generically implemented as a feature of an Object Relational Database. Metadata can be automatically populated for images stored in the database to facilitate subsequent retrieval, updates, and miscellaneous application support. This plus user-provided metadata can be used to provide virtual collections of objects in the database.

[0045] In another embodiment of the present invention, the database is modified to recognize and store implicit associations between objects through virtual joins based on metadata collected from various data sources, such as weather databases, GPS databases, and personal information database. In addition, the database can export a protocol through which public content (which is possibly indexed by a hosting service for collaboration and sharing) can be exposed. (This is similar to exposing text content to search engines).

Hosting Technology

[0046] In one embodiment of the present invention, the image search capability can be a part of an enterprise search application, as well as a stand-alone application. As part of an enterprise search application, the image search capability will provide users the capability to search through image files on the basis of typed in keywords. Each search result (image) can be appended with the list of virtual folders (albums) to which this image belongs. This allows the user to navigate to those albums to look for related images.

[0047] As a stand-alone application, the image search capability can provide search capability based on keywords similar to the enterprise search application discussed above. However, in this application, the user can also obtain an album view of the entire image collection. The user will then be able to modify the albums and the image metadata. Note that the image collection which is available to the user depends on the current credentials of the user, as well as the credentials associated with each image/album. For example if a user X is logged into the application and user Y has specified that,

[0048] All the images in my public album are available to everyone,

[0049] All the images in my album G-album are available to members of group G (of which X is a member),

[0050] All the images in my album X-album are available to X.

[0051] Then, user X can view these three albums and the images contained in them. User Y can also specify which images/albums are view only and which can be modified by others. This stand alone application also provides the capability to be able to upload an image and to add keywords to an image, which will help facilitate subsequent searches.
Photo Database with Embedded Metadata

[0052] FIG. 2 illustrates a photo database 200 with metadata 110 in accordance with an embodiment of the present invention. Database 200 can include any type of system for storing data in non-volatile storage. This includes, but is not limited to, systems based upon magnetic, optical, and magneto-optical storage devices, as well as storage devices based on flash memory and/or battery-backed up memory. Database 200 includes images table 202 and attributes table 204. As images, such as digital image 108 are stored in images table 202, the associated metadata, such as metadata 110, is stored in attributes table 204. This facilitates indexing and fast retrieval of images from images table 202 via the associated metadata in attributes table 204.

Process of Storing Digital Images

[0053] FIG. 3 illustrates the process of storing digital images in accordance with an embodiment of the present invention. The system starts when user 100 stores digital image 108 into images table 202 in database 200 (step 302). The system then retrieves embedded metadata 110 from digital image 108 (step 304) and stores metadata 110 in attributes table 204 (step 306).

[0054] The system also uses the metadata 110 to identify additional metadata associated with digital image 108 (step 308). Note that this could include connecting to a web service, querying a database, scanning a flat file, examining a PIM application, etc. For example, the system could lookup the name of a location associated with GPS coordinates in metadata 110. Finally, the system stores the additional metadata into database 200 (step 310). Note that it is not necessary to store the additional metadata in database 200, and the additional metadata can be identified at a later time. However, it may be necessary to store the additional metadata into database 200 to facilitate indexing and faster queries against the data a later time.

[0055] The foregoing descriptions of embodiments of the present invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.