Method and apparatus for automatic photograph annotation

[0001] 1. Field of the Invention

[0002] This invention relates to the field of photograph annotation, and, more specifically, to a system, method, and apparatus for automatically annotating digital photographs based upon identification information stored within an information tag device located near a site at which a photograph was taken.

[0003] 2. Background of the Invention

[0004] There are digital cameras and personal digital assistants (“PDAs”) that allow a user to add annotations to photographs. For example, some digital cameras allow a user to type in an annotation/caption for a photograph immediately after the photograph has been taken. However, when a user takes many pictures, it is very burdensome for the user to manually write/type a caption for each photograph.

[0005] Some digital cameras utilize a global positioning system (“GPS”). GPS can be used to determine the latitude and longitude coordinates of the location where a photograph was taken. Some cameras can annotate photographs with GPS coordinates. However, if several photographs are taken at different sites at around the same latitude/longitude coordinates, the user might not remember which photographs were taken at which site. Also, GPS does not work at all indoors—it requires an unobstructed view of the sky. GPS can sometimes also take a relatively long time to pinpoint the user's location.

[0014] The preferred embodiment of the present invention is a system for automatically annotating photographs taken by a digital camera. The user may take a photograph of anything near an information tag device, the information tag device being a small device for holding identification information concerning the location of the information tag device or of a nearby scene of interest, such as latitude/longitude coordinates or the like. The camera acquires the identification information from the information tag device when the user controls the camera to “read”, or acquire the information from, the information tag device through the use of a tag-reader. The information stored within the information tag device is utilized to acquire annotation information from an annotation provider over a network, such as the Internet, for example.

[0015] FIG. 1 illustrates a user 100 taking a photograph of a scenic place 115 near an information tag device 120 with a camera 105 according to an embodiment of the invention. In the preferred embodiment, unique identification information is stored within the information tag device 120. The camera 105 has a tag-reader 110, for reading the identification information stored in and transmitted by the information tag device 120. In the preferred embodiment, the tag-reader 110 is physically a part of the camera 105. In other embodiments, the tag-reader 110 and the camera 105 may be separate devices. In other embodiments, a camera 105 for taking still photographs need not be used. For example, a video camera, an audio recorder, or any other suitable device may be used.

[0016] In the preferred embodiment, the information tag device 120 is a Radio Frequency Identification (RFID) tag, and the tag-reader 110 is an RFID tag reader. RFID is a data collection technology that utilizes a electronic tag device 120 to store identification data and a wireless transmission method to capture data from the tag device 120. A small RFID tag device 120 can be a small semiconductor chip in which 2 kilobits of data are stored, for example. The semiconductor chip can be a memory such as a random access memory (RAM), an electrically erasable programmable read only memory (EEPROM), a flash memory, or the like.

[0017] The preferred embodiment utilizes a “passive” RFID technology, in which the RFID tag-reader 110 transmits a high frequency signal that is received by the RFID information tag device 120. A typical frequency of the high frequency signal is 13 MHz, an international standard.

[0018] The RFID information tag device 120 is a small semi-conductor attached to a coil antenna. The RFID information tag device 120 is typically small, around one or two inches square. The RFID information tag device 120 can usually be affixed to any non-metallic material. When the RFID tag-reader 110 transmits a high-frequency signal to the information tag device 120, the signal provides energy to “turn on” the information tag device 120. In other words, the high-frequency signal from the RFID tag-reader 110 is used to power the RFID information tag device 120. Once the RFID information tag device 120 has power, it transmits a signal to the tag-reader 110. The identification information stored in its memory is transmitted in the signal. For an RFID information tag device 120 that is an inch or two in diameter, the RFID tag-reader 110 must be close in proximity to the RFID information tag device 120 during data transmission, because the read range is just a couple of inches.

[0019] In other embodiments, an “active” RFID information tag device 120 is utilized. An active RFID information tag device 120 has its own power source, such as a battery. An active information tag device 120 does not require a high frequency signal from an RFID tag-reader 110 to supply power for transmitting data. The active information tag device 120 has more transmitting power than the passive information tag device 120, and can be read from a distance of a few meters, for example. In one embodiment, an active information tag device 120 may continuously transmit a signal containing the identification information stored in the information tag device 120. In another embodiment, the active information tag device 120 may transmit a signal containing the identification information stored in the information tag device 120 only after the RFID tag-reader 110 transmits a signal to the information tag device 120, the signal causing the information tag device 120 to begin data transmission.

[0020] In an RFID embodiment, the identification information stored in the information tag device 120 may be the positional location of the information tag device 120. This positional data may be the latitude and longitude coordinates of the information tag device 120. In other embodiments, the data stored in the memory in the information tag device 120 may be American Standard Code for Information Interchange (ASCII) characters corresponding to a description or location of the site at which the information tag device 120 has been placed. For example, a information tag device 120 located near Old Faithful in Yellowstone National Park could contain the ASCII characters “OLD FAITHFUL”.

[0021] In other embodiments, technology other than RFID may be employed. For example, in an alternative embodiment, the information tag device 120 may be a barcode, and the tag-reader 110 could be a barcode reader. The barcode reader may be a laser diode scanner, which emits laser beams at a barcode and extracts information from the bar code based upon reflected laser beams.

[0022] Alternatively, the information tag device 120 may be a digital watermark. An operator or owner of a museum or a poster in the museum may add a digital watermark to the poster The digital watermark may be a non-random pattern embedded within, or superimposed on top of, the poster. The non-random pattern contains information identifying its location. Such a non-random noise pattern is not noticeable to the user's eye. However, a photograph is taken of a poster having such a non-random pattern, would contain an image of the poster with the non-random pattern. When the image is processed, the non-random pattern may be extracted from the photograph and identification information recorded therein may be determined.

[0023] Another embodiment utilizes a Global Positioning System (GPS). In a GPS system, an information tag device 120 is not required. Instead, through the use of the GPS system, the tag-reader 110 pinpoints the location of the user 100 when a photograph is taken.

[0024] An additional embodiment may read an information tag device 120 via infrared signals. The information tag device 120 in an infrared embodiment may actively transmit a signal that is readable by a tag-reader 110 in the camera 105.

[0025] FIG. 2A illustrates a process of the user 100 taking photographs and then reading an information tag device 120 according to an embodiment of the invention. First, the user 100 brings 200 a camera 105 to a place having an information tag device 120. Next, the user 100 takes 205 a photograph with the camera 105. The camera 105 saves 210 the photograph in a memory, preferably along with the time that the photograph was taken. In the preferred embodiment, a real-time clock is stored within the camera 105. In other embodiments, a stopwatch-type device or program may be used to calculate a time difference between when the last photograph was taken and when the current photograph was taken. The user 100 may then take 215 another photograph, if desired. The user 100 may continue taking photographs until the user 100 desires to move to a new location. If additional photographs are taken, the same process is repeated, with respect to the taking and saving of photographs.

[0026] When the user 100 has finished taking photographs of the scene 115, the user 220 may “read” the information tag device 120 with the tag-reader 110. After the information tag device 120 has been read, the camera 105 saves 225 the information from the information tag device 120 in a memory in the camera 105, along with the time of day that the information tag device 120 was read.

[0027] FIG. 2B illustrates the process of the user 100 reading an information tag device 120 and then taking photographs according to an embodiment of the invention. The process described in FIG. 2B differs from that shown in FIG. 2A only in that in FIG. 2B, the user 100 first reads 220 the information tag device 120 before taking 205 any photographs.

[0028] In an additional embodiment, the user 100 may take 205 pictures, read 220 the identification tag device 120, and then continue taking more pictures 205 of the same scenery.

[0029] FIG. 3 illustrates a general overview of digital photographs being transferred from the camera 105 and acquiring annotations according to an embodiment of the invention. As shown, the camera 105 is connected to a computer 300. A program executed by the computer 300 is preferably used to determine which photographs were taken near which information tag devices 120.

[0030] The computer 300 may contact a database of caption providers 305. In the preferred embodiment, the information read from the information tag device 120 is used to locate a provider of captions for that information tag device 120. When an information tag device 120 is read, a unique number, preferably stored therein, is saved in memory of the camera 105. When the camera 105 is connected to the computer 300, this number is transmitted to the database of caption providers 305, which then returns an Internet website address, or the like, of a caption provider for the particular information tag device 120. A program may be executed by the computer 300 to automatically open a browser pointed to the caption provider's website 310, and the user 100 may acquire the captions for the photographs (to be further discussed below).

[0031] FIG. 4 illustrates a timeline showing when photographs were taken and when two information tag devices 120 were read according to an embodiment of the invention. X and W denote the time at which two separate information tag devices 120 were read. Times at which photographs were taken are represented by “v”, “y” and “z”. In an embodiment where photographs were taken over an 11-minute time span, a program executed by the computer 300 determines which photographs were taken near each information tag device 120. In a program designed so that all photographs taken within 3 minutes of an information tag device 120 being read, for example, are associated with that information tag device 120, the photographs represented by “y” taken between the 6th and 10 minute are associated with information tag device 120 W. The photographs represented by “z” taken between the 10th and 11th minutes are not associated with either information tag device 120 X or W, because they were taken more than 3 minutes after X and W were read. In alternative embodiment, another suitable timing parameter may be utilized. However, any other suitable method of associating photographs with tag devices 120 may be utilized.

[0032] In this example, information tag devices 120 X and W were read less than 6 minutes apart, and some photographs were taken within 3 minutes of each information tag device 120 being read. In such instances, the program may be configured so that the photographs are associated with the information tag device 120 read within the smallest time interval. In this case, the photographs represented by “y” taken between the 5th and the 7th minute are associated with information tag device 120 W because they were taken closer in time to W being read than X being read. Similarly, the photographs represented by “v” taken between the 3rd and the 5th minute are associated with information tag device 120 X because they were taken closer in time to X being read than W being read. Also, the photographs represented by “v” taken during the 1st minute the 3rd minute are associated with information tag device 120 X because they were taken within 3 minutes of X being read.

[0033] FIG. 5 illustrates a second timeline showing when photographs were taken and when two information tag devices 120 were read according to another embodiment of the invention. In this embodiment, the program does not associate photographs with information tag devices 120 based upon a predetermined time threshold. Instead, the program looks for “clusters” of photographs being taken and assumes that all of the photographs in a cluster were taken near the same information tag device 120. The program looks for clusters of photographs taken around the time that an information tag device 120 is read. As shown in FIG. 5, the photographs represented by “a” are associated with information tag device 120 B. The photographs represented by “d” are associated with information tag device 120 E. The photographs represented by “c” are not associated with information tag device 120 B or E, since they are not located in a cluster. A “cluster” is generally a group of photographs taken around the time that an information tag device 120 is read. Clusters are determined by the amount of time elapsing between an information tag 120 being read and photographs being taken. The amount of time elapsing between photographs being taken is also considered when locating each cluster of photographs.

[0034] As shown in FIG. 5, the photographs represented by “c” were taken closer in time to information tag device 120 B being read than the time between some of the photographs represented by “d” (specifically, those taken after the 10th minute) being read and information tag device 120 E being read. However, all of the photographs represented by “d” are associated with information tag device 120 E because they are clearly clustered around that information tag device E 120. Conversely, the photographs represented by “c” are not associated with information tag devices 120 B or E because there was a gap of time between when those photographs were taken and when the “a” and “d” photographs were taken. In other words, the photographs represented by “c” are not clustered around either information tag devices 120 B or E.

[0035] FIG. 6 illustrates a process by which digital photographs receive annotations according to an embodiment of the invention. First, the user 100 uploads 600 the photograph information to the computer 300 (as discussed above with respect to FIG. 3). Next, the program associates 605 photographs with particular information tag devices 120. The program, or another program, contacts 610 the database of caption providers 305. The database of caption providers supplies 615 the caption provider's 310 network address or Internet address to the program. The program then contacts 620 the caption provider's Internet website. For example, at the caption provider's 310 Internet website, the user 100 is provided the option of downloading 625 captions from the caption provider 310 relating to an information tag device 120. After the user 100 chooses a caption, the program annotates 630 the photographs with the selected captions.

[0036] In the preferred embodiment, the caption provider 310 provides the user 100 with at least two caption choices. The first choice is to get a free basic caption listing the name of the place where the photograph was taken. In such an instance, the provider could put advertising underneath the caption of the photograph. The second choice is for the user to pay a fee, such as a flat $5 fee for 100 captions, for example. The user 100 would then receive detailed captions, with information similar to what might be found in a guidebook. Also, the user 100 would not have any advertising placed within the caption.

[0037] FIG. 7 illustrates a digital photograph to which two different annotations have been added according to an embodiment of the invention. First, the user 100 takes a photograph 700 of a house, for example. Next, the user 100 is given the choice of the free basic caption or the more detailed caption that costs money. If the user chooses the free caption, the annotated photograph 710 lists the name of the photographed scene plus advertising. In this example, the caption is “John Smith's house.” The advertising is “Visit http://www.caption_provider.com”.

[0038] Alternatively, the user 100 may choose to pay for the caption. The annotated photograph 715 reads:

• • [0039] John Smith's house.
  • [0040] Colonial style. Built in 1902
  • [0041] Located at:
  • [0042] 1234 West Kissel Boulevard
  • [0043] Springfield, Mass. 12345
  • [0044] John Smith built this house from oak trees growing on his Estate over 100 years ago.

[0045] In other embodiments, more than two caption options may be presented to the user 100. Also, instead of being automatically directed to one caption provider, the user 100 may be given the option of choosing between caption providers for a particular information tag device 120. In other embodiments, the caption information may be contained totally within the information tag device 120. In such a case, it is not necessary to access the database of caption providers 305 and choose from options at the caption provider's 310 Internet website. Alternatively, the network address or Internet website address of the caption provider 310 may be stored within the information tag device 120, rather than simply a number that has to be matched with the caption provider's network address or Internet website at the database of caption providers 305.

[0046] The description above has mainly described embodiments in which the source of data is an RFID tag and the tag reader 120 is an RFID tag reader. However, any of the following technologies are also suitable:

[0047]

	Source of data	Data reader
	Plurality of GPS satellites	GPS receiver
	Infrared transmitter	Infrared receiver
	RFID tag	RFID tag reader
	Barcode tag	Barcode reader
	A printed image having a	Imaging device that
	digital watermark	records the printed image,
		coupled with digital
		watermark extractor
	Radio transmitter	Radio receiver

[0048] While the description above refers to particular embodiments of the present invention, it will be understood that many modifications may be made without departing from the spirit thereof. The accompanying claims are intended to cover such modifications as would fall within the true scope and spirit of the present invention. The presently disclosed embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims, rather than the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.