System and method for acquisition of related graphical material in a digital graphics album

[0002] The invention relates generally to the field of digital graphics albums, and, more particularly, to the searching, acquisition, and insertion of related digital images and video clips into a digital graphics album.

[0003] With the advent of digital imaging in the marketplace, and the broad acceptance of digital cameras, as well as broad utilization of digital images in Internet applications, the volume of commercial and consumer held digital image files has increased dramatically in recent years. Much like the problem of a closet full of shoe boxes of randomly filed photographic prints, in the digital image and graphics environment, there are computer “shoe boxes” full of randomly stored digital graphic images. This leads to problems in locating and categorizing digital graphics files.

[0004] A traditional photo album is a place to store photographic prints. Pictures were placed in the album and often times notes and annotations were added to identify pertinent information about the pictures. In recent times, the electronic photo album has been developed to store digital graphic image files much in the same way as the traditional photo album. Eastman Kodak Company has developed digital graphic album applications, such as the Kodak Picture Page software, which allow a user to organize digital graphics images into album pages. Typically, users organize images by dates and times, places, subject and so forth. Such software allows the user to annotate the images by entering text, or other data, describing the image.

[0005] Traditionally, picture albums contained photographs taken by a user, and perhaps photographs given to them by others. In the digital environment, however, the sources of digital graphic materials is much more diverse. Of course, digital graphic materials may come from a user's digital camera. They may also come from scanned photographic prints that are converted to digital format. They may also be created electronically, such as with graphic software or photo editing software. In addition, there are private and commercial sources of graphical images. These include stock photo agencies, and shared photo resources that provide the sharing of images over networks, such as the Internet. User can now access digital graphical images on the Internet and download them into their personal computer. These images can be placed in a digital graphic album and be made an integral part of the album.

[0006] As a user builds a digital graphic album, there are many choices as to how the images will be organized and annotated. Naturally, digital graphic album software applications allow the user to do this manually. But because of the power of computers and software, software suppliers have added features which make organization of images in digital graphic albums more automated, easier and more flexible. In addition, the kinds of things that can be stored in a digital graphics album has increased. For example, video clips can be placed in the album as well as still images, computer generated graphics, and other digital materials. In the case of a video image, typically a key frame is selected for static display, identifying the video. When a user desires to watch the video, the key frame is selected and this causes the software application to play the video clip.

[0007] Modern camera systems have evolved and some now provide a means of generating annotation data for digital graphic images. Cameras may have a built in clock which time stamps the images. Some allow entry of textual data that can be associated with the digital images. Some even include a global position systems (GPS) receiver which can mark images with the geographic location of the camera at the time the image is exposed. Some allow for voice annotation. All of these kinds of information can be fed to the digital graphics albuming application to be used to annotate the digital graphics materials.

[0008] When the digital graphics materials are added to the digital graphics albuming applications, users can manually add annotation data to the graphics materials. These, combined with the aforementioned automatic annotation materials allow the final, arranged, album to be more rich and full in its context and presentation, which prompts a viewer's memory and greatly expands the enjoyment of viewing such an album. Of course, the arrangement of such a wealth of information can be cumbersome and time consuming. Digital graphics albuming application suppliers have responded by providing features which automatically arrange digital graphics materials within such albums. Kodak Picture Page software is an example of this. The arrangement decisions can be based on the classical indicia, such as time, place, event, and so forth. Much of this information is derived from the annotation materials associated with particular graphical materials. After the graphical material is preliminarily arranged by the software application, the user can make modifications to suit personal tastes.

[0009] An aspect of the subsequent arrangements that a user may make to a photo album is that the user may desire to add additional images to complete the album. As was discussed earlier, the sources are many and varied. This presents a problem to the user because the user may know what kind of images are desired, but not know where to obtain such images. For example, suppose a user has returned from a vacation in France and has a collection of images and videos from the vacation. These are placed in the digital graphics album, annotated and arranged. Upon review, the user realizes that there are several images of the user in the vicinity of the Eiffel Tower, but no images of the Tower itself. Or perhaps the user knows that during the vacation, a major news story broke about France, and the users desires a video clip for the album. Through some amount of search, the user may find such digital graphics materials, but such searching is cumbersome and time consuming.

[0010] Consequently, a need exists in the art for an automatic way of identifying, searching and selecting digital graphical materials for use in supplementing digital graphics albums.

[0011] The need in the art is addressed by the apparatus and methods of the present invention. In an illustrative embodiment of the present invention, a method of adding graphical material to a digital graphics album is disclosed. The method includes specifying reference material in a digital graphics album and extracting annotation data from said reference material. Then, processing the extracted annotation data by a natural language processor to produce search keywords. User directive data is then received and processed by the natural language processor to produce additional keywords. Both the keywords and additional keywords are prioritized followed by querying a graphical material database through a network connection in accordance with the keywords. Then, receiving from the database at least one resultant graphical material and selecting one or more of the resultant graphical material for insertion into the digital graphics album. However, if none of the resultant graphical materials is selected, specifying at least one reference graphic material indicative of a desired search result and processing the reference graphical material to produce search criteria that are image content descriptors. Using the image content descriptors, querying an image content database through a network connection, and receiving from the image content database at least one resultant image. Having received the resultant image or images, selecting at least one of the resultant images, and inserting the selected resultant image in the digital graphics album.

[0012] In addition to the method described above, an illustrative apparatus of the present invention is disclosed. The apparatus is a digital graphics albuming system for searching a graphical material database, and the apparatus includes a display, an input device, a network interface, a memory, and a processor coupled to the display, the input device, the network interface and the memory. The processor, which is inherently a general purpose machine is programmed to implement the forgoing function of the present invention, and, in one illustrative embodiment, the processor is operable to store a digital graphics album in the memory, and the processor is operable to recall a portion of the digital graphics album and display the portion of the digital graphics album on the display. The processor is operable to receive a selection of reference material from the input device, and the processor is operable to process this reference material to produce search criteria. The processor is operable to couple the search criteria to the network interface and access a graphical material database there through, and thereby enabling a search that returns at least one resultant graphical material. The processor is operable to receive at least one resultant graphical material from the network interface, and the processor is operable to display the resultant graphical material on the display and to receive a selection input specifying at least one of the resultant graphical materials from the input device. The processor is operable to store the selected resultant graphical material in the memory as a part of the digital graphics album.

[0013] These and other aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from a review of the following detailed description of the preferred embodiments and appended claims, and by reference to the accompanying drawings.

[0014] FIG. 1 is a diagram of the apparatus of an illustrative embodiment of the present invention.

[0015] FIG. 2 is a functional block diagram of the software applications in an illustrative embodiment of the present invention.

[0016] FIG. 3 is a diagram of the display when reference material is selected in the digital graphics album.

[0017] FIG. 4 is a diagram of the display when a search is selected in an illustrative embodiment of the present invention.

[0018] FIG. 5 is a diagram of the display requesting user directives for a search in an illustrative embodiment of the present invention.

[0019] FIG. 6 is a diagram of the display when resultant graphical material are displayed in an illustrative embodiment of the present invention.

[0020] FIG. 7 is a diagram of the display of a user menu in an illustrative embodiment of the present invention.

[0021] FIG. 8 is a diagram of the display requesting an refined search in an illustrative embodiment of the present invention.

[0022] FIG. 9 is a page one of three of a flow diagram of an illustrative embodiment of the present invention.

[0023] FIG. 10 is a page two of three of a flow diagram of an illustrative embodiment of the present invention.

[0024] FIG. 11 is a page three of three of a flow diagram of an illustrative embodiment of the present invention.

[0025] While the present invention is described herein with reference to illustrative embodiments for particular applications, it should be understood that the invention is not limited thereto. Those having ordinary skill in the art and access to the teachings provided herein will recognize additional modifications, applications, and embodiments within the scope thereof and additional fields in which the present invention would be of significant utility.

[0026] Reference is directed to FIG. 1 which is a diagram of the apparatus of an illustrative embodiment of the present invention. The apparatus and methods utilize a computer 2 that has an input device 3, which is typically a keyboard, but could be other input devices such as a touch screen, character recognition system, mouse, track ball, touch pad, or other human interface device or peripheral. The input device 3 is coupled to processing unit 4 which includes a processor, memory, power supply, input/output circuits, mass storage devices and other circuits and devices typically found in a computer. The processing unit may be a personal computer, such as an IBM PC compatible personal computer. The processing unit 4 is coupled to a display 5 which serves as a video display for visual user interface to the user. Any of the commonly used computer visual display device may be utilized, including, but not limited to, cathode ray tubes, matrix displays, LCD displays, TFT displays, and so forth.

[0027] The processing unit 4 is also coupled to a network interface 6. The network interface 6 is here illustrated as being outside of the processing unit 4, but could be located inside the processing unit 4, as well. The network interface can be any device, or even a simple conductive circuit, to interface the processing unit 4 to an external network 7 or device. This illustrative embodiment illustrates the Internet 7 as the external network, however, the networks utilized could be a private network, an intranet, the Internet, a commercial network, or other network which hosts a database of graphical data. Respecting the network interface device 6, this could be a conventional dial-up modem, an ADSL model, an ISDN interface, a cable modem, direct hardwire, a radio modem, an optical modem or any other device suitable for interconnecting the processing unit 4 to an external network 7, as described previously.

[0028] While not a part of this illustrative embodiment, the present invention relies upon access to a graphical material database 8 as the source of graphical material used to supplement the digital graphics album. The requirements of such a database 8 will be more fully discussed hereinafter.

[0029] References is directed to FIG. 2 which is a functional block diagram of the software applications in an illustrative embodiment of the present invention. The digital albuming application 10, as was discussed earlier runs on the processor 4. In addition, the digital graphic album search application 12 runs on the same processor 4. In this illustrative embodiment, the two are shown as running on the same processor 18. It is the image search application 12, in this illustrative embodiment, that executes a portion of the claimed methods in the present invention. The image search application communicates to the external graphical material databases. In the first example, it is a digital image database 14 and/or a digital video database 16. Such a database comprises a number of digital graphical materials that are accessible by a search function. Typically, the database is a relational database indexed by a plurality of indices. The conventional approach to search such a database is to provide one or more prioritized keywords. The database responds to such a request with a search result that lists a number of hits. In the case of a graphical material database, these are called resultant graphical materials.

[0030] It is understood by those skilled in the art that databases such as 14 and 16 may use more sophisticated indexing strategies and that any such database would be applicable to the present invention. For example, the graphical images may be indexed based on image content descriptors, rather than keywords. Where keywords may describe the circumstances surrounding the image, things such as the who, what, where, when, and why parameters, content descriptors actually describe the data within the digital graphical material. Such factors are derived from the image itself and may include a color histogram, texture data, resolution, brightness, contrast and so forth.

[0031] The source of the graphical material databases may be existing databases such as stock photo databases or private databases. Where a stock database is used, the user will be required to pay a copyright fee for access to the graphical materials. This is a basic subscription transaction that is well understood by those skilled in the art. In the case of a private database, the supplier of the albuming software or search software may provide access for users to a database of graphical materials. This could be handled as a subscription or as an incentive to users to purchase the software. It is also foreseeable that public sites will develop for dissemination of graphical materials.

[0032] Reference is directed to FIG. 3 which is a diagram of the display in which a user specifies reference material in the digital graphics album. The display 20 appears on the screen of a personal computer. The display 20 has a pull-down menu 24 in this illustrative embodiment. The albuming application has multiple album pages 22 that appear on the screen 20. On the front page, in this example, four graphic materials appear 28 and 26, each of which as some annotation 27 associated therewith. In one illustrative embodiment, if the graphic materials are digital photographs, and the annotation is a brief description of the event in the photograph.

[0033] The user reviews the graphic materials and may decide that additional materials would improve the album, especially if the additional materials logically augmented some of the existing materials. The user specifies some of the graphical materials in the album as reference materials by highlighting them as shown 28. By doing this, the user is establishing a reference from which new materials may be searched for and added to the album.

[0034] Reference is directed to FIG. 4 which is a diagram of the display used when a user initiates a search for additional graphical materials. In the display 20 is the pull-down menu 24, which is selected by the user to initiate a search. The menu, in this illustrative embodiment, allows the user to select an image search 30 or a video search 32. It would also be reasonable to select a unified search for both digital images and video clips if desired. In this illustrative embodiment of the present invention, separate graphical material databases are used for digital images and video clips. For this reason, the search is directed to one of two of these sources of digital graphical materials. It will be understood by those skilled in the art that a single search could access a plurality of database and that the selection of the search type does not necessarily need to limit the search to a single database. Once the user has selected the search type, the content of the reference graphical materials, which is digital images 28 and their respective annotation 27 in this example, are subsequently used to direct the search to produce results consistent with the user selection. The process of generating keywords and annotation of images is discussed in U.S. Pat. No. 5,493,677 to Balogh et al. For GENERATING, ARCHIVING, AND RETRIEVAL OF DIGITAL IMAGES WITH EVOKED SUGGESTION-SET CAPTIONS AND NATURAL LANGUAGE INTERFACE, the substance of which is hereby incorporated by reference.

[0035] Reference is directed to FIG. 5. After the user has initiated a search, the processor of this illustrative embodiment of the present invention processes that reference material to produce search criteria. This aspect will be more fully discussed hereinafter, but in this present example, the user has selected three digital images with annotations so the processor extracts the annotation data and processes it to produce a set of search keywords. These keywords establish the criteria of the subsequent search. However, these keywords are augmented in FIG. 5 by the addition of user directives. Essentially, a pop-up display 34 is overlaid on screen 20 and offers a range of options 36 to the users. Since a search may produce very large amounts of relevant search result, the pop-up display 34 also allows the user to limit the number of resultant graphical materials at option 38. In this example, eighteen resultant graphical materials.

[0036] Considering again the range of options 36 offered to the user, in this example the options are: MORE IMAGES LIKE THESE which will cause the processor to prioritize and augment the search to produce results similar to the annotation keywords; IMAGES WITH MORE DETAILS which will cause the processor to prioritize and augment the keywords to produce search results producing detailed images similar to those references selected; IMAGES WITH WIDER VIEWS which will cause the processor to prioritize and augment the keywords to produce resultant images with more expansive views; and IMAGES THAT CONTRAST which will cause the processor to prioritize and augment the keywords to produce search results that are in contrast with the selected reference materials. Another option is VIDEO CLIP SEARCH in this example, which will cause the processor to direct a search to video clips related to the selected digital images. In this case, the video clip can be represented by key frames that have been extracted a prior, for content-based matching. It should be understood that the foregoing list is not exhaustive. Other criteria for contouring the search to the user directives can be defined and would be understood by those skilled in the art. In another illustrative embodiment, a text window can be offered for the user to enter additional keywords. In the example of wanting a picture of the Eiffel Tower, the user might merely type in “get me a picture of the Eiffel Tower” and the processor would process this request along with the selected reference materials to search for an image of the Eiffel tower that is consistent the reference graphical materials selected.

[0037] Reference is directed to FIG. 6 which is a diagram of the display after the database search has been conducted in an illustrative embodiment of the present invention. In this example the screen 40 displays a plurality of resultant graphical materials 42 that may have associated annotation data 44, if the search database offered such information. In addition, a menu option 46 is present. The user reviews the images to determine if one or more of them are deemed suitable for inclusion into the user's digital photo album. If so, the images are selected and placed in the album. This placement can be by any means, and may include a drag and drop interface for convenience.

[0038] In some applications, the image resolution may be high and therefore the search may produce initial resultant materials that are lower resolution, or thumbnail, representations of the final image for review. This has the advantage of increasing the speed of the search and reducing the download time of images that may not ultimately be selected. Once an image is selected, based on the thumbnail representation, a full resolution version can be downloaded for inclusion in the album.

[0039] Reference is directed to FIG. 7 which is a diagram of the display after the search results have been received. In this instance, the user has reviewed the resultant graphical materials and determined that none of the images is desirable for inclusion in the album. The screen 40 includes menu 46 which offers a choice 48 to refine the search. The user selects this and the process proceeds to the screen depicted in FIG. 8. At this point, in FIG. 8, the present invention contemplates that an offer to refine the search be presented to the user. On top of the previous screen 40, a pop-up menu 50 offers additional choices 52. Since another search will be conducted to refine the results, the user is again offered a choice 54 to limit the number resultant graphical materials from the search. The user may choose to more narrowly construe the keywords previously produced by selecting IMAGES LIKE SELECTED, or may enter specific keywords to refine the search by selecting KEYWORD. Also, the user may elect to conduct a video clip search by selecting VIDEO CLIP.

[0040] However, there may be situations where the keyword data is insufficient to produce the results desired by the user. This may be because of insufficient annotation materials, or because the desired results are not easily described by words. In such situations, the user can select that CONTENT SEARCH option in pop-up menu 50. A content search is not based on keywords, but rather on the content of the reference images themselves, sometimes called the visual characteristics. If selected, the processor analyzes the images to produce metrics of the image content which may include a color histogram, textural data, contrast, brightness, geometric information and other data. Details of such a search and cataloging approach can be found in U.S. Pat. No. 5,751,286 to Barber et al. For IMAGE QUERY SYSTEM AND METHOD, the substance of which is hereby incorporated by reference. This option affords an alternative to the keyword search. It should be understood that the option to conduct a content search may be manually selected, as described in this illustrative embodiment, or may be accomplished automatically by the processor which the initial search results have been otherwise deemed inadequate.

[0041] The results of the content search are offered to the users in a manner as was previously described.

[0042] Reference is directed to FIG. 9 which is the first of three figures depicting a flow chart of an illustrative embodiment of the present invention. The process begins at step 60 and immediately proceeds to step 62 where the user browses an albuming application and decides that additional graphical materials are needed and therefore selects certain reference material from the album that may be in the form of digital images, video clip key frames, of other graphical materials that may reside in the album. At step 64, the processor extracts the annotation data from the selected reference materials as the initial search will be a keyword search.

[0043] At step 66, the processor processes the annotation data in accordance with a natural language processing routine. Natural language processing routines are known in the art and in this instance are used to extract the who, where, what, why, and when metrics of the annotation data, as these are the kinds of metrics that are useful in association with graphical information in a digital graphics album. For a more detailed discussion of natural language processing, see U.S. Pat. No. 5,963,940 to Liddy et al. For NATURAL LANGUAGE INFORMATION RETRIEVAL SYSTEM AND METHOD; and U.S. Pat. No. 5,590,039 to Ikeda et al. for NATURAL LANGUAGE PROCESSING APPARATUS AND METHOD FOR PERFORMING PROCESSING TO ANALYZE THE MEANING OF AN INPUT SENTENCE ENTERED IN THE FORM OF A NATURAL LANGUAGE; and U.S. Pat. No. 6,052,656 to Suda et al. for NATURAL LANGUAGE PROCESSING SYSTEM AND METHOD FOR PROCESSING INPUT INFORMATION BY PREDICTING KIND THEREOF, each of which is hereby incorporated by reference.

[0044] The result of the natural language processing is a list of prioritized keywords that are used to drive the search of one or more relational databases. Continuing in FIG. 9, at step 68, the processor inquires as to whether the desired search is a digital image search or a video clip search. If a video clip search is selected, a video search routine is run at step 70. The video clip routine closely parallels the image search routine which will be discussed next. One difference is that a separate data base is searched. Another difference is that the video clip is not reference material in and of itself, it may be annotated so the keyword aspects are the same as with digital image searches. In the video clip circumstance, the video clip is represented by a key frame. This is a single frame, selected from the numerous frames in the video clip, that is used to represent the video in the album display presentation. Therefore, the video clip routine is not detailed in this figure as it can deduced by those skilled in the art by referring to the image routine. It should also be noted that a search for a digital image and a search for a digital graphic is identical, differing only in the original source of the image information, and not in the storage and retrieval of them.

[0045] Continuing in FIG. 9, if an image search is selected at step 68, the processor presents the user directive input screen at step 72 for entry of user directives. At step 74 the processor tests to determine what, if any, user directives have been received. If user directives have been selected, the user directives are processed with the natural language processor at step 76 to produce additional keywords, called directive keywords, in this example. The keywords derived from the annotations and the keywords derived from the directives are prioritized and augmented at step 78 to produce a set of search keywords more closely tailored to the user's intent. In the event the user has not entered directives at step 74, or if there were directives which were processed through to step 78, the processor initiates an image search with the available keywords at step 80. Process then flows through node 82 in FIG. 9 to node 82 in FIG. 10.

[0046] Referring back to FIG. 9, block 80, where the processor initiates the graphical material database search, it should be noted that a decision on what database or databases to search must be made. The user will have control over this aspect of the search in another portion of the present invention, which is not illustrated. The database may be a commercial stock photograph database, or it could be a private database accessible only to users of the present invention. Or, other kinds of database services may evolve to meet the demand for this kind of service. In any event, the user will have specified the databases beforehand so that the processor identifies the appropriate database and is given a path to access it. Such a path may be an Internet IP address, a uniform resource locator (URL), or any other address in a public or private network.

[0047] The process continues in FIG. 10, which is page two of three of the flow diagram. The database search will typically return a plurality of graphical material which are received by the processor at step 84. If the images are in reduced resolution, or thumbnail, form, they are present to the user at step 86 where they are displayed on the display for review. Of course, if the images are provided in full resolution form, they are displayed in full resolution form at step 86. The user reviews the displayed graphical materials and at step 88 may select one or more of them for inclusion in the album.

[0048] At step 88, if the user selects one or more graphical materials, and if thumb nails where provided as the search results, then the processor requests full resolution versions of the graphical materials from the database at step 90. The graphical material is then presented to the albuming application at step 92 where it is inserted into the album. If the user selects another graphical material at step 94, the flow recirculates to step 90 where the download and insertion are reiterated on an image by image basis.

[0049] Once all of the graphical materials are inserted at step 94, or if the user failed to select any graphical materials at step 88, the processor inquires with the user as to whether the search results were satisfactory or not at step 96. If the answer is yes, the process finishes at step 98. On the other hand, if the search was not satisfactory at step 96, the process requests further directives from the user at step 100 for use in refining the search. If the user provides additional keywords to this request at step 102, the flow continues through node 104 back to step 76 in FIG. 9 to reiterate the previously described search procedure, but with the newly added keywords from step 100. On the other hand, if further keywords were not provided at step 100, step 102 directs flow through node 106 to step 108 in FIG. 11.

[0050] Referring to FIG. 11, which is page three of three of the flow diagram, the processor test to determine if the user has requested a content search at step 108. If no content search was requested, the process is finished at step 98. On the other hand if a content search was requested, flow continues to step 110 where images, or thumbnail images from the album are presented to the user for selection of a reference image or images. At this point, the user is changing the focus of the search from using keywords to describe what is being search for to using existing image content to define what is being searched for. At step 112, if the user has selected a thumbnail image, the processor obtains a full resolution image at step 114. On the other hand, at step 112, if the user has not selected thumbnail images, the user may select an image from the album at step 116. If no image is selected, the process is finished at step 98.

[0051] If full images are obtained from the database at step 114 or if the user selects images from the album at step 116, the processor extracts content information at step 118. The process was described hereinbefore. The content data is then provided to a database which is indexed by content at step 120. Upon receipt of resultant graphical material at step 120, the flow re-circulates to step 84, through node 82, where the search results are handled as was previously described.

[0052] Thus, the present invention has been described herein with reference to a particular embodiment for a particular application. Those having ordinary skill in the art and access to the present teachings will recognize additional modifications, applications and embodiments within the scope thereof. For example,

[0053] It is therefore intended by the appended claims to cover any and all such applications, modifications and embodiments within the scope of the present invention.