Contact sheet based image management

[0001] Individuals are rapidly accumulating large collections of digital image content, including still images, text, graphics, animated graphics, and full-motion video images. This content may be presented individually or combined in a wide variety of different forms, including documents, presentations, still photographs, commercial videos, home movies, and meta data describing one or more associated digital content files. As these collections grow in number and diversity, individuals increasingly will require systems and methods for organizing and presenting the digital content in their collections. To meet this need, a variety of different systems and methods for organizing and presenting digital image content have been proposed, including manual digital image albuming systems that enable users to create digital photo albums manually and digital image albuming systems that provide various levels of automated image layout functionality.

[0002] Users typically find the process of managing a large collection of digital image content to be tedious and time consuming. Recognizing this common experience, efforts have been made to improve the interfaces through which users can manage their digital image content collections. Many image management systems are implemented by software programs that create graphical user interfaces that allow users to perform various image management tasks using a keyboard and a pointing device, such as a computer mouse. To operate these systems, however, users typically must have access to their personal computer systems. This is not always possible, especially when users are away from their homes.

[0003] Hewlett-Packard Company recently has developed a line of all-in-one printer/scanner/copier machines that allow users to print contact sheets containing thumbnail versions of the images that are stored in a memory card that is plugged into the machines. The thumbnail images (or simply “thumbnails”) are distributed uniformly on the contact sheets. Adjacent to each thumbnail is a series of fields that provide options, such as the number of prints of an image to be printed and the size and type of paper on which the prints are to be printed. Users can fill in the fields with a marking instrument, such as a pencil. A user scans a marked contact sheet into the printer/scanner/copier machine and the machine automatically reads the handwritten information that was entered into the fields and prints images from the memory card in accordance with the entered information (e.g., prints the right numbers and sizes of only the requested images).

[0004] In one aspect, the invention features an image management method in accordance with which images are segmented into image clusters. A page layout for a contact sheet is generated. The contact sheet comprises thumbnail clusters. Each of the thumbnail clusters comprises thumbnails of the images in respective ones of the image clusters. The page layout divides at least a portion of the contact sheet into regions each containing a respective one of the thumbnail clusters and at least one associated user input field. A description of the page layout is stored on a machine-readable medium.

[0005] The invention also features a system and machine-readable instructions for implementing the above-described electronic document processing method.

[0006] In another aspect, the invention features an image management apparatus that comprises a contact sheet, clusters of thumbnail images arranged in respective cluster regions on the contact sheet, and at least one user input field located in each of the cluster regions of the contact sheet.

[0007] In another aspect, the invention features an image management method in accordance with which user inputs are extracted from a contact sheet. The contact sheet comprises thumbnail images and one or more annotation user input fields respectively associated with at least one of the thumbnail images. Whether one or more of the extracted user inputs correspond to respective ones of the annotation user input fields is determined. In response to a determination that a given one of the extracted user inputs corresponds to a given one of the annotation user input fields, a link between the given user input and at least one of the thumbnail images associated with the given annotation user input field is generated.

[0008] In another aspect, the invention features an image management method in accordance with which user inputs are extracted from a contact sheet. The contact sheet comprises thumbnail images, one or more action user input fields respectively associated with at least one of the thumbnail images, and one or more destination user input fields. Whether one or more of the extracted user inputs correspond to respective ones of the destination user input fields is determined. In response to a determination that a given one of the extracted user inputs corresponds to a given one of the action user input fields, an image of the given user input is associated with the at least one of the thumbnail images associated with the given action user input field.

[0009] Other features and advantages of the invention will become apparent from the following description, including the drawings and the claims.

[0010] FIG. 1 is a diagrammatic view of an embodiment of a contact sheet based image management system.

[0011] FIG. 2 is a block diagram of an embodiment of a contact sheet generation system.

[0012] FIG. 3 is a flow diagram of an embodiment of a method of generating a contact sheet for managing a collection of images.

[0013] FIG. 4 is a front view of an embodiment of a page layout for a contact sheet that is generated in accordance with the method of claim 2.

[0014] FIG. 5 is a block diagram of an embodiment of a contact sheet processing system.

[0015] FIG. 6 is a flow diagram of an embodiment of a method of processing a marked contact sheet.

[0016] FIG. 7 is a front view of a scanned image of an embodiment of a marked contact sheet containing user input.

[0017] In the following description, like reference numbers are used to identify like elements. Furthermore, the drawings are intended to illustrate major features of exemplary embodiments in a diagrammatic manner. The drawings are not intended to depict every feature of actual embodiments nor relative dimensions of the depicted elements, and are not drawn to scale.

[0018] The embodiments that are described in detail below provide improved contact sheet based interfaces that allow users to manage the digital image content in their collections. These embodiments automatically segment images into clusters and present corresponding clusters of thumbnails of the images on one or more contact sheets. A contact sheet may contain user-selectable options for managing the digital image content at the cluster level. A contact sheet also may contain fields for receiving handwritten annotations and storage destinations that will be linked to associated image clusters. In this way, these embodiments provide a simple, easy-to-use interface for managing digital image content that avoids the complexity and tedium often associated with digital image content management approaches that require a user to make decisions at the individual image level. This approach is particularly well-suited for performing a first level sort and management of a user's digital image content, especially when the user does not have access to tools (e.g., digital image management software tools) that enable the user to manage digital image content directly.

[0019] FIG. 1 shows an embodiment of an image management system 10 that includes a contact sheet based interface system 12 and a computer 14. The computer 14 is connected a network 16, which allows the computer 14 to communicate with a wide variety of different service providers, including providers of a print service 18, an electronic mail (e-mail) service 20, an archive service 22, and an image management service 24.

[0020] In the illustrated implementation, the interface system 12 is implemented by an all-in-one printer/scanner/copier (e.g., an HP Officejet® 9100 available from Hewlett-Packard Company of Palo Alto, Calif., U.S.A.). In other implementations, the interface system 12 may be implemented by separate printing and scanning devices. The printing device may be a portable printer or a desktop printer (e.g., a LaserJet® printer available from Hewlett-Packard Company of Palo Alto, Calif., U.S.A.). The scanning device may be a scanner, such as a desktop optical scanner (e.g., a ScanJet® scanner available from Hewlett-Packard Company of Palo Alto, Calif., U.S.A.), a portable scanner (e.g., a CapShare® portable scanner available from Hewlett-Packard Company of Palo Alto, Calif., U.S.A.), or a facsimile machine.

[0021] In the illustrated implementation, the computer 14 is a server computer. In general, the computer 14 may be any type of personal computer, portable computer, or workstation computer that includes a processing unit, a system memory, and a system bus that couples the processing unit to the various components of the computer. The processing unit may include one or more processors, each of which may be in the form of any one of various commercially available processors. Generally, each processor receives instructions and data from a read-only memory and/or a random access memory. The system memory typically includes a read only memory (ROM) that stores a basic input/output system (BIOS) that contains start-up routines for the computer, and a random access memory (RAM). The computer 14 also may include a hard drive, a floppy drive, and CD ROM drive that contain respective computer-readable media disks that provide non-volatile or persistent storage for data, data structures and computer-executable instructions.

[0022] In operation, a user 26 transfers copies 28 of images that are stored on a portable image storage device 29 to the interface system 12. The images 28 may be complete or partial versions of any type of digital or electronic image, including: an image that was captured by an image sensor (e.g., a video camera, a still image camera, or an optical scanner) or a processed (e.g., filtered, reformatted, enhanced or otherwise modified) version of such an image; a computer-generated bitmap or vector graphic image; a textual image; and an iconographic image. The portable image storage device 29 may be any type of portable electronic device that is capable of storing images. Exemplary implementations of the portable image storage device 29 include a digital still image camera, a digital video camera, a mobile phone, a laptop or notebook computer, a personal digital assistant, and a portable music player. In the illustrated implementation, the portable image storage device 29 is a mobile phone and the images 28 correspond to images that were captured by an integrated camera module of the mobile phone. The portable image storage device 29 may transfer the images 28 to the interface system 12 in any one of a wide variety of different ways. For example, in some implementations, the images 28 are stored on a removable memory card (e.g., a compact flash memory card, a smart card, or a memory stick) and the interface system 12 includes a module for reading images that are stored on the removable memory card. In other implementations, the images 28 may be transferred over a wired connection or a wireless connection that is established between the portable image storage device 29 and the interface system 12.

[0023] After the image copies 28 have been transferred from the portable image storage device 29, the interface system 12 generates at least one contact sheet 30. Each contact sheet 30 includes thumbnails of respective ones of the images 28. The thumbnails are arranged on the contact sheet 30 in clusters that correspond to a logical clustering of the images 28. The contact sheet 30 includes for each thumbnail cluster at least one associated user input field for receiving user input specifying actions that the image management system 10 will perform with respect to corresponding ones of the images 28.

[0024] The user 26 marks the contact sheet 30 with user input that specifies actions to be performed by the image management system 10. In some implementations, the user 26 also may mark the contact sheet 30 with cluster-specific annotations that the image management system 10 will associate with corresponding clusters of the images 28. In some implementations, the user 26 also may specify on the contact sheet 30 one or more custom destinations where the image management system 10 will store user-selected ones of the clusters of the images 28 in a logical file system.

[0025] After the user 26 has marked the contact sheet 30, the user feeds the contact sheet into the interface system 12. The interface system 12 scans the marked contact sheet to generate a scanned image. The interface system 12 extracts the user input from the scanned image and performs various actions with respect to the images 28 in accordance with the extracted user input. For example, the image management system 10 may transmit respective ones of the images 28 to the print service 18 for printing, to the e-mail service 20 for e-mailing, to the archive service 22 for archiving, and to the image management service 24 for performing one or more image management operations. In this way, the user 26 can perform a wide variety of different image management tasks simply by marking the contact sheet 30 with handwritten user input.

II. GENERATING A CONTACT SHEET

[0026] FIG. 2 shows an embodiment of a contact sheet generation system 40 that includes an image segmentation module 42, a page layout generation module 44, and a print module 46. These modules 42-46 may be implemented in any computing or processing environment, including in digital electronic circuitry or in computer hardware, firmware, or software. In some embodiments, the image segmentation module 42, the page layout generation module 44, and the print module 46 are implemented by one or more respective software modules that are executed on the interface system 12, on the computer 14, or distributed across the interface system 12 and the computer 14. Computer process instructions for implementing the modules 42-46 and the data generated by the modules 42-46 are stored in one or more machine-readable media. Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROMs.

[0027] FIG. 3 shows an embodiment of a method by which the contact sheet generation system 40 generates the contact sheet 30. The image segmentation module 42 segments the images 28 into clusters (block 48). The images 28 may be clustered in any one of a wide variety of ways, including by spatial, temporal or other properties of the images 28. The images 28 may be clustered using, for example, k-means clustering or some other clustering method. In general, the clustering process attempts to form clusters that contain images that are related in some way (e.g., they correspond to the same event, the same location, or the same time period). This feature makes it likely that users will want to manage the images that are in the same cluster in the same way.

[0028] In the illustrated embodiment, the image segmentation module 42 clusters the images 28 in accordance with timestamps that are associated with the images 28. In this process, the image segmentation module 42 sorts the images 28 chronologically. The image segmentation module 42 then computes the geometric mean of all the time differences between adjacent ones of the sorted images 28. The image segmentation module 42 iterates through the sorted images 28 to build clusters of the images 28. If the time difference between a current image and a previous image is less than the geometric mean, the current image is assigned to the current cluster; otherwise, the current image is assigned to a new cluster.

[0029] In other embodiments, the image segmentation module 42 may assign images to clusters based on a fixed cluster time period. Alternatively, the image segmentation module 42 may cluster images based on computed measures of relatedness between ones of the images 28.

[0030] After the images 28 have been clustered by the image segmentation module 42 (block 48), the page layout generation module 44 generates page layouts for one or more of the contact sheets 30 based on the segmented image clusters (block 50). In this process, the page layout generation module 44 generates thumbnail versions of the images 28 and segments the thumbnails into clusters corresponding to the clusters of images 28 that were built by the image segmentation module 42. The page layout generation module 44 lays out the thumbnail clusters on the contact sheets in accordance with the page layouts, which divide respective ones of the contact sheets into different regions.

[0031] FIG. 4 shows an exemplary implementation of a page layout that is generated by the page layout generation module 44 for a exemplary set of images 28. In this implementation, the page layout divides the contact sheet 30 into a header region 52, an index region 54, an interface region 56, and a barcode region 58.

[0032] The header region 52 contains a descriptive title for the contact sheet 30 (e.g., “Recent photos for User”) and a set of labels (e.g., “destination”, “comment”, and “cluster action”) that describe the index region 54 and the user input fields of the interface region 56. The header region 52 additionally includes a user input field 60 for receiving user input specifying an image management action that will be applied to all of the images 28 that correspond to the thumbnails appearing on the contact sheet 30.

[0033] The index region 54 includes a list of action codes 62 that designate respective machine-implementable actions that can be performed by the image management system 10. The index region 54 includes a set of predefined action codes each of which is associated with an iconographic representation 64 of the corresponding machine-implementable action.

[0034] In the illustrated implementation, the index region 54 includes the following predefined or default action codes: “T”, “F”, “A”, “P”, and “X”. The “T” action code stands for “Talk” and is intended to be used to designate image clusters that a user may want to discuss with others. The “F” action code stands for “Favorites” and is intended to be used to designate image clusters that are to be stored in an image file location labeled “Favorites”. The “A” action code stands for “Annotation” and is intended to be used to designate image clusters that a user would like to annotate at a later time. The “P” action code stands for “Print” and is intended to be used to designate image clusters that are to be placed in a print queue that is associated with the user. The “X” action code stands for “Delete” and is intended to be used to designate image clusters that are to be deleted by the image management system 10.

[0035] In the illustrated implementation, the index region 54 additionally includes a set of action codes (e.g., “1”, “2”, . . . , “8”) that are associated with respective destination user input fields 66. Each of the destination user input fields 66 is intended to be used to specify a respective storage location in a logical file system for user-selected clusters of the images 28. In some implementations, the storage locations are labeled with icons corresponding to bitmap images of the user input that is contained in the destination user input fields 66. In other implementations, the image management system 10 may perform optical character recognition on the user input that is contained in the destination user input fields 66 to extract names for the corresponding storage locations.

[0036] The interface region 56 is subdivided into horizontal rows 68. In the illustrated implementation, adjacent ones of the rows 68 are demarcated by different background colors. Each row 68 contains a respective one of the thumbnail clusters 70. As shown in FIG. 4, each thumbnail cluster 70 includes one or more thumbnails that are arranged in a left-aligned array. Superimposed over each of the thumbnails is a delete field 72. The image management system 10 responds to user input in a delete field 72 that is associated with a given thumbnail by performing a delete operation with respect to the image 28 corresponding to the given thumbnail. For example, in some implementations, the image management system 10 may delete the corresponding one of the images 28 from the portable image storage device 29.

[0037] Each row 68 additionally includes a cluster-specific annotation (or comment) user input field 74 and two cluster-specific action user input fields 76, 78. The annotation user input fields 74 are intended to be used to specify respective labels for the corresponding image clusters. In some implementations, the images corresponding to a given thumbnail cluster 70 are linked to a bitmap image of the user input contained in the associated annotation user input fields 74. The link may be stored in a header (e.g., an EXIF header) of the images or in a separate link data structure. In other implementations, the image management system 10 may perform optical character recognition on the user input contained in the annotation user input fields 74 to extract text that may be used to derive file names for the images in the corresponding clusters. The action user input fields 76, 78, on the other hand, are intended to be used to specify respective ones of the action codes 62 that are listed in the index region 54. The image management system 10 performs optical character recognition on the user input that is contained in the action user input fields 76, 78 to determine which of the action codes 62 that are listed in the index region 54 corresponds to the user input.

[0038] The barcode region 58 includes a barcode 80 that encodes information about the page layout of the contact sheet 30. Among the various types of information that may be encoded in the barcode are: a unique identifier for the contact sheet 30; a unique identifier for the user; the storage location address of a description of the page layout; and the storage location addresses of the ones of the images 28 corresponding to the thumbnails appearing on the contact sheet. The encoded information may be presented in any type of one-dimensional or two-dimensional barcode format provided it has sufficient capacity to store the required information in the space available in the barcode region 58.

[0039] Referring back to FIG. 3, after the page layout generation module 44 generates the page layouts for the one or more contact sheets (block 50), the page layout generation module 44 stores descriptions 81 of the page layouts on a machine-readable medium (block 82). The page layout descriptions 81 may be described in XML (extensible Markup Language). Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROMs.

[0040] The print module 46 prints the contact sheets in accordance with the page layout descriptions that are generated by the page layout generation module 44 (block 84).

III. PROCESSING A MARKED CONTACT SHEET

[0041] FIG. 5 shows an embodiment of a system 90 for processing a marked contact sheet 88 containing user input. The contact sheet processing system 90 includes a scan module 92, a user input extraction module 94, and an action execution module 96. These modules 92-96 may be implemented in any computing or processing environment, including in digital electronic circuitry or in computer hardware, firmware, or software. In some embodiments, the scan module 92, the user input extraction module 94, and the action execution module 96 are implemented by one or more respective software modules that are executed on the interface system 12, on the computer 14, or distributed across the interface system 12 and the computer 14. Computer process instructions for implementing the modules 92-96 and the data generated by the modules 92-96 are stored in one or more machine-readable media. Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM.

[0042] FIG. 6 shows an embodiment of a method by which the contact sheet processing system 90 processes the marked contact sheet 88. In accordance with this method, the scan module 92 generates a scanned image of the marked contact sheet 88 (block 98). FIG. 7 shows a scanned image of an exemplary implementation of the marked contact sheet 88.

[0043] The user input extraction module 94 detects and decodes the barcode 80 in the scanned image (block 100). The user input extraction module 94 then retrieves the page layout description 81 from the storage location address that is encoded in the barcode 80 (block 102). The user input extraction module 94 determines the locations of all of the user input fields (i.e., user input fields 60, 66, 72, 74, 76, and 78) in the scanned image of the marked contact sheet 88 based on the retrieved page layout description 81 (block 104). The user input extraction module 94 then extracts the user inputs from the scanned image of the marked contact sheet 88 (block 106). In this process, the user input extraction module 94 may perform optical character recognition on areas of the scanned image corresponding to the user input fields. The user input extraction module 94 also may copy bitmap images from the areas of the scanned image corresponding to the annotation and destination user input fields 74, 66.

[0044] The action execution module 96 performs actions with respect to ones of the images 28 corresponding to the thumbnails appearing on the marked contact sheet 88.

[0045] If there are any user-specified annotations contained in the annotation user input fields 74 (block 108), the action execution module 96 links bitmap images of the user-specified annotations to ones of the images 28 in the corresponding clusters (block 110). These annotations may be presented together with the images in the corresponding cluster by a suitably configured image presentation system.

[0046] If there are any user-specified actions contained in the action input fields 60, 76, 78 (block 112), the action execution module 96 performs the user-specified actions (block 114). Among the exemplary types of actions that may be performed by the action execution module 96 are the following:

• • [0047] 1. Move user-designated ones of the image clusters to a folder in a logical file system. The folder may be a predefined folder (e.g., a “Talk” folder, a “Favorites” folder, or an “Annotation” folder) or the folder may be defined by user input that is contained in the destination user field 66. The user-defined folder may be associated with an icon corresponding to a bitmap image of the user input contained in the destination user input field 66.
  • [0048] 2. Move user-designated ones of the image clusters to a print queue.
  • [0049] 3. Delete user-designated ones of the image clusters.
  • [0050] 4. Transmit user-designated ones of the image clusters to one or more of the service providers 18-24.

[0051] After all of the user-specified actions have been performed, the contact sheet processing method terminates (block 120).

IV. CONCLUSION

[0052] The embodiments that are described in detail above provide a simple, easy-to-use interface for managing digital image content that avoids the complexity and tedium often associated with digital image content management approaches that require a user to make decisions at the individual image level. This approach is particularly well-suited for performing a first level sort and management of a user's digital image content, especially when the user does not have access to tools (e.g., digital image management software tools) that enable the user to management digital image content directly.

[0053] Other embodiments are within the scope of the claims.

[0054] For example, in the above-described embodiments, images are transferred to the interface system 12 from the portable image storage device 29. In other embodiments, images may be transferred to the interface system 12 from any type of device or system that is capable of storing images, including non-portable devices and systems.