Method and apparatus for processing an image, and storage

[0002] 1. Field of the Invention

[0003] The present invention relates to an image processing apparatus having the capability of retrieving and sorting images input from an input device, such as a digital camera, according to a condition such as a keyword, and also to a method thereof.

[0004] 2. Description of the Related Art

[0005] In conventional image processing apparatus, a keyword used in retrieving or sorting of images is provided by a user for direct entry through a keyboard or the like.

[0006] To provide a user with information about which keywords are assigned to respective images, a keyword is generally displayed together with a corresponding image one by one.

[0007] When keywords are assigned to a plurality of images in accordance with the places and situations in which the respective images were taken, it is required that the user observe the images one by one.

[0008] However, when a significant number of images are dealt with as in the case where images are taken by a modern sophisticated digital camera system, the entering of keywords through a keyboard from one image to another becomes difficult.

[0009] Furthermore, when a plurality of images are displayed in the form of a list of thumbnail images, it is difficult to display keywords associated with the respective images in a limited display area and thus the user cannot know which keywords are assigned to the respective images.

[0010] To solve the above problems, the invention provides an image processing apparatus including storage means for storing a plurality of image data, and control means for controlling the process of displaying a plurality of image data stored in the storage means in the form of a list of thumbnail images so that labels corresponding to the keywords of the image data are displayed near the respective image data.

[0011] FIG. 1 is a block diagram illustrating an embodiment of an image processing apparatus according to the present invention and also illustrating a system including the image processing apparatus;

[0012] FIG. 2 is an illustration of images displayed on a screen according to the present invention;

[0013] FIG. 3 is an illustration of a label window according to the present invention;

[0014] FIG. 4 is an illustration of an image displayed on the screen in another mode according to the present invention;

[0015] FIG. 5 is a flow chart illustrating a process of displaying an image in a calender mode;

[0016] FIG. 6 is a flow chart illustrating a process of grouping a plurality of reduced-size images;

[0017] FIG. 7 is a block diagram illustrating the image processing apparatus according to an embodiment of the invention;

[0018] FIG. 8 is an illustration of images displayed on the screen according to an embodiment of the invention;

[0019] FIG. 9 is a flow chart illustrating a process of performing retrieval using labels.

[0020] The present invention is described in further detail below with reference to a preferred embodiment.

[0021] FIG. 7 illustrates the construction of an image processing apparatus according to the present invention.

[0022] The image processing apparatus shown in FIG. 7 corresponds to that denoted by reference numeral 4 in FIG. 1.

[0023] In FIG. 7, similar elements to those in FIG. 1 are denoted by similar reference numerals.

[0024] In the image processing apparatus shown in FIG. 7, a CRT display device 6 displays image data taken by a digital camera or the like and also various kinds of information such as message menus. The CRT display device 6 may be replaced with a liquid crystal display device. A video RAM (VRAM) 8 is used to store character and image data to be displayed on the screen of the CRT display device 6.

[0025] A bit move unit (BMU) 9 controls data transfer among memories or between memories and various devices. A keyboard 10 including various keys is used to enter information required for editing images or for other operations. A pointing device (PD) 11 is used to point to image data or an icon on the screen of the display device 6. A CPU 5 controls various parts of the image processing apparatus in accordance with a control program stored in a ROM 12 (which includes a program for editing images and other programs for performing various processes according to the invention).

[0026] The image processing apparatus also includes: a RAM 13 used by the CPU 5 as a working area during the operation of performing various programs and also used to temporarily store data when an error occurs; a hard disk drive (HDD) 14; and a floppy disk drive (FDD) 15. The hard disk drive 14 and the floppy disk drive 15 are used to store programs, data and libraries which will be described later.

[0027] Furthermore, there is provided a network interface (NET-I/F) 16 for controlling and diagnosing data in the operation of transferring and receiving data to and from another image processing apparatus via a network. The units described above are connected to each other through an I/O bus (including an address bus, a data bus, and a control bus) 17.

[0028] In this embodiment, the program stored in the ROM 12 may also be stored on another storage medium such as a hard disk (HD) or a floppy disk (FD) directly connected to the image processing apparatus or may be stored on another apparatus connected through a network to the image processing apparatus.

[0029] Referring to FIG. 1, an image processing means for processing image data taken via an image pickup apparatus such as a digital camera is described below. The related system construction and operation are also described.

[0030] Image data taken by an image pickup apparatus 1 shown in FIG. 1 is stored in a memory 2 provided therein. Date/time when the image is taken is measured by an internal clock 3 and the resultant date/time information is also stored as associated data in the memory 2. Furthermore, a reduced-size image (thumbnail image) corresponding to each image data is generated and stored in connection with the corresponding image data. The generation of reduced-size images may be accomplished using a known technique, and thus it is not described in further detail herein.

[0031] Although in this specific embodiment, it is assumed that the associated data is date/time information, the associated data may also be other types of information such as the identification code of the image pickup apparatus and information about operating conditions such as a shutter speed, a zooming ratio, and an f-stop number under which the image is taken by the image pickup apparatus. In the case where operating conditions are stored as the associated data, it is possible to retrieve image data using any operating condition as a keyword. However, the present embodiment is described only for the case where the associated data includes only date/time information associated with the respective images, and the operation of retrieving image data in terms of image taking conditions is not described.

[0032] The image data stored in the memory 2 is transferred to the image processing apparatus 4 via a removable external memory or via a cable, and the image data is displayed on the display device 6 of the image processing apparatus 4.

[0033] In the image processing apparatus 4, a microprocessor 5 generates images to be displayed on the screen of the display device 6 in accordance with the image data and associated data thereof stored in the memory 2 using the application program and associated data thereof stored in a memory 7 provided in the image processing apparatus 4. The resultant images are displayed on the display device 6.

[0034] A cursor pointer K is also displayed on the display device 6. When images are displayed in the form of a list of thumbnail images, keywords are assigned to images, or retrieval or sorting is performed, the position of the cursor pointer K is controlled via the pointing device 11 so as to achieve the purpose.

[0035] The present embodiment of the invention has the feature that when a plurality of reduced-size images corresponding to image data are displayed in the form of a list of thumbnail images, it is possible to easily see which keywords are assigned to the respective image data. This feature is described in further detail below.

[0036] FIG. 2 illustrates a screen on which images and associated data are displayed in a special manner characterizing the present invention.

[0037] When the application program according to the present embodiment is started, a sheet window is displayed so that it has a size within the screen of the display device 6 and so that the sheet size is equal to the final sheet size at the end of the previous operation. A desired sheet size can be selected by clicking a proper one of buttons “SMALL SIZE” (212), “MIDDLE SIZE” (213) or “LARGE SIZE” (214) shown in FIG. 2. In the example shown in FIG. 2, the sheet is displayed in the middle size. This occurs when the final size at the end of the previous operation was middle or when the middle size (button 213) is selected by a user after starting the application program.

[0038] After the sheet appears on the screen, if an icon 219 located on the upper side of the screen is clicked, image data stored in the memory of the image pickup apparatus is transferred to the image processing apparatus and reduced-size images 201 are displayed in the form of a list of thumbnail images together with date/time information 203, as shown in FIG. 2. At this stage, labels 202 corresponding to keywords are not assigned.

[0039] A particular label 202 can be given to a desired reduced-size image by dragging a label from a label window 204 and dropping it onto the reduced-size image. FIG. 2 illustrates a state in which the respective reduced-size images have their own labels 202 assigned in the above-described manner.

[0040] Each label has a particular color. For example, a label “seasons” 205 is colored with orange, “people” 206 with yellow, “landscapes” 207 with green, “animals” 208 with yellow green, “plants ” 209 with blue, “businesses” 210 with purple, “travel” 211 with pink, and so on. Colored labels are placed near images having keywords corresponding to the labels so that the user can see, from the color of the labels, which keywords are assigned to the respective images.

[0041] That is, when a plurality of images are displayed in the form of a list of thumbnail images, the user can visually recognize the keywords assigned to the respective images.

[0042] For example, in FIG. 2, the image on the upper and left side of the screen has orange, yellow and green labels which indicate that keywords “seasons”, “people”, and “landscapes” have been assigned to this image. This makes it possible for the user to easily know at a glance which keywords are given to the respective images displayed in the form of a list of thumbnail images.

[0043] If keywords have been once defined for the respective labels in the label window 204, then any desired keyword can be assigned to a particular image simply by dragging a label and dropping it onto the reduced-size image as shown in FIG. 2. Thus, the invention provides a very easy way of assigning a keyword to image data.

[0044] Referring now to FIG. 3, the label window is briefly described.

[0045] It is possible to set the display condition via the window menu shown in FIG. 2 so that the label window 300 is either displayed or not displayed on the screen.

[0046] Using the label window 300, a particular color can be assigned to each keyword. If a button at the right side of each label is clicked via the pointing device, sublabels appear in a drop-down window. The user can define arbitrary keywords for sublabels. The keywords defined for the respective sublabels are displayed in relation to the corresponding sublabels in the drop-down window. For example, when a keyword “animals” is defined for a certain label, further detailed keywords may be defined such as “bear”, “lion”, “giraffe”, and so on for its sublabels. The user can assign a desired keyword to a particular image simply by dragging a sublabel and dropping it onto the reduced-size image.

[0047] Although in the specific example described above, color information is used to identify the respective labels, the labels may also be identified by symbols such as a triangle (Δ), a rectangle ([□]), a circle (∘), etc. If desired, both colors and symbols may be employed to identify the labels.

[0048] Referring to FIGS. 8 and 9, the retrieving process using a label is described below.

[0049] FIG. 8 illustrates an example of an image screen according to the invention.

[0050] FIG. 9 is a flow chart associated with the retrieving process using labels.

[0051] In step S901 in FIG. 9, it is determined whether the cursor is on a label such as label 202 shown in FIG. 8. If it is concluded in step S901 that the cursor is on a label, then the process goes to step S902 in which the keyword represented by that label is displayed. For example, a message “Season: Autumn” is displayed in a message box 226 on the screen as shown in FIG. 8.

[0052] In step S903, it is determined whether the user has performed an operation via the pointing device on the message box in which the keyword (“Season: Autumn”) is displayed.

[0053] If a double-click is detected in step S903, then it is further determined in step S904 whether there is another label selected. If another label is not selected, then the process goes to step S908, and retrieval is performed using the keyword represented by the label. The retrieval result is displayed in the form of a list of thumbnail images as shown in FIG. 8.

[0054] In the case where it is concluded in step S904 that another label is selected, then the process goes to step S905 and retrieval is performed in terms of a logical OR of keywords represented by the selected labels. The retrieval result is displayed in the form of a list of thumbnail images as shown in FIG. 8.

[0055] In the case where a click is detected in step S903, the process goes to step S907 in which the selection of labels other than the label currently pointed to by the cursor is canceled, and the label currently pointed to by the cursor is selected. If “SHIFT KEY+click” is detected in step S903, the label currently pointed to by the cursor is selected while maintaining the existing selected labels in the selected state.

[0056] As described above, if the cursor is placed on a label in the vicinity of a reduced-size image, the keyword represented by the label is displayed on the screen. Therefore, the user can easily see which keyword is assigned to the image.

[0057] If a label is double-clicked, reduced-size images having the keyword represented by that label are retrieved. If the above process is repeated, it is possible to easily perform retrieval in terms of a logical AND of a plurality of keywords. If retrieval is executed after selecting a plurality of labels using a “SHIFT KEY+click” command, then retrieval is performed in terms of a logical OR of a plurality of keywords.

[0058] In the process described above, the mode of operation of the pointing device is checked in step S903. Alternatively, the operation of the keyboard may be checked in step S903. Furthermore, detection by the pointing device may be performed for various combinations of double-click, click, and SHIFT+click.

[0059] The switching of the display mode is described below.

[0060] The sheet may be displayed in various sizes. If one of buttons “SMALL SIZE” 212, “MIDDLE SIZE” 213, and “LARGE SIZE” 214 is clicked, then the corresponding size is selected and the sheet is displayed in the selected size.

[0061] If the display mode is switched to a calender mode (by clicking the calender button 216), the reduced-size images corresponding to the respective images are sorted in terms of time on the basis of date/time information representing when the images were taken, and the resultant images are displayed along a time axis for a particular period of time such as a week, month or year (that is, in units of weeks, months or years).

[0062] FIG. 4 illustrates an example in which images are displayed along the time axis for a particular week selected (in units of weeks).

[0063] The process of displaying images in the calender mode as shown in FIG. 4 is described below referring to the flow chart shown in FIG. 5.

[0064] In step S101, the microprocessor 5 reads image data and date/time information from the memory 2. Then in step S102, reduced-size images are produced from the image data read from the memory. In the case where reduced-size images have already been produced in the image pickup apparatus 1 in the manner described above, those reduced-size images may be directly displayed. In step S103, the display positions on the screen of the display device 6 are calculated from the date/time information. Then in step S104, the reduced-size images are displayed at the calculated display positions.

[0065] By performing the above process, the images are displayed on-the screen as shown in FIG. 4.

[0066] In FIG. 4, reference numeral 400 denotes a window having a form in which images are displayed in units of weeks, and 401 denotes a time axis. When images are displayed in unit of weeks, the time axis is displayed in the full scale of 24 hours. On the other hand, when images are displayed in units of months, a calender of a month is displayed. If images are displayed in units of years, a calender of a year is displayed.

[0067] When a plurality of reduced-size images are displayed in the form of a stack as denoted by reference numeral 402, it is possible to assign a keyword to all of these images at one time. When a plurality of reduced-size images are displayed in the form of a stack, they were most likely taken at the same event such as at an exhibition or during travel. Therefore, the capability of assigning a keyword to all images at one time is very useful.

[0068] More specifically, a stack of images as that denoted by 402 may be selected via the cursor pointer or the like, and a keyword such as “travel to . . .” or “exhibition of . . .” may be assigned to the stack of images. In this case, the operation is similar to that of assigning a keyword to a single image described above, and thus the operation is not described in further detail herein. It is also possible to assign particular keywords such as “ipeople” or “season” individually to the respective images in the stack.

[0069] FIG. 6 is a flow chart illustrating the process of grouping a plurality of reduced-size images.

[0070] In step S201, the microprocessor 5 reads a set of image data and associated date/time information from the memory 2. In step S202, reduced-size images are sorted in the order of image taking date/time on the basis of the date/time information read from the memory. Then in step S203, the date/time information is read record by record in the sorted order. In step S204, the date/time information of the current image data is compared with that of the immediately previous image date, and it is determined whether the difference in time is less than a predetermined value. If it is concluded in step S204 that the difference is less than the predetermined value, then the process goes to step S205 in which the same attribute as that of the immediately previous image data is assigned to the current image data. On the other hand, if it is concluded in step S204 that the difference is greater than the predetermined value, then the process returns to step S203 and the date/time information of the next image data is read.

[0071] In the process described above, if the date/time information of the current image data has a small difference within the predetermined range relative to that of the immediately previous image data, the current image data is incorporated into the same group as that of the previous image data. In this way, a plurality of image data are grouped. Furthermore, a particular keyword may be assigned to a particular group.

[0072] The predetermined value used as the reference in the above process to determine the difference in date/time is set to a particular default value. If desired, the user can set the predetermined value to an arbitrary value.

[0073] Once keywords have been assigned to images, it is possible to retrieve a desired image from a large number of images in terms of a particular keyword. It is also possible to produce an image album in terms of a particular theme.

[0074] For example, a keyword “people” is assigned to portraits, and a keyword “spring”, “summer”, “autumn”, or “winter” is assigned to pictures characterized by such a season. If it is desired to retrieve a picture of children playing with a dog on a summer vacation, then the picture can be obtained by performing retrieval using keywords “people-child”, “summer”, and “animals-dog”.

[0075] The retrieval result is displayed in such a manner that pictures obtained via the retrieval are displayed in an array fashion in a new window.

[0076] As described above, the technique of the present embodiment provides an easy and high-efficiency method of assigning a keyword to image data.

[0077] Furthermore, when reduced-size images are displayed in the form of a list of thumbnail images, labels representing the keywords assigned to the respective images are indicated in small limited areas in a clear and systematic fashion.

[0078] A series of stored images may be displayed along the time axis in the order of image taking date/time so that the places and situations in which the image were taken can be easily determined thus making it easy to classify the images and assign keywords to the respective images.

[0079] When images are displayed along the time axis in the order of image taking date/time, it is possible to group the images in such a manner that a plurality of images which are very near in terms of the image taking date/time are taken into the same group.

[0080] Furthermore, the objects of the present invention may also be achieved by supplying a storage medium, in which a software program implementing the functions of any of the embodiments described above is stored, to a system or an apparatus whereby a computer (CPU or MPU) in the system or apparatus reads and executes the program code stored in the storage medium.

[0081] In this case, it should be understood that the program code read from the storage medium implements the functions of invention and thus the storage medium storing the program code falls within the scope of present invention.

[0082] Storage media which can be preferably employed in the present invention to supply the program code include a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, non-volatile memory card, and ROM.

[0083] Furthermore, the scope of the present invention includes not only such a system in which the functions of any embodiment described above is implemented simply by reading and executing a program code on a computer, but also a system in which a part of or the whole of a process instructed by the program code is performed using an OS (operating system) on the computer.

[0084] Furthermore, the scope of the present invention also includes a system in which a program code is transferred once from a storage medium into a memory provided in a function extension board inserted in a computer or provided in a function extension unit connected to the computer, and then a part of or the whole of the process instructed by the program code is performed by a CPU or the like in the function extension board or the function extension unit thereby implementing the functions of any embodiment described above.

[0085] Furthermore, the scope of the present invention also includes a system in which in response to a request from a requester, a program implementing the functions of any embodiment is read from a storage medium and sent to the requester via a communication network such as a personal computer communication network.

[0086] As described above, the present invention provides a technique of easily and efficiently assigning a keyword to image data.

[0087] Reduced-size images can be displayed in the form of a list of thumbnail images so that the labels representing the keywords assigned to the respective images are indicated in limited small areas in a clear and systematic fashion.

[0088] Furthermore, a series of stored images can be displayed along the time axis in the order of image taking date/time so that the places and situations in which the image were taken can be easily determined thus making it easy to classify the images and assign keywords to the respective images.

[0089] Still furthermore, when images are displayed along the time axis in the order of image taking date/time, it is possible to group the images in such a manner that a plurality of images which are very near in terms of the image taking date/time are assigned to the same group.