IMAGE PROCESSING USING A PROFILE SELECTION BASED ON PHOTOGRAPHING CONDITION

[0001] 1. Field of the Invention

[0002] The present invention relates to an image processing apparatus and method and, more particularly, to an image processing apparatus and method of performing color matching processing on an input image photographed by a digital still camera or a digital video camera.

[0003] 2. Description of the Related Art

[0004] In multimedia systems centering around host computers, a color matching system (CMS) for performing image data color matching processing between input and output devices is being extensively developed.

[0005] ColorSync of Apple Computer, Inc. as a representative CMS framework realizes a common CMS on a system by converting an image signal in a color space (Device Dependent Color Space) dependent on an input device into an image signal in a color space (Device Independent Color Space) independent of the device and further converting this image signal in the device independent color space into an image signal in a color space dependent on an output device.

[0006] Conversion characteristics for this conversion process are stored in data called a profile having Inter Color Profile Format. Profiles are prepared in a host computer in a one-to-one correspondence with devices. In performing conversion, one profile is automatically or manually selected, and the color space of an image signal is converted in accordance with the conversion characteristics of the selected profile.

[0007] In the above system, an image input device such as a flat bed scanner is directly connected to a host computer. If the characteristics of this input device remain unchanged, the effect of the CMS can be expected by using the conversion characteristics of a single profile of the host computer.

[0008] Unfortunately, the color characteristics of images such as those photographed by a digital still camera or a digital video camera are largely influenced by the photographing conditions. Therefore, color matching cannot be performed because the photographing conditions cannot be specified. Accordingly, the effect of the CMS cannot be expected when a single profile is prepared for an image input device such as a digital still camera or a digital video camera. In addition, if the photographing conditions and conversion characteristics do not match, unpreferable conversion is performed. As the photographing conditions, it is possible to prepare a plurality of profiles corresponding to the colors, i.e., white balances or exposure amounts of illuminating light during photographing. However, it is impractical to prepare profiles corresponding to all possible photographing conditions.

[0009] In an image input device represented by a flat bed scanner, a profile can be formed by the following method. That is, an image input device for forming a profile is used to scan or photograph a color target having a plurality of color chips. Signal values corresponding to these color chips in a device independent color space are compared with the signal values, obtained by scanning or photographing the color target, of the color chips in a color space dependent on the image input device. In this manner, color conversion characteristics or a profile for converting the color signal from the color space dependent on the image input device to the device independent color space is acquired.

[0010] In the above method of deriving color conversion characteristics by scanning or photographing a color target, the color target is inexpensive, and any user can easily perform the work. Therefore, the method has a great merit for the user.

[0011] In the following description, signal values in a device independent color space will be sometimes referred to as colorimetric values. Similarly, signal values in a device dependent color space will be simply referred to as signal values in some cases.

[0012] The above color conversion characteristics deriving method is based on the assumption that an original has no brighter color than the brightest white of the color target. The object of a device such as a flat bed scanner is to scan a paper original and store or copy the original image. Such devices are originally so designed as to read paper white as the brightest color. Hence, the above assumption is practical in the color reproduction range of paper as a medium.

[0013] A cloud contained in scenery sometimes has a brighter color than the brightest white of the color target. Accordingly, image input devices such as a digital still camera and a digital video camera are so designed as to read a color with lightness much higher than that of paper white as the brightest color.

[0014] However, even when the color target is photographed by a digital still camera or a digital video camera, it is impossible to obtain a color signal having brightness larger than a certain value. Even if color conversion characteristics are acquired from the calorimetric values of color chips of the color target and signal values obtained by photographing the color target, no accurate color conversion characteristics can be obtained for an object such as a cloud including a high-lightness color.

[0015] It is an object of the present invention to provide an image processing apparatus and method suited to performing color matching processing corresponding to the photographing conditions of each individual input image.

[0016] To achieve the above object, one preferred embodiment of the present invention discloses an image processing apparatus comprising: input means for inputting a color image and a photographing condition of the color image; selecting means for selecting a profile corresponding to the photographing condition from a plurality of profiles stored in a memory; and processing means for performing color matching processing on the input color image on the basis of the selected profile.

[0017] It is another object of the present invention to provide an image processing apparatus and method suited to performing color matching processing on an image photographed by a digital still camera or a digital video camera with accuracy higher than that of the digital still camera or the digital video camera.

[0018] To achieve the above object, another preferred embodiment of the present invention discloses an image processing apparatus comprising: input means for inputting a photographed color image and a photographing condition of the color image; selecting means for selecting a profile corresponding to the photographing condition from a plurality of profiles stored in a memory; and processing means for storing the input color image and the selected profile in a medium.

[0019] It is still another object of the present invention to provide an image processing apparatus and method capable of easily obtaining accurate color conversion characteristics for an image input device such as a digital still camera or a digital video camera. It is still another object of the present invention to provide an image processing apparatus and method of performing color matching processing on the basis of the color conversion characteristics obtained for the image input device.

[0020] To achieve the above objects, other preferred embodiments of the present invention provide an image processing apparatus comprising: a memory for storing reference image data obtained by photographing a reference image including a plurality of color chips, and color data in a device independent color space of the color chips; selecting means for selecting reference image data corresponding to a photographing condition of a color image; and calculating means for calculating, on the basis of photographing data of the color chips obtained from the selected reference image data and color data of the color chips, a conversion characteristic for converting the photographing data into the color data and the image processing apparatus further comprising: input means for inputting a color image and a photographing condition of the color image; and processing means for performing color matching processing on the input color image on the basis of the conversion characteristic calculated by said calculating means.

[0021] Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.

[0022] FIG. 1 is a block diagram showing the arrangement of an image processing system according to an embodiment of the present invention;

[0023] FIG. 2 is a block diagram showing the arrangement of a host computer shown in FIG. 1;

[0024] FIG. 3 is a view showing the correspondence between groups of photographing conditions and profiles;

[0025] FIG. 4 is a flow chart showing the procedure of color matching processing in the first embodiment;

[0026] FIG. 5 is a flow chart showing details of a step of selecting an input profile;

[0027] FIG. 6 is a view showing a table containing input profiles to be selected in accordance with the photographing conditions;

[0028] FIG. 7 is a flow chart showing a color conversion characteristic deriving process in the second embodiment; and

[0029] FIG. 8 is a flow chart showing the procedure of color matching processing in the second embodiment;

[0030] Image processing in image processing apparatuses according to preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0031] Arrangement

[0032] FIG. 1 is a block diagram showing the system configuration of this embodiment.

[0033] This system comprises a camera 11 as an input device, a host computer 10 for performing image processing including color matching processing and edit processing on image data, and a monitor 12 and a printer 13 as output devices. Note that the host computer 10 can also be a personal computer for executing software having functions to be described later or dedicated hardware having similar functions.

[0034] The camera 11 is an image or picture input device such as a digital still camera or a digital video camera and can take various environmental conditions and adjustment states in photographing. Therefore, the camera 11 varies its photographing conditions (input characteristics) from one photographing to another, so the input characteristics cannot be specified. Also, the camera 11 can process both motion and still images.

[0035] The monitor 12 is an image display device such as a CRT display or a liquid crystal panel display. The monitor 12 displays images and information which a user requires to input data to the system or check the operation of the system. The printer 13 is a printing apparatus which is an image output device for forming a visual image on a recording medium such as a recording sheet by using a recording material such as ink or toner.

[0036] An external storage 14 includes a recording medium such as a hard disk, a magnetooptical disk, or a floppy disk, and a drive for reading and writing the recording medium. The host computer 10 causes the external storage 14 to store image data supplied from the camera 11 and performs image processing on the stored image data. Also, the host computer 10 supplies the stored unprocessed or processed image data to the monitor 12 or the printer 13 to display or print an image corresponding to the image data.

[0037] Furthermore, the host computer 10 causes the external storage 14 to store photographing conditions supplied from the camera 11 or input by a user in connection with object image data. In this manner the host computer 10 can read out image data and photographing conditions corresponding to the image data from the external storage 14.

[0038] The external storage 14 also stores profiles necessary for color matching processing. If necessary, the host computer 10 reads out a profile stored in the external storage 14 and performs color matching processing.

[0039] Image processing programs and device control programs according to the present invention are usually stored in the host computer 10. However, these programs can also be stored in the external storage 14. That is, in accordance with instructions from a user, the host computer 10 can read out a program stored in the external storage 14 and execute image processing and device control according to the embodiment of the present invention.

[0040] A keyboard 15 and a mouse 16 are operated, where necessary, by a user to input instructions or data to the host computer 10.

[0041] Host Computer

[0042] FIG. 2 is a block diagram showing the arrangement of the host computer 10. Referring to FIG. 2, a console 21 including the keyboard 15, the mouse 16, and the monitor 12 is connected to a CPU 17 through an I/O interface 22 and a CPU bus 25. The external storage 14 is connected to the CPU 17 through an external storage interface 26 and the CPU bus 25.

[0043] The CPU 17 has a system controller 171 for controlling the entire system, a processor 172 for executing various arithmetic operations, and a work memory 173 for temporarily storing necessary data during arithmetic operations. On the basis of programs stored in a ROM 18 or a RAM 19, the CPU 17 performs image processing such as color matching processing on image data, read out by the host computer 10 and stored in the RAM 19, and outputs the processed image data to an output device.

[0044] A video RAM 23 stores the pictures to be displayed on the monitor 12.

[0045] Color Matching Processing

[0046] In this system, the following input device-output device combinations are possible. Note that combination (4) is a combination for performing so-called preview. This combination is used to display an output image from the printer 13 on the monitor 12 to allow a user to check the image.

[0047] (1) Camera 11Monitor 12

[0048] (2) Camera 11Printer 13

[0049] (3) Monitor 12Printer 13

[0050] (4) Printer 13Monitor 12

[0051] To perform color matching processing corresponding to any of the above combinations, the CPU 17 reads out an input profile corresponding to the input device and an output profile corresponding to the output device from the external storage 14. Each profile stores data representing the relationship between a device dependent color space and a device independent color space. Therefore, the input profile stores data for converting an image signal in a color space dependent on the input device into an image signal in a device independent color space. On the other hand, the output profile stores data for converting the image signal in the device independent color space into an image signal in a color space dependent on the output device. The data of the output profile sometimes further contains color space compression for compressing input image data into the range of color reproduction of the output device, in addition to the color space conversion.

[0052] That is, on the basis of the input device-output device combination, the CPU 17 first reads out an input profile from the external storage 14. By using this input profile, the CPU 17 converts an image signal in a color space dependent on the input device into an image signal in a device independent color space.

[0053] Next, the CPU 17 reads out an output profile corresponding to the output device from the external storage 14. By using this output profile, the CPU 17 converts the image signal in the device independent color space into an image signal in a color space dependent on the output device.

[0054] As described above, a profile used in the color matching processing is data based on the characteristics of a device.

[0055] The phosphor or color filter and the use environment of the monitor 12 can be specified. Analogously, the recording medium, recording material, and image formation method of the printer 13 can be previously specified. Therefore, the output characteristics of the monitor 12 and the printer 13 can be previously known. Accordingly, data based on the specified output characteristics can be used as profiles.

[0056] In contrast, the characteristics of images photographed by the camera 11 vary from one image to another. Therefore, various photographing conditions of the camera 11 are classified into several representative groups having a one-to-one correspondence with input profiles. That is, one profile corresponds to photographing conditions included in the same group.

[0057] FIG. 3 is a view showing the correspondence between photographing condition groups and profiles. Photographing condition groups (to be referred to as reference photographing conditions hereinafter) and corresponding input profiles are stored in the external storage 14 or the camera 11 so that the host computer 10 can read out both data.

[0058] More specifically, it is only necessary to form a plurality of input profiles on the basis of a plurality of images photographed by the camera 11 under a plurality of representative photographing conditions (e.g., colors, white balances, and exposure amounts of illuminating light during photographing).

[0059] Procedure

[0060] FIG. 4 is a flow chart showing the procedure of the color matching processing in this embodiment executed by the CPU 17.

[0061] In step S31, the CPU 17 stores input image data from the camera 11 into the external storage 14. In step S32, the CPU 17 stores an input photographing condition of the image data from the camera 11 into the external storage 14.

[0062] In steps S33 and S34, the CPU 17 selects input and output profiles used in the color matching processing. In step S35, the CPU 17 performs the color matching processing on the image data. In step S36, the CPU 17 outputs the image data subjected to the color matching processing to an output device, i.e., the monitor 12 or the printer 13.

[0063] FIG. 5 is a flow chart showing details of step S33 for selecting an input profile.

[0064] In step S331, of the input profiles stored in the external storage 14 and corresponding to the camera 11, the CPU 17 sets an input profile that is currently being looked up as P. In step S332, the CPU 17 substitutes a maximum value Dmax of a photographing condition difference into dmin. For example, the photographing condition is color temperature, a white point and the like. In step S333, the CPU 17 sets the difference between the photographing condition of the input image data and the reference photographing condition of the input profile P as d. In step S334, the CPU 17 compares d with dmin. If ddmin, the flow jumps to step S337. If d<dmin, the flow advances to step S335, and the CPU 17 sets the input profile P as an input profile selection candidate Ps. In step S336, the CPU 17 substitutes d into dmin.

[0065] In step S337, the CPU 17 sets a value indicating lookup done in a flag corresponding to the input profile P. In step S338, the CPU 17 checks whether the value indicating lookup done is set in flags of all of the input profiles stored in the external storage 14 and corresponding to the camera 11. If YES in step S338, the CPU 17 sets the input profile set as the selection candidate Ps as a selected input profile and returns to the main processing flow in FIG. 4.

[0066] If input profiles which correspond to the camera 11 and are not looked up yet remain in the external storage 14, the flow advances to step S339, and the CPU 17 changes the input profile P to be referred to. After that, the flow returns to step S333.

[0067] As described earlier, the output characteristics of the monitor 12 and the printer 13 as output devices can be specified. In step S34, therefore, the CPU 17 need only select an output profile corresponding to the characteristics of an output device designated by a user.

[0068] In this embodiment as described above, in accordance with the photographing condition of input image data from the camera 11, an input profile best suited to the photographing condition can be selected from a limited number of input profiles fewer than possible photographing conditions. Consequently, appropriate color space conversion can be performed.

[0069] A table as shown in FIG. 6 can also be formed and stored in the external storage 14 by previously calculating input profiles to be selected for individual corresponding photographing conditions, i.e., calculating the approximations of individual photographing conditions to the reference photographing conditions of input profiles stored in the external storage 14. An input profile can be selected in step S33 on the basis of this table.

[0070] A profile used in the above embodiment can be data representing a transformation function such as a 33 matrix or data of a transformation table. Also, a profile can of course have a format such as Inter Color Profile Format based on an arbitrary standard.

[0071] In the above embodiment, color matching processing is performed on color image data by using a profile having closest relevance to the photographing condition. However, the present invention is not limited to the above embodiment. For example, color image data and a profile having closest relevance to the data can be paired and stored in a medium such as a tape or disk medium, the main body of a host computer, an external storage, or a video camera. This advantageously reduces the load when stored color images are processed later.

[0072] Image processing for deriving color conversion characteristics according to the present invention will be described below. The arrangement of an image processing apparatus for realizing the image processing of the second embodiment is the same as in the first embodiment, so a detailed description thereof will be omitted. However, since conversion characteristics are calculated on the basis of the colorimetric values and signal values of color chips of a color target, an external storage 14 stores the calorimetric values of color chips of a color target and an image obtained by photographing the color target.

[0073] Method of Deriving Color Conversion Characteristics

[0074] A method of deriving conversion characteristics from the calorimetric values (standard values) of color chips of a color target and signal values obtained by photographing the color target will be described below. Note that the calorimetric values and signal values of color chips previously stored in the external storage 14 are represented by an XYZ colorimetric system and an RGB colorimetric system, respectively, and that a signal value can be converted into a calorimetric value by: $\begin{array}{cc}\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=M\left[\begin{array}{c}R\\ G\\ B\end{array}\right]& \left(1\right)\end{array}$

[0075] Calculating optimum color conversion characteristics is to optimize a matrix M of equation (1), i.e., to substitute the signal values of color chips obtained by photographing the color target into equation (1) to determine the elements of the matrix M by which errors between the calculated colorimetric values of the color chips and the colorimetric values of the color chips stored in the external storage 14 are minimized. In other words, this matrix M represents the color conversion characteristics as the relationships between the signal values and the colorimetric values.

[0076] Equation (1) is applicable to a color not included in the color target. Therefore, a color having higher lightness as a colorimetric value than that of the brightest white in the color target can be converted. However, digital still cameras are so designed that signal values RGB obtained by photographing a gray scale (neutral colors) having a calorimetric value Y in the color target have relationships represented using constants _R, _G, _B, _R, _G, _B, _R, _G, and _B by $\begin{array}{cc}\begin{array}{c}R={{}_R\left(\frac{Y}{{}_R}\right)}^{{}_R}\\ G={{}_G\left(\frac{Y}{{}_G}\right)}^{{}_G}\\ B={{}_B\left(\frac{Y}{{}_B}\right)}^{{}_B}\end{array}& \left(2\right)\end{array}$

[0077] A gray scale is a group of neutral colors. Therefore, calorimetric values X and Z corresponding to the calorimetric value Y have relationships represented using constants _X and _Z as

[0078] Accordingly, by calculating the constants in equations (2) and (3) from the colorimetric values of a gray scale in a color target and the signal values obtained by photographing the gray scale, conversion coefficients with higher accuracy than when equation (1) is used can be obtained for the colors in the gray scale. In addition, it is possible by the use of the obtained conversion coefficients to estimate, from the gray scale of the color target, the colorimetric value of white having higher lightness as a colorimetric value than white contained in the color target. A signal value which can be obtained by photographing the estimated white is estimated by extrapolation or the like. The calorimetric value and the signal value of the estimated white are used in an arithmetic operation of color correction coefficients. This decreases errors of color correction coefficients with respect to such high-lightness colors. Consequently, it is possible to obtain a profile having accurate color conversion characteristics with respect to a high-lightness color not contained in a color target, particularly white. By applying this profile to a CMS, high-accuracy color matching can be performed on an image or picture photographed by an image input device such as a digital still camera or a digital video camera.

[0079] Procedure

[0080] FIG. 7 is a flow chart showing the process of deriving color conversion characteristics in this embodiment executed by a CPU 17.

[0081] In step S301, the CPU 17 reads out, from the external storage 14, image data of a color target photographed under the same photographing conditions as or photographing conditions which can be regarded as the same as the photographing conditions of an object in image data to be subjected to color matching. In step S302, the CPU 17 extracts the signal values of color chips from the readout image data of the color target. In step S303, the CPU 17 extracts the signal values of a gray scale from the extracted signal values of the color chips.

[0082] In step S304, the CPU 17 reads out the calorimetric values of the color chips in the color target. In step S305, the CPU 17 extracts the colorimetric values of the gray scale from the readout calorimetric values of the color chips.

[0083] In step S306, from the extracted signal values and calorimetric values of the gray scale, the CPU 17 estimates the calorimetric value of a color having higher lightness than that of a white chip contained in the color target and a signal value which can be obtained when this high-lightness color is photographed. In step S307, the CPU 17 adds the estimated signal value to the extracted signal values of the color chips. In step S308, the CPU 17 adds the estimated colorimetric value to the readout calorimetric values of the color chips.

[0084] In step S309, the CPU 17 calculates the relationships between the signal values and the colorimetric values on the basis of the signal values and the colorimetric values of the color chips. In step S310, the CPU 17 forms a profile on the basis of the calculated relationships between the colorimetric values and the signal values. The obtained profile is stored in the external storage 14 or a RAM 19.

[0085] Note that in step S301, photographing conditions are designated or image data of a color target is selected by a user or on the basis of data indicating photographing conditions added to image data to be subjected to the color matching processing by using data indicating the photographing conditions of a profile to be formed, e.g., a formed profile. Also, the processing shown in FIG. 3 can be performed on image data of a color target photographed under different conditions. Alternatively, it is possible to group nearly identical photographing conditions and perform the processing shown in FIG. 3 on image data of a color target photographed under a representative photographing condition of the group. Profiles corresponding to these photographing conditions can be previously formed and stored in the external storage 14.

[0086] FIG. 8 is a flow chart showing the color matching processing in this embodiment executed by the CPU 17.

[0087] In step S321, the CPU 17 inputs image data to be subjected to the color matching processing from, e.g., a camera 11 to a host computer 10. In step S322, the CPU 17 selects an input profile from profiles stored in the external storage 14. This selection is done by a user on the basis of the photographing conditions of the input image data or done on the basis of data indicating photographing conditions added to the input image data.

[0088] In step S323, of the profiles stored in the external storage 14, the CPU 17 selects, as an output profile, a profile corresponding to an output device (e.g., a monitor 12 or a printer 13) desired by the user as an output destination of the image data and the set conditions of the desired output device.

[0089] In step S324, the CPU 17 performs color matching processing on the input image data on the basis of the selected input and output profiles. In step S325, the CPU 17 sends the image data thus subjected to the color matching processing to an output device as an output destination.

[0090] In this embodiment as described above, by using devices and works similar to those in a method of forming a profile in a flat bed scanner, accurate profiles can be formed for image input devices such as a digital still camera and a digital video camera. Therefore, it is possible to perform high-accuracy color matching processing on images and pictures such as scenery photographed by these image input devices and obtain good color matching results.

[0091] The color conversion characteristics deriving process of forming a profile and the color matching processing are separately described with reference to FIGS. 7 and 8. However, it is of course possible to execute these processes as a series of processes.

[0092] Also, in the second embodiment, the relationships between calorimetric values and signal values, i.e., color conversion characteristics are profiled on the basis of the assumption that a CMS framework such as ColorSync is used. However, the present invention includes a case wherein color conversion is performed without profiling. That is, effects similar to those in the second embodiment can be obtained when color matching is performed for pixels of image data on the basis of the color conversion characteristics obtained by the process shown in FIG. 6.

[0093] Furthermore, profiles obtained by the procedure shown in FIG. 6 can be used for image data photographed by a normal photographing mode. On the other hand, when image data photographed by a macro photographing mode is to be processed, whether the process of estimating a color having higher lightness than that of a white chip in a color target, which is contained in the procedure shown in FIG. 6, is to be performed is determined in accordance with the setting by a user. If no estimation processing is to be performed, steps S303 and 305 to S308 in FIG. 6 are passed through.

[0094] As many apparently widely different embodiments of the present invention can be made without departing from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments thereof except as defined in the appended claims.