Image processing method and apparatus

[0002] 1. Field of the Invention

[0003] This invention relates to an image processing method and apparatus. This invention particularly relates to an image processing method and apparatus, wherein sharpness of an image is enhanced, while graininess (high frequency noise) of the image is being suppressed. This invention also relates to an image processing method and apparatus, wherein graininess of an image is suppressed.

[0004] 2. Description of the Related Art

[0005] With techniques for reading out an image (or a color image) and reproducing the image as a visible image, an image having been recorded on photographic film, a print, or the like, is photoelectrically read out with a sensor, such as a charge coupled device (CCD) image sensor, and an image signal is thereby obtained. Various image processing operations are carried out on the image signal, and a visible image is reproduced on a print, a cathode ray tube (CRT) display device, or the like, from the image signal, which has been obtained from the image processing. Examples of such image processing include processing in frequency domain, which acts upon predetermined frequency components contained in an image, and gradation processing, which acts upon image density, or the like.

[0006] Examples of the frequency processing in frequency domain include sharpness enhancement processing for suppressing blurr of a contour in an image, and graininess suppression processing, such as smoothing processing, for suppressing noise (graininess) due to grains of a photosensitive material, or the like.

[0007] As the sharpness enhancement processing, there have heretofore been known unsharp masking processing, high-band enhancement filter processing, processing disclosed in, for example, U.S. Pat. No. 5,739,922, wherein high frequency components of an image are enhanced and middle frequency components of the image are suppressed, and the like. As the graininess suppression processing, there have heretofore been known median filter processing, hysteresis smoothing processing, noise removal processing with iteration, graininess suppression processing utilizing a morphology operation, and the like.

[0008] With the sharpness enhancement processing, the sharpness of an image can be enhanced, but the problems occur in that the graininess of the image is also enhanced and a rough feeling of the image remains. With the graininess suppression processing, the graininess of an image can be suppressed and a rough feeling of the image can be reduced, but the problems occur in that the extent of the sharpness of the image becomes low.

[0009] Therefore, a need exists for an image processing method, with which the graininess can be suppressed and, at the same time, the sharpness can be enhanced. There have heretofore been proposed various image processing methods, which aim at simultaneously achieving graininess suppression and sharpness enhancement. (Such image processing methods have been proposed in, for example, U.S. Pat. No. 4,812,903, and Japanese Unexamined Patent Publication Nos. 63(1988)-26783 and U.S. Pat. No. 5,739,922.) However, with the proposed image processing methods, actually, both the graininess suppression and the sharpness enhancement cannot simultaneously be achieved efficiently.

[0010] For example, the technique disclosed in U.S. Pat. No. 5,739,922 aims at suppressing the graininess and enhancing the sharpness by carrying out processing for enhancing high frequency components, which affect the image sharpness, and suppressing middle frequency components, which affect the graininess that gives a rough feeling. The disclosed technique utilizes the characteristics in that the degree of correlation between colors is low at a grainy area and is high at a not-grainy area (edge area). Also, with the disclosed processing, an image is spread in the frequency domain, intensities of middle and high frequency components are controlled, and the extent of them is kept low with respect to an area having a low level of color correlation. In this manner, graininess suppression is carried out. Specifically, with the disclosed processing method, since the factor of the connection between adjacent pixels in an image is not taken into consideration, the extent of the intensities of frequency components changes sharply at a certain area in the image, and it cannot be felt that the graininess has been suppressed. Therefore, the effects of graininess suppression are small in comparison with the effects of the sharpness enhancement. From the aspect of the effects, the disclosed processing is actually the sharpness enhancement processing, instead of being the graininess suppression and sharpness enhancement processing.

[0011] The primary object of the present invention is to provide an image processing method, wherein graininess suppression and sharpness enhancement for an image are capable of being simultaneously achieved efficiently.

[0012] Another object of the present invention is to provide an image processing method, wherein graininess suppression and sharpness enhancement for an image are capable of being simultaneously achieved efficiently, and graininess suppression processing is not carried out with respect to a predetermined image area that is not to be suppressed.

[0013] A further object of the present invention is to provide a graininess suppression image processing method, wherein positive graininess and negative graininess are suppressed accurately.

[0014] The specific object of the present invention is to provide an apparatus for carrying out the image processing method.

[0015] A first image processing method in accordance with the present invention is characterized by carrying out sharpness enhancement image processing and graininess suppression image processing, respectively, on an original image signal, carrying out processing on image signal components of two image signals having been obtained from the sharpness enhancement image processing and the graininess suppression image processing, which image signal components represent corresponding pixels in two images represented by the two image signals, and thereby obtaining a processed image signal.

[0016] Specifically, the present invention provides a first image processing method, wherein predetermined processing is carried out on an original image signal (or each of original image signals representing an original color image) fin, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal fout is thereby obtained, the method comprising the steps of:

[0017] i) carrying out graininess suppression image processing on the original image signal fin, a graininess suppression processing image signal f1 being obtained from the graininess suppression image processing,

[0018] ii) carrying out sharpness enhancement image processing on the original image signal fin, a sharpness enhancement processing image signal f2 being obtained from the sharpness enhancement image processing, and

[0019] iii) carrying out processing for obtaining the processed image signal fout in accordance with the graininess suppression processing image signal f1 and the sharpness enhancement processing image signal f2, the processed image signal fout being obtained with operation processing carried out on image signal components of the graininess suppression processing image signal f1 and the sharpness enhancement processing image signal f2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal f1 and the sharpness enhancement processing image signal f2.

[0020] Details of the operation processing may vary for the pixels or may be common to all of the pixels. Also, the details of the operation processing may be selected from a plurality of kinds of predetermined operations and in accordance with, for example, relationship with the value of the original image signal fin, the kind of the image, or the position of a picture element. Alternatively, information representing the details of the operation processing may be inputted from the exterior each time the processing is to be carried out.

[0021] Specifically, as the operation processing described above, the operation processing defined with Formula (1) shown below may be employed:

[0022] fout=w1·f1+w2·f2  (1)

[0023] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0024] Specifically, with the operation processing defined with Formula (1), the processed image signal fout is obtained by weighting the graininess suppression processing image signal f1 and the sharpness enhancement processing image signal f2 and adding the weighted image signals to each other. In the operation processing defined with Formula (1) shown above, the values of the weight factors w1 and w2 may be altered in accordance with the relationship with the value of the original image signal, or the like. Alternatively, the weight factors common to all of the pixels may be set.

[0025] Each of the weight factors w1 and w2 should preferably take a value falling within the range of 0.2 to 0.8, should more preferably take a value falling within the range of 0.3 to 0.7, and should most preferably take a value falling within the range of 0.4 to 0.6.

[0026] Alternatively, as the operation processing described above, the operation processing defined with Formula (2) shown below may be employed:

[0027] fout={|f1−fin|(f1−fin)+|f2−fin|(f2−fin)}/{|f1−fin|+|f2−fin|}+fin  (2)

[0028] wherein fin represents the original image signal, fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0029] Specifically, with the operation processing defined with Formula (2), as for a picture element, at which the value of the graininess suppression processing image signal is equal to the value of the original image signal, the value of the sharpness enhancement processing image signal is taken as the value of the processed image signal. As for a picture element, at which the value of the sharpness enhancement processing image signal is equal to the value of the original image signal, the value of the graininess suppression processing image signal is taken as the value of the processed image signal. As for a picture element, at which the value of the graininess suppression processing image signal and the value of the sharpness enhancement processing image signal are not equal to the value of the original image signal, the addition through weighting with the difference between the graininess suppression processing image signal or the sharpness enhancement processing image signal, whichever has a larger difference with respect to the original image signal, and the original image signal is carried out, and the value obtained from the weighting addition is taken as the value of the processed image signal. The operation processing defined with Formula (2) is one of the examples, wherein the processing is selected for each of the pixels in accordance with the relationship with the value of the original image signal (fin).

[0030] As the graininess suppression image processing (and the graininess suppression image processing carried out by a graininess suppression processing means, which will be described later), it is possible to employ median filter processing, hysteresis smoothing processing, noise removal processing with iteration, graininess suppression processing (smoothing processing) utilizing a morphology operation, or the like. As the sharpness enhancement image processing, it is possible to employ unsharp masking processing, high-band enhancement filter processing, processing disclosed in, for example, U.S. Pat. No. 5,739,922, wherein an image signal is separated into low frequency components, middle frequency components, and high frequency components, wherein enhancement and suppression processing is carried out such that the high frequency components may be enhanced and the middle frequency components may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another, or the like. As the graininess suppression image processing and the sharpness enhancement image processing, there should preferably be employed a combination of the graininess suppression processing (smoothing processing), which utilizes a morphology operation, and the sharpness enhancement processing (disclosed in, for example, U.S. Pat. No. 5,739,922, wherein the enhancement and suppression processing is carried out such that the high frequency components of an image may be enhanced and the middle frequency components of the image may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another.

[0031] In general, the morphology operation processing is expanded as the theory of sets in an N-dimensional space. Also, the morphology operation processing is often applied to an image in a two-dimensional space (as described in, for example, U.S. Ser. No. 08/623,223, U.S. Ser. No. 08/818,039 and Japanese Unexamined Patent Publication No. 9(1997)-91421). The morphology operation processing will hereinbelow be described briefly by taking a gray level image as an example.

[0032] The gray level image is considered as a space, in which a point having coordinates (x, y) has a height corresponding to an image density value f(x, y). Also, a one-dimensional function f(x) corresponding to the cross-section of the gray level image is considered. It is assumed that a structure element g used in the morphology operation processing is a symmetric function of Formula (5) shown below, which is symmetric with respect to the origin.

[0033] g^s(x)=g(−x)  (5)

[0034] It is also assumed that the value is 0 in a domain of definition G, which is represented by Formula (6) shown below.

[0035] G={−m, −m+1, . . . , −1, 0, 1, . . . , m−1, m}  (6)

[0036] In such cases, the fundamental forms of the morphology operation are very simple operations carried out with Formulas (7), (8), (9), and (10) shown below.

[0037] dilation; [f⊕G^s](i)=max {f(i−m), . . . , f(i), . . . , f(i+m)}  (7)

[0038] erosion; [f⊖G^s](i)=min {f(i−m), . . . , f(i), . . . , f(i+m)}  (8)

[0039] opening; f_g=(f⊖g^s)⊕g  (9)

[0040] closing; f^g=(f⊕g^s)⊖g  (10)

[0041] Specifically, as illustrated in FIG. 4A, the dilation processing is the processing for retrieving the maximum value in the region of a width of ±m (which width is the value determined in accordance with the structure element B and corresponds to the mask size shown in FIG. 4A) having its center at a picture element of interest. As illustrated in FIG. 4B, the erosion processing is the processing for retrieving the minimum value in the region of the width of ±m having its center at the picture element of interest. The opening processing is equivalent to the processing in which the dilation processing is carried out after the erosion processing, i.e., the processing in which the maximum value is searched after the searching of the minimum value. Also, the closing processing is equivalent to the processing in which the erosion processing is carried out after the dilation processing, i.e., the processing in which the minimum value is searched after the searching of the maximum value.

[0042] More specifically, as illustrated in FIG. 4C, the opening processing is equivalent to the processing for smoothing the image density curve f(x) from the low density side, and removing a convex image density fluctuating area (i.e., the area at which the density is higher than that of the surrounding areas), which fluctuates in a region spatially narrower than the mask size of 2 m.

[0043] Also, as illustrated in FIG. 4D, the closing processing is equivalent to the processing for smoothing the image density curve f(x) from the high density side, and removing a concave image density fluctuating area (i.e., the area at which the density is lower than that of the surrounding areas), which fluctuates in the region spatially narrower than the mask size of 2 m.

[0044] In cases where the image signal representing the image density value f(x) is a high image density-high signal level type of image signal, in which a high image density is represented by a high image signal level, the relationship between the image density value f(x) and the image signal value becomes reverse to the relationship between the image density value f(x) and the image signal value in the high luminance-high image signal level type of image signal. Therefore, the dilation processing, which is carried out on the high image density-high signal level type of image signal, coincides with the erosion processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 4B. The erosion processing, which is carried out on the high image density-high signal level type of image signal, coincides with the dilation processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 4A. The opening processing, which is carried out on the high image density-high signal level type of image signal, coincides with the closing processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 4D. Also, the closing processing, which is carried out on the high image density-high signal level type of image signal, coincides with the opening processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 4C.

[0045] In the manner described above, the opening processing or the closing processing with the morphology operation processing may be carried out on the image signal representing an original image. As a result, graininess (i.e., noise in the image signal) in the image can be suppressed (or removed). (Such features are described in, for example, “Morphology” by Kobatake, published by Corona Co.).

[0046] The morphology operation processing, which is preferable in the image processing method in accordance with the present invention, will be described later.

[0047] The sharpness enhancement processing (disclosed in, for example, U.S. Pat. No. 5,739,922, wherein the enhancement and suppression processing is carried out such that the high frequency components of an image may be enhanced and the middle frequency components of the image may be suppressed, will hereinbelow be described briefly.

[0048] With the disclosed sharpness enhancement processing, an image signal is separated into low frequency components, middle frequency components, and high frequency components, and enhancement and suppression processing is carried out such that the high frequency components may be enhanced and the middle frequency components may be suppressed. The high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are then combined with one another, and a processed image signal is thereby obtained.

[0049] FIG. 5 is a graph showing how low, middle, and high frequency components are distributed. The term “low frequency components, middle frequency components, and high frequency components of an image signal” as used herein means the frequency components, which are distributed in the patterns shown in FIG. 5. The middle frequency components are distributed such that they may have a peak in the vicinity of ⅓ of the Nyquist frequency of the output in the reproduction of a visible image from the signal having been processed. The low frequency components are distributed such that they may have a peak at a frequency, at which the Nyquist frequency of the output becomes equal to 0. The high frequency components are distributed such that they may have a peak at the Nyquist frequency of the output. At each frequency, the sum of the low, middle, and high frequency components becomes equal to 1.

[0050] In the disclosed sharpness enhancement processing, the processing should preferably be constituted such that, after the separation, luminance components may be extracted from the high frequency components and the middle frequency components, and the enhancement and suppression processing and the combining may be carried out in accordance with only the luminance components.

[0051] Also, in the disclosed sharpness enhancement processing, the processing should preferably be constituted such that a specific color region may be extracted from the given image, and the enhancement and suppression processing may be carried out such that the middle frequency components, which correspond to the specific color region, may be suppressed even further.

[0052] Further, in the disclosed sharpness enhancement processing, the processing should preferably be constituted such that, after the separation, a calculation may be made to find a rating value of the middle frequency components and/or the high frequency components, and the enhancement and suppression processing may be carried out such that the middle frequency components corresponding to pixels, which are associated with the rating values smaller than a predetermined threshold value, may be more suppressed than the middle frequency components corresponding to pixels, which are associated with the rating values larger than the predetermined threshold value, are. In such cases, the high frequency components corresponding to the pixels, which are associated with the rating values smaller than the predetermined threshold value, should preferably be less enhanced than the high frequency components corresponding to the pixels, which are associated with the rating values larger than the predetermined threshold value, are.

[0053] The term “rating value” as used herein means a value, such as a correlation value between at least a single set of two colors, which are among red, green, and blue three colors, as will be described later, or a local area limited variance value of the frequency components of the image signal.

[0054] The rating value should preferably be a correlation value between at least a single set of two colors, which are among red, green, and blue three colors, the correlation value being calculated from the middle frequency components and/or the high frequency components, which are of the respective colors and correspond to the same picture element.

[0055] Also, the rating value may be subjected to a filter processing with a median filter, and thereafter the enhancement and suppression processing may be carried out by using the predetermined threshold value and in accordance with the rating value, which has been obtained from the filter processing with the medial filter.

[0056] Further, the calculation of the middle frequency components and/or the high frequency components and the calculation of the rating value may be carried out respectively in accordance with different colors, which are among red, green, and blue three colors.

[0057] Furthermore, the degrees of the enhancement and the suppression in the enhancement and suppression processing should preferably be determined by being selected from a plurality of enhancement and suppression processing conditions, which have been determined previously in accordance with image reproducing conditions in the reproduction of a visible image from the processed image signal.

[0058] The term “image reproducing conditions” as used herein means the conditions, which affect the characteristics of the reproduced visible image, such as the kind of the recording medium on which the given image was recorded (e.g., negative film or reversal film), the size of the print on which the visible image is reproduced, or a correction amount which is specified by an operator from a keyboard such that a desired image processing may be carried out.

[0059] As described above, with the graininess suppression in the enhancement and suppression processing disclosed in U.S. Pat. No. 5,739,922, since the factor of the connection between adjacent pixels in an image is not taken into consideration, it cannot be felt that the graininess has been suppressed. However, the graininess suppression processing utilizing the morphology operation is the processing on the image plane utilizing the isolation characteristics of graininess (graininess has low continuity of image signal (image density) with adjacent pixels). Therefore, with the graininess suppression processing utilizing the morphology operation, the connection between adjacent pixels can be kept good, the extent of graininess suppression can be felt visually, and large effects of graininess suppression can be obtained.

[0060] The present invention also provides a first image processing apparatus for carrying out the first image processing method in accordance with the present invention. Specifically, the present invention also provides a first image processing apparatus, wherein predetermined processing is carried out on an original image signal, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal is thereby obtained, the apparatus comprising:

[0061] i) a graininess suppression processing means for carrying out graininess suppression image processing on the original image signal, a graininess suppression processing image signal being obtained from the graininess suppression image processing,

[0062] ii) a sharpness enhancement processing means for carrying out sharpness enhancement image processing on the original image signal, a sharpness enhancement processing image signal being obtained from the sharpness enhancement image processing, and

[0063] iii) a signal processing means for carrying out processing for obtaining the processed image signal in accordance with the graininess suppression processing image signal, which has been obtained from the graininess suppression processing means, and the sharpness enhancement processing image signal, which has been obtained from the sharpness enhancement processing means, the processed image signal being obtained with operation processing carried out on image signal components of the graininess suppression processing image signal and the sharpness enhancement processing image signal, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal and the sharpness enhancement processing image signal.

[0064] The operation processing, which is carried out by the signal processing means, may vary for the pixels or may be common to all of the pixels. However, the operation processing should preferably be set so as to vary for the pixels. In such cases, graininess can be suppressed appropriately and sharpness can be enhanced appropriately in accordance with details of a structure embedded in the image, or the like.

[0065] Also, as the operation processing, which is carried out by the signal processing means, the operation processing described above, which is defined with Formula (1) or Formula (2) shown above, should preferably be employed.

[0066] The first image processing method in accordance with the present invention should preferably be modified such that, after the graininess suppression image processing and the sharpness enhancement image processing have respectively been carried out on the original image signal, threshold value processing with a difference between the original image signal and the graininess suppression processing image signal may be carried out, and details of subsequent processing may be altered in accordance with the results of the threshold value processing. In cases where the degree of the graininess suppression is high (i.e., in cases where the difference is larger than a threshold value), graininess suppression processing may not be carried out. In cases where the degree of the graininess suppression is low (i.e., in cases where the difference is smaller than the threshold value), the operation processing described above may be carried out. (Such a modification constitutes a second image processing method in accordance with the present invention.)

[0067] Specifically, the present invention further provides a second image processing method, wherein predetermined processing is carried out on an original image signal (or each of original image signals representing an original color image) fin, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal fout is thereby obtained, the method comprising the steps of:

[0068] i) carrying out graininess suppression image processing on the original image signal fin, a graininess suppression processing image signal f1 being obtained from the graininess suppression image processing,

[0069] ii) carrying out sharpness enhancement image processing on the original image signal fin, a sharpness enhancement processing image signal f2 being obtained from the sharpness enhancement image processing,

[0070] iii) calculating a difference |f1−fin| between image signal components of the graininess suppression processing image signal f1 and the original image signal fin, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal f1 and the original image signal fin,

[0071] iv) comparing the calculated difference and a predetermined threshold value T with each other,

[0072] v) with respect to pixels, which have been found, as a result of the comparison, to be associated with the difference smaller than the predetermined threshold value (|f1−fin|<T), obtaining the processed image signal fout with the operation processing described above, i.e., with operation processing carried out on image signal components of the graininess suppression processing image signal f1 and the sharpness enhancement processing image signal f2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal f1 and the sharpness enhancement processing image signal f2, and

[0073] vi) with respect to pixels, which have been found, as a result of the comparison, to be associated with the difference larger than the predetermined threshold value (|f1−fin|>T), taking the original image signal fin, an enhancement processing signal Fin′, which is obtained by carrying out enhancement processing on the original image signal fin, or the sharpness enhancement processing image signal f2 as the processed image signal fout.

[0074] In the second image processing method in accordance with the present invention, with respect to pixels, which are associated with the difference |f1−fin| equal to the threshold value T (|f1−fin|=T), the operation processing employed with respect to the pixels, which are associated with the difference smaller than the threshold value T, may be carried out. Alternatively, with respect to such pixels, the processing employed with respect to the pixels, which are associated with the difference larger than the threshold value T, may be carried out. However, in such cases, the former processing (i.e., the operation processing employed with respect to the pixels, which are associated with the difference smaller than the threshold value T) should preferably be employed.

[0075] Also, the details of the operation processing may be selected from a plurality of kinds of predetermined operations and in accordance with, for example, the kind of the image or the position of a picture element. Alternatively, information representing the details of the operation processing may be inputted from the exterior each time the processing is to be carried out.

[0076] Specifically, in the second image processing method in accordance with the present invention, as the operation processing, the operation processing described above, which is defined with Formula (1) shown below, may be employed:

[0077] fout=w1·f1+w2·f2  (1)

[0078] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0079] Specifically, with the operation processing defined with Formula (1), the processed image signal fout is obtained by weighting the graininess suppression processing image signal f1 and the sharpness enhancement processing image signal f2 and adding the weighted image signals to each other. In the operation processing defined with Formula (1) shown above, the values of the weight factors w1 and w2 may be altered in accordance with the kind of the image, the position of a picture element, the relationship with the value of the original image signal, or the like. Alternatively, the weight factors common to all of the pixels may be set.

[0080] Each of the weight factors w1 and w2 should preferably take a value falling within the range of 0.2 to 0.8, should more preferably take a value falling within the range of 0.3 to 0.7, and should most preferably take a value falling within the range of 0.4 to 0.6.

[0081] Alternatively, in the second image processing method in accordance with the present invention, as the operation processing, the operation processing described above, which is defined with Formula (2) shown below, may be employed:

[0082] fout={|f1−fin|(f1−fin)+|f2−fin|(f2−fin)}/{|f1−fin|+|f2−fin|}+fin  (2)

[0083] wherein fin represents the original image signal, fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0084] Further, as in the first image processing method in accordance with the present invention, as the graininess suppression image processing, it is possible to employ median filter processing, hysteresis smoothing processing, noise removal processing with iteration, graininess suppression processing (smoothing processing) utilizing a morphology operation, or the like. As the sharpness enhancement image processing, it is possible to employ unsharp masking processing, high-band enhancement filter processing, processing disclosed in, for example, U.S. Pat. No. 5,739,922, wherein an image signal is separated into low frequency components, middle frequency components, and high frequency components, wherein enhancement and suppression processing is carried out such that the high frequency components may be enhanced and the middle frequency components may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another, or the like. As the graininess suppression image processing and the sharpness enhancement image processing, there should preferably be employed a combination of the graininess suppression processing (smoothing processing), which utilizes a morphology operation, and the sharpness enhancement processing (disclosed in, for example, U.S. Pat. No. 5,739,922, wherein the enhancement and suppression processing is carried out such that the high frequency components of an image may be enhanced and the middle frequency components of the image may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another.

[0085] The predetermined threshold value T, which is compared with the difference |f1−fin| between the graininess suppression processing image signal f1 and the original image signal fin, varies in accordance with, for example, the kind of the recording medium, on which the original image is recorded, and details (frequency, and the like) of an image area, which is not to be subjected to graininess suppression, such as a catch lights area, which is reflected in the pupil of the eye of a human body and is considered to be essential to portraits. From the results of experiments carried out by the inventors and experience of the inventors, the threshold value T should preferably be equal to a value, α×D (where 0.01≦α≦0.20), which falls within the range of 1% to 20% of the number D of gradation levels, which the original image signal fin can take. (In cases where the original image signal can take 10-bit gradation levels, D=2¹⁰=1,024 gradation levels, which correspond to signal values of 0 to 1,023.) The threshold value T should more preferably be equal to a value, α×D (where 0.02≦α≦0.10), which falls within the range of 2% to 10% of the number D of gradation levels, which the original image signal fin can take. In cases where the threshold value T takes a value falling within the range described above, the image area, which is not to be subjected to graininess suppression, such as a catch lights area, can be prevented from being subjected to the graininess suppression processing, and a processed image can be obtained, in which both the graininess suppression and the sharpness enhancement have been achieved more efficiently.

[0086] The present invention still further provides a second image processing apparatus for carrying out the second image processing method in accordance with the present invention. Specifically, the present invention still further provides a second image processing apparatus, wherein predetermined processing is carried out on an original image signal, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal is thereby obtained, the apparatus comprising:

[0087] i) a graininess suppression processing means for carrying out graininess suppression image processing on the original image signal, a graininess suppression processing image signal being obtained from the graininess suppression image processing,

[0088] ii) a sharpness enhancement processing means for carrying out sharpness enhancement image processing on the original image signal, a sharpness enhancement processing image signal being obtained from the sharpness enhancement image processing,

[0089] iii) a comparison means for calculating a difference between image signal components of the graininess suppression processing image signal, which has been obtained from the graininess suppression processing means, and the original image signal, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal and the original image signal fin, and comparing the calculated difference and a predetermined threshold value with each other, and

[0090] iv) a signal processing means for carrying out processing, which comprises the steps of:

[0091] a) with respect to pixels, which have been found, as a result of the comparison made by the comparison means, to be associated with the difference smaller than the predetermined threshold value, obtaining the processed image signal by carrying out operation processing on image signal components of the graininess suppression processing image signal and the sharpness enhancement processing image signal having been obtained from the sharpness enhancement processing means, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal and the sharpness enhancement processing image signal, and

[0092] b) with respect to pixels, which have been found, as a result of the comparison, to be associated with the difference larger than the predetermined threshold value, taking the original image signal, an enhancement processing signal, which is obtained by carrying out enhancement processing on the original image signal, or the sharpness enhancement processing image signal as the processed image signal.

[0093] In the second image processing apparatus in accordance with the present invention, as the operation processing, which is carried out by the signal processing means, the operation processing described above, which is defined with Formula (1) or Formula (2) shown below, should preferably be employed.

[0094] fout=w1·f1+w2·f2  (1)

[0095] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0096] fout={|f1−fin|(f1−fin)+|f2−fin|(f2−fin)}/{|f1−fin|+|f2−fin|}+fin  (2)

[0097] wherein fin represents the original image signal, fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0098] The predetermined threshold value T, which is compared with the difference |f1−fin| between the graininess suppression processing image signal f1 and the original image signal fin, should preferably be equal to a value, α×D (where 0.01≦α≦0.20), which falls within the range of 1% to 20% of the number D of gradation levels, which the original image signal fin can take. The threshold value T should more preferably be equal to a value, α×D (where 0.02≦α≦0.10), which falls within the range of 2% to 10% of the number D of gradation levels, which the original image signal fin can take.

[0099] Third and fourth image processing methods in accordance with the present invention are characterized by carrying out opening processing in accordance with a morphology operation and closing processing in accordance with a morphology operation, respectively, on an original image signal, carrying out operation processing on image signal components of two image signals having been obtained from the opening processing and the closing processing, which image signal components represent corresponding pixels in two images represented by the two image signals, and thereby obtaining a processed image signal (in this case, a graininess suppression processing image signal). The third and fourth image processing methods in accordance with the present invention are appropriate for the graininess suppression image processing utilizing the morphology operation described above.

[0100] Specifically, the present invention also provides a third image processing method, wherein predetermined processing is carried out on an original image signal (or each of a luminance signal and color difference signals, each of original image signals representing an original color image, or the like) fin, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal (in this case, a graininess suppression processing image signal) fout is thereby obtained, the method comprising the steps of:

[0101] i) carrying out opening processing in accordance with a morphology operation on the original image signal fin, an opening processing image signal fopn being obtained from the opening processing,

[0102] ii) carrying out closing processing in accordance with the morphology operation on the original image signal fin, a closing processing image signal fcls being obtained from the closing processing, and

[0103] iii) carrying out processing for obtaining the processed image signal (in this case, the graininess suppression processing image signal) fout in accordance with the opening processing image signal fopn and the closing processing image signal fcls, the processed image signal fout being obtained with operation processing g(fopn, fcls) (=fout) carried out on image signal components of the opening processing image signal fopn and the closing processing image signal fcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal fopn and the closing processing image signal fcls, the operation processing being defined with Formula (3): $\begin{array}{cc}\mathrm{fout}=\{\begin{array}{cc}\mathrm{fopn}& \left(\mathrm{if}\mathrm{fcls}=\mathrm{fin}\right)\\ \mathrm{fcls}& \left(\mathrm{if}\mathrm{fopn}=\mathrm{fin}\right)\\ \mathrm{w1}·\mathrm{fopn}+\mathrm{w2}·\mathrm{fcls}& \left(\mathrm{else}\right)\end{array}& \left(3\right)\end{array}$

[0104] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fin represents the original image signal, fout represents the processed image signal (in this case, the graininess suppression processing image signal), fopn represents the opening processing image signal, and fcls represents the closing processing image signal.

[0105] Specifically, with the operation processing defined with Formula (3), as for a picture element, at which the value of the closing processing image signal fcls is equal to the value of the original image signal fin, the value of the opening processing image signal fopn is taken as the value of the processed image signal fout. As for a picture element, at which the value of the opening processing image signal fopn is equal to the value of the original image signal fin, the value of the closing processing image signal fcls is taken as the value of the processed image signal fout. As for a picture element, at which the value of the closing processing image signal fcls and the value of the opening processing image signal fopn are not equal to the value of the original image signal fin, the addition processing through the weighting with the weight factors w1 and w2 is carried out on the closing processing image signal fcls and the opening processing image signal fopn.

[0106] As for how to determine the weight factors w1 and w2, one of various techniques for determination may be employed. Each of the weight factors w1 and w2 should preferably take a value falling within the range of 0.2 to 0.8, and should more preferably take a value falling within the range of 0.3 to 0.7. Each of the weight factors w1 and w2 should even further preferably take a value falling within the range of 0.4 to 0.6, and should most preferably take a value of 0.5.

[0107] The present invention further provides a fourth image processing method, wherein predetermined processing is carried out on an original image signal (or each of a luminance signal and color difference signals, each of original image signals representing an original color image, or the like) fin, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal (in this case, a graininess suppression processing image signal) fout is thereby obtained, the method comprising the steps of:

[0108] i) carrying out opening processing in accordance with a morphology operation on the original image signal fin, an opening processing image signal fopn being obtained from the opening processing,

[0109] ii) carrying out closing processing in accordance with the morphology operation on the original image signal fin, a closing processing image signal fcls being obtained from the closing processing, and

[0110] iii) carrying out processing for obtaining the processed image signal (in this case, the graininess suppression processing image signal) fout in accordance with the opening processing image signal fopn and the closing processing image signal fcls, the processed image signal fout being obtained with operation processing g(fopn, fcls) (=fout) carried out on image signal components of the opening processing image signal fopn and the closing processing image signal fcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal fopn and the closing processing image signal fcls, the operation processing being defined with Formula (4):

[0111] fout={|fopn−fin|(fopn−fin)+|fcls−fin|(fcls−fin)}/{|fopn−fin|+|fcls−fin|}+fin  (4)

[0112] Specifically, with the operation processing defined with Formula (4), as for a picture element, at which the value of the opening processing image signal fopn is equal to the value of the original image signal fin, the value of the closing processing image signal fcls is taken as the value of the processed image signal fout. As for a picture element, at which the value of the closing processing image signal fcls is equal to the value of the original image signal fin, the value of the opening processing image signal fopn is taken as the value of the processed image signal fout. As for a picture element, at which the value of the opening processing image signal fopn and the value of the closing processing image signal fcls are not equal to the value of the original image signal fin, the addition through weighting with the difference (|fopn−fin| or |fcls−fin|) between the opening processing image signal fopn or the closing processing image signal fcls, whichever has a larger difference with respect to the original image signal fin, (i.e., fopn or fcls) and the original image signal fin is carried out, and the value obtained from the weighting addition is taken as the value of the processed image signal fout.

[0113] The present invention still further provides a third image processing apparatus for carrying out the third image processing method in accordance with the present invention. Specifically, the present invention still further provides a third image processing apparatus, wherein predetermined processing is carried out on an original image signal, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal (in this case, a graininess suppression processing image signal) is thereby obtained, the apparatus comprising:

[0114] i) a morphology operation processing means for carrying out opening processing and closing processing in accordance with a morphology operation, respectively, on the original image signal, an opening processing image signal being obtained from the opening processing, a closing processing image signal being obtained from the closing processing, and

[0115] ii) a signal processing means for carrying out processing for obtaining the processed image signal (in this case, the graininess suppression processing image signal) in accordance with the opening processing image signal and the closing processing image signal, which have been obtained from the morphology operation processing means, the processed image signal being obtained with operation processing carried out on image signal components of the opening processing image signal and the closing processing image signal, which image signal components represent corresponding pixels in two images represented by the opening processing image signal and the closing processing image signal, the operation processing being defined with Formula (3) shown above.

[0116] In the third image processing apparatus in accordance with the present invention, as in the third image processing method in accordance with the present invention, one of various techniques for determining the weight factors w1 and w2, may be employed. Also, each of the weight factors w1 and w2 should preferably take a value falling within the range described above.

[0117] The present invention also provides a fourth image processing apparatus for carrying out the fourth image processing method in accordance with the present invention. Specifically, the present invention also provides a fourth image processing apparatus, wherein predetermined processing is carried out on an original image signal, which is made up of a series of image signal components defining values of pixels constituting an original image, and a processed image signal (in this case, a graininess suppression processing image signal) is thereby obtained, the apparatus comprising:

[0118] i) a morphology operation processing means for carrying out opening processing and closing processing in accordance with a morphology operation, respectively, on the original image signal, an opening processing image signal being obtained from the opening processing, a closing processing image signal being obtained from the closing processing, and

[0119] ii) a signal processing means for carrying out processing for obtaining the processed image signal (in this case, the graininess suppression processing image signal) in accordance with the opening processing image signal and the closing processing image signal, which have been obtained from the morphology operation processing means, the processed image signal being obtained with operation processing carried out on image signal components of the opening processing image signal and the closing processing image signal, which image signal components represent corresponding pixels in two images represented by the opening processing image signal and the closing processing image signal, the operation processing being defined with Formula (4) shown above.

[0120] In the third and fourth image processing methods and the third and fourth image processing apparatuses in accordance with the present invention, in order for the processing to be simplified, the structure element employed in the opening processing and the structure element employed in the closing processing should preferably have the same size and the same shape.

[0121] With the first image processing method and the first image processing apparatus in accordance with the present invention, each of the sharpness enhancement image processing and the graininess suppression image processing is carried out on the original image signal, and the operation processing is carried out on the image signal components of the two image signals having been obtained from the sharpness enhancement image processing and the graininess suppression image processing, which image signal components represent corresponding pixels in the two images represented by the two image signals. In this manner, the processed image signal is obtained. Therefore, the graininess suppression and the sharpness enhancement for the image can be simultaneously achieved more efficiently than with the conventional various methods or apparatuses.

[0122] With the first image processing method and the first image processing apparatus in accordance with the present invention, wherein the details of the operation processing vary for the pixels, instead of being set to be common to all of the pixels, graininess can be suppressed appropriately and sharpness can be enhanced appropriately in accordance with the details of structure patterns embedded in the image.

[0123] With the first image processing method and the first image processing apparatus in accordance with the present invention, wherein the operation processing defined with Formula (1) is employed, the weighted mean value of the graininess suppression processing image signal, in which graininess has been suppressed by the graininess suppression image processing, and the sharpness enhancement processing image signal, in which sharpness has been enhanced by the sharpness enhancement image processing, is calculated. In this manner, sharpness can be enhanced, while graininess is being suppressed.

[0124] With the first image processing method and the first image processing apparatus in accordance with the present invention, wherein the operation processing defined with Formula (2) is employed, as for a picture element, at which the value of the graininess suppression processing image signal is equal to the value of the original image signal, the value of the sharpness enhancement processing image signal is taken as the value of the processed image signal. As for a picture element, at which the value of the sharpness enhancement processing image signal is equal to the value of the original image signal, the value of the graininess suppression processing image signal is taken as the value of the processed image signal. As for a picture element, at which the value of the graininess suppression processing image signal and the value of the sharpness enhancement processing image signal are not equal to the value of the original image signal, the addition through weighting with the difference between the graininess suppression processing image signal or the sharpness enhancement processing image signal, whichever has a larger difference with respect to the original image signal, and the original image signal is carried out, and the value obtained from the weighting addition is taken as the value of the processed image signal. Therefore, as for a picture element corresponding to a flat image density area (i.e., an image area having little change in image density or luminance), at which graininess is more perceptible than sharpness is, the value of the graininess suppression processing image signal is taken as the value of the image signal representing the picture element, and graininess can thereby be suppressed. Also, as for a picture element corresponding to an image edge area (i.e., an image area having a sharp change in image density or luminance), at which sharpness is more perceptible than graininess is, the value of the sharpness enhancement processing image signal is taken as the value of the image signal representing the picture element, and sharpness can thereby be suppressed.

[0125] As the graininess suppression image processing and the sharpness enhancement image processing, there should preferably be employed a combination of the graininess suppression processing, which utilizes the morphology operation, and the sharpness enhancement processing (disclosed in U.S. Pat. No. 5,739,922, wherein the enhancement and suppression processing is carried out such that the high frequency components of the image may be enhanced and the middle frequency components of the image may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another. In such cases, the effects described below can be obtained.

[0126] Specifically, in particular, in cases where the operation processing defined with Formula (2) is employed, if graininess suppression has not been carried out on a sharpness enhancement area, and if sharpness enhancement is carried out extremely, graininess will also be enhanced, and the difference with respect to the original image signal will become larger than the difference between the graininess suppression processing image signal and the original image signal, and consequently graininess suppression will not be effected. Therefore, in cases where Formula (2) is employed, the sharpness enhancement processing disclosed in U.S. Pat. No. 5,739,922, in which the effects of graininess suppression can be obtained to a certain extent also with the sharpness enhancement processing, should preferably be employed.

[0127] Also, in cases where the image edge area has become dull due to the graininess suppression processing, such results affect the results of the sharpness enhancement processing. Therefore, the graininess suppression processing utilizing the morphology operation, with which graininess suppression can be carried out such that the image edge area may not become dull, should preferably be employed.

[0128] As described above, as the graininess suppression image processing, graininess suppression processing having the edge retaining effects, such as the graininess suppression processing utilizing the morphology operation, may be employed. As the sharpness enhancement image processing, sharpness enhancement processing having a certain extent of graininess suppression effects, such as the sharpness enhancement processing disclosed in U.S. Pat. No. 5,739,922, may be employed. Particularly, in cases where the operation processing defined with Formula (2) is employed, the graininess suppression processing and the sharpness enhancement processing described above should preferably be employed.

[0129] With the second image processing method and the second image processing apparatus in accordance with the present invention, each of the sharpness enhancement image processing and the graininess suppression image processing is carried out on the original image signal, and the operation processing is carried out on the image signal components of the two image signals having been obtained from the sharpness enhancement image processing and the graininess suppression image processing, which image signal components represent corresponding pixels in the two images represented by the two image signals. In this manner, the processed image signal is obtained. Therefore, the graininess suppression and the sharpness enhancement for the image can be simultaneously achieved more efficiently than with the conventional various methods or apparatuses. Also, before the operation processing is carried out, a calculation is made to find the difference between the image signal components of the graininess suppression processing image signal and the original image signal, which image signal components represent corresponding pixels in the two images represented by the graininess suppression processing image signal and the original image signal, and the calculated difference and the predetermined threshold value are compared with each other. With respect to pixels, which have been found, as a result of the comparison, to be associated with the difference larger than the predetermined threshold value, i.e. with respect to the pixels at which the degree of graininess suppression is high, the processing containing the aforesaid graininess suppression processing is not carried out, and the original image signal, an enhancement processing signal, which is obtained by carrying out enhancement processing on the original image signal, or the sharpness enhancement processing image signal is taken as the processed image signal. Therefore, the problems can be prevented from occurring in that, for example, the image area, which is not to be subjected to graininess suppression, such as a catch lights area, which is reflected in the pupil of the eye of a human body and is considered to be essential to portraits, is subjected, by mistake, to graininess suppression with the processing containing the aforesaid graininess suppression processing.

[0130] With the second image processing method and the second image processing apparatus in accordance with the present invention, wherein the operation processing defined with Formula (1) is employed, the weighted mean value of the graininess suppression processing image signal, in which graininess has been suppressed by the graininess suppression image processing, and the sharpness enhancement processing image signal, in which sharpness has been enhanced by the sharpness enhancement image processing, is calculated. In this manner, sharpness can be enhanced, while graininess is being suppressed.

[0131] With the second image processing method and the second image processing apparatus in accordance with the present invention, wherein the operation processing defined with Formula (2) is employed, the value, which is obtained from the weighting addition of the difference between the graininess suppression processing image signal and the original image signal and the difference between the sharpness enhancement processing image signal and the original image signal through weighting with the respective differences, is taken as the value of the processed image signal. Therefore, with respect to the image signal component representing a picture element corresponding to a flat image density area (i.e., an image area having little change in image density or luminance), at which graininess is more perceptible than sharpness is, the rate of contribution of the graininess suppression processing becomes high. Also, with respect to the image signal component representing a picture element corresponding to an image edge area (i.e., an image area having a sharp change in image density or luminance), at which sharpness is more perceptible than graininess is, the rate of contribution of sharpness enhancement processing becomes high. Accordingly, graininess can be suppressed such that the image edge area may not become dull.

[0132] As the graininess suppression image processing and the sharpness enhancement image processing, there should preferably be employed a combination of the graininess suppression processing, which utilizes the morphology operation, and the sharpness enhancement processing (disclosed in U.S. Pat. No. 5,739,922, wherein the enhancement and suppression processing is carried out such that the high frequency components of the image may be enhanced and the middle frequency components of the image may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another. In such cases, the effects described below can be obtained.

[0133] Specifically, in particular, in cases where the operation processing defined with Formula (2) is employed, if graininess suppression has not been carried out on a sharpness enhancement area, and if sharpness enhancement is carried out extremely, graininess will also be enhanced, and the difference with respect to the original image signal will become larger than the difference between the graininess suppression processing image signal and the original image signal, and consequently graininess suppression will not be effected. Therefore, in cases where Formula (2) is employed, the sharpness enhancement processing disclosed in U.S. Pat. No. 5,739,922, in which the effects of graininess suppression can be obtained to a certain extent also with the sharpness enhancement processing, should preferably be employed.

[0134] With the third and fourth image processing methods and the third and fourth image processing apparatuses in accordance with the present invention, each of the opening processing and the closing processing in accordance with the morphology operation is carried out on the original image signal. The opening processing image signal, in which a convex image density change area (hereinbelow referred to as the positive noise or positive graininess) has been suppressed, is obtained from the opening processing. Also, the closing processing image signal, in which a concave image density change area (hereinbelow referred to as the negative noise or negative graininess) has been suppressed, is obtained from the closing processing. The operation processing is then carried out on the image signal components of the two image signals having been obtained from the opening processing and the closing processing, which image signal components represent corresponding pixels in the two images represented by the two image signals, and the processed image signal (in this case, the graininess suppression processing image signal) is thereby obtained. Therefore, positive noise and negative noise contained in the original image can be suppressed accurately.

[0135] Also, with the third and fourth image processing methods and the third and fourth image processing apparatuses in accordance with the present invention, wherein the details of the operation processing vary for the pixels, instead of being set to be common to all of the pixels, noise can be suppressed appropriately in accordance with the details of structure patterns embedded in the image. Therefore, graininess can be suppressed more accurately, and the degree of visual graininess suppression can be kept higher than with conventional processing technique in which the mean value of the opening processing image signal and the closing processing image signal is employed for every picture element (i.e., the technique described in “Detail-Preserving Smoothing with Morphology and Fuzzy Reasoning,” Journal of Electronic Imaging, Vol. 5(3),pp. 396-401, Jul. 1996). Also, graininess can be suppressed with the technique simpler than a conventional technique in which a large number of reference tables are utilized (i.e., the technique referred to as the “Tsukasamoto's Fuzzy Models” and reported in the literature described above, wherein the difference between a closing processing image signal and an opening processing image signal is compared with a certain table, the corresponding area is judged as being grainy in cases where the difference is larger than a certain threshold value, the corresponding area is judged as being not grainy and being a fine structure pattern in the image in cases where the difference is smaller than the threshold value, and the area having been judged as being grainy is processed).

[0136] With the third image processing method and the third image processing apparatus in accordance with the present invention, wherein the operation processing defined with Formula (3) is employed, as for a picture element representing negative noise, which is not suppressed by the opening processing (i.e., the picture element, at which the value of the opening processing image signal is equal to the value of the original image signal), the value of the closing processing image signal is taken as the value of the processed image signal. Therefore, as for such a picture element, the processed image signal is obtained wherein negative noise has been suppressed. As for a picture element representing positive noise, which is not suppressed by the closing processing (i.e., the picture element, at which the value of the closing processing image signal is equal to the value of the original image signal), the value of the opening processing image signal is taken as the value of the processed image signal. Therefore, as for such a picture element, the processed image signal is obtained wherein positive noise has been suppressed. Accordingly, both the positive noise and the negative noise can be suppressed. As for a picture element, at which the value of the opening processing image signal and the value of the closing processing image signal are not equal to the value of the original image signal, the weighted mean value of the opening processing image signal and the closing processing image signal is taken as the value of the processed image signal. In this manner, positive noise and negative noise can be suppressed appropriately. In particular, in cases where the weighting ratio in the calculation of the weighted mean value is set to be 5:5, optimum graininess suppression can be achieved.

[0137] With the fourth image processing method and the fourth image processing apparatus in accordance with the present invention, wherein the operation processing defined with Formula (4) is employed, as for a picture element representing negative noise, which is not suppressed by the opening processing (i.e., the picture element, at which the value of the opening processing image signal is equal to the value of the original image signal), the value of the closing processing image signal is taken as the value of the processed image signal. Therefore, as for such a picture element, the processed image signal is obtained wherein negative noise has been suppressed. As for a picture element representing positive noise, which is not suppressed by the closing processing (i.e., the picture element, at which the value of the closing processing image signal is equal to the value of the original image signal), the value of the opening processing image signal is taken as the value of the processed image signal. Therefore, as for such a picture element, the processed image signal is obtained wherein positive noise has been suppressed. Accordingly, both the positive noise and the negative noise can be suppressed. As for a picture element, at which the value of the opening processing image signal and the value of the closing processing image signal are not equal to the value of the original image signal, the addition through weighting with the difference between the opening processing image signal fopn or the closing processing image signal fcls, whichever has a larger difference with respect to the original image signal fin, and the original image signal fin is carried out, and the value obtained from the weighting addition is taken as the value of the processed image signal. In this manner, positive noise and negative noise can be suppressed appropriately.

[0138] FIG. 1 is a flow chart showing how processing is carried out in a first embodiment of the image processing method in accordance with the present invention,

[0139] FIG. 2 is a block diagram showing an image processing apparatus for carrying out the first embodiment of FIG. 1,

[0140] FIG. 3A is a graph showing a distribution of an original image signal Rin,

[0141] FIG. 3B is a graph showing a distribution of a graininess suppression processing image signal R1, which is obtained from the original image signal Rin,

[0142] FIG. 3C is a graph showing a distribution of a sharpness enhancement processing image signal R2, which is obtained from the original image signal Rin,

[0143] FIG. 3D is a graph showing a distribution of a processed image signal Rout,

[0144] FIG. 4A is a graph showing how a dilation processing, which is one of fundamental morphology operations, is carried out,

[0145] FIG. 4B is a graph showing how an erosion processing, which is one of fundamental morphology operations, is carried out,

[0146] FIG. 4C is a graph showing how an opening processing, which is one of fundamental morphology operations, is carried out,

[0147] FIG. 4D is a graph showing how a closing processing, which is one of fundamental morphology operations, is carried out,

[0148] FIG. 5 is a graph showing how low, middle, and high frequency components are distributed,

[0149] FIG. 6 is a flow chart showing how processing is carried out in a third embodiment of the image processing method in accordance with the present invention,

[0150] FIG. 7 is a block diagram showing an image processing apparatus for carrying out the third embodiment of FIG. 6,

[0151] FIG. 8A is a graph showing a distribution of an original image signal Rin,

[0152] FIG. 8B is a graph showing a distribution of a graininess suppression processing image signal R1, which is obtained from the original image signal Rin,

[0153] FIG. 8C is a graph showing a distribution of a sharpness enhancement processing image signal R2, which is obtained from the original image signal Rin,

[0154] FIG. 8D is a graph showing the results of first operation processing,

[0155] FIG. 8E is a graph showing the results of second operation processing,

[0156] FIG. 8F is a graph showing a distribution of a processed image signal Rout, which is obtained ultimately,

[0157] FIG. 9 is a flow chart showing how processing is carried out in a fourth embodiment of the image processing method in accordance with the present invention,

[0158] FIG. 10 is a flow chart showing how processing is carried out in a sixth embodiment of the image processing method in accordance with the present invention,

[0159] FIG. 11 is a block diagram showing an image processing apparatus for carrying out the sixth embodiment of FIG. 10,

[0160] FIG. 12A is a graph showing a distribution of an original image signal Rin,

[0161] FIG. 12B is an enlarged view showing high frequency noise (an area A) in the original image signal Rin shown in FIG. 12A,

[0162] FIG. 12C is an enlarged view showing an area of an opening processing image signal Ropn obtained from the original image signal Rin, which area corresponds to the area A,

[0163] FIG. 12D is an enlarged view showing an area of a closing processing image signal Rcls obtained from the original image signal Rin, which area corresponds to the area A,

[0164] FIG. 12E is an enlarged view showing an area of a processed image signal Rout, which area corresponds to the area A,

[0165] FIGS. 13A, 13B, 13C, and 13D are explanatory views showing rectilinear structure elements, which constitute a multi-structure element,

[0166] FIG. 14 is a flow chart showing how processing is carried out in an eighth embodiment of the image processing method in accordance with the present invention,

[0167] FIG. 15 is a block diagram showing an image processing apparatus for carrying out the eighth embodiment of FIG. 14,

[0168] FIGS. 16A, 16B, 16C, and 16D are explanatory views showing rectilinear structure elements, which are utilized in opening processing and closing processing carried out by a morphology operation processing means 310 of the image processing apparatus shown in FIG. 15,

[0169] FIGS. 16E through 16L are explanatory views showing curvilinear structure elements, which are utilized in the opening processing and the closing processing carried out by the morphology operation processing means 310 of the image processing apparatus shown in FIG. 15, and

[0170] FIGS. 17A through 17H are explanatory views showing how a character pattern “A” and graininess are suppressed by rectilinear structure elements and a curvilinear structure element.

[0171] The present invention will hereinbelow be described in further detail with reference to the accompanying drawings.

[0172] FIG. 1 is a flow chart showing how processing is carried out in a first embodiment of the image processing method in accordance with the present invention. FIG. 2 is a block diagram showing an image processing apparatus for carrying out the first embodiment of FIG. 1.

[0173] With reference to FIGS. 1 and 2, the image processing apparatus comprises a graininess suppression processing means 11 for receiving RGB image signals Rin, Gin, and Bin, which represent a color image and have been acquired from photographing with a digital still camera or have been photoelectrically detected from a color photograph print or color film, and carrying out graininess suppression image processing on each of the RGB image signals Rin, Gin, and Bin. Graininess suppression processing image signals R1, G1, and B1 are obtained from the graininess suppression processing carried out by the graininess suppression processing means 11. The image processing apparatus also comprises a sharpness enhancement processing means 12 for carrying out sharpness enhancement image processing on each of the RGB image signals Rin, Gin, and Bin. Sharpness enhancement processing image signals R2, G2, and B2 are obtained from the sharpness enhancement processing carried out by the sharpness enhancement processing means 12. The image processing apparatus further comprises a signal processing means 20 for carrying out processing for obtaining processed image signals Rout, Gout, and Bout in accordance with the graininess suppression processing image signals R1, G1, and B1 and the sharpness enhancement processing image signals R2, G2, and B2. The processed image signal Rout is obtained with operation processing g(R1, R2) carried out on image signal components of the graininess suppression processing image signal R1 and the sharpness enhancement processing image signal R2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal R1 and the sharpness enhancement processing image signal R2. The processed image signal Gout is obtained with operation processing g(G1, G2) carried out on image signal components of the graininess suppression processing image signal G1 and the sharpness enhancement processing image signal G2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal G1 and the sharpness enhancement processing image signal G2. Also, the processed image signal Bout is obtained with operation processing g(B1, B2) carried out on image signal components of the graininess suppression processing image signal B1 and the sharpness enhancement processing image signal B2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal B1 and the sharpness enhancement processing image signal B2.

[0174] By way of example, as each operation processing g carried out by the signal processing means 20, the weighting addition processing represented by Formula (1) shown below may be employed.

[0175] fout=w1·f1+w2·f2  (1)

[0176] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0177] Specifically, g(f1, f2)=w1·f1+w2·f2.

[0178] The values of the weight factors w1 and w2 are set to vary for different pixels and are determined in accordance with, for example, the original image signals Rin, Gin, and Bin.

[0179] Specifically, as for a picture element corresponding to a flat image density area (i.e., an image area having little change in image density or luminance), at which graininess is more perceptible than sharpness is, the value of the weight factor w1 for the graininess suppression processing image signal is set to be larger than the value of the weight factor w2 for the sharpness enhancement processing image signal. Also, as for a picture element corresponding to an image edge area (i.e., an image area having a sharp change in image density or luminance), at which sharpness is more perceptible than graininess is, the value of the weight factor w2 for the sharpness enhancement processing image signal is set to be larger than the value of the weight factor w1 for the graininess suppression processing image signal.

[0180] In this embodiment, median filter processing is employed as the graininess suppression processing carried out by the graininess suppression processing means 11, and unsharp masking (USM) processing is employed as the sharpness enhancement processing carried out by the sharpness enhancement processing means 12.

[0181] How this embodiment of the image processing apparatus operates will be described hereinbelow by taking the R image signal Rin, which is one of the original image signals and has the distribution shown in FIG. 3A, as an example.

[0182] Firstly, each of the RGB image signals Rin, Gin, and Bin, which represent the color image, is fed from the digital still camera, or the like, into the graininess suppression processing means 11 and the sharpness enhancement processing means 12. As described above, the image signal Rin has the distribution shown in FIG. 3A. Each of the other image signals Gin and Bin also has a predetermined distribution (not shown).

[0183] The original image signal Rin represents an image edge area, at which the image density changes comparatively largely, and a flat image density area, at which the change in image density is comparatively small. High frequency noise is superposed upon the image density distribution. The high frequency noise is visually perceptible particularly at the flat image density area.

[0184] The graininess suppression processing means 11 carries out the median filter processing on each of the received original image signals Rin, Gin, and Bin and thereby calculates the graininess suppression processing image signals R1, G1, and B1. FIG. 3B shows the distribution of the graininess suppression processing image signal R1, which has been obtained by carrying out the graininess suppression processing on the original image signal Rin.

[0185] The sharpness enhancement processing means 12 carries out the USM processing on each of the received original image signals Rin, Gin, and Bin and thereby calculates the sharpness enhancement processing image signals R2, G2, and B2. FIG. 3C shows the distribution of the sharpness enhancement processing image signal R2, which has been obtained by carrying out the sharpness enhancement processing on the original image signal Rin.

[0186] As illustrated in FIG. 3B, in the graininess suppression processing image signal R1, graininess in the original image signal Rin has been suppressed. The graininess suppression processing image signal R1 is thus a smoothed signal. Also, as illustrated in FIG. 3C, in the sharpness enhancement processing image signal R2, the image density change areas in the original image signal Rin have been enhanced.

[0187] In each of the other graininess suppression processing image signals G1 and B1, as in the graininess suppression processing image signal R1, graininess has been suppressed and the signal has been smoothed. In each of the other sharpness enhancement processing image signals G2 and B2, as in the sharpness enhancement processing image signal R2, the image density change areas have been enhanced.

[0188] The graininess suppression processing image signals R1, G1, and B1, which have been calculated by the graininess suppression processing means 11, and the sharpness enhancement processing image signals R2, G2, B2, which have been calculated by the sharpness enhancement processing means 12, are fed into the signal processing means 20.

[0189] The signal processing means 20 carries out the weighting addition processing in accordance with Formula (1) shown above and on the received graininess suppression processing image signal and the received sharpness enhancement processing image signal, which correspond to each of the R, G, and B colors. The weighting addition processing is carried out on the image signal components of the graininess suppression processing image signal and the sharpness enhancement processing image signal corresponding to each color, which image signal components represent corresponding pixels in the two images represented by the graininess suppression processing image signal and the sharpness enhancement processing image signal. In this manner, the processed image signal is calculated with respect to each color. At this time, as described above, in Formula (1), as for a picture element corresponding to a flat image density area, at which graininess is more perceptible than sharpness is, the value of the weight factor w1 for the graininess suppression processing image signal is set to be larger than the value of the weight factor w2 for the sharpness enhancement processing image signal. Also, as for a picture element corresponding to an image edge area, at which sharpness is more perceptible than graininess is, the value of the weight factor w2 for the sharpness enhancement processing image signal is set to be larger than the value of the weight factor w1 for the graininess suppression processing image signal.

[0190] FIG. 3D shows the distribution of the processed image signal Rout, which is obtained by carrying out the weighting addition processing in accordance with Formula (1) and on the graininess suppression processing image signal R1, which is shown in FIG. 3B, and the sharpness enhancement processing image signal R2, which is shown in FIG. 3C.

[0191] As illustrated in FIG. 3D, with respect to the original image signal Rin shown in FIG. 3A, the processed image signal Rout has the characteristics described below. Specifically, at the flat image density area, at which graininess is comparatively perceptible, graininess has been suppressed. Also, at the image edge area, at which sharpness is comparatively perceptible, sharpness has been enhanced. Therefore, the graininess suppression and the sharpness enhancement can be simultaneously achieved efficiently.

[0192] As in the processed image signal Rout, each of the other processed image signals Gout and Bout can be obtained such that the graininess suppression and the sharpness enhancement can be achieved efficiently.

[0193] As described above, with this embodiment of the image processing apparatus, each of the graininess suppression processing and the sharpness enhancement processing is carried out on the given original image signal. The graininess suppression processing image signal, in which graininess has been suppressed, and the sharpness enhancement processing image signal, in which sharpness has been enhanced, are thereby obtained. Also, the operation processing is carried out on the image signal components of the graininess suppression processing image signal and the sharpness enhancement processing image signal, which image signal components represent corresponding pixels in the two images represented by the two image signals. In this manner, the processed image signal is obtained. Therefore, graininess in the original image can be suppressed efficiently, and sharpness can be enhanced efficiently.

[0194] In the first embodiment described above, as each operation processing g carried out by the signal processing means 20, the weighting addition processing represented by Formula (1) is employed. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to the use of the operation processing represented by Formula (1), and one of various other appropriate operation processings may be employed.

[0195] For example, in a second embodiment of the image processing apparatus in accordance with the present invention, the operation processing represented by Formula (2) shown below may be employed.

[0196] fout={|f1−fin|(f1−fin)+|f2−fin|(f2−fin)}/{|f1−fin|+|f2−fin|}+fin  (2)

[0197] wherein fin represents the original image signal, fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0198] Specifically, with the operation processing represented by Formula (2), as for a picture element, at which the value of the graininess suppression processing image signal R1 is equal to the value of the original image signal Rin, the value of the sharpness enhancement processing image signal R2 is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the sharpness enhancement processing image signal R2 is equal to the value of the original image signal Rin, the value of the graininess suppression processing image signal R1 is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the graininess suppression processing image signal R1 and the value of the sharpness enhancement processing image signal R2 are not equal to the value of the original image signal Rin, the addition through weighting with the difference (|R1−Rin| or |R2−Rin|) between the graininess suppression processing image signal R1 or the sharpness enhancement processing image signal R2, whichever has a larger difference (|R1−Rin| or |R2−Rin|) with respect to the original image signal Rin, and the original image signal Rin is carried out, and the value obtained from the weighting addition is taken as the value of the processed image signal Rout. As for each of the other G and B color image signals, processing is carried out in the same manner.

[0199] With the second embodiment, wherein the operation processing represented by Formula (2) is employed, as in the first embodiment, wherein the operation processing represented by Formula (1) is employed, graininess in the original image can be suppressed efficiently, and sharpness can be enhanced efficiently.

[0200] In the second embodiment, wherein the operation processing represented by Formula (2) is employed, there should preferably be employed a combination of the graininess suppression processing, which utilizes the morphology operation, and the sharpness enhancement processing (disclosed in U.S. Pat. No. 5,739,922, wherein the enhancement and suppression processing is carried out such that the high frequency components of the image may be enhanced and the middle frequency components of the image may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another.

[0201] FIG. 6 is a flow chart showing how processing is carried out in a third embodiment of the image processing method in accordance with the present invention. FIG. 7 is a block diagram showing an image processing apparatus for carrying out the third embodiment of FIG. 6.

[0202] With reference to FIGS. 6 and 7, the image processing apparatus comprises a graininess suppression processing means 111 for receiving RGB image signals Rin, Gin, and Bin (each having 1,024 gradation levels), which represent a color image and have been acquired from photographing with a digital still camera or have been photoelectrically detected with a scanner, or the like, from a color photograph print or color film, and carrying out graininess suppression image processing on each of the RGB image signals Rin, Gin, and Bin. Graininess suppression processing image signals R1, G1, and B1 are obtained from the graininess suppression processing carried out by the graininess suppression processing means 111. The image processing apparatus also comprises a sharpness enhancement processing means 112 for carrying out sharpness enhancement image processing on each of the RGB image signals Rin, Gin, and Bin. Sharpness enhancement processing image signals R2, G2, and B2 are obtained from the sharpness enhancement processing carried out by the sharpness enhancement processing means 112. The image processing apparatus further comprises a comparison means 130. With respect to the R signal, the comparison means 130 calculates a difference |R1−Rin| between image signal components of the graininess suppression processing image signal R1 and the original image signal Rin, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal R1 and the original image signal Rin. Also, with respect to the G and B signals, the comparison means 130 calculates differences |G1−Gin|, |B1−Bin| between image signal components of the graininess suppression processing image signals G1, B1 and the corresponding original image signals Gin, Bin, which image signal components represent corresponding pixels in the images represented by the graininess suppression processing image signals G1, B1 and the original image signals Gin, Bin. The comparison means 130 further compares each of the calculated differences and a corresponding threshold value, which is among threshold values α1·1,023, α2·1,023, and α3·1,023 having been set previously for the R, G, and B signals, with each other. The comparison means 130 feeds out information, which represents the results of the comparison. The image processing apparatus still further comprises a signal processing means 120 for carrying out predetermined operation processing in accordance with the results of the comparison made by the comparison means 130.

[0203] Each of the threshold values α1·1,023, α2·1,023, and α3·1,023, which have been set previously and stored in the comparison means 130, is the value corresponding to the number of gradation levels of each of the original image signals Rin, Gin, and Bin. In this embodiment, the coefficients α1, α2, and α3 are set to be 0.04 (i.e., 4% of the number of gradation levels). The coefficients α1, α2, and α3 need not necessarily take the same value and may take values varying for R, G, and B signals in accordance with the kind of the image represented by the original image signals, or the like. Also, the values of the coefficients α1, α2, and α3 are not limited to 0.04. However, ordinarily, the coefficients α1, α2, and α3 should preferably take values falling within the range of 0.01 to 0.20, and should more preferably take values falling within the range of 0.02 to 0.10.

[0204] With the signal processing means 120, the predetermined operation processing is carried out in the manner described below. Specifically, with respect to pixels, which have been found, as a result of the comparison made by the comparison means 130 with respect to all of the R, G, and B colors, to be associated with the differences smaller than or equal to the corresponding threshold values, first operation processing is carried out. With the first operation processing, processing is carried out for obtaining processed image signals Rout, Gout, and Bout in accordance with the graininess suppression processing image signals R1, G1, and B1 and the sharpness enhancement processing image signals R2, G2, and B2. The processed image signal Rout is obtained with operation processing g(R1, R2) carried out on image signal components of the graininess suppression processing image signal R1 and the sharpness enhancement processing image signal R2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal R1 and the sharpness enhancement processing image signal R2. The processed image signal Gout is obtained with operation processing g(G1, G2) carried out on image signal components of the graininess suppression processing image signal G1 and the sharpness enhancement processing image signal G2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal G1 and the sharpness enhancement processing image signal G2. Also, the processed image signal Bout is obtained with operation processing g(B1, B2) carried out on image signal components of the graininess suppression processing image signal B1 and the sharpness enhancement processing image signal B2, which image signal components represent corresponding pixels in two images represented by the graininess suppression processing image signal B1 and the sharpness enhancement processing image signal B2. With respect to pixels, which have been found, as a result of the comparison with respect to the R color, the G color, or the B color, to be associated with the difference larger than the threshold value, second operation processing is carried out. With the second operation processing, the sharpness enhancement processing image signals R2, G2, and B2 are taken, respectively, as the processed image signals Rout, Gout, and Bout.

[0205] By way of example, as the first operation processing g carried out by the signal processing means 120, the weighting addition processing represented by Formula (1) shown below may be employed.

[0206] fout=w1·f1+w2·f2  (1)

[0207] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0208] Specifically, g(f1, f2)=w1·f1+w2·f2.

[0209] The values of the weight factors w1 and w2 are set to vary for different pixels and are determined in accordance with, for example, the original image signals Rin, Gin, and Bin.

[0210] Specifically, as for a picture element corresponding to a flat image density area (i.e., an image area having little change in image density or luminance), at which graininess is more perceptible than sharpness is, the value of the weight factor w1 for the graininess suppression processing image signal f1 is set to be larger than the value of the weight factor w2 for the sharpness enhancement processing image signal f2. Also, as for a picture element corresponding to an image edge area (i.e., an image area having a sharp change in image density or luminance), at which sharpness is more perceptible than graininess is, the value of the weight factor w2 for the sharpness enhancement processing image signal f2 is set to be larger than the value of the weight factor w1 for the graininess suppression processing image signal f1.

[0211] In the third embodiment, the median filter processing is employed as the graininess suppression processing carried out by the graininess suppression processing means 111, and the unsharp masking (USM) processing is employed as the sharpness enhancement processing carried out by the sharpness enhancement processing means 112.

[0212] How the third embodiment of the image processing apparatus operates will be described hereinbelow by taking the R image signal Rin, which is among the R, G, and B signals, as an example.

[0213] Firstly, each of the RGB image signals Rin, Gin, and Bin, which represent the color image, is fed from the digital still camera, or the like, into the graininess suppression processing means 111 and the sharpness enhancement processing means 112. The image signal Rin has the distribution shown in FIG. 8A. Each of the other image signals Gin and Bin also has a predetermined distribution (not shown).

[0214] The original image signal Rin represents an image edge area, at which the image density changes comparatively largely, and a flat image density area, at which the change in image density is comparatively small. High frequency noise is superposed upon the image density distribution. The high frequency noise is visually perceptible particularly at the flat image density area. Also, at the flat image density area (e.g., the pattern of the pupil of the eye of a human body), a catch lights area, which is reflected in the pupil of the eye, is embedded.

[0215] The graininess suppression processing means 111 carries out the median filter processing on each of the received original image signals Rin, Gin, and Bin and thereby calculates the graininess suppression processing image signals R1, G1, and B1. FIG. 8B shows the distribution of the graininess suppression processing image signal R1, which has been obtained by carrying out the graininess suppression processing on the original image signal Rin.

[0216] The sharpness enhancement processing means 112 carries out the USM processing on each of the received original image signals Rin, Gin, and Bin and thereby calculates the sharpness enhancement processing image signals R2, G2, and B2. FIG. 8C shows the distribution of the sharpness enhancement processing image signal R2, which has been obtained by carrying out the sharpness enhancement processing on the original image signal Rin.

[0217] As illustrated in FIG. 8B, in the graininess suppression processing image signal R1, graininess in the original image signal Rin has been suppressed. The graininess suppression processing image signal R1 is thus a smoothed signal. Also, as illustrated in FIG. 8C, in the sharpness enhancement processing image signal R2, the image density change areas in the original image signal Rin have been enhanced.

[0218] In each of the other graininess suppression processing image signals G1 and B1, as in the graininess suppression processing image signal R1, graininess has been suppressed and the signal has been smoothed. In each of the other sharpness enhancement processing image signals G2 and B2, as in the sharpness enhancement processing image signal R2, the image density change areas have been enhanced.

[0219] The graininess suppression processing image signals R1, G1, and B1, which have been calculated by the graininess suppression processing means 111, are fed into the comparison means 130 together with the original image signals Rin, Gin, and Bin.

[0220] With respect to the R signal, the comparison means 130 calculates the difference |R1−Rin| between the image signal components of the graininess suppression processing image signal R1 and the original image signal Rin, which image signal components represent corresponding pixels in the two images represented by the graininess suppression processing image signal R1 and the original image signal Rin. Also, with respect to the G and B signals, the comparison means 130 calculates the differences |G1−Gin|, |B1−Bin| between the image signal components of the graininess suppression processing image signals G1, B1 and the corresponding original image signals Gin, Bin, which image signal components represent corresponding pixels in the images represented by the graininess suppression processing image signals G1, B1 and the original image signals Gin, Bin. The comparison means 130 further compares each of the calculated differences |R1−Rin|, |G1−Gin|, |B1−Bin| and the corresponding threshold value, which is among the threshold values α1·1,023 (for the R signal), α2·1,023 (for the G signal), and α3·1,023 (for the B signal) having been set previously for the R, G, and B signals, with each other. The comparison means 130 feeds the information, which represents the results of the comparison, into the signal processing means 120. At the grainy area shown in FIG. 8A, the values of the aforesaid differences are smaller than the threshold values. At the catch lights area shown in FIG. 8A, the values of the aforesaid differences are larger than the threshold values.

[0221] The signal processing means 120 carries out the first operation processing g or the second operation processing described above in accordance with the information representing the results of the comparison, which has been received from the comparison means 130.

[0222] In the cases of the first operation processing g, the signal processing means 120 carries out the weighting addition processing in accordance with Formula (1) shown above and on the received graininess suppression processing image signal and the received sharpness enhancement processing image signal, which correspond to each of the R, G, and B colors. The weighting addition processing is carried out on the image signal components of the graininess suppression processing image signal and the sharpness enhancement processing image signal corresponding to each color, which image signal components represent corresponding pixels in the two images represented by the graininess suppression processing image signal and the sharpness enhancement processing image signal. In this manner, the processed image signal is calculated with respect to each color. At this time, as described above, in Formula (1), as for a picture element corresponding to a flat image density area, at which graininess is more perceptible than sharpness is, the value of the weight factor w1 for the graininess suppression processing image signal is set to be larger than the value of the weight factor w2 for the sharpness enhancement processing image signal. Also, as for a picture element corresponding to an image edge area, at which sharpness is more perceptible than graininess is, the value of the weight factor w2 for the sharpness enhancement processing image signal is set to be larger than the value of the weight factor w1 for the graininess suppression processing image signal.

[0223] FIG. 8D shows the distribution of the processed image signal Rout, which is obtained by carrying out the weighting addition processing in accordance with Formula (1) and on the graininess suppression processing image signal R1, which is shown in FIG. 8B, and the sharpness enhancement processing image signal R2, which is shown in FIG. 8C.

[0224] As illustrated in FIG. 8D, with respect to the original image signal Rin shown in FIG. 8A, the processed image signal Rout has the characteristics described below. Specifically, at the flat image density area, at which graininess is comparatively perceptible, graininess has been suppressed. Also, at the image edge area, at which sharpness is comparatively perceptible, sharpness has been enhanced. Therefore, the graininess suppression and the sharpness enhancement can be simultaneously achieved efficiently. As in the processed image signal Rout, each of the other processed image signals Gout and Bout can be obtained such that the graininess suppression and the sharpness enhancement can be achieved efficiently.

[0225] If the catch lights area, which is to be processed with the second operation processing in the manner described later, were processed with the first operation processing g, the results indicated by the broken line in FIG. 8D would be obtained. In FIG. 8D, the catch lights area is assumptively illustrated for comparison with the results of the second operation processing.

[0226] In the cases of the second operation processing, with respect to each of the R, G, and B colors, the signal processing means 120 feeds out the sharpness enhancement processing image signal as the processed image signal. Therefore, as illustrated in FIG. 8E, as for the area corresponding to the catch lights area, the sharpness enhancement processing image signal is fed out as the processed image signal.

[0227] As illustrated in FIG. 8F, the processed image signal, which has been obtained from the aforesaid first operation processing or the aforesaid second operation processing carried out with respect to each picture element, is fed out from the signal processing means 120.

[0228] As described above, with the third embodiment of the image processing apparatus, graininess in the original image can be suppressed efficiently, and sharpness can be enhanced efficiently. Also, with the threshold value processing carried out by the comparison means 130, the predetermined image area, such as the catch lights area, can be prevented from being subjected to the graininess suppression processing as in the grainy area (as indicated by the broken line in FIG. 8D) and can thereby be retained accurately in the image after being processed.

[0229] In the third embodiment of the image processing apparatus, as illustrated in FIG. 6, as the technique for changing over whether the signal processing means 120 should carry out the first operation processing or the second operation processing on the received signal, the RGB batch change-over technique is employed. With the RGB batch change-over technique, in accordance with the results of the comparison may by the comparison means 130 with respect to each of the R, G, and B color image signals, with respect to a picture element, at which the aforesaid difference is larger than the threshold value with respect to at least one of the R, G and B color image signals, the second operation processing is carried out with respect to all of the R, G, and B color image signals. Also, with respect to a picture element, at which the aforesaid difference is smaller than the threshold value with respect to all of the R, G and B color image signals, the first operation processing is carried out with respect to all of the R, G, and B color image signals. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to the use of the change-over technique wherein, with respect to a single picture element, the same processing is employed for all of the R, G, and B color image signals. It is also possible to employ a change-over technique for utilizing different operation processings for the R, G, and B color image signals such that, for example, the first operation processing may be utilized for the R signal, and the second operation processing may be utilized for the G signal and the B signal.

[0230] Specifically, as in a fourth embodiment shown in FIG. 9, in accordance with the results of the comparison may by the comparison means 130 with respect to each of the R, G, and B color image signals, the operation processing may be changed over to the first operation processing or the second operation processing with respect to each of the R, G, and B color image signals. In this manner, with respect to a single picture element, different operation processings can be carried out for the R, G, and B color image signals.

[0231] Also, in the third embodiment of the image processing apparatus, the image processing is carried out on the color image signals, and the graininess suppression processing, the sharpness enhancement processing, and the threshold value processing, and the like, are carried out on each of the R, G, and B color image signals. However, the image processing method and the image processing apparatus in accordance with the present invention are also applicable when an image signal representing a gray level image is processed.

[0232] In the third embodiment described above, as the first operation processing g carried out by the signal processing means 120, the weighting addition processing represented by Formula (1) is employed. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to the use of the operation processing represented by Formula (1), and one of various other appropriate operation processings may be employed.

[0233] For example, in a fifth embodiment of the image processing apparatus in accordance with the present invention, the operation processing represented by Formula (2) shown below may be employed.

[0234] fout={|f1−fin|(f1−fin)+|f2−fin|(f2−fin)}/{|f1−fin|+|f2−fin|}+fin  (2)

[0235] wherein fin represents the original image signal, fout represents the processed image signal, f1 represents the graininess suppression processing image signal, and f2 represents the sharpness enhancement processing image signal.

[0236] Specifically, with the operation processing represented by Formula (2), the value, which is obtained from the weighting addition of the difference |R1−Rin| between the graininess suppression processing image signal R1 and the original image signal Rin and the difference |R2−Rin| between the sharpness enhancement processing image signal R2 and the original image signal Rin through weighting with the respective differences |R1−Rin|, |R2−Rin|, is taken as the value of the processed image signal Rout. Therefore, with respect to the image signal component representing a picture element corresponding to the flat image density area, at which graininess is more perceptible than sharpness is, the rate of contribution of the graininess suppression processing becomes high. Also, with respect to the image signal component representing a picture element corresponding to the image edge area, at which sharpness is more perceptible than graininess is, the rate of contribution of sharpness enhancement processing becomes high. Accordingly, graininess can be suppressed such that the image edge area may not become dull.

[0237] In the fifth embodiment, wherein the operation processing represented by Formula (2) is employed, there should preferably be employed a combination of the graininess suppression processing, which utilizes the morphology operation, and the sharpness enhancement processing (disclosed in U.S. Pat. No. 5,739,922, wherein the enhancement and suppression processing is carried out such that the high frequency components of the image may be enhanced and the middle frequency components of the image may be suppressed, and wherein the high frequency components and the middle frequency components, which have been obtained from the enhancement and suppression processing, and the low frequency components are combined with one another.

[0238] Experiments were carried out with respect to the coefficient (corresponding to the coefficients α1, α2, and α3) defining the threshold value T by taking a catch lights area in a pupil pattern of a human body image, which was photographed with film described in each of (1) and (2) below, as an example.

[0239] (1) An image was photographed with color film “G ACE800” (supplied by Fuji Photo Film Co., Ltd.) in a −2 under mode. With respect to a G signal Gin having 256 gradation levels (8 bits), which was acquired from the image, the threshold value T to be compared with |G1−Gin| was set to be 13. In such cases, the catch lights area could be prevented from being subjected to graininess suppression by mistake, and graininess suppression and sharpness enhancement on the image could simultaneously be achieved efficiently. At this time, the percentage of the threshold value T (=13) with respect to the number of gradation levels (=256) of the G signal was approximately 5%.

[0240] (2) An image was photographed with color film “G ACE400” (supplied by Fuji Photo Film Co., Ltd.) in a normal mode. With respect to a G signal Gin having 256 gradation levels (8 bits), which was acquired from the image, the threshold value T to be compared with |G1−Gin| was set to be 26. In such cases, the catch lights area could be prevented from being subjected to graininess suppression by mistake, and graininess suppression and sharpness enhancement on the image could simultaneously be achieved efficiently. At this time, the percentage of the threshold value T (=26) with respect to the number of gradation levels (=256) of the G signal was approximately 10%.

[0241] In the experiments described above, only the center picture element in the catch lights area was processed as the catch lights area. In the actual image, it is necessary that, besides the center of the pixel in the catch lights area, pixels adjacent to the center of the pixel in the catch lights area are also processed as the catch lights area, when necessary. Therefore, the threshold value was set at various different values in accordance with the results of the experiments described above, and an optimum threshold value was thereby found. Also, the coefficient value α appropriate for the number of gradation levels was set at a value falling within the range of 0.01 (1%) to 0.20 (20%), and more preferably at a value falling within the range of 0.02 (2%) to 0.10 (10%).

[0242] FIG. 10 is a flow chart showing how processing is carried out in a sixth embodiment of the image processing method in accordance with the present invention. FIG. 11 is a block diagram showing an image processing apparatus for carrying out the sixth embodiment of FIG. 10.

[0243] With reference to FIGS. 10 and 11, the image processing apparatus comprises a morphology operation processing means 210 for receiving RGB image signals Rin, Gin, and Bin, which represent a color image and have been acquired from photographing with a digital still camera or have been photoelectrically detected from a color photograph print or color film, and carrying out opening processing and closing processing in accordance with a morphology operation on each of the RGB image signals Rin, Gin, and Bin. Opening processing image signals Ropn, Gopn, and Bopn are obtained from the opening processing carried out by the morphology operation processing means 210. Also, closing processing image signals Rcls, Gcls, and Bcls are obtained from the closing processing carried out by the morphology operation processing means 210. The image processing apparatus also comprises a signal processing means 220 for carrying out processing for obtaining processed image signals (in this case, graininess suppression processing image signals) Rout, Gout, and Bout in accordance with the opening processing image signals Ropn, Gopn, and Bopn and the closing processing image signals Rcls, Gcls, and Bcls. The processed image signal Rout is obtained with operation processing g(Ropn, Rcls) carried out on image signal components of the opening processing image signal Ropn and the closing processing image signal Rcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal Ropn and the closing processing image signal Rcls. The processed image signal Gout is obtained with operation processing g(Gopn, Gcls) carried out on image signal components of the opening processing image signal Gopn and the closing processing image signal Gcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal Gopn and the closing processing image signal Gcls. Also, the processed image signal Bout is obtained with operation processing g(Bopn, Bcls) carried out on image signal components of the opening processing image signal Bopn and the closing processing image signal Bcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal Bopn and the closing processing image signal Bcls.

[0244] By way of example, as each operation processing g carried out by the signal processing means 220, the operation processing represented by Formula (3) shown below may be employed. $\begin{array}{cc}\mathrm{fout}=\{\begin{array}{cc}\mathrm{fopn}& \left(\mathrm{if}\mathrm{fcls}=\mathrm{fin}\right)\\ \mathrm{fcls}& \left(\mathrm{if}\mathrm{fopn}=\mathrm{fin}\right)\\ \mathrm{w1}·\mathrm{fopn}+\mathrm{w2}·\mathrm{fcls}& \left(\mathrm{else}\right)\end{array}& \left(3\right)\end{array}$

[0245] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fin represents the original image signal, fout represents the processed image signal (in this case, the graininess suppression processing image signal), fopn represents the opening processing image signal, and fcls represents the closing processing image signal.

[0246] The symbol “f” in Formula (3) is an ordinary representation of the signal value. In cases where the color image signals are to be processed in this embodiment, the symbol “f” serving as the ordinary representation is replaced by each of R, G, and B representing the color image signals.

[0247] Specifically, with the operation processing defined with Formula (3), as for a picture element, at which the value of the closing processing image signal Rcls is equal to the value of the original image signal Rin, the value of the opening processing image signal Ropn is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the opening processing image signal Ropn is equal to the value of the original image signal Rin, the value of the closing processing image signal Rcls is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the opening processing image signal Ropn and the value of the closing processing image signal Rcls are not equal to the value of the original image signal Rin, the addition processing through the weighting with the weight factors w1 and w2 is carried out on the opening processing image signal Ropn and the closing processing image signal Rcls. With respect to each of the other G and B image signals, the operation processing is carried out in the same manner.

[0248] In this embodiment, the weight factors w1 and w2 are set to be 0.5.

[0249] As a structure element (i.e., a mask) utilized for each of the opening processing and the closing processing carried out by the morphology operation processing means 210, a multi-structure element, which is constituted of four rectilinear structure elements shown in FIGS. 13A, 13B, 13C, and 13D, may be employed. (The multi-structure element has a mask size of 5 pixels×5 pixels. Alternatively, a multi-structure element, which is constituted of curvilinear structure elements that will be described later, or a multi-structure element, which is constituted of a combination of rectilinear structure elements and curvilinear structure elements, may be employed.) In the opening processing utilizing the multi-structure element, opening processing steps, each utilizing one of the structure elements constituting the multi-structure element, are carried out. Also, in the closing processing utilizing the multi-structure element, closing processing steps, each utilizing one of the structure elements constituting the multi-structure element, are carried out. In the cases of the opening processing, the maximum value of the values obtained from the opening processing steps is taken up. In the cases of the closing processing, the minimum value of the values obtained from the closing processing steps is taken up. The opening processing and the closing processing are useful when the signal values distributed in two-dimensional directions, as in the signal values representing an image, are to be processed. The mask size (i.e., the size of the structure element) may be altered in accordance with the size of the grainy area to be deleted. As for ordinary images, a multi-structure element having a size falling within the range of 3 pixels×3 pixels to 13 pixels×13 pixels should preferably be employed. A multi-structure element having a size falling within the range of 3 pixels×3 pixels to 7 pixels×7 pixels should more preferably be employed.

[0250] As an aid in facilitating the explanation, the processing will hereinbelow be described by taking a one-dimensional signal distribution, which corresponds to a cross-section of a two-dimensional image, as an example. Therefore, the processing with one of the structure elements constituting the multi-structure element will be described hereinbelow.

[0251] How the sixth embodiment of the image processing apparatus operates will be described hereinbelow by taking the R image signal Rin, which is one of the original image signals and has the distribution shown in FIG. 12A, as an example.

[0252] Firstly, each of the RGB image signals Rin, Gin, and Bin, which represent the color image, is fed from the digital still camera, or the like, into the morphology operation processing means 210. As described above, the image signal Rin has the distribution shown in FIG. 12A. Each of the other image signals Gin and Bin also has a predetermined distribution (not shown).

[0253] The original image signal Rin has an image density distribution with a comparatively large period, and noise (an area A in FIG. 12A) having frequency higher than the frequency of the image density distribution is superposed upon the image density distribution. FIG. 12B is an enlarged view showing high frequency noise (the area A) in the original image signal Rin shown in FIG. 12A.

[0254] The morphology operation processing means 210 carries out the opening processing and the closing processing in accordance with the morphology operation, in which the predetermined structure element is utilized, on each of the received original image signals Rin, Gin, and Bin and thereby calculates the opening processing image signals Ropn, Gopn, Bopn and the closing processing image signals Rcls, Gcls, Bcls. FIG. 12C is an enlarged view showing an area of the opening processing image signal Ropn obtained from the opening processing carried out on the original image signal Rin, which area corresponds to FIG. 12B (i.e., the area A in FIG. 12A). FIG. 12D is an enlarged view showing an area of the closing processing image signal Rcls obtained from the closing processing carried out on the original image signal Rin, which area corresponds to FIG. 12B (i.e., the area A in FIG. 12A).

[0255] As illustrated in FIG. 12C, in the opening processing image signal Ropn, convex image density change areas (positive noise) in the original image signal Rin (indicated by the broken lines in FIG. 12C) have been suppressed. Also, as illustrated in FIG. 12D, in the closing processing image signal Rcls, concave image density change areas (negative noise) in the original image signal Rin (indicated by the broken lines in FIG. 12D) have been suppressed.

[0256] In each of the other opening processing image signals Gopn and Bopn, as in the opening processing image signal Ropn, positive noise has been suppressed. In each of the other closing processing image signals Gcls and Bcls, as in the closing processing image signal Rcls, negative noise has been suppressed.

[0257] The opening processing image signals Ropn, Gopn, Bopn and the closing processing image signals Rcls, Gcls, Bcls, which have been calculated by the morphology operation processing means 210, are fed into the signal processing means 220.

[0258] The signal processing means 220 carries out the operation processing in accordance with Formula (3) shown above and on the received opening processing image signal and the received closing processing image signal, which correspond to each of the R, G, and B colors. The operation processing is carried out on the image signal components of the opening processing image signal and the closing processing image signal corresponding to each color, which image signal components represent corresponding pixels in the two images represented by the opening processing image signal and the closing processing image signal. In this manner, the processed image signal is calculated with respect to each color.

[0259] FIG. 12E shows the distribution of the processed image signal Rout, which is obtained by carrying out the operation processing in accordance with Formula (3) and on the opening processing image signal Ropn, which is shown in FIG. 12C, and the closing processing image signal Rcls, which is shown in FIG. 12D.

[0260] As illustrated in FIG. 12E, in the processed image signal Rout, both of positive noise and negative noise in the original image signal Rin (indicated by the broken lines in FIG. 12E) have been suppressed accurately. In this embodiment, the values of the weight factors w1 and w2 are set to be 0.5. However, the values of the weight factors w1 and w2 are not limited to 0.5, and may be set at various different values falling within an appropriate range.

[0261] As in the processed image signal Rout, each of the other processed image signals Gout and Bout can be obtained such that both of positive noise and negative noise may be suppressed accurately.

[0262] As described above, with the sixth embodiment of the image processing apparatus, each of the opening processing and the closing processing in accordance with the morphology operation is carried out on the given original image signal. The opening processing image signal, in which the positive noise has been suppressed, is obtained from the opening processing. Also, the closing processing image signal, in which the negative noise has been suppressed, is obtained from the closing processing. The operation processing is then carried out on the image signal components of the two image signals having been obtained from the opening processing and the closing processing, which image signal components represent corresponding pixels in the two images represented by the two image signals, and the processed image signal (in this case, the graininess suppression processing image signal) is thereby obtained. Therefore, positive noise and negative noise contained in the original image can be suppressed accurately.

[0263] In the sixth embodiment described above, as each operation processing g carried out by the signal processing means 220, the operation processing represented by Formula (3) is employed. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to the use of the operation processing represented by Formula (3), and one of various other appropriate operation processings may be employed.

[0264] For example, in a seventh embodiment of the image processing apparatus in accordance with the present invention, the operation processing represented by Formula (4) shown below may be employed.

[0265] fout={|fopn−fin|(fopn−fin)+|fcls−fin|(fcls−fin)}/{|fopn−fin|+|fcls−fin|}+fin  (4)

[0266] Specifically, with the operation processing defined with Formula (4), as for a picture element, at which the value of the opening processing image signal Ropn is equal to the value of the original image signal Rin, the value of the closing processing image signal Rcls is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the closing processing image signal Rcls is equal to the value of the original image signal Rin, the value of the opening processing image signal Ropn is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the opening processing image signal Ropn and the value of the closing processing image signal Rcls are not equal to the value of the original image signal Rin, the addition through weighting with the difference (|Ropn−Rin| or |Rcls−Rin|) between the opening processing image signal Ropn or the closing processing image signal Rcls, whichever has a larger difference with respect to the original image signal Rin, (i.e., Ropn or Rcls) and the original image signal Rin is carried out, and the value obtained from the weighting addition is taken as the value of the processed image signal (in this case, the graininess suppression processing image signal) Rout. As for each of the other G and B color image signals, processing is carried out in the same manner.

[0267] With the seventh embodiment, wherein the operation processing represented by Formula (4) is employed, as in the sixth embodiment, wherein the operation processing represented by Formula (3) is employed, positive noise and negative noise can be suppressed accurately.

[0268] In the sixth embodiment, the R, G, and B image signals representing a color image are processed. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to the processing of the color image signals and are also applicable when various other signals, such as a luminance signal and color difference signals, are processed.

[0269] FIG. 14 is a flow chart showing how processing is carried out in an eighth embodiment of the image processing method in accordance with the present invention. FIG. 15 is a block diagram showing an image processing apparatus for carrying out the eighth embodiment of FIG. 14.

[0270] With reference to FIGS. 14 and 15, the image processing apparatus comprises a morphology operation processing means 310. The morphology operation processing means 310 receives RGB image signals Rin, Gin, and Bin, which represent a color image and have been acquired from photographing with a digital still camera or have been photoelectrically detected from a color photograph print or color film, and carries out the opening processing and the closing processing in accordance with the morphology operation on each of the RGB image signals Rin, Gin, and Bin. In the opening processing, opening processing steps, each utilizing one of rectilinear structure elements, which are shown in FIGS. 16A, 16B, 16C, and 16D, and curvilinear structure elements, which are shown in FIGS. 16E through 16L, are carried out. Also, in the closing processing, closing processing steps, each utilizing one of the structure elements shown in FIGS. 16A through 16L, are carried out. Signals Ro1, . . . , Ro12, Go1, . . . , Go12, Bo1, . . . , Bo12 are obtained from the opening processing steps. Also, signals Rc1, . . . , Rc12, Gc1, . . . , Gc12, Bc1, . . . , Bc12 are obtained from the opening processing steps. The maximum signal among the signals Ro1, . . . , Ro12, the maximum signal among the signals Go1, . . . , Go12, and the maximum signal among the signals Bo1, . . . , Bo12 are extracted, respectively, as opening processing image signals Ropn, Gopn, and Bopn. Also, the minimum signal among the signals Rc1, . . . , Rc12, the minimum signal among the signals Gc1, . . . , Gc12, and the minimum signal among the signals Bc1, . . . , Bc12 are extracted, respectively, as closing processing image signals Rcls, Gcls, and Bcls. The image processing apparatus also comprises a signal processing means 320 for carrying out processing for obtaining processed image signals (in this case, graininess suppression processing image signals) Rout, Gout, and Bout in accordance with the thus extracted opening processing image signals Ropn, Gopn, and Bopn and the thus extracted closing processing image signals Rcls, Gcls, and Bcls. The processed image signal Rout is obtained with operation processing g(Ropn, Rcls) carried out on image signal components of the opening processing image signal Ropn and the closing processing image signal Rcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal Ropn and the closing processing image signal Rcls. The processed image signal Gout is obtained with operation processing g(Gopn, Gcls) carried out on image signal components of the opening processing image signal Gopn and the closing processing image signal Gcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal Gopn and the closing processing image signal Gcls. Also, the processed image signal Bout is obtained with operation processing g(Bopn, Bcls) carried out on image signal components of the opening processing image signal Bopn and the closing processing image signal Bcls, which image signal components represent corresponding pixels in two images represented by the opening processing image signal Bopn and the closing processing image signal Bcls.

[0271] By way of example, as each operation processing g carried out by the signal processing means 320, the operation processing represented by Formula (11) shown below may be employed.

[0272] fout=w1·fopn+w2·fcls  (11)

[0273] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fout represents the processed image signal (in this case, the graininess suppression processing image signal), fopn represents the opening processing image signal, and fcls represents the closing processing image signal.

[0274] The symbol “f” in Formula (11) is an ordinary representation of the signal value. In cases where the color image signals are to be processed in this embodiment, the symbol “f” serving as the ordinary representation is replaced by each of R, G, and B representing the color image signals.

[0275] Specifically, with the operation processing defined with Formula (11), the addition processing through the weighting with the weight factors w1 and w2 is carried out on the opening processing image signal Ropn and the closing processing image signal Rcls. An addition processing image signal, which has thus been obtained, is taken as the processed image signal fout.

[0276] As for how to determine the weight factors w1 and w2, one of various techniques for determination may be employed. In this embodiment, the weight factors w1 and w2 are set at the most preferable values, i.e. w1=w2=0.5. With respect to each of the other G and B image signals, the weight factors w1 and w2 are set in the same manner. the operation processing is carried out in the same manner.

[0277] As the structure elements (i.e., the masks) utilized for each of the opening processing and the closing processing carried out by the morphology operation processing means 310, the four rectilinear structure elements, which are shown in FIGS. 16A, 16B, 16C, and 16D, and the eight curvilinear structure elements, which are shown in FIGS. 16E through 16L, are employed. Each of these structure elements is set to have a size falling within the range of 5 pixels (along the column in the array of the pixels)×5 pixels (along the row in the array of the pixels). The size of the structure element may be altered in accordance with the size of the grainy area to be deleted. As for ordinary images, a structure element having a size falling within the range of 3 pixels×3 pixels to 13 pixels×13 pixels should preferably be employed.

[0278] As an aid in facilitating the explanation, the processing will hereinbelow be described by taking a one-dimensional signal distribution, which corresponds to a cross-section of a two-dimensional image, as an example, except for a certain part.

[0279] How the eighth embodiment of the image processing apparatus operates will be described hereinbelow by taking the R image signal Rin, which is one of the original image signals and has the distribution shown in FIG. 12A, as an example.

[0280] Firstly, each of the RGB image signals Rin, Gin, and Bin, which represent the color image, is fed from the digital still camera, or the like, into the morphology operation processing means 310. As described above, the image signal Rin has the distribution shown in FIG. 12A. Each of the other image signals Gin and Bin also has a predetermined distribution (not shown).

[0281] The original image signal Rin has an image density distribution with the comparatively large period, and noise (the area A in FIG. 12A) having frequency higher than the frequency of the image density distribution is superposed upon the image density distribution. FIG. 12B is an enlarged view showing high frequency noise (the area A) in the original image signal Rin shown in FIG. 12A.

[0282] The morphology operation processing means 310 carries out the opening processing and the closing processing in accordance with the morphology operation, in which each of the structure elements shown in FIGS. 16A through 16L are utilized, on each of the received original image signals Rin, Gin, and Bin.

[0283] By way of example, the opening processing image signals, which have been obtained by carrying out the opening processing on the original image signals Rin, Gin, and Bin by using the rectilinear structure element shown in FIG. 16A, are represented by Ro1, Go1, and Bo1. The opening processing image signals, which have been obtained by carrying out the opening processing on the original image signals Rin, Gin, and Bin by using the rectilinear structure element shown in FIG. 16B, are represented by Ro2, Go2, and Bo2. The opening processing image signals, which have been obtained by carrying out the opening processing on the original image signals Rin, Gin, and Bin by using the structure elements shown in FIGS. 16C through 16L, are represented by Ro3˜Ro12, Go3˜Go12, and Bo3˜Bo12. Also, the closing processing image signals, which have been obtained by carrying out the closing processing on the original image signals Rin, Gin, and Bin by using the rectilinear structure element shown in FIG. 16A, are represented by Rc1, Gc1, and Bc1. The closing processing image signals, which have been obtained by carrying out the closing processing on the original image signals Rin, Gin, and Bin by using the rectilinear structure element shown in FIG. 16B, are represented by Rc2, Gc2, and Bc2. The closing processing image signals, which have been obtained by carrying out the closing processing on the original image signals Rin, Gin, and Bin by using the structure elements shown in FIGS. 16C through 16L, are represented by Rc3˜Rc12, Gc3˜Gc12, and Bc3˜Bc12.

[0284] Thereafter, with respect to each of the R, G, and B colors, the morphology operation processing means 310 extracts the maximum signal among the signals Ro1˜Ro12, the maximum signal among the signals Go1˜Go12, and the maximum signal among the signals Bo1˜Bo12, respectively, as the opening processing image signals Ropn, Gopn, and Bopn. Also, the morphology operation processing means 310 extracts the minimum signal among the signals Rc1˜Rc12, the minimum signal among the signals Gc1˜Gc12, and the minimum signal among the signals Bc1˜Bc12, respectively, as the closing processing image signals Rcls, Gcls, and Bcls.

[0285] FIG. 12C shows the area of the opening processing image signal Ropn, which area corresponds to FIG. 12B (i.e., the area A in FIG. 12A). FIG. 12D shows the area of the closing processing image signal Rcls, which area corresponds to FIG. 12B (i.e., the area A in FIG. 12A).

[0286] As illustrated in FIG. 12C, in the opening processing image signal Ropn, convex image density change areas (positive noise) in the original image signal Rin (indicated by the broken lines in FIG. 12C) have been suppressed. Also, as illustrated in FIG. 12D, in the closing processing image signal Rcls, concave image density change areas (negative noise) in the original image signal Rin (indicated by the broken lines in FIG. 12D) have been suppressed.

[0287] In each of the other opening processing image signals Gopn and Bopn, as in the opening processing image signal Ropn, positive noise has been suppressed. In each of the other closing processing image signals Gcls and Bcls, as in the closing processing image signal Rcls, negative noise has been suppressed.

[0288] The opening processing image signals Ropn, Gopn, Bopn and the closing processing image signals Rcls, Gc1s, Bcls, which have been calculated by the morphology operation processing means 310, are fed into the signal processing means 320.

[0289] The signal processing means 320 carries out the weighting addition processing in accordance with Formula (11) shown above and on the received opening processing image signal and the received closing processing image signal, which correspond to each of the R, G, and B colors. The operation processing is carried out on the image signal components of the opening processing image signal and the closing processing image signal corresponding to each color, which image signal components represent corresponding pixels in the two images represented by the opening processing image signal and the closing processing image signal. In this manner, the processed image signal is calculated with respect to each color.

[0290] FIG. 12E shows the distribution of the processed image signal Rout, which is obtained by carrying out the operation processing in accordance with Formula (11) and on the opening processing image signal Ropn, which is shown in FIG. 12C, and the closing processing image signal Rcls, which is shown in FIG. 12D.

[0291] As illustrated in FIG. 12E, in the processed image signal Rout, both of positive noise and negative noise in the original image signal Rin (indicated by the broken lines in FIG. 12E) have been suppressed accurately. In this embodiment, the values of the weight factors w1 and w2 are set to be 0.5. However, the values of the weight factors w1 and w2 are not limited to 0.5, and may be set at various different values falling within an appropriate range.

[0292] As in the processed image signal Rout, each of the other processed image signals Gout and Bout can be obtained such that both of positive noise and negative noise may be suppressed accurately.

[0293] Also, with the eighth embodiment of the image processing apparatus, in each of the opening processing and the closing processing carried out by the morphology operation processing means 310, besides the processing steps with the rectilinear structure elements shown in FIGS. 16A, 16B, 16C, and 16D, the processing steps with a plurality of the curvilinear structure elements shown in FIGS. 16E through 16L are also carried out. Therefore, an image area having a large curvature, such as a character pattern, which will be suppressed as graininess by mistake with the processing using only the rectilinear structure elements, can be prevented from being suppressed. Accordingly, the graininess suppression processing can be carried out more accurately.

[0294] Specifically, a character pattern “A” shown in FIG. 17A and a grainy area shown in FIG. 17B may be embedded in an original image to be processed with this embodiment of the image processing apparatus. (In FIG. 17B, the grainy area is indicated by the rectangular contour.) In cases where the grainy area is to be suppressed and the character pattern “A” is to be retained, as illustrated in FIG. 17D, with the processing using only the rectilinear structure elements shown in FIGS. 16A, 16B, 16C, and 16D (5 pixels×5 pixels), the grainy area smaller than the structure element can be suppressed. However, in such cases, as illustrated in FIG. 17C, the end area having a large curvature in the character pattern is also suppressed. As illustrated in FIG. 17E, such that the end area of the character patter may be prevented from being suppressed, a smaller rectilinear structure element (which is set in the range of, for example, 4 pixels×4 pixels) may be employed. In such cases, suppression of the end area of the character pattern can be prevented. However, in such cases, as illustrated in FIG. 17F, graininess suppression cannot be effected.

[0295] With this embodiment of the image processing apparatus, besides the processing with the rectilinear structure elements, the processing with the curvilinear structure elements is also carried out. Therefore, for example, as illustrated in FIG. 17G, with the processing using the curvilinear structure element shown in FIG. 16L, the end area of the character pattern can be prevented from being suppressed. Further, as illustrated in FIG. 17H, graininess can be suppressed.

[0296] As described above, with the eighth embodiment of the image processing apparatus, each of the opening processing and the closing processing in accordance with the morphology operation, in which the rectilinear structure elements and the curvilinear structure elements are utilized, is carried out on the given original image signal. The opening processing image signal, in which the positive noise has been suppressed, is obtained from the opening processing. Also, the closing processing image signal, in which the negative noise has been suppressed, is obtained from the closing processing. The operation processing is then carried out on the image signal components of the two image signals having been obtained from the opening processing and the closing processing, which image signal components represent corresponding pixels in the two images represented by the two image signals, and the processed image signal (in this case, the graininess suppression processing image signal) is thereby obtained. Therefore, positive noise and negative noise contained in the original image can be suppressed accurately. Further, an image area having a large curvature, such as a character pattern, which will be suppressed as graininess by mistake with the processing using only the rectilinear structure elements, can be prevented from being suppressed. Accordingly, the graininess suppression processing can be carried out more accurately.

[0297] In the eighth embodiment described above, as each operation processing g carried out by the signal processing means 320, the operation processing represented by Formula (11) is employed. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to the use of the operation processing represented by Formula (11), and one of various other appropriate operation processings may be employed.

[0298] For example, in a ninth embodiment of the image processing apparatus in accordance with the present invention, the operation processing represented by Formula (3) shown below may be employed. $\begin{array}{cc}\mathrm{fout}=\{\begin{array}{cc}\mathrm{fopn}& \left(\mathrm{if}\mathrm{fcls}=\mathrm{fin}\right)\\ \mathrm{fcls}& \left(\mathrm{if}\mathrm{fopn}=\mathrm{fin}\right)\\ \mathrm{w1}·\mathrm{fopn}+\mathrm{w2}·\mathrm{fcls}& \left(\mathrm{else}\right)\end{array}& \left(3\right)\end{array}$

[0299] wherein w1 and w2 represent weight factors, w1+w2=1 (0<w1<1, 0<w2<1), fin represents the original image signal, fout represents the processed image signal (in this case, the graininess suppression processing image signal), fopn represents the opening processing image signal, and fcls represents the closing processing image signal.

[0300] Specifically, with the operation processing defined with Formula (3), as for a picture element, at which the value of the closing processing image signal Rcls is equal to the value of the original image signal Rin, the value of the opening processing image signal Ropn is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the opening processing image signal Ropn is equal to the value of the original image signal Rin, the value of the closing processing image signal Rcls is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the opening processing image signal Ropn and the value of the closing processing image signal Rcls are not equal to the value of the original image signal Rin, the addition processing through the weighting with the weight factors w1 and w2 is carried out on the opening processing image signal Ropn and the closing processing image signal Rcls. With respect to each of the other G and B image signals, the operation processing is carried out in the same manner.

[0301] With the ninth embodiment, wherein the operation processing represented by Formula (3) is employed, as in the eighth embodiment, wherein the operation processing represented by Formula (11) is employed, positive noise and negative noise can be suppressed accurately. Also, an image area having a large curvature, such as a character pattern, can be prevented from being suppressed. Accordingly, the graininess suppression processing can be carried out more accurately.

[0302] Further, as each operation processing g carried out by the signal processing means 320, the operation processing represented by Formula (4) shown below may be employed.

[0303] fout={|fopn−fin|(fopn−fin)+|fcls−fin|(fcls−fin)}/{|fopn−fin|+|fcls−fin|}+fin  (4)

[0304] Specifically, with the operation processing defined with Formula (4), as for a picture element, at which the value of the opening processing image signal Ropn is equal to the value of the original image signal Rin, the value of the closing processing image signal Rcls is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the closing processing image signal Rcls is equal to the value of the original image signal Rin, the value of the opening processing image signal Ropn is taken as the value of the processed image signal Rout. As for a picture element, at which the value of the opening processing image signal Ropn and the value of the closing processing image signal Rcls are not equal to the value of the original image signal Rin, the addition through weighting with the difference (|Ropn−Rin| or |Rcls−Rin|) between the opening processing image signal Ropn or the closing processing image signal Rcls, whichever has a larger difference with respect to the original image signal Rin, (i.e., Ropn or Rcls) and the original image signal Rin is carried out, and the value obtained from the weighting addition is taken as the value of the processed image signal (in this case, the graininess suppression processing image signal) Rout. As for each of the other G and B color image signals, processing is carried out in the same manner.

[0305] In cases where the operation processing represented by Formula (4) is employed, as in the eighth embodiment, wherein the operation processing represented by Formula (11) is employed, positive noise and negative noise can be suppressed accurately. Also, an image area having a large curvature, such as a character pattern, can be prevented from being suppressed. Accordingly, the graininess suppression processing can be carried out more accurately.

[0306] In the eighth embodiment, the R, G, and B image signals representing a color image are processed. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to the processing of the color image signals and are also applicable when various other signals, such as a luminance signal and color difference signals, are processed.

[0307] Also, in the eighth embodiment, the steps using the rectilinear structure elements and the curvilinear structure elements are carried out in both of the opening processing and the closing processing. However, the image processing method and the image processing apparatus in accordance with the present invention are not limited to such an embodiment, and the rectilinear structure elements and the curvilinear structure elements may be utilized in at least either one of the opening processing and the closing processing.

[0308] The image processing method and the image processing apparatus in accordance with the present invention are applicable to the processing of an image signal, which has been obtained by photoelectrically reading out an image recorded on photographic film, a print, or the like, and an image signal, which has been acquired from photographing with a digital still camera.

[0309] The image processing method and the image processing apparatus in accordance with the present invention are also applicable to the processing of an image signal having been obtained from an image, which has been formed with an image forming method utilizing a predetermined photosensitive material, which is proposed in Japanese Patent Application No. 9(1997)-56551 (or European Patent Publication No. 800.114A). It has been found that the image processing method and the image processing apparatus in accordance with the present invention are particularly efficient for suppressing graininess due to grainy constituents remaining in a silver image, which graininess is inherent to such an image signal.

[0310] In the image forming method proposed in Japanese Patent Application No. 9(1997)-56551, a photosensitive material is utilized, which contains photosensitive silver halide grains and forms an image corresponding to a latent image having been recorded through exposure when it is overlaid upon a predetermined processing material and heated. A latent image is recorded through exposure on the photosensitive material. The photosensitive material and the processing material are then overlaid one upon the other and heated. In this manner, the image, which corresponds to the latent image having been recorded on the photosensitive material, is formed on the photosensitive material. The photosensitive material, on which the image has thus been formed, is separated from the processing material. The image having been formed on the photosensitive material is read out by a scanner, and an image signal representing the image is thereby obtained. Predetermined image processing is then carried out on the image signal, and a digital image signal capable of reproducing an image is thereby formed. The photosensitive material is the one capable of developing the recorded latent image with the so-called non-wet type of processing step (a small amount of water may be used). By way of example, the photosensitive material may be a coloring agent-containing heat developable photosensitive material comprising a substrate and at least three kinds of photosensitive layers overlaid on the substrate. The photosensitive layers contain at least photosensitive silver halide particles, a binder, and a coloring agent having functions for releasing or diffusing a diffusing dye image-wise. The photosensitive layers have different wavelength regions to which they are sensitive, and exhibit different hues after development processing of the coloring agent. The processing material may have the characteristics such that, when it is overlaid upon the coloring agent-containing heat developable photosensitive material and heated, it may form the image on the coloring agent-containing heat developable photosensitive material. The processing material may be a mordant-containing processing material, which comprises a substrate and a layer containing at least a mordant and being overlaid upon the substrate. The processing material should preferably be the material capable of removing at least part of the diffusing dye, which is released from the coloring agent-containing heat developable photosensitive material due to heating, from the photosensitive material and thereby forming an image of at least three colors on the photosensitive material.

[0311] Alternatively, the photosensitive material may be a heat developable photosensitive material comprising a substrate and at least three kinds of photosensitive layers overlaid on the substrate. The photosensitive layers contain at least photosensitive silver halide grains, a binder, a color developing agent, and a dye-donating coupler. The photosensitive layers have different wavelength regions to which they are sensitive, and exhibit different hues of dyes, which are formed from an oxidant of the color developing agent and the dye-donating coupler. The processing material may have the characteristics such that, when it is overlaid upon the heat development photosensitive material and heated, it may form the image on the heat developable photosensitive material. With heating, an image of at least three colors may be formed on the heat developable photosensitive material.

[0312] As another alternative, the photosensitive material may be a photosensitive material comprising a transparent substrate and at least three kinds of photosensitive layers overlaid on the substrate. The photosensitive layers contain at least photosensitive silver halide grains, a color developing agent, a coupler, and a binder. The photosensitive layers have different wave length regions to which they are sensitive, and exhibit different absorption wavelength regions of dyes, which are formed from an oxidant of the color developing agent and the coupler. The processing material may comprise a substrate and a processing layer containing at least a base and/or a base precursor and being overlaid upon the substrate. The photosensitive member and the processing member are overlaid one upon the other such that the photosensitive layers of the photosensitive material and the processing layer of the processing material may stand facing each other. In this state, the photosensitive material and the processing material are heated in the presence of water in an amount corresponding to a ratio between 0.1 and 1 with respect to the amount, which is required to effect the maximum swelling of all coating layers, except for backing layers of the photosensitive material and the processing material. In this manner, an image of at least three colors based upon non-diffusing dyes may be formed on the photosensitive material.