Signal processing apparatus and method for reducing generation of false color by adaptive luminance interpolation

[00002] The present invention relates to a signal processing apparatus and method and, more particularly, to a signal processing apparatus and method for reducing generation of a false color in adaptive luminance interpolation.

[00003] FIG. 3 is a block diagram showing the arrangement of a conventional single-CCD digital camera, and especially shows a signal processor in detail.

[00004] A signal from a CCD image sensing element 501 is adjusted in white gain by a WB (White Balance) circuit 502, and sent to a luminance notch circuit 510. The luminance notch circuit 510 reduces the gain of color signal components near the Nyquist frequency in the vertical direction by using a VLPF (Vertical Low-Pass Filter), and also reduces the gain in the horizontal direction by using an HLPF (Horizontal Low-Pass Filter). These filters are called luminance notch filters. Then, an HBPF (Horizontal BandPass Filter) circuit 511 and VBPF (Vertical BandPass Filter) circuit 514 shift up frequency components slightly lower than the Nyquist frequency decreased by the notch filters.

[00005] PP (aperture Peak) gain circuits 512 and 515 adjust the amplitude in both the horizontal and vertical directions, and BC (Base Clip) circuits 513 and 516 cut small-amplitude components and remove noise. An adder 517 adds the horizontal and vertical components, an APC (APerture Control) main gain circuit 518 applies a main gain to the resultant signal, and an adder 519 adds a baseband signal to the signal. A γ conversion circuit 520 γ-converts the signal, and a luminance correction (YCOMP) circuit 521 corrects the luminance signal level by color.

[00006] Assume that an output from an image sensing element having filters of a checkered Bayer layout as shown in FIG. 4 is processed. In particular, primary color filters achieve good color separation. Therefore, as shown in FIG. 5A, the notch filter method cannot absorb gain differences between different color filters by using only LPFs at the edge of an image having opposite hues, e.g., red and blue in left and right halves. The edge staircases or becomes jaggy, which degrades the image quality of a playback image. This will be explained with reference to FIG. 5B.

[00007] FIG. 5B is a view for explaining an output level from each pixel of the image sensing element. In FIG. 5B, a pixel outputting a relatively large value is blank, and a pixel having an output of almost 0 is hatched for descriptive convenience. Signal level differences between different color filters are large at an edge between opposite hues, and appear as jaggies. Further, the jaggies are enhanced by edge enhancement which is performed to increase the resolution (MTF: Modulation Transfer Function) which has been decreased by LPFs.

[00008] To prevent jaggies, the following adaptive interpolation luminance signal generation method has been proposed. The correlation between signals of upper, lower, right, and left pixels of the pixel to be interpolated is detected to determine based on the correlation whether this pixel corresponds to a vertical or horizontal stripe. Interpolation is performed using signals of upper and lower pixels for a vertical stripe, and using signals of right and left pixels for a horizontal stripe, thereby preventing jaggies of a luminance signal.

[00009] The adaptive interpolation luminance signal generation method will be described with reference to the flow chart of FIG. 8.

[00010] A green signal is interpolated first. For example, in the image shown in FIG. 6A, to interpolate pixels P1 to P9 (each parenthesized character represents a chroma signal obtained from the pixel, and corresponds to the color of the filters), a green signal at the pixel P5 (P5(G)) is interpolated as follows.

[00011] 1. The absolute values (HDiff and VDiff) of the differences between upper and lower pixels of the pixel to be interpolated and between right and left pixels of the pixel to be interpolated are calculated by equation (1) (step S101):
HDiff=|P4(G)−P6(G)|, VDiff=|P2(G)−P8(G)|  (1)

[00013] 2. The interpolation method is changed based on the calculated absolute values (step S102).

[00014] If VDiff<HDiff, the green signal P5(G) is interpolated using signals of adjoining pixels in the vertical direction by equation (2) (step S103):
P5(G)=(P2(G)+P8(G))/2  (2)

[00016] If VDiff>HDiff, the green signal P5(G) is interpolated using signals of adjoining pixels in the horizontal direction by equation (3) (step S104):
P5(G)=(P4(G)+P6(G))/2  (3)

[00018] Green signals are interpolated in this way for pixels which output signals other than green signals. Thereafter, red and blue signals are interpolated. Red signals are calculated by
P2(R)=((P1(R)−P1(G))+(P3(R)−P3(G)))/2+P2(G)
P4(R)=((P1(R)−P1(G))+(P7(R)−P7(G)))/2+P4(G)
P5(R)=((P1(R)−P1(G))+(P3(R)−P3(G))+(P7(R)−P7(G))+(P9(R)−P9(G)))/4+P5(G)  (4)
Blue signals can also be obtained by the same calculation. Accordingly, signals of three, R, G, and B colors can be attained for each pixel. Further, a luminance signal Y is calculated by
Y=0.3×R+0.59×G+0.11×B  (5)

[00024] On the other hand, a color interpolation circuit 503 interpolates a chroma signal so as to give all color pixel values to all pixels (step S105). A color conversion MTX (MaTriX) circuit 504 converts the R, G, and B signals to the chroma signal. A CSUP (Chroma SUPpress) circuit 505 suppresses the chroma gain in low and high luminance regions, and a CLPF (Chroma Low-Pass Filter) circuit 506 limits the band of the chroma signals. A γ conversion circuit 507 converts the band-limited chroma signal into R, G, and B signals and at the same time γ-converts the R, G, and B signals. The γ-converted R, G, and B signals are converted into a color difference signal again. A C gain knee (Chroma gain knee) circuit 508 adjusts the chroma gain of the color difference signal, and an LCMTX (Linear Clip MaTriX) circuit 509 corrects the hue. The resultant signal is output together with the luminance signal.

[00025] Matrix operation by the color conversion MTX circuit 504 satisfies conditions that no false color is generated in the vertical direction when an image of an achromatic object is sensed.

[00026] However, in matrix operation performed by the color conversion MTX circuit 504, a false color is readily generated at a high frequency around the Nyquist frequency.

[00027] The present invention has been made in consideration of the above situation, and has as its object to easily reduce a false color to be generated as chroma signals of a high-frequency image and form a higher-quality image.

[00028] It is another object of the present invention to adjust the tendency of a false color to be appeared in the vertical and horizontal directions thereby flexibly adjusting the degree of the false color without decreasing the resolution upon a change in the characteristics of an optical LPF.

[00029] According to the present invention, the foregoing object is attained by providing a signal processing apparatus for acquiring an image signal from an image sensing element having color filters and processing the image signal, comprising: correlation operation means for determining correlation of the image signal in vertical and horizontal directions; first interpolation means for interpolating a chroma signal of the image signal; first signal processing means for performing signal processing of suppressing a false color in the vertical direction using a first matrix operation; second signal processing means for performing signal processing of suppressing a false color in the horizontal direction using a second matrix operation; and selection means for selecting said first or second signal processing means on the basis of the correlation obtained by said correlation operation means, wherein the chroma signal interpolated by said first interpolation means is processed by said first or second signal processing means selected by said selection means.

[00030] According to the present invention, the foregoing object is also attained by providing a signal processing method of acquiring an image signal from an image sensing element having color filters and processing the image signal, comprising: a correlation operation step of determining correlation of the image signal in vertical and horizontal directions; a first interpolation step of interpolating a chroma signal of the image signal; a selection step of selecting either of the vertical and horizontal directions on the basis of the correlation obtained in said correlation operation step; and a signal processing step of performing signal processing of suppressing a false color in the vertical or horizontal direction selected in said selection step for the chroma signal interpolated in the first interpolation step by performing matrix operation.

[00031] Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.

[00041] A preferred embodiment of the present invention will be described in detail in accordance with the accompanying drawings.

[00042] FIG. 1 is a block diagram showing the arrangement of an image signal processing unit according to the embodiment of the present invention. A signal from an image sensing element 100 is adjusted in WB by a WB (White Balance) circuit 101, delayed by a delay circuit 130, and interpolated using signals of adjoining pixels in the vertical or horizontal direction by an adaptive luminance interpolation circuit 111. Details of the interpolation method will be described later. A determination signal from an interpolation direction detection circuit 112 serves as a control signal for a color conversion MTX (MaTriX) switching circuit 103.

[00043] A luminance signal undergoes edge enhancement (aperture circuit). The adaptively interpolated luminance signal passes through a horizontal bandpass filter 113, vertical bandpass filter 116, PP (aperture Peak) gain circuits 114 and 117, and BC (Base Clip) circuits 115 and 118. An adder 119 adds the high-frequency signals of the horizontal and vertical components and outputs the resultant signal to an APC (APerture Control) main gain circuit 120. An adder 121 adds an output signal from the APC main gain circuit 120 and the baseband signal of the luminance signal from the adaptive luminance interpolation circuit 111, and sends the resultant signal to a γ conversion circuit 122. A luminance correction (YCOMP) circuit 123 corrects the luminance of the γ-converted signal.

[00044] The signals interpolated by a color interpolation circuit 102 or the adaptive luminance interpolation circuit 111 is converted into R, G, and B signals by a color conversion MTX circuit 104 or 105. The color conversion MTX circuits 104 and 105 execute different matrix operations (as will be described later). An interpolation direction determination signal from the interpolation direction detection circuit 112 switches the color conversion MTX switching circuit 103 to select either of the color conversion MTX circuits 104 and 105. An output signal from the color conversion MTX circuit 104 or 105 is converted into a color difference signal. A CSUP (Chroma SUPress) circuit 106 suppresses the chroma gain in high and low luminance regions. A CLPF (Chroma Low-Pass Filter) circuit 107 limits the band of the signal and converts the signals into R, G, and B signals. A γ conversion circuit 108 γ-converts the R, G, and B signals. The γ-converted R, G, and B signals are converted into color difference signals again. A C gain knee (Chroma gain knee) circuit 109 adjusts the chroma gain of the color difference signals, and an LCMTX (Linear Clip MaTriX) circuit 110 corrects the hue. The resultant signals are output together with the luminance signal.

[00045] The adaptive interpolation method according to the embodiment in the image signal processing block having the above arrangement will be explained. According to the basic method, a green signal is adaptively interpolated, then red and blue signals are interpolated based on the green signal, and a luminance signal is generated. This method is executed by the adaptive luminance interpolation circuit 111. In this embodiment, interpolation is switched between the horizontal and vertical directions in accordance with the determination result of a vertical or horizontal stripe in interpolating a green signal. The method will be described with reference to the flow chart of FIG. 7.

[00046] Green signals are interpolated for all pixels other than green pixels, i.e., for red and blue pixels. To interpolate a green signal at a pixel P5 (P5(G)) in FIG. 2 (P1 to P9 represent pixel positions, and each character in parentheses represents a chroma signal obtained from the pixel and corresponds to the filter color), the absolute values of the differences between signals of upper and lower pixels of the pixel to be interpolated and between signals of right and left pixels of the pixel to be interpolated are calculated (correlation operation: step S1):
HDiff=|P4(G2)−P6(G2)|, VDiff=|P2(G1)−P8(G1)|  (6)

[00048] The difference between HDiff and VDiff is set as an interpolation direction determination signal MatSw:
MatSw=HDiff−VDiff  (7)

[00050] If the difference is negative, horizontal correlation is higher (step S2), and interpolation is performed using signals of adjoining pixels in the horizontal direction (step S4). If the difference is positive (step S2), interpolation is performed using signals of adjoining pixels in the vertical directions (step S3). All blue and red pixels are also interpolated by the same method.

[00051] Red signals are interpolated by using the interpolated green signals for all pixels except for red pixels. Green pixels are interpolated by using right and left or upper and lower red signals and the green signals of these red pixels, and blue pixels are interpolated by using four peripheral red pixels by
P2(R)=((P1(R)−P1(G))+(P3(R)−P3(G)))/2+P2(G)
 P4(R)=((P1(R)−P1(G))+(P7(R)−P7(G)))/2+P4(G)
P5(R)=((P1(R)−P1(G))+(P3(R)−P3(G))+(P7(R)−P7(G))+(P9(R)−P9(G)))/4+P5(G)  (8)

[00055] Blue signals can also be obtained by the same calculation. As a result, signals of three, R, G, and B colors can be attained for each pixel. Further, a luminance signal Y is calculated by
Y=0.3×R+0.59×G+0.11×B  (9)
Note that the coefficients of the respective terms in equation (9) can be properly changed.

[00058] The color interpolation circuit 102 performs interpolation so as to give signals R, G1, G2, and B to all pixels (step S5).

[00059] As for interpolation of R and B signals, the interpolated R and B signals calculated by the adaptive luminance interpolation circuit 111 may be used.

[00060] The signals R, G1, G2, and B obtained by interpolation are input to the color conversion MTX circuit 104 or 105 via the MTX switching circuit 103 which is controlled by the interpolation direction detection circuit 112.

[00061] The interpolation direction detection circuit 112 determines vertical or horizontal interpolation on the basis of the interpolation direction determination signal MatSw obtained by the adaptive luminance interpolation circuit 111 (step S6). If vertical interpolation is determined, the interpolation direction detection circuit 112 controls the MTX switching circuit 103 to connect it to the color conversion MTX circuit 104 for suppressing a vertical false signal (step S7). If horizontal interpolation is determined, the interpolation direction detection circuit 112 controls the MTX switching circuit 103 to connect it to the color conversion MTX circuit 105 for suppressing a horizontal false signal (step S8).

[00062] The color conversion MTX circuits 104 and 105 will be described.

[00063] When the image sensing element 100 has filters of a primary color Bayer layout as shown in FIG. 4, the color conversion MTX circuit 104 computes $\begin{array}{cc}\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=\left[a_{\mathrm{ij}}\right]\left[\begin{array}{c}R\left(\lambda \right)\\ \mathrm{G1}\left(\lambda \right)\\ \mathrm{G2}\left(\lambda \right)\\ B\left(\lambda \right)\end{array}\right]& \left(10\right)\end{array}$
where λ is the wavelength of light.

[00065] The coefficient {a_ij} of the linear matrix in equation (10) must meet the following conditions. That is, when an image of an achromatic object is output via filters of a primary color Bayer layout as shown in FIG. 6A, output signals R(λ), G1(λ), G2(λ), and B(λ) corresponding to the respective filters are converted by the matrix [a_ij] (i=3, j=4). The converted R, G, and B signals are
R′(λ)=a11R(λ)+a12G1(λ)+a13G2(λ)+a14B(λ)
 G′(λ)=a21R(λ)+a22G1(λ)+a23G2(λ)+a24B(λ) B′(λ)=a31R(λ)+a32G1(λ)+a33G2(λ)+a34B(λ)  (11)

[00068] At this time, for example, the R and G1 filter positions shown in FIG. 2 are assumed to be in dark portions of an object to be sensed, and the G2 and B positions are assumed to be in bright portions of the object. Hence, any interpolation filters satisfy
R(λ)=αG1(λ)=V1(λ), B(λ)=βG2(λ)=V2(λ)  (12)
where α and β are parameters which depend on the color temperature of the object to be sensed.

[00071] From equations (11) and (12),
R′(λ)=(a11+a12/α)V1(λ)+(a13/βa14)V2(λ)
G′(λ)=(a21+a22/α)V1(λ)+(a23/βa24)V2(λ)
B′(λ)=(a31+a32/α)V1(λ)+(a33/βa34)V2(λ)  (13)

[00075] If R′(λ)=G′(λ)=B′(λ) does not hold in equation (13) in sensing an image of an achromatic object, a vertical false color is generated. To prevent this vertical false color, the following conditions must be satisfied:
a11+a12/α=a21+a22/α=a31+a32/αa13/β+a14=a23/β+a24=a33/β+a34  (14)

[00077] Since α and β change depending on the color temperature of the object to be sensed, α and β of an achromatic object are measured at a plurality of color temperatures in advance. If conversion is done by substituting G1′(λ) and G2′(λ) in equation (1) by the products obtained by multiplied the output G1(λ) by a and the output G2(λ) by β, equation (4) is rewritten as
a11+a12=a21+a22=a31+a32a13+a14=a23+a24=a33+a34  (15)
which are vertical false color suppression conditions. Coefficients which satisfy these conditions are obtained in advance and held in the color conversion MTX circuit 104.

[00080] On the other hand, the color conversion MTX circuit 105 can easily realize matrix operation which satisfies horizontal false color suppression conditions, by exchanging G1(λ) and G2(λ) in equation (10) in terms of the above description and the layout of respective color filters.

[00081] More specifically, using matrix coefficients which meet the conditions of equation (15), the color conversion MTX circuit 105 performs matrix operation: $\begin{array}{cc}\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=\left[a_{\mathrm{ij}}\right]\left[\begin{array}{c}R\left(\lambda \right)\\ \mathrm{G2}\left(\lambda \right)\\ \mathrm{G1}\left(\lambda \right)\\ B\left(\lambda \right)\end{array}\right]& \left(16\right)\end{array}$

[00082] Alternatively, different coefficients may be held without exchanging G1 and G2. In this case, for example, if the R and G2 filter positions shown in FIG. 2 are assumed to correspond to dark portions of an object to be sensed, and the G1 and B positions are assumed to correspond to bright portions of the object, then any interpolation filters satisfy
R(λ)=αG2(λ)=V1(λ), B(λ)=βG1(λ)=V2(λ)  (17)
Hence, by the same calculation as that described above,
a11+a13=a21+a23=a31+a33a12+a14=a22+a24=a32+a34  (18)
can be defined as horizontal false color suppression conditions.

[00087] In this manner, the interpolation direction in adaptive interpolation is determined. Matrix operation is done by the color conversion MTX circuit 104 having matrix coefficients which meet vertical false color suppression conditions for a pixel interpolated in the vertical direction based on the determination signal. Matrix operation is done by the color conversion MTX circuit 105 having matrix coefficients which meet horizontal false color suppression conditions for a pixel interpolated in the horizontal direction. In other words, R, G, and B signals in the color processing system are generated by switching color conversion matrix operation so as to suppress a false color in a direction in which such false color is readily generated.

[00088] In a method of determining a vertical or horizontal stripe and adaptively switching interpolation between the vertical and horizontal directions, like the prior art, a color conversion matrix circuit to be used is switched based on an interpolation direction determining signal. Only this switching can easily reduce a false color particularly generated in a high-frequency image and form a higher-quality image.

[00089] The threshold is adjusted by introducing an offset value into the interpolation direction determination signal MatSw:
MatSw=HDiff−VDiff−Offset
Matrix switching between vertical and horizontal false color suppression conditions can be adjusted. The tendency of a false color to be appeared in the vertical and horizontal directions can be adjusted, so that the degree of the false color can be flexibly adjusted without decreasing the resolution even upon a change in the characteristics of an optical LPF.

[00092] As has been described above, according to this embodiment, a false color particularly generated in a high-frequency image can be easily reduced to form a higher-quality image using the method of determining a vertical or horizontal stripe and adaptively switching interpolation between the vertical and horizontal directions.

[00093] By adjusting the tendency of a false color to be appeared in the vertical and horizontal directions, the degree of the false color can be flexibly adjusted without decreasing the resolution upon a change in the characteristics of the optical LPF.

[00094] <Other Embodiment>

[00095] The present invention can be applied to a system constituted by a plurality of devices (e.g., host computer, interface, reader, camera head) or to an apparatus comprising a single device (e.g., copying machine, facsimile machine, digital camera).

[00096] Further, the object of the present invention can also be achieved by providing a storage medium storing program codes for performing the aforesaid processes to a computer system or apparatus (e.g., a personal computer), reading the program codes, by a CPU or MPU of the computer system or apparatus, from the storage medium, then executing the program.

[00097] In this case, the program codes read from the storage medium realize the functions according to the embodiment, and the storage medium storing the program codes constitutes the invention.

[00098] Further, the storage medium, such as a floppy disk, a hard disk, an optical disk, a magneto-optical disk, CD-ROM, CD-R, a magnetic tape, a non-volatile type memory card, and ROM can be used for providing the program codes.

[00099] Furthermore, besides aforesaid functions according to the above embodiment are realized by executing the program codes which are read by a computer, the present invention includes a case where an OS (operating system) or the like working on the computer performs a part or entire processes in accordance with designations of the program codes and realizes functions according to the above embodiment.

[00100] Furthermore, the present invention also includes a case where, after the program codes read from the storage medium are written in a function expansion card which is inserted into the computer or in a memory provided in a function expansion unit which is connected to the computer, CPU or the like contained in the function expansion card or unit performs a part or entire process in accordance with designations of the program codes and realizes functions of the above embodiment.

[00101] In a case where the present invention is applied to the aforesaid storage medium, the storage medium stores program codes corresponding to the flowchart in FIG. 7 described in the embodiment.

[00102] The present invention is not limited to the above embodiments and various changes and modifications can be made within the spirit and scope of the present invention. Therefore to apprise the public of the scope of the present invention, the following claims are made.