Interpolation device, process and recording medium on which interpolation processing program is recorded

[0002] The disclosures of the following priority applications are herein incorporated by reference: Japanese Patent Application No. 9-261359, filed Sep. 26, 1997 and Japanese Patent Application No. 9-293912, filed Oct. 27, 1997.

[0003] 1. Field of Invention

[0004] The present invention relates to an interpolation device and process that perform interpolation of an image in which grid points that output a predetermined color and empty grid points that do not output the color are arrayed in a checkered configuration, and to a recording medium in which an interpolation processing program for use with such an image is recorded.

[0005] 2. Description of Related Art

[0006] There are electronic cameras that use a single imaging element to produce an image signal of a color image. An imaging element like this, for example, as shown in FIG. 16A, is structured with color filters of the three colors of RGB (red color, green color, blue color) arranged in a Bayer array.

[0007] Referring to the green color (G), it can be said that the imaging element is structured as an array having some grid points, which are pixels that output green color as shown in FIG. 16B, and other empty grid points, which are pixels that do not output green color. These grid points are arrayed in a checkered configuration.

[0008] Furthermore, when using a color filter having a structure in which two imaging elements comprising only green color (G) as shown in FIG. 17A are shifted relative to each other, as shown in FIG. 17B, it can be said that imaginary empty grid points that do not output green color, and grid points that output green color are structured so as to be arrayed in a checkered configuration.

[0009] Furthermore, when using a two-plate-type of color filter, one imaging element is formed by G only, and the other imaging element is structured such that pixels that output red color and pixels that output blue color are arrayed in a checkered configuration. That is, the pixels that output blue color become empty grid points with respect to red color, and the pixels that output red color become empty grid points with respect to blue color.

[0010] Conventionally, in methods that perform interpolation of the color image that corresponds to the image signals formed by such imaging elements, the average value of the image (pixel) output of the grid points adjacent to empty grid points is used as the pixel output (interpolation amount) of the empty grid points.

[0011] U.S. Pat. No. 5,373,322 discloses a process that estimates the directionality of the image output of the empty grid points by using chrominance gradients as a technique for interpolating parts of an image, such as edges, having a spatially strong directivity.

[0012] FIG. 18 illustrates a conventional interpolation process disclosed in U.S. Pat. No. 5,373,322. In FIG. 18, G11, G13, . . . , G66, and so forth, show the pixel output of green color (G), R12, R14, . . . , R56, and so forth, show the pixel output of red color (R), and B21, B23, . . . , B67, and so forth, show the pixel output of blue color (B). In U.S. Pat. No. 5,373,322, as described above, when the interpolation amount G34 of green color of the pixel that outputs R34 is calculated, the information HDiff related to the chrominance gradients in the horizontal direction is obtained as

[0013] HDiff=|((R32+R36)/2)−R34|

[0014] and the information VDiff related to the chrominance gradients in the vertical direction is obtained as

[0015] VDiff=|((R14+R54)/2)−R34|.

[0016] The calculation method of G34 varies from case to case, based upon the relationship between HDiff and VDiff, which were thus obtained. That is, in the case of HDiff<VDiff (when the correlation in the horizontal direction is strong), G34 is obtained by

[0017] G34=(G33+G35)/2

[0018] and in the case of VDiff<HDiff (when the correlation in the vertical direction is strong), G34 is obtained by

[0019] G34=(G24+G44)/2

[0020] and in the case of HDiff=VDiff, G34 is obtained by

[0021] G34=(G24+G44+G33+G35)/4.

[0022] That is, the information related to the chrominance gradients of green color is estimated in this prior art technique based upon the pixel output of red color irrespective of the process that obtains the interpolation amount of green color.

[0023] Estimating the spatial similarity of the empty grid points in order to accurately predict the interpolation amount of the empty grid points is effective. However, the information related to the chrominance gradients of green color is estimated in the prior art based upon the pixel output of red color irrespective of the process that obtains the interpolation amount of green color. Because of this, it was not possible to accurately estimate the directivity of the pixel output of green color based upon the pixel output of red color in an image in which the chrominance gradients of green color and the chrominance gradients of red color are not constant, or in an image in which the component of red color is low, or the like.

[0024] Furthermnore, the directivity of the pixel output of the empty grid points is not always symmetrically right and left or symmetrically up and down. Because of this, for reasons similar to the prior art described above, when the information related to the chrominance gradients was obtained in only the horizontal and vertical directions, it was difficult to accurately estimate the directivity of the appropriate pixel output.

[0025] For example, when each pixel output in FIG. 18 has the value shown in FIG. 19, G34 has a strong correlation in the vertical direction (particularly in the upper direction).

[0026] U.S. Pat. No. 5,373,322 uses the following calculations:

[0027] HDiff=|((R32+R36)/2)−R34|

[0028] VDiff=|((R14+R54)/2)−R34|

[0029] In this example, HDiff=VDiff is established, and it is estimated that the correlation in the vertical and horizontal directions is the same. In such a situation, where HDiff=VDiff=0.5, it is preferable for G34=1 to be satisfied. However, in this situation, since HDiff=VDiff: $\begin{array}{c}\mathrm{G34}=\left(\mathrm{G24}+\mathrm{G44}+\mathrm{G33}+\mathrm{G35}\right)/4\\ =0.75\end{array}$

[0030] That is, G34=0.75, rather than G34=1. Thus, when the interpolation is performed based upon the directivity of the pixel output as conventionally estimated, there is a possibility of damaging the clearness of the image, or destroying the image, in the worst case. Because of this, there is a possibility that the sharpness and the smoothness of the border parts can be lost.

[0031] Thus, one object of the invention is to provide an interpolation device and process that estimates the spatial directivity of empty grid points in detail, and accurately obtains the interpolation amount of the empty grid points.

[0032] Another object of the invention is to provide a recording medium on which is recorded an interpolation processing program that estimates the spatial directivity of the empty grid points in detail and that accurately obtains the interpolation amount of the empty grid points.

[0033] An interpolation device according to one aspect of the invention performs interpolation of an image having grid points that output a predetermined color and at least one empty grid point that does not output the predetermined color, the grid points being oriented in a checkered configuration, and obtains an interpolation amount that becomes a pixel output of the empty grid point. A correlation amount calculator determines a correlation between individual pixel outputs of at least two grid points adjacent to the empty grid point and pixel outputs of each one of grid points oriented in shifted positions in the four directions of upper, lower, right, and left directions, respectively, with respect to the at least two grid points, and calculates a correlation amount showing the correlation strength in each direction. A similar direction detector identifies a direction having a strong correlation amount based upon the correlation amount for each direction calculated by the correlation amount calculator. An interpolation amount calculator determines an interpolation amount of the empty grid point based upon a pixel output of at least one grid point in a direction where correlation discerned by the similar direction detector is strong.

[0034] An interpolation device according to another aspect of the invention performs interpolation of an image having grid points that output a predetermined color and at least one empty grid point that does not output the predetermined color, the grid points being oriented in a checkered configuration, and obtains an interpolation amount that becomes a pixel output of the empty grid point. A correlation amount calculator determines a correlation between individual pixel outputs of at least two grid points adjacent to the empty grid point and pixel outputs of each one of grid points oriented in shifted positions in the four directions of upper, lower, right, and left directions, respectively, with respect to the at least two grid points. A weighting factor calculator calculates a weighting factor for each direction based upon the correlation amount for each direction calculated by the correlation amount calculator. An interpolation amount calculator calculates the product of the weighting factor for each direction calculated by the weighting factor calculator and the pixel outputs of grid points corresponding to each weighting factor, and assigns a sum of those products as an interpolation amount of the empty grid point.

[0035] According to another aspect of the invention, an interpolation device includes a first similarity calculator that calculates a first vertical direction similarity indicative of a similarity of two grid points that are adjacent in a vertical direction to an empty grid point and a first horizontal direction similarity indicative of a similarity of two grid points that are adjacent in a horizontal direction to the empty grid point. A second similarity calculator calculates a second vertical direction similarity indicative of a similarity of at least one of the grid points that are adjacent in a horizontal direction to the empty grid point and a second horizontal direction similarity indicative of a similarity of at least one of the grid points that are adjacent in a vertical direction to the empty grid point. A similarity detector calculates a similarity in a vertical direction of the empty grid point based on the first vertical direction similarity and the second vertical direction similarity and determines a similarity in a horizontal direction of the empty grid point based on the first horizontal direction similarity and the second horizontal direction similarity. An interpolation calculator calculates an interpolation amount of the empty grid point by selecting at least one pixel output of the grid points that are positioned in a direction where the similarity is strong according to the similarity determined by the similarity detector or by weighted addition of at least one pixel output of the grid points that are positioned in the vertical direction and at least one pixel output of the grid point positioned in the horizontal direction, according to the similarity.

[0036] The invention also relates to the processes performed by the interpolation device and to a recording medium, such as, for example, a CD-ROM or a carrier wave, that stores a computer readable program or data signal representative of the program to perform the interpolation process.

[0037] The invention will be described in conjunction with the following drawings in which like reference numerals designate like elements and wherein:

[0038] FIGS. 1A-1C explain one calculation method of the correlation amount according to an aspect of the invention;

[0039] FIG. 2 is a block diagram of a first embodiment of the invention;

[0040] FIG. 3 is an operational flow chart of the first embodiment;

[0041] FIG. 4 is an operational flow chart of the interpolation processing of the first embodiment;

[0042] FIG. 5 shows the pixel output of the grid points and the interpolation amount of the empty grid points;

[0043] FIGS. 6A-6B show examples of the pixel outputs of the grid points;

[0044] FIG. 7 is a block diagram of second, third, and fourth embodiments of the invention;

[0045] FIG. 8 is an operational flow chart of the interpolation processing of the second, third, and fourth embodiments;

[0046] FIG. 9 is a chart showing the relationship between function f and variable c;

[0047] FIGS. 10A-10D explain the calculation method of the correlation amount according to an aspect of the invention;

[0048] FIG. 11 is an operational flow chart of the interpolation processing of a fifth embodiment;

[0049] FIG. 12 explains one calculation method of the correlation amount;

[0050] FIG. 13 is an operational flow chart of the interpolation processing of a sixth embodiment;

[0051] FIG. 14 is a block diagram of the seventh embodiment;

[0052] FIGS. 15A and 15B are flow charts applicable to the seventh embodiment;

[0053] FIGS. 16A-16B show one possible structure of an imaging element;

[0054] FIGS. 17A-17B show another possible structure of an imaging element;

[0055] FIG. 18 explains conventional interpolation processing;

[0056] FIG. 19 explains an example of pixel outputs;

[0057] FIG. 20 is a block diagram of an eighth embodiment;

[0058] FIG. 21 explains the flow of data of an interpolation processor of FIG. 20;

[0059] FIG. 22 shows an empty grid point and pixel outputs of other grid points;

[0060] FIG. 23 is an operational flow chart of the eighth embodiment;

[0061] FIG. 24 is an operational flow chart of the interpolation processing of the eighth embodiment;

[0062] FIGS. 25A-25B explain a method of calculating similarity in the eighth embodiment;

[0063] FIG. 26 is a diagram showing a shape of a graph of f(x);

[0064] FIGS. 27A-27B explain a method of calculating similarity in a ninth embodiment;

[0065] FIGS. 28A-28B explain a method of calculating similarity in a tenth embodiment;

[0066] FIGS. 29A-29B explain a method of calculating similarity in an eleventh embodiment; and

[0067] FIG. 30 is a block diagram of a twelfth embodiment.

[0068] Hereafter, preferred embodiments are explained with reference to the drawings. The embodiments are described with reference to an electronic (digital) still camera in which the interpolation device concerning the invention is adopted. The invention is applicable to devices other than still cameras, such as, for example, moving image cameras and imaging devices (e.g., scanners) in general.

[0069] FIG. 2 is a block diagram of the first embodiment. In FIG. 2, the output of a controller 40 is connected to a shooting optical system 41, an OLPF (Optical Low Pass Filter) 42, an imaging element 43, an A/D converter 44, an image buffer memory 45, a signal processor 46, and a memory card driver 51. The signal processor 46 includes an interpolation processor 47, which is composed of a correlation amount calculator 48, a similar direction detector 49 and an interpolation amount calculator 50.

[0070] The imaging element 43 can be, for example, a two-dimensional Charge-Coupled-Device (CCD). Imaging element 43 can alternatively include, for example, a PSD (Photo-Sensitive-Diode) or a CMOS device.

[0071] In addition, the imaging element 43, as shown in FIG. 16A, includes a three-colored (RGB) color filter, which is Bayer-arranged. The image buffer memory 45 is composed by three areas that correspond to these three colors.

[0072] The optical image obtained at the shooting optical system 41 is filtered by the OLPF 42, and is given to the imaging element 43. The output of the imaging element 43 is connected to the A/D converter 44. The output of the A/D converter 44 is connected to the image buffer memory 45. The output of the image buffer memory 45 is connected to the signal processor 46 and to the memory card driver 51.

[0073] The output of the signal processor 46 is connected to the image buffer memory 45, and to the memory card 52 when it is loaded in the memory card driver 51.

[0074] FIG. 3 is an operational flow chart of the first embodiment. FIG. 4 is an operational flow chart of the interpolation processing of the first embodiment. In the examples described for the present embodiment and for each later mentioned embodiment, the pixels that output green are grid points, and the pixels that output red or blue are empty grid points (empty with respect to green). In other words, the interpolation processing in the examples for the present embodiment and each later-mentioned embodiment corresponds to the processing performed to obtain the interpolation amount that corresponds to the green pixel output of these empty grid points.

[0075] In order to simplify the later explanation, the position of each pixel of the imaging element 43 is indicated by a coordinate system composed of a horizontal axis X in which the right direction is positive and a vertical axis Y in which the downward direction is positive. In addition, the pixel output of a grid point is indicated by G(x, y) and the interpolated amount of an empty grid point is indicated by G′(x, y).

[0076] FIG. 5 shows this kind of pixel output of the grid points and the interpolated amount of the empty grid points. In FIG. 5, X_max represents the maximum value of the X component, and Y_max represents the maximum value of the Y component.

[0077] Hereafter, the operation of the first embodiment is explained with reference to FIG. 2 through FIG. 5.

[0078] When the main power is turned on, the optical image provided via the shooting optical system 41 and the OLPF 42 forms an image on the optical converting surface (not shown) of the imaging element 43, and signal charges are formed by photoelectric conversion. The imaging element 43 forms the image signal by scanning the signal charges, and sends it to the A/D converter 44.

[0079] The A/D converter 44 forms the image data by converting the image signal provided in this manner (step S1). In the present embodiment and each later-mentioned embodiment, the image data corresponds to the pixel outputs of the imaging element 43. Such image data is divided into three colors of RGB, and stored in areas in the buffer memory 45 which correspond to each color (step S2). This kind of image data formation (step S1) and processing and storing (step S2) are repeatedly performed based on the timing control by the controller 40.

[0080] The area in the image buffer memory 45 that corresponds to green includes an area that stores the green image data (corresponding to the pixel output of the grid points) formed by the A/D converter 44 and an area that stores the green image data (corresponding to the interpolation amount of the empty grid points) obtained by the later-mentioned interpolation processing. These green image data (pixel outputs of the grid points and the interpolation amount of the empty grid points), as shown in FIG. 5, are stored in correlation with coordinates that are indicated by the abovementioned X-Y coordinate system.

[0081] The controller 40 evaluates whether the release button (not shown), which is provided in the main body of the electronic still camera, is pressed (step S3). When the controller 40 recognizes that the release button is not pressed, it repeatedly performs step S1 and step S2 at the above-mentioned timing. When the controller 40 recognizes that the release button is pressed, the later-mentioned execution of the interpolation processing by the interpolation processor 47 in the signal processor 46 is designated (step S4). In other words, in the present embodiment, the image data stored in the image buffer memory 45 becomes the object of the interpolation processing when the release button is pressed.

[0082] In addition, for performing the image display, it is also acceptable to constantly perform the interpolation processing without regard to when the release button is pressed. In addition, it is also acceptable to repeatedly perform the processing of step S1 and the subsequent steps after release occurs.

[0083] The controller 40 records the image data for which the interpolation processing is performed and that is stored in the image buffer memory 45 to the memory card 52 via the memory card driver 51 when the interpolation processing is completed (step S5).

[0084] Hereafter, the operation of the interpolation processing is explained. The interpolation processor 47 selects an empty grid point which is to be interpolated when the execution of the interpolation processing is indicated by the controller 40 (step S11). For example, in FIG. 5, when [k+p=odd number] is established for the arbitrary coordinates (k, p), an empty grid point corresponds to these coordinates. Because of this relationship, in the present embodiment, the coordinates that correspond to each pixel are retrieved in order from (1, 1) to (X_max, Y_max), and the coordinates for which the sum of the X element and Y element is an odd number are made to be empty grid points. (Of course, other relationships are possible, and other techniques can be used to determine whether a grid point is an empty grid point for which interpolation is needed.)

[0085] When the correlation amounts in the later-mentioned four different directions are calculated, it is desirable that at least three grid points exist in each of the four directions of upper, lower, left and right of the empty grid point. For example, an evaluation of whether the correlation amounts of four different directions are possible corresponds to an evaluation of whether [4≦k≦X_max−3] and [4≦p≦Y_max−3] are established for the arbitrary coordinates (k, p). The interpolation processor 47, by performing this kind of evaluation, evaluates whether calculation of the correlation amounts in the four directions is possible based on the coordinates of the selected empty grid point (step S12).

[0086] When the interpolation processor 47 determines that calculation of the correlation amounts in the four directions is impossible by performing this kind of evaluation, it takes the average value of the pixel outputs of the grid points that are adjacent to the empty grid point as the interpolation amount (step S13).

[0087] When the interpolation processor 47 determines that calculation of the correlation amounts in the four directions is possible, it activates the correlation calculator 48, the similar direction detector 49 and the interpolation amount calculator 50.

[0088] The correlation calculator 48, for the coordinates (k, p) of the empty grid point calculates the correlation amounts, C-up, C-down, C-left and C-right that correspond to the four directions of up, down, left and right as follows: $\begin{array}{c}C\text{-}\mathrm{up}=\left|G\left(k,p-1\right)-G\left(k,p-1-n\right)\right|·\mathrm{tu1}+\\ \left|G\left(k,p+1\right)-G\left(k,p+1-n\right)\right|·\mathrm{tu2}+\\ \left|G\left(k-1,p\right)-G\left(k-1,p-n\right)\right|·\mathrm{tu3}+\\ \left|G\left(k+1,p\right)-G\left(k+1,p-n\right)\right|·\mathrm{tu4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down}=\left|G\left(k,p-1\right)-G\left(k,p-1+n\right)\right|·\mathrm{td1}+\\ \left|G\left(k,p+1\right)-G\left(k,p+1+n\right)\right|·\mathrm{td2}+\\ \left|G\left(k-1,p\right)-G\left(k-1,p+n\right)\right|·\mathrm{td3}+\\ \left|G\left(k+1,p\right)-G\left(k+1,p+n\right)\right|·\mathrm{td4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{left}=\left|G\left(k,p-1\right)-G\left(k-n,p-1\right)\right|·\mathrm{tl1}+\\ \left|G\left(k,p+1\right)-G\left(k-n,p+1\right)\right|·\mathrm{tl2}+\\ \left|G\left(k-1,p\right)-G\left(k-1-n,p\right)\right|·\mathrm{tl3}+\\ \left|G\left(k+1,p\right)-G\left(k+1-n,p\right)\right|·\mathrm{tl4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right}=\left|G\left(k,p-1\right)-G\left(k+n,p-1\right)\right|·\mathrm{tr1}+\\ \left|G\left(k,p+1\right)-G\left(k+n,p+1\right)\right|·\mathrm{tr2}+\\ \left|G\left(k-1,p\right)-G\left(k-1+n,p\right)\right|·\mathrm{tr3}+\\ \left|G\left(k+1,p\right)-G\left(k+1+n,p\right)\right|·\mathrm{tr4}\end{array}$

[0089] (Step S14).

[0090] In the present embodiment, the correlation amounts C-up, C-down, C-left and C-right that correspond to the four directions of up, down, left and right are calculated with tu1−tu4, td1−td4, tl1−tl4 and tr1−tr4 having the value 1.

[0091] Here, when n=2, the correlation amount C-up of the upper direction corresponds to the total sum of the absolute values of the differences between the pixel outputs of the four grid points that are indicated as ◯ and the four grid points that are indicated as □ as shown in FIG. 1A.

[0092] The similar direction detector 49 identifies the direction having the strongest correlation by selecting the smallest value (while accepting overlap) among the correlation amounts C-up, C-down, C-left and C-right that are calculated by the correlation calculator 48 (step S15).

[0093] The interpolation amount calculator 50 then takes the pixel output of the grid point in the direction of the strongest correlation among the pixel outputs (G(k, p−1), G(k, p+1), G(k−1, p), G(k+1, p)) of the grid points that are adjacent to the empty grid point as the interpolation amount (step S16).

[0094] For example, when the correlation in the upper direction is strong, G′(k, p) is G(k, p−1). When the correlation in the lower direction is strong, G′(k, p) is G(k, p+1). When the correlation in the left direction is strong, G′(k, p) is G(k−1, p). When the correlation in the right direction is strong, G′(k, p) is G(k+1, p). When there are a plurality of directions having strong correlation, it is appropriate to use the average value of the pixel outputs of the grid points of these directions as the interpolation amount of the empty grid point.

[0095] When the interpolation amount of an empty grid point is thus determined, the interpolation processor 47 determines whether the interpolation processing of all the empty grid points is completed (step S17). For example, this kind of evaluation corresponds to determining whether [k>X_max−3] and [p>Y_max−3] are satisfied for the arbitrary coordinates (k, p) in the process of retrieving the coordinates that correspond to each pixel as mentioned above. The interpolation processor 47 repeatedly performs the above-mentioned processing (the processing of step S11 and after) until the interpolation processing of all the empty grid points is completed (YES in step S17).

[0096] Thus, in the present embodiment, the spatial directivity for an empty grid point in the four directions of up, down, left and right can be estimated, and the pixel output of the grid points that are adjacent in the direction which has the strongest directivity becomes the interpolation amount of the empty grid point. Moreover, this kind of estimation of the directivity is performed based on the directivity of the color (pixel output of the grid points) in which the interpolation processing is being performed.

[0097] Accordingly, according to the present embodiment, the spatial directivity of empty grid points can be estimated in detail and accurately. In addition, when the directivity of one direction is extremely strong, the accuracy of the interpolation processing can be reliably improved. For example, when the pixel outputs of the grid points in FIG. 1A are the values which are shown in FIG. 6A, C-up=0, C-down=2, C-left=1 and C-right=3. Alternatively, when the pixel outputs of the grid points in FIG. 1A are the values shown in FIG. 6B, C-up=0, C-down=2.5, C-left=0.5 and C-right=1.5. In either case, the fact that the correlation of the upper direction is the strongest is identified in FIG. 6A and FIG. 6B, and G′(k, p)=G(k, p−1)=1.

[0098] The correlation amounts of the vertical direction and the horizontal direction with respect to the empty grid point can be obtained based on the red color gradient as disclosed in U.S. Pat. No. 5,373,322. However, it can be obtained based on the pixel outputs of the grid points by using the correlation amounts of C-up, C-down, C-left and C-right that correspond to the four directions of up, down, left and right.

[0099] Here, when the correlation amount in the vertical direction is C-up+C-down, and the correlation amount in the horizontal direction is C-left+C-right, if the pixel outputs of the grid points are the values shown in FIG. 6A, C-up+C-down=2 and C-left+C-right=4, and the fact that the correlation in the vertical direction is strong can be estimated. Therefore, the interpolation amounts of the empty grid points can be obtained. However, when the pixel outputs of the grid points are the values shown in FIG. 6B, with the process of U.S. Pat. No. 5,373,322, even when the correlation in the vertical direction is strong, C-up+C-down=2.5 and C-left+C-right=2, and the correlation in the horizontal direction becomes strong. Therefore, it is possible that the image will degenerate through the interpolation.

[0100] In other words, in the present embodiment, high-accuracy interpolation can be performed with respect to an image that has a possibility of deterioration through interpolation by the correlation amounts of the vertical direction and the horizontal direction.

[0101] In addition, in the present embodiment, when it is recognized that the calculation of the correlations of four directions is impossible, the average value of the pixel outputs of the grid points that are adjacent to the empty grid point is the interpolation amount. However, when at least two directions exist in which the calculation of the correlation amount is possible, it is also acceptable to obtain this kind of correlation amount by the above-mentioned equations, and to obtain the interpolation amounts of the empty grid point based on the pixel output of the grid a point that is adjacent in the direction having the strong correlation. For example, in FIG. 5, when G′(2, 3) is calculated, the correlation amounts C-down and C-right of the lower direction and the right direction can be obtained as follows: $\begin{array}{c}C\text{-}\mathrm{down}=\left|G\left(2,2\right)-G\left(2,4\right)\right|+\\ \left|G\left(2,4\right)-G\left(2,6\right)\right|+\\ \left|G\left(1,3\right)-G\left(1,5\right)\right|+\\ \left|G\left(3,3\right)-G\left(3,5\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right}=\left|G\left(2,2\right)-G\left(4,2\right)\right|+\\ \left|G\left(2,4\right)-G\left(4,4\right)\right|+\\ \left|G\left(1,3\right)-G\left(3,3\right)\right|+\\ \left|G\left(3,3\right)-G\left(5,3\right)\right|.\end{array}$

[0102] The results can then be used to determine which of the two directions (down and right) has the strongest correlation.

[0103] FIG. 7 is a block diagram relating to second, third and fourth embodiments of the invention. In FIG. 7, devices that function the same as devices in the block diagram of the first embodiment shown in FIG. 2 are shown with the same symbols, and the explanation of such structure is omitted. The differences between the second, third and fourth embodiments and the first embodiment are that the interpolation processor 53 is provided instead of the interpolation processor 47. The interpolation processor 53 includes a correlation amount calculator 54, a weighting factor calculator 55 and an interpolation amount calculator 56.

[0104] FIG. 8 is an operational flow chart of the interpolation processing of the second, third and fourth embodiments. Hereafter, the operation of the second embodiment is explained with reference to FIG. 7 and FIG. 8. The different characteristics of the present embodiment lie in the operation of the interpolation processing. Therefore, the explanations of other operations are omitted since they are the same as in the first embodiment.

[0105] When the controller 40 recognizes that the release button is pressed, it designates the execution of the interpolation processing to the interpolation processor 53 within the signal processor 46. The interpolation processor 53, when thus instructed, selects the empty grid point which is to be the object of the interpolation processing, just like in the first embodiment (step S21).

[0106] In the second-fourth embodiments, when the correlation amounts of the later mentioned four directions are calculated, it is desirable that at least two pixels exist in the four directions of upper, lower, left and right of the empty grid point. For example, the evaluation of whether the calculations of the correlation amounts in four directions are possible corresponds to an evaluation of whether [3≦k≦X_max−2] and [3≦p≦Y_max−2] are established. The interpolation processor 53, by performing this kind of evaluation, evaluates whether the calculations of the correlation amounts in four directions is possible (step S22).

[0107] When the interpolation processor 53, by doing this kind of evaluation, recognizes that the calculations of the correlation amounts in the four directions are impossible, the average value of the pixel outputs of the grid points that are adjacent to the empty grid point becomes the interpolation amount (step S23). When the interpolation processor 53 recognizes that the calculations of the correlation amounts in the four directions are possible, it activates the correlation amount calculator 54, the weighting factor calculator 55 and the interpolation amount calculator 56.

[0108] The correlation amount calculator 54, for the coordinates (k, p) of the empty grid point, calculates the correlation amounts C-up, C-down, C-left and C-right that correspond to the four directions of up, down, left and right (step S24) as follows: $\begin{array}{c}C\text{-}\mathrm{up}=\left|G\left(k,p+1\right)-G\left(k,p+1-n\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1,p-n\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1,p-n\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down}=\left|G\left(k,p-1\right)-G\left(k,p-1+n\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1,p+n\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1,p+n\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{left}=\left|G\left(k,p-1\right)-G\left(k-n,p-1\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k-n,p+1\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1-n,p\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right}=\left|G\left(k,p-1\right)-G\left(k+n,p-1\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k+n,p+1\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1+n,p\right)\right|\end{array}$

[0109] (Step S24).

[0110] For example, when n=2, the correlation amount C-up in the upper direction is the sum of the absolute values of the differences between the pixel outputs of the three grid points that are indicated as ◯ and the three grid points that are indicated as □ as shown in FIG. 1B. The correlation amount C-left in the left direction corresponds to the sum of the absolute values of the differences of the pixel outputs of the three grid points that are indicated as ◯ and the pixel outputs of the three grid points that are indicated as Δ as shown in FIG. 1C. The correlation amount in the lower direction (or the right direction) corresponds to the sum of the absolute values of the differences between the pixel outputs of different combinations of three grid points that are adjacent to the empty grid point and the pixel outputs of grid points arranged in positions shifted from these grid points by 2 pixels in the lower direction (or the right direction).

[0111] The weighting factor calculator 55, as shown in FIG. 9, for f(0)=1, and for the function f whose value approaches 0 in accordance with the increase of the variable number (according to the functions f₁, f₂and f₃), calculates the weighting factors α-down, α-up, α-left and α-right that correspond to the four directions of up, down, left and right (step S25) as follows:

[0112] α-up=f(C-up)/sf

[0113] α-down=f(C-down)/sf

[0114] α-left=f(C-left)/sf

[0115] α-right=f(C-right)/sf.

[0116] Here, sf=f(C-up)+f(C-down)+f(C-left)+f(C-right).

[0117] For example, in the present embodiment, as one example of the function f, f(C)=1/(1+q×C). Here, q=⅓.

[0118] Thus, once the correlation amounts C-up, C-down, C-left and C-right are determined, weighting factors α-up, α-down, α-left and α-right can be determined based on the respective correlation amounts. Since the function used to determine the weighting factor has a value that decreases from 1 toward 0 as the input variable of the function increases, the weighting factor has a value of 1 when correlation is strong, and decreasing values as correlation weakens.

[0119] The interpolation amount calculator 56, for the weighting factors α-down, α-up, α-left and α-right that are calculated by the weighting factor calculator 55, calculates the interpolation amount G′(k, p) of the empty grid point as follows:

[0120] G′(k, p)=α-up×G(k, p−1)+α-down×G(k, p+1)+α-left×G(k−1, p)+α-right×G(k+1, p)

[0121] (Step S26).

[0122] When the interpolation amount of the empty grid point is thus determined, the interpolation processor 53 evaluates whether the interpolation processing of all the empty grid points is completed (step S27). For example, this kind of evaluation, in the process of searching for the coordinates that correspond to the above-mentioned empty grid points, is equivalent to the evaluation of whether [k>X_max−2] and [p>Y_max−2] are established for the arbitrary coordinates (k, p). The interpolation processor 53, by doing this kind of evaluation, repeatedly performs the above-mentioned processing (processing of step S21 and after) until it determines that the interpolation processing of all empty grid points is completed.

[0123] Thus, in the present embodiment, the spatial directivity of the empty grid points in the four directions of up, down, left and right can be estimated, and the weighting factor to be used when calculating the interpolation amounts can be obtained based on these directivities. Accordingly, in the present embodiment, when the spatial directivities of empty grid points are mutually related, since interpolation processing suitable to this kind of directivity can be reliably performed, the accuracy of the interpolation processing can be reliably improved.

[0124] Furthermore, in the present embodiment, when it is recognized that the calculation of the correlation amounts in the four directions is impossible, the average value of the pixel outputs of the grid points adjacent to the empty grid point is used as the interpolation amount. Alternatively, when one or more directions exist where the calculation of the correlation amounts is possible, it is also acceptable to obtain the interpolation amount of the empty grid point by obtaining the correlation amounts of these directions by the above-mentioned formula.

[0125] For example, in FIG. 5, when G′(2, 3) is calculated, the correlation amounts C-up, C-down, and C-right of the upper direction, the lower direction, and the right direction, respectively, can be obtained as follows: $\begin{array}{c}C\text{-}\mathrm{up}=\left|G\left(2,4\right)-G\left(2,2\right)\right|+\\ \left|G\left(1,3\right)-G\left(1,1\right)\right|+\\ \left|G\left(3,3\right)-G\left(3,1\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down}=\left|G\left(2,2\right)-G\left(2,4\right)\right|+\\ \left|G\left(1,3\right)-G\left(1,5\right)\right|+\\ \left|G\left(3,3\right)-G\left(3,5\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right}=\left|G\left(2,2\right)-G\left(4,2\right)\right|+\\ \left|G\left(2,4\right)-G\left(4,4\right)\right|+\\ \left|G\left(1,3\right)-G\left(3,3\right)\right|\end{array}$

[0126] FIGS. 10A-10D are diagrams explaining the method of calculating the correlation amounts in the third embodiment. The following explains the operation of the third embodiment. Furthermore, the different characteristics of the third embodiment lie in the operation of interpolation processing performed by the weighting factor calculator 55 and the interpolation amount calculator 56. As the other operations are the same as the second embodiment, the explanation thereof is omitted.

[0127] As shown in FIGS. 10A-10D, the correlation amount calculator 54 calculates the correlation amounts by dividing three grid points adjacent to the empty grid point into two pairs. For example, this is equivalent to up1 and up2 in FIG. 10A, left1 and left2 in FIG. 10B, down1 and down2 in FIG. 10C and right1 and right2 in FIG. 10D.

[0128] That is, with respect to the coordinates (k, p) of the empty grid point, the correlation amount calculator 54 calculates the correlation amounts C-up1, C-up2, C-down1, C-down2, C-left1, C-left2, C-right1, and C-right2 corresponding to the four directions of up, down, left and right of each pair as follows: $\begin{array}{c}C\text{-}\mathrm{up1}=\left|G\left(k,p+1\right)-G\left(k,p+1-n\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1,p-n\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{up2}=\left|G\left(k,p+1\right)-G\left(k,p+1-n\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1,p-n\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down1}=\left|G\left(k,p-1\right)-G\left(k,p-1+n\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1,p+n\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down2}=\left|G\left(k,p-1\right)-G\left(k,p-1+n\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1,p+n\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{left1}=\left|G\left(k,p+1\right)-G\left(k-n,p+1\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1-n,p\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{left2}=\left|G\left(k,p-1\right)-G\left(k-n,p-1\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1-n,p\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right1}=\left|G\left(k,p-1\right)-G\left(k+n,p-1\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1+n,p\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right2}=\left|G\left(k,p+1\right)-G\left(k+n,p+1\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1+n,p\right)\right|\end{array}$

[0129] (A Variant of Step S24).

[0130] With respect to function f described for the second embodiment (defined here as f(C)=1/(1+q×C), q=⅓), the weighting factor calculator 55 calculates the weighting factors α-up1, α-up2, α-down1, α-down2, α-left1, α-left2, α-right1, and α-right2 as follows:

[0131] α-up1=f(C-up1)/sf

[0132] α-up2=f(C-up2)/sf

[0133] α-down1=f(C-down1)/sf

[0134] α-down2=f(C-down2)/sf

[0135] α-left1=f(C-left1)/sf

[0136] α-left2=f(C-left2)/sf

[0137] α-right1=f(C-right1)/sf

[0138] α-right2=f(C-right2)/sf

[0139] (A Varient of Step S25). $\begin{array}{c}\mathrm{Here},\mathrm{sf}=f\left(C\text{-}\mathrm{up1}\right)+f\left(C\text{-}\mathrm{up2}\right)+f\left(C\text{-}\mathrm{down1}\right)+f\left(C\text{-}\mathrm{down2}\right)+\\ f\left(C\text{-}\mathrm{left1}\right)+f\left(C\text{-}\mathrm{left2}\right)+f\left(C\text{-}\mathrm{right1}\right)+f\left(C\text{-}\mathrm{right2}\right)\end{array}$

[0140] With respect to the weighting factors α-up1, α-up2, α-down1, α-down2, α-left1, α-left2, α-right1, and α-right2, the interpolation amount calculator 56 calculates the interpolation amount G′(k, p) of the empty grid point as follows: $\begin{array}{c}{G}^{\prime }\left(k,p\right)=\left(\alpha \text{-}\mathrm{up1}+\alpha \text{-}\mathrm{up2}\right)\times G\left(k,p-1\right)+\\ \left(\alpha \text{-}\mathrm{down1}+\alpha \text{-}\mathrm{down2}\right)\times G\left(k,p+1\right)+\\ \left(\alpha \text{-}\mathrm{left1}+\alpha \text{-}\mathrm{left2}\right)\times G\left(k-1,p\right)+\\ \left(\alpha \text{-}\mathrm{right1}+\alpha \text{-}\mathrm{right2}\right)\times G\left(k+1,p\right)\end{array}$

[0141] (A Variant of Step S26).

[0142] Thus, in the third embodiment, it is possible to improve the accuracy of the interpolation processing just like in the second embodiment.

[0143] The following explains the operation of the fourth embodiment. The different characteristics of the fourth embodiment lie in the operation of interpolation processing performed by the correlation amount calculator 54, the weighting factor calculator 55, and the interpolation amount calculator 56. The other operations are the same as the second embodiment. Thus, the explanation thereof is omitted.

[0144] The correlation amount calculator 54 calculates the correlation amounts corresponding to the four directions of up, down, right, and left for each of three grid points adjacent to the empty grid point. The grid point adjacent to the empty grid point in the upper direction is designated by U, the grid point adjacent to the empty grid point in the lower direction is designated by D, the grid point adjacent to the empty grid point in the left direction is designated by L, and the grid point adjacent to the empty grid point in the right direction is designated by R.

[0145] That is, with respect to the coordinates (k, p) of the empty grid point, the correlation amount calculator 54 calculates the correlation amounts C-up_D, C-up_L, C-up_R, C-down_U, C-down_L, C-down_R, C-left_U, C-left_D, C-left_R, C-right_U, C-right_D, and C-right_L corresponding to the four directions of upper, lower, right, and left of three grid points adjacent to the empty grid point as follows: $C\text{-}\mathrm{up\_D}=\left|G\left(k,p+1\right)-G\left(k,p+1-n\right)\right|$ $C\text{-}\mathrm{up\_L}=\left|G\left(k-1,p\right)-G\left(k-1,p-n\right)\right|$ $C\text{-}\mathrm{up\_R}=\left|G\left(k+1,p\right)-G\left(k+1,p-n\right)\right|$ $C\text{-}\mathrm{down\_U}=\left|G\left(k,p-1\right)-G\left(k,p-1+n\right)\right|$ $C\text{-}\mathrm{down\_L}=\left|G\left(k-1,p\right)-G\left(k-1,p+n\right)\right|$ $C\text{-}\mathrm{down\_R}=\left|G\left(k+1,p\right)-G\left(k+1,p+n\right)\right|$ $C\text{-}\mathrm{left\_U}=\left|G\left(k,p-1\right)-G\left(k-n,p-1\right)\right|$ $C\text{-}\mathrm{left\_D}=\left|G\left(k,p+1\right)-G\left(k-n,p+1\right)\right|$ $C\text{-}\mathrm{left\_R}=\left|G\left(k+1,p\right)-G\left(k+1-n,p\right)\right|$ $C\text{-}\mathrm{right\_U}=\left|G\left(k,p-1\right)-G\left(k+n,p-1\right)\right|$ $C\text{-}\mathrm{right\_D}=\left|G\left(k,p+1\right)-G\left(k+n,p+1\right)\right|$ $C\text{-}\mathrm{right\_L}=\left|G\left(k-1,p\right)-G\left(k-1+n,p\right)\right|$

[0146] (A Variant of Step S24).

[0147] With respect to function f shown in the second embodiment, the weighting factor calculator 55 calculates the weighting factors α-up_D, α-up_L, α-up_R, α-down_U, α-down_L, α-down_R, α-left_U, α-left_D, α-left_R, α-right_U, α-right_D, and α-right_L corresponding to each correlation amount as follows:

[0148] α-up_D=f(C-up_D)

[0149] α-up_L=f(C-up_L)

[0150] α-up_R=f(C-up_R)

[0151] α-down_U=f(C-down_U)

[0152] α-down_L=f(C-down_L)

[0153] α-down_R=f(C-down_R)

[0154] α-left_U=f(C-left_U)

[0155] α-left_D=f(C-left_D)

[0156] α-left_R=f(C-left_R)

[0157] α-right_U=f(C-right_U)

[0158] α-right_D=f(C-right_D)

[0159] α-right_L=f(C-right_L)

[0160] (A Variant of Step S25).

[0161] The interpolation amount calculator 56 calculates the interpolation amount G′(k, p) of the empty grid point as follows: $\begin{array}{c}{G}^{\prime }\left(k,p\right)=\alpha \text{-}\mathrm{up\_D}\times \left(\alpha \text{-}\mathrm{up\_L}+\alpha \text{-}\mathrm{up\_R}\right)/\mathrm{sf}\times G\left(k,p-1\right)+\\ \alpha \text{-}\mathrm{down\_U}\times \left(\alpha \text{-}\mathrm{down\_L}+\alpha \text{-}\mathrm{down\_R}\right)/\mathrm{sf}\times \\ G\left(k,p+1\right)+\alpha \text{-}\mathrm{left\_R}\times \left(\alpha \text{-}\mathrm{left\_U}+\alpha \text{-}\mathrm{left\_D}\right)/\mathrm{sf}\times \\ G\left(k-1,p\right)+\alpha \text{-}\mathrm{right\_L}\times \left(\alpha \text{-}\mathrm{right\_U}+\alpha \text{-}\mathrm{right\_D}\right)/\mathrm{sf}\times \\ G\left(k+1,p\right)\end{array}$

[0162] with respect to the weighting factors α-up_D, α-up_L, α-up_R, α-down_U, α-down_L, α-down_R, α-left_U, α-left_D, α-left_R, α-right_U, α-right_D, and α-right_L calculated by the weighting factor calculator 55 (a variant of step S26).

[0163] Here, $\begin{array}{c}\mathrm{sf}=\alpha \text{-}\mathrm{up\_D}\times \left(\alpha \text{-}\mathrm{up\_L}+\alpha \text{-}\mathrm{up\_R}\right)+\\ \alpha \text{-}\mathrm{down\_U}\times \left(\alpha \text{-}\mathrm{down\_L}+\alpha \text{-}\mathrm{down\_R}\right)+\\ \alpha \text{-}\mathrm{left\_R}\times \left(\alpha \text{-}\mathrm{left\_U}+\alpha \text{-}\mathrm{left\_D}\right)+\\ \alpha \text{-}\mathrm{right\_L}\times \left(\alpha \text{-}\mathrm{right\_U}+\alpha \text{-}\mathrm{right\_D}\right)\end{array}$

[0164] Thus, in the present embodiment, just like the second embodiment, it is possible to improve the accuracy of the interpolation processing.

[0165] FIG. 11 is an operational flow chart of the interpolation processing of a fifth embodiment. The structure of the hardware of the fifth embodiment is the same as the block diagram of the first embodiment shown in FIG. 2.

[0166] The following explains the operation of the fifth embodiment with reference to FIGS. 2 and 11. The different characteristics of the fifth embodiment lie in the operation of the interpolation processing. As the other operations are the same as the first embodiment, an explanation thereof is omitted.

[0167] When the controller 40 recognizes that the release button has been pressed, just like the first embodiment, it instructs the execution of the interpolation processing to the interpolation processor 47 within the signal processor 46. When thus instructed, the interpolation processor 47 selects an empty grid point which becomes the object of interpolation processing, just like the first embodiment (step S31).

[0168] When calculating the correlation amounts of the eight directions, which will be discussed later, it is desirable that at least three grid points exist in the four directions of up, down, right, and left of the empty grid point. For example, the determination as to whether or not the calculation of the correlation amount in the eight directions is possible can be made by the determination as to whether or not [4≦k≦X_max−3] and [4≦p≦Y_max−3] are established with respect to the arbitrary coordinates (k, p). Thus, in step S32, the interpolation processor 47 determines whether or not the calculation of the correlation amounts in the eight directions are possible based upon the coordinates of the selected empty grid point.

[0169] When the interpolation processor 47 recognizes that the calculation of the correlation amount in the eight directions is impossible by such a determination, the average value of the pixel outputs of the grid points adjacent to the empty grid point is defined as the interpolation amount (step S33).

[0170] When the interpolation processor 47 recognizes that the calculation of the correlation amounts in the eight directions is possible, it activates the correlation amount calculator 48, the similar direction detector 49, and the interpolation amount calculator 50. The correlation amount calculator 48 calculates the correlation amounts C-up, C-down, C-left, C-right, C-up_right, C-down_right, C-up_left, and C-down_left corresponding to the eight directions of up, down, left, right, upper right, lower right, upper left, and lower left with respect to the coordinates (k, p) of the empty grid points as follows: $\begin{array}{c}C\text{-}\mathrm{up}=\left|G\left(k,p-1\right)-G\left(k,p-1-2\mathrm{xm}\right)\right|·\mathrm{tu1}+\\ \left|G\left(k,p+1\right)-G\left(k,p+1-2\mathrm{xm}\right)\right|·\mathrm{tu2}+\\ \left|G\left(k-1,p\right)-G\left(k-1,p-2\mathrm{xm}\right)\right|·\mathrm{tu3}+\\ \left|G\left(k+1,p\right)-G\left(k+1,p-2\mathrm{xm}\right)\right|·\mathrm{t4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down}=\left|G\left(k,p-1\right)-G\left(k,p-1+2\mathrm{xm}\right)\right|·\mathrm{td1}+\\ \left|G\left(k,p+1\right)-G\left(k,p+1+2\mathrm{xm}\right)\right|·\mathrm{td2}+\\ \left|G\left(k-1,p\right)-G\left(k-1,p+2\mathrm{xm}\right)\right|·\mathrm{td3}+\\ \left|G\left(k+1,p\right)-G\left(k+1,p+2\mathrm{xm}\right)\right|·\mathrm{td4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{left}=\left|G\left(k,p-1\right)-G\left(k-2\mathrm{xm},p-1\right)\right|·\mathrm{tl1}+\\ \left|G\left(k,p+1\right)-G\left(k-2\mathrm{xm},p+1\right)\right|·\mathrm{tl2}+\\ \left|G\left(k-1,p\right)-G\left(k-1-2\mathrm{xm},p\right)\right|·\mathrm{tl3}+\\ \left|G\left(k+1,p\right)-G\left(k+1-2\mathrm{xm},p\right)\right|·\mathrm{tl4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right}=\left|G\left(k,p-1\right)-G\left(k+2\mathrm{xm},p-1\right)\right|·\mathrm{tr1}+\\ \left|G\left(k,p+1\right)-G\left(k+2\mathrm{xm},p+1\right)\right|·\mathrm{tr2}+\\ \left|G\left(k-1,p\right)-G\left(k-1+2\mathrm{xm},p\right)\right|·\mathrm{tr3}+\\ \left|G\left(k+1,p\right)-G\left(k+1+2\mathrm{xm},p\right)\right|·\mathrm{tr4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{up\_right}=\left|G\left(k,p-1\right)-G\left(k+m,p-1-m\right)\right|·\mathrm{tur1}+\\ \left|G\left(k,p+1\right)-G\left(k+m,p+1-m\right)\right|·\mathrm{tur2}+\\ \left|G\left(k-1,p\right)-G\left(k-1+m,p-m\right)\right|·\mathrm{tur3}+\\ \left|G\left(k+1,p\right)-G\left(k+1+m,p-m\right)\right|·\mathrm{tur4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down\_right}=\left|G\left(k,p-1\right)-G\left(k+m,p-1+m\right)\right|·\mathrm{tdr1}+\\ \left|G\left(k,p+1\right)-G\left(k+m,p+1+m\right)\right|·\mathrm{tdr2}+\\ \left|G\left(k-1,p\right)-G\left(k-1+m,p+m\right)\right|·\mathrm{tdr3}+\\ \left|G\left(k+1,p\right)-G\left(k+1+m,p+m\right)\right|·\mathrm{tdr4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{up\_left}=\left|G\left(k,p-1\right)-G\left(k-m,p-1-m\right)\right|·\mathrm{tul1}+\\ \left|G\left(k,p+1\right)-G\left(k-m,p+1-m\right)\right|·\mathrm{tul2}+\\ \left|G\left(k-1,p\right)-G\left(k-1-m,p-m\right)\right|·\mathrm{tul3}+\\ \left|G\left(k+1,p\right)-G\left(k+1-m,p-m\right)\right|·\mathrm{tul4}\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down\_left}=\left|G\left(k,p-1\right)-G\left(k-m,p-1+m\right)\right|·\mathrm{tdl1}+\\ \left|G\left(k,p+1\right)-G\left(k-m,p+1+m\right)\right|·\mathrm{tdl2}+\\ \left|G\left(k-1,p\right)-G\left(k-1-m,p+m\right)\right|·\mathrm{tdl3}+\\ \left|G\left(k+1,p\right)-G\left(k+1-m,p+m\right)\right|·\mathrm{tdl4}\end{array}$

[0171] (Step S34).

[0172] In the present embodiment, tu1−tu4, td1−td4, tl1−tl4, tr1−tr4, tur1−tur4, tdr1−tdr4, tul1−tul4, and tdl1−tdl4 are equal to 1 and the correlation amounts C-up, C-down, C-left, C-right, C-up_right, C-down_right, and C-up_left, and C-down_left corresponding to the eight directions are calculated. For example, when m=1 is established, the correlation amount C-up_right of the upper right direction is the sum of the absolute values of the differences between the pixel outputs of the four grid points shown by ◯ and the pixel outputs of the four grid points shown by ☆ in FIG. 12.

[0173] The similar direction detector 49 identifies the direction where the correlation is the strongest by selecting the smallest value while accounting for equality among the correlation amounts C-up, C-down, C-left, C-right, C-up_right, C-down_right, C-up_left, and C-down_left calculated by the correlation amount calculator 48 (step S35).

[0174] The interpolation amount calculator 50 determines whether or not the direction identified by the similar direction detector 49 is any of the directions of up, down, right, and left (step S36). When the interpolation amount calculator 50 determines that the correlation of any of the directions of up, down, right, and left is strong by this kind of determination (YES side of step S36), the pixel output of the grid point adjacent in the direction where the correlation is strong is the interpolation amount of the empty grid point, just like in the first embodiment (step S37). When the interpolation amount calculator 50 determines that the correlation of a diagonal direction is strong by the above-described determination (NO side of step S36), the average value of the pixel output of two grid points among the grid points adjacent to the empty grid point closest to that direction is defined as the interpolation amount of the empty grid point (step S38).

[0175] For example, when the correlation of the upper right direction is strong, the average value between the pixel output G(k, p−1) of the grid point adjacent in the upper direction and the pixel output G(k+1, p) of the grid point adjacent in the right direction is defined as G′(k, p). That is, G′(k, p) is calculated as follows:

[0176] G′(k, p)=(G(k, p−1)+G(k+1, p))/2.

[0177] Furthermore, when the correlation of the lower right direction is strong, G′(k, p) is calculated as follows:

[0178] G′(k, p)=(G(k, p+1)+G(k+1,p))/2,

[0179] when the correlation of the upper left direction is strong, G′(k, p) is calculated as follows:

[0180] G′(k, p)=(G(k, p−1)+G(k−1, p))/2,

[0181] and when the correlation of the lower left direction is strong, G′(k, p) is calculated as follows:

[0182] G′ (k, p)=(G(k, p+1)+G(k−1, p))/2.

[0183] Furthermore, when there are a plurality of directions where the correlation is strong, the average value of the pixel outputs of the grid points of these directions can be defined as the interpolation amount of the empty grid point.

[0184] When the interpolation amount of the empty grid point is thus determined, the interpolation processor 47 determines whether or not the interpolation processing of all the empty grid points is completed, just like in the first embodiment (step S39). The interpolation processor 47 repeatedly performs the above-described processing (the process of step S31 and after) until it determines that the interpolation processing of all the empty grid points is completed.

[0185] Thus, in the present embodiment, it is possible to estimate the spatial directivity of the empty grid point for not only the upper, lower, right, and left directions, but also for the diagonal directions. Therefore, according to the present embodiment, it is possible to further improve the accuracy of interpolation processing as compared to the first embodiment. In addition, in the present embodiment and the first embodiment, which was described above, the direction where the correlation is the strongest is identified by the similar direction detector 49, but, for example, it is also acceptable to predetermine the threshold value of the correlation amount which can be compared to the pixel output or the like and identify the direction corresponding to a correlation amount lower than the threshold value. Furthermore, it is also acceptable to identify two directions where the correlation is strong, and obtain the interpolation amount of the empty grid point based upon the average value of the pixel outputs of the grid points corresponding to these directions.

[0186] FIG. 13 is an operational flow chart of the interpolation processing of a sixth embodiment. The structure of the hardware of the sixth embodiment is the same as the block diagram shown in FIG. 7.

[0187] The following explains the operation of the sixth embodiment with reference to FIGS. 7 and 13. The different characteristics of the sixth embodiment lie in the operation of the interpolation processing. As the other operations are the same as the second embodiment, the explanation thereof is omitted.

[0188] The controller 40 instructs interpolation processor 53 within the signal processor 46 to execute the interpolation processing when it determines that the release button has been pressed, just like in the second embodiment. When thus instructed, the interpolation processor 53 selects the empty grid point which will become the object of the interpolation processing, just like in the second embodiment (step S41).

[0189] When the interpolation amount in the eight directions, which will be discussed later, is calculated, it is desirable that at least two pixels exist for the four directions of up, down, right, and left of the empty grid point. For example, the determination as to whether or not the calculation of the correlation amounts in the eight directions is possible can be performed by making the determination as to whether or not [3≦k≦X_max−2] and [3≦p≦Y_max−2] are established with respect to the arbitrary coordinates (k, p). Therefore, the interpolation processor 53 determines whether or not the calculation of the interpolation amounts in the eight directions is possible based upon the coordinates of the selected empty grid point (step S42).

[0190] When the interpolation processor 53 determines that the calculation of the correlation amounts in the eight directions is impossible by this kind of determination, the average value of the pixel outputs of the grid points adjacent to the empty grid point is defined as the interpolation amount (step S43).

[0191] When the interpolation processor 53 determines that the calculation of the correlation amounts in the eight directions is possible, the correlation amount calculator 54 activates the weighting factor calculator 55 and the interpolation amount calculator 56. The correlation amount calculator 54 calculates the correlation amounts C-up, C-down, C-left, C-right, C-up_right, C-down_right, C-up_left,and C-down_left corresponding to the eight directions as follows: $\begin{array}{c}C\text{-}\mathrm{up}=\left|G\left(k,p+1\right)-G\left(k,p+1-2\mathrm{xm}\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1,p-2\mathrm{xm}\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1,p-2\mathrm{xm}\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down}=\left|G\left(k,p-1\right)-G\left(k,p-1+2\mathrm{xm}\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1,p+2\mathrm{xm}\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1,p+2\mathrm{xm}\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{left}=\left|G\left(k,p-1\right)-G\left(k-2\mathrm{xm},p-1\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k-2\mathrm{xm},p+1\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1-2\mathrm{xm},p\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{right}=\left|G\left(k,p-1\right)-G\left(k+2\mathrm{xm},p-1\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k+2\mathrm{xm},p+1\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1+2\mathrm{xm},p\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{up\_right}=\left|G\left(k,p-1\right)-G\left(k+m,p-1-m\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k+m,p+1-m\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1+m,p-m\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1+m,p-m\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down\_right}=\left|G\left(k,p-1\right)-G\left(k+m,p-1+m\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k+m,p+1+m\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1+m,p+m\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1+m,p+m\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{up\_left}=\left|G\left(k,p-1\right)-G\left(k-m,p-1-m\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k-m,p+1-m\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1-m,p-m\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1-m,p-m\right)\right|\end{array}$ $\begin{array}{c}C\text{-}\mathrm{down\_left}=\left|G\left(k,p-1\right)-G\left(k-m,p-1+m\right)\right|+\\ \left|G\left(k,p+1\right)-G\left(k-m,p+1+m\right)\right|+\\ \left|G\left(k-1,p\right)-G\left(k-1-m,p+m\right)\right|+\\ \left|G\left(k+1,p\right)-G\left(k+1-m,p+m\right)\right|\end{array}$

[0192] (Step S44).

[0193] Then, the weighting factor calculator 55 calculates the weighting factors α-up, α-down, α-left, α-right, α-up_right, α-down_right, α-up_left, and α-down_left corresponding to the eight directions with respect to function f (here, f(C)=1/(1+q×C), q=⅓) just like in the second embodiment as follows:

[0194] α-up=f(C-up)/sf

[0195] α-down=f(C-down)/sf

[0196] α-left=f(C-left)/sf

[0197] α-right=f(C-right)/sf

[0198] α-up_right=f(C-up_right)/sf