Correlation Operational Apparatus for Multi-Dimensional Images

1. Field of the Invention

The present invention relates to a correlation operational apparatus for multi-dimensional images which performs a correlation operation on multi-dimensional images such as color images, and which can be used in stereo matching or image recognition.

2. Description of the Prior Art

In image recognition or stereo matching techniques utilized in extracting contours from aerial photographs, it is essential to improve the accuracy of correlation operations conducted on images. Various techniques of improving the accuracy of these correlation operations in stereo matching have been introduced in "Measurements of Three-Dimensional Surfaces for Two-Dimensional Images" from page 31 to page 36 of 17th Image Engineering Conference Papers 2-1, issued in 1986. These techniques involve the provision of geometrical constraints, verification of a three-dimensional position obtained from two images by means of a third eye, or tracking of a corresponding point by a slowly-moving camera for searching one of the images for an area which corresponds to an object area in the other image.

Although any of these techniques is effective in improving accuracy, most of them are not practical because they are impossible to produce or because the algorithms they use too complicated. Further, the above-described literature has introduced only techniques for variable density images, and does not propose techniques for improving the accuracy of correlation operations by handling multi-dimensional images such as color images which increase the amount of information.

Conventionally, since only techniques for conducting correlation operations on variable density images are known, multi-dimensional images which are colored must be handled as variable density images when correlation operations are conducted thereon. However, handling multi-dimensional images as variable density images does not ensure optimal correlation operation of multi-dimensional images which are thought to contain much more information than variable density images. Further, if the three primary color components that constitute a color image are separately handled and the results of correlation operations on the consequent three variable density images are added, the quantities of calculations are enormously increased whereas the accuracy of the correlations remains substantially the same. Most of the conventional techniques used in stereo matching for improving the accuracy of correlation operations are difficult to carry out, as stated previously

Accordingly, an object of the present invention is to provide an improved correlation operational apparatus for multi-dimensional images which can obviate the above-described problems of the known image correlation operation techniques, which is capable of greatly improving the accuracy as well as decreasing the quantities of calculations when compared with the handling of variable density images, and which is therefore very useful.

To this end, the present invention provides a correlation operational apparatus for multi-dimensional images which comprises: a device for reducing at least two-dimensional components of the multi-dimensional images, composed of at least two components, by at least one component by projecting a vector representing the images onto a space which is perpendicular to a vector that represents a large variance of distribution of the multi-dimensional image in a vector space whose orthogonal axes represent the individual dimensional component values of the multi-dimensional image: and a device for performing a correlation operation for each component on the multi-dimensional images whose dimensional components have been reduced by at least one.

In consequence, images having relatively a high number of spatial frequency components are extracted so as to enable a correlation operation to be conducted with a high degree of accuracy. Further, since correlation calculations are conducted on multi-dimensional images whose multi-dimensional components have been reduced by at least one, so as to make their components effective in the correlation operation, the quantities of calculations can be decreased.

FIG. 1 is a flowchart of a basic processing of a correlation operation which is conducted on two components by a correlation operational apparatus for color images according to the present invention;

FIG. 2 illustrates a positional relationship between a color image and its unit vectors in a three-dimensional R, G and B space;

FIG. 3 is a block diagram of a first embodiment of the present invention;

FIG. 4 is a block diagram of a correlation operational unit, showing an example of its structure;

FIGS. 5 and 5A are block diagram of the correlation operational unit, showing another example of its structure;

FIG. 6 is a block diagram of the correlation operational unit, showing still another example of its structure;

FIG. 7 is a block diagram of a second embodiment of the present invention;

FIG. 8 is a block diagram of a third embodiment of the present invention;

FIG. 9 is a block diagram of a fourth embodiment of the present invention;

FIG. 10 is a block diagram of a fifth embodiment of the present invention;

FIG. 11 is a block diagram of a seventh embodiment of the present invention;

FIG. 12 is a block diagram of an eighth embodiment of the present invention;

FIG. 13 is a block diagram of a ninth embodiment of the present invention;

FIG. 14 is a block diagram of a tenth embodiment of the present invention;

FIGS. 15A-B are a block diagram of an eleventh embodiment of the present invention;

FIG. 16 is a block diagram of a twelfth embodiment of the present invention; and

FIG. 17 is a block diagram of a thirteenth embodiment of the present invention.

The basic operation of a typical correlation operation which is conducted on two component values of color images by a correlation operational apparatus for multi-dimensional images according to the present invention will be described below with reference to FIG. 1. First, two digital color images f(r) and g(r), which are each composed of three-dimensional components red (R), green (G), and blue (B) and which are expressed by Equations (1) below, are input.

Next, in one [f(r) in this case] of the two color images f(r) and g(r), a unit vector n=(n.sub.1, n.sub.2, n.sub.3) which is the closest to the largest variance direction (main component direction) in the three-dimensional R, G and B space is obtained. Subsequently, the component values of the color images f(r) and g(r) are projected onto a plane (hereinafter referred to as a projection plane) perpendicular to the unit vector n in the R, G, and B space (this projection being hereinafter referred to as a projection transformation) so as to express the color images f(r) and g(r) as two component values.

This transformation is expressed using the equations in the manner described below. First, let the images obtained by the projection transformation of the color images f(r) and g(r) be f'(r) and g'(r), and let these projection-transformed images be expressed by:

To express the projection-transformed images f'(r) and g'(r) as two components, n.sub.x =(x.sub.1, x.sub.2, x.sub.3) and n.sub.y =y.sub.1, y.sub.2, 0) are used for unit vectors perpendicular to each other on the projection plane, and the component values of these vectors n.sub.x and n.sub.y are expressed by using the component values (n.sub.1, n.sub.2, n.sub.3) of the unit vector n as follows: ##EQU1##

The projection-transformed images f'(r) and g'(r) are represented by the unit vectors n.sub.x and n.sub.y on the projecting plane so as to express them as two components [f.sub.x (r), f.sub.y (r)] and [g.sub.x (r), g.sub.y (r)] as follows:

FIG. 2 illustrates the positional relationship between these vectors in the three-dimensional R, G, and B space.

A correlation operation expressed by Equation (5) below is performed on the thus-obtained projection transformed images f'(r) and g'(r) expressed as two components, and the results are output. ##EQU2##

If a correlation operation is to be conducted on a one-component value, the thus-obtained two component values are first transformed into a one-dimensional component by linear combination, and a correlation operation is then conducted thereon.

Thus, since the input color images f(r) and g(r) are projected onto a plane perpendicular to a vector which is closest to the main component direction, images having a relatively high number of high spatial frequency components are extracted so as to enable a highly accurate correlation operation. Further, since the three primary color components of a color image are transformed into two or one component which is effective for a correlation operation and a correlation operation is then performed on the consequent two or one component, the quantities of calculations can be decreased.

Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

FIG. 3 is a block diagram of a first embodiment of the present invention. This correlation operational apparatus for color images includes: an image inputting unit 20 consisting of a color image inputting device 1 and color image memories 2a and 2b; a main component analyzing unit 21 consisting of a covariance matrix operator 4, a memory 5, an eigenvalue operator 6, and an eigenvector operator 7; an image transforming unit 22 consisting of multiplication/addition operators 8a and 8b; a correlation operating unit 23 consisting of correlation operators 9a and 9b; an output image memory 24; and a controller 25 comprised of, for example, a CPU for controlling the operations of these constituents.

The structure of each constituent of the apparatus and the operation thereof will be described below. Two color images 0 are input to the color image inputting device 1 where they are converted into digital color images f(r) and g(r) having R, G and B three primary color components and suitable size and gradient, and the results are stored in the color image memories 2a and 2b, respectively. The color image inputting device 1 may be a drum scanner, if the input color images 0 are silver salt photographs. It may be a unit comprised of a combination of a matrix circuit and an A/D converter, if an NTSC signal is to be input. For R, G and B signals, it may be an A/D converter. It may alternatively be a unit comprised of a timing converter and a bus converter for digital R, G, and B signals.

Subsequently, the three component values [R.sub.f (r), G.sub.f (r), B.sub.f (r)] of the color image f(r) which has been stored in the color image memory 2b are input to the covariance matrix operator 4. The covariance matrix operator 4 is comprised by a squarer, a multiplier, an adder, and an accumulator, and is adapted to calculate covariance values .sigma..sub.RR.sup.2, .sigma..sub.GG.sup.2, .sigma..sub.BB.sup.2, .sigma..sub.RG.sup.2, .sigma..sub.GB.sup.2, .sigma..sub.BR.sup.2 of each of the three component values [R.sub.f (r), G.sub.f (r), B.sub.f (r)].

A covariance matrix V; is defined here as follows: ##EQU3##

N: the number of picture elements of an object image The covariance matrix elements are stored in the memory 5. They are also input to the eigenvalue operator 6 to solve a proper equation expressed by Equation (7).

where A=(A.sub.1, A.sub.2, A.sub.3).sup.t : an eigenvector and .alpha.: an eigenvalue.

The eigenvalue operator 6 is a circuit that calculates only the largest root of Equation (8) for obtaining the eigenvalue .alpha., and which is comprised of a multiplier, a divider, an adder and a table conversion memory. In Equation (8), E denotes a unit matrix. ##EQU4##

Subsequently, the covariance matrix elements which have been stored in the memory 5 and the output .alpha. from the eigenvalue operator 6 are both input to the eigenvector operator 7, where the components (n.sub.1, n.sub.2, n.sub.3) of an eigenvector n for the eigenvalue .alpha.are calculated on the basis of Equation (9), and the results are stored in a register provided in the eigenvector operator 6.

From (V-.alpha.E)n=0, ##EQU5##

Subsequently, the three primary color signals [R.sub.g (r), G.sub.g (r), B.sub.g (r)] and [R.sub.f (r), G.sub.f (r), B.sub.f (r)] of the color images g(r) and f(r) which have been respectively stored in the color image memories 2a and 2b are input to the multiplication/addition operators 8a and 8b, respectively. At the same time, the output n=(n.sub.1, n.sub.2, n.sub.3) from the eigenvector operator 7 is input to both of the multiplication/addition operators 8a and 8b so that a projection transformation expressed by Equation (10) is performed. ##EQU6##

The color images g'(r) and f'(r), which have been transformed into images composed of two components by the multiplication/addition operators 8a and 8b, are input for each component to the correlation operators 9a and 9b where correlation operations are conducted on these images. The outputs from the correlation operators 9a and 9b are input to the adder 10 where they are added, and the results of the addition are stored in the image memory 24 as a correlated image. Each of the correlation operators 9a and 9b may be a multiplication/addition operator which consists of a multiplier, an adder, and a memory, as shown in FIG. 4, and which is adapted to perform a correlation operation expressed by Equation (11). The correlation operator 9a or 9b may be constructed in such a manner that it incorporates a circuit which performs a fast Fourier transform (FFT), as shown in FIG. 5, so as to enable a correlation operation such as that expressed by Equation (12a) or a phase correlation such as that expressed by Equation (12b) to be performed on frequency images. Alternatively, it may be the one such as that shown in FIG. 6 in which a standard deviation for each component value is calculated beforehand so that each component value is divided by the standard deviation before a multiplication/addition operation is conducted thereon, as shown in Equation (13). ##EQU7##

A correlation operator such as that shown in FIG. 5A may also be employed for effecting a correlation operation. This correlation operator is constructed in the manner in which the correlation operator shown in FIG. 5 is structured with its absolute value converter, multiplier that multiplies the output of the absolute value converter and divider that divides the output of the multiplier being removed therefrom.

The correlated image which has been stored in the image memory 24 is utilized to attain various objects by being output to another apparatus or being analyzed by the controller.

In this embodiment, a vector which represents the largest variance in the three-dimensional R, G, and B space is obtained by actually performing main component analysis. In consequence, the most desirable projection transformation of the color images is enabled, enabling a correlation operation to be performed with a high degree of accuracy.

FIG. 7 is a block diagram of a second embodiment of the present invention. The correlation operational apparatus of this embodiment is comprised in the same manner as in the first embodiment with the exception that the main component analyzing unit 21 is replaced by a mean color direction operating unit 26. The mean color direction operating unit 26 is comprised of the low-pass filter 3, adders 11a, 11b, and 11c, memories 12a, 12b, and 12c, and a vector operator 13.

Regarding the operation of the apparatus of this embodiment, the three primary color component values (R.sub.f, G.sub.f, B.sub.f) of one image f(r) of the two color images f(r) and g(r) which have been input to the image inputting unit 20 are input to the low-pass filter 3 in the mean color direction operating unit 26 where the low spatial frequency components thereof are extracted. The three primary color values (R.sub.L, G.sub.L, B.sub.L) of the low spatial frequency area which are output from the low-pass filter 3 are input to the adders 11a to 11c, respectively, where they are added to the accumulated values of those of the previously input picture elements that have been stored in the memories 12a to 12c, and the results of the calculations are again stored in the memories 12a to 12c.

After this addition has been performed on all the picture elements of the color image f(r) the resultant values (S.sub.R, S.sub.G, S.sub.B) in the memories 12a to 12c are input to the vector operator 13 where three component values (n.sub.1, n.sub.2, n.sub.3) of a vector n which determines projection transformation are calculated. More specifically, the following equations are calculated: ##EQU10## The three component values (n.sub.1, n.sub.2, n.sub.3) of the vector n are stored in a register provided in the vector operator 13. The output n=(n.sub.1, n.sub.2, n.sub.3) from the vector operator 13 is input to the image transforming unit 22, as in the first embodiment, where it is used for projection transforming the input color images as the preliminary step of correlation operations conducted by the correlation operational unit 23.

This embodiment utilizes the fact that the R G B primary color images of a color image have a correlation, and that the distribution of the image in the three-dimensional R, G and B space tends to be largely in the lightness direction represented by Y=C.sub.1 R+C.sub.2 G+C.sub.3 B. This means that the main component direction actually obtained in the first embodiment and the mean color direction of the low spatial frequency area obtained in this embodiment are very close to each other. Therefore, in this embodiment, the same function and effect as obtained in the first embodiment are ensured by simply and separately adding the three primary color component values of the low spatial frequency area of the color image. Thus, according to this embodiment, the calculations can be simplified, thereby simplifying the apparatus which performs correlation operations with a high degree of accuracy.

FIG. 8 is a block diagram of a third embodiment of the present invention. The apparatus of this embodiment does not include the low-pass filter 3 that is incorporated in the mean color direction operating unit 26 of the second embodiment. More specifically, in this embodiment, the three primary color component values (R.sub.f, G.sub.f, B.sub.f) of one image f(r) of the two images f(r) and g(r) which have been input to the image inputting unit 20 are directly input to the adders 11a to 11c which perform additions on all the picture elements of the color image f(r) in cooperation with the memories 12a to 12c. The values (S.sub.R, S.sub.G, S.sub.B) which are respectively stored in the memories 12a to 12c are input to the vector operator 13, as in the second embodiment, where the three component values of a vector n that determines the projection transformation are calculated. Thereafter, the projection transformations and the correlation operations are performed in the same manner as in any of the previous embodiments.

This embodiment utilizes the facts that an image has in general a very high number of components of low spatial frequency area, and that calculation of the mean color direction using the components as they are, as in this embodiment, provides substantially the same effect as that obtained by calculating the mean color direction using the low-pass filtered components, as in the second embodiment. Therefore, in this embodiment, the substantially same effect as that obtained in the first and second embodiments can be obtained by simpler calculations of separately adding the three primary color component values of the color image. Thus, according to the third embodiment, the calculations are further simplified, thereby simplifying the apparatus which performs correlation operations with a high degree of accuracy.

FIG. 9 is a block diagram of a fourth embodiment of the present invention. In this embodiment, the two-dimensional components of the color images f'(r) and g'(r) which are produced by the image transforming unit in the first to third embodiments are transformed into one-dimensional components, and the correlation operation is then performed thereon. More specifically, an image transforming unit 28 of this embodiment consists of multiplication/addition operators 8a and 8b, and subtracters 14a and 14b, as shown in FIG. 9. The two-dimensional signals (g.sub.x, g.sub.y) and (f.sub.x, f.sub.y) which are respectively produced by the multiplication/addition operators 8a and 8b are input to the subtracters 14a and 14b, respectively, where one-dimensional signals are calculated as follows:

Originally, the two components of the color images f'(r) and g'(r) are linearly combined to obtain one-dimensional signals by:

the coefficients (c.sub.x, c.sub.y) in Equation (16) being set to the optimal values in accordance with the images. However, it is in general sufficient to set these coefficients to 1, -1, respectively, and the calculations are facilitated in this way. Therefore, the present embodiment employs the subtracters 14a and 14b to obtain the one-dimensional signals by Equation (15).

The outputs g.sub.z and f.sub.z from the subtracters 14a and 14b are input to the correlation operator 9 where a correlation operation is performed thereon, and the output of the correlation operator 9 is stored in the image memory 24.

Thus, in this embodiment, the two components which are effective for correlation operations and which are calculated from the three primary color components of the color images are linearly combined to obtain one-dimensional signals and a correlation operation is then conducted thereon. In consequence, the quantities of calculations can be further decreased, thereby reducing the size of the apparatus which is capable of performing a correlation operation with a high degree of accuracy.

FIG. 10 is a block diagram of a fifth embodiment of the present invention. The apparatus of this embodiment includes a correlation operating unit 23 which is constructed in a different way from that in which the correlation operating unit 23 used in the first to third embodiments is structured. In the correlation operating unit 23 of this embodiment the outputs g.sub.x, g.sub.y, f.sub.x, f.sub.y from the image transforming unit 22 are first input to FFT operators 101a and 101b, respectively, as complex number signals g.sub.c =g.sub.x +ig.sub.y and f.sub.c=f.sub.x +if.sub.y. The FFT operators 101a and 101b conduct Fourier transform on the complex number signals g.sub.c and f.sub.c, and output consequent complex number signals G.sub.c =G+iG' and F.sub.c =F+iF' to image memories 102a and 102b, respectively, where these complex number signals G.sub.c and F.sub.c are stored as real parts and imaginary parts.

At the next stage, the real part G and the real part F are input to a multiplier 103a from the image memories 102a and 102b, respectively, the imaginary part G' and the imaginary part F' are input to a multiplier 103b from the image memories 102a and 102b, respectively, the imaginary part G' and the real part F are input to a multiplier 103c from the image memories 102a and 102b, respectively, and the real part G and the imaginary part F' are input to a multiplier 103d from the image memories 102a and 102b, respectively, so that multiplications are performed on these values in the multipliers 103a, 103b, 103c, and 103d. The outputs from the multipliers 103a and 103b are input to an adder 104a where U=FG+F'G' is calculated, and the outputs from the mulitpliers 103c and 103d are input to a subtracter 104b so as to calculate V=F'G-FG'. The outputs of the adder 104a and the subtracter 104b are input to an inverse FFT operator 105 which regards (U+iV) as a complex number signal and which performs an inverse Fourier transform thereon. The real part of the output of the inverse FFT operator 105 is then output as the results of the inverse Fourier transform to the image memory 24.

The operation of this embodiment will now be described using equations. Transformations performed by the FFT operators 101a and 101b are expressed as follows:

where

{f.sub.x }=F.sub.x =F.sub.xR +i F.sub.xI

{f.sub.y }=F.sub.y =F.sub.yR +i F.sub.yI

{g.sub.x }=G.sub.x =G.sub.xR +i G.sub.xI

{g.sub.y }=G.sub.y =G.sub.yR +i G.sub.yI

In Equation (17), let the real and imaginary parts be F and F', respectively, and set:

F=F.sub.xR -F.sub.yI, and F'=F.sub.xI +F.sub.yR.

Similarly, in Equation (18), let the real and imaginary parts be G and G', respectively and set:

G=G.sub.xR -G.sub.yI, and G'=G.sub.xI +G.sub.yR.

Correlation between Equations (17) and (18) gives: ##EQU11## The four multiplications of FG, F'G', F'G, and FG' in the above Equation (19) are done by the multipliers 103a, 103b, 103c, and 103d shown in FIG. 10 and an addition of (FG+F'G') and subtraction of (F'G-FG') are performed by the adder 104a and subtracter 104b, respectively

Equation (19) can also be written as follows:

Inverse Fourier transform of Equation (20) gives:

where is an operator representing a correlation operation.

From Equation (21), it is clear that the real part (f.sub.x g.sub.x +f.sub.y g.sub.y) represents a correlation between the two-dimensional images (f.sub.x, f.sub.y) and (g.sub.x, g.sub.y).

Thus, in this embodiment, since the two components of the two-dimensional image signal are respectively treated as the real part and the imaginary part of a complex number during the fast Fourier transform (FFT) performed in the correlation operations, the size of the apparatus can be greatly reduced. The quantities of calculations performed can also be greatly reduced so as to enable correlation operations to be conducted at a high speed.

A sixth embodiment of the present invention employs three-dimensional color signals other than (R, G, B) signals as input color image signals 0 in any of the first to fifth embodiments shown in FIGS. 3 to 10. The color signals other than (R, G, B) signals may be (Y, I, Q) signals in an NTSC color TV system. In this embodiment, the flow of the (R, G, B) signals in any of the first to fifth embodiments is replaced by that of (Y, I, Q) signals, if (Y, I, Q) signals are to be employed.

In this way, correlation operations can be performed on any three-dimensional color signals such as color signals transmitted in the NTSC system without transforming them into (R, G, B) signals. This enables the present invention to be applied to any three-dimensional color signals without increasing the size of the apparatus.

This embodiment involves correlation operations performed only on two signals of three-dimensional signals, such as (I, Q) signals in (Y, I, Q) signals in the NTSC system which are composed of a Y signal representing brightness and biaxial components (I, Q) defined in planes perpendicular to the brightness, i.e., in planes representing hue and saturation, in color space. The above-described color space is defined to be a space whose orthogonal three axes represent the three-dimensional components of a color.

FIG. 11 is a block diagram of this embodiment. The input image signals 0 are assumed to be (Y, I, Q) signals. The two input image signals 0 are first input to the color image inputting device 1 where they are converted into digital image signals which have suitable size and gradient and which are each composed of two components of (I, Q). The two types of image signals g(r) and f(r) which are output from the color image inputting device 1 as (I, Q) two-dimensional images are stored in the image memories 2a and 2b, respectively. The two-component images which are represented by the two types of image signals g(r) and f(r) and which are stored in the image memories 2a and 2b are input to the correlation operators 9a and 9b where correlation operations are conducted for each of the I and Q components so as to obtain correlated signals .phi..sub.I (t) and .phi..sub.Q (t). The output signals .phi..sub.I (t) and .phi..sub.Q (t) from the correlation operators 9a and 9b are added to each other by the adder 10, and the results are stored in the image memory 24. The above-described operation of this embodiment is controlled by the controllers 25.

This embodiment utilizes the fact that in a color image, variance of components representing brightness is in general large, and involves correlation operations performed only on two components of a three-dimensional image, such as (I, Q) signals in (Y, I, Q) signals composed of a Y signal representing brightness in color space and two component (I, Q) signals on planes perpendicular thereto. Therefore, highly accurate correlation operations can be performed with a simple structure on signals composed of brightness and two components in planes (hue and saturation planes) perpendicular to brightness.

The present embodiment involve correlation operations performed on one-dimensional components obtained from the two components of three-dimensional signals, such as (I, Q) components of (Y, I, Q) signals composed of a Y signal representing brightness and biaxial components (I, Q) defined in planes perpendicular to brightness in color space. FIG. 12 is a block diagram of this embodiment. As shown in FIG. 12, the two component image signals [g.sub.I (r), g.sub.Q (r)] and [f.sub.I (r), f.sub.Q (r)] of the two-dimensional image signals g(r) and f(r) which have been input to and stored in the color image memories 2a and 2b in the same manner as in the seventh embodiment are input to the subtracters 14a and 14b, respectively, where subtractions are performed between the components of the signals, i.e., g.sub.Z (r)=g.sub.I (r)-g.sub.Q (r) and f.sub.Z (r)=f.sub.I (r)-f.sub.Q (r) are calculated. The outputs g.sub.Z (r) and f.sub.Z (r) of the subtracters 14a and 14b are input to the correlation operator 9, where a correlation operation is performed thereon, and the results of the operation is stored in the image memory 24. The above-described operation is controlled by the controller 25.

In this embodiment, since the (I, Q) signals of (Y, I, Q) signals which are thought to be effective for correlation operations are linearly combined into one-dimensional signals g.sub.Z =c.sub.1 g.sub.I +c.sub.2 g.sub.Q and f.sub.Z =c.sub.1 f.sub.I +c.sub.2 f.sub.Q and a correlation operation is performed thereon, the quantities of calculations can be further decreased. Although the coefficients (c.sub.1, c.sub.2) for the above linear combinations may be set to optimal values in accordance with images, it is in general sufficient to set them to (1, -1), the calculations being simplified in this way. Therefore, the present embodiment employs one-dimensional signals g.sub.Z =g.sub.I -g.sub.Q and f.sub.Z =f.sub.I -f.sub.Q which are obtained with the coefficients set to (1, -1).

A linear operator comprised by a multiplication/addition operator or by a look up table memory and an adder may be used to obtain more appropriate one-dimensional signals. A correlation operation conducted only on I or Q signals may ensure substantially the same effect as that of this embodiment.

As will be understood from the foregoing description, according to this embodiment, image signals each of which is composed of brightness and biaxial components in planes perpendicular to the brightness in color space are transformed into one-dimensional signals by linear combination, and a correlation operation is then performed thereon. Therefore, the quantities of calculations can be decreased, thereby reducing the size of the apparatus which is capable of performing the correlation operations with a high degree of accuracy.

Whereas, in the seventh and eighth embodiments, input signals such as (Y, I, Q) signals which are used in the NTSC system and are composed of a signal representing brightness and two component signals defined in planes perpendicular to the brightness in color space are transformed into two- or one-dimensional signals and correlation operations are then performed thereon, the ninth embodiment has a function of transforming input image signals 0 which are color three-dimensional signals, such as (R, G, B) signals, into two-dimensional signals defined in planes perpendicular to a signal representing brightness in color space, so as to ensure the same operation as those of seventh and eighth embodiments, this function being provided by a matrix operator incorporated in the color image inputting device 1.

FIG. 13 is a block diagram of this embodiment. The input color (R, G, B) three-dimensional signals 0 are stored in a buffer memory 110 in the color image inputting device 1. The (R, G, B) three-dimensional signals stored in the buffer memory 110 are concurrently input to a matrix operator 111 comprised by a multiplication/addition operator or by a look up table memory and an adder, where linear operation is conducted between the three-dimensional signals so as to obtain two-dimensional (I, Q) signals. The two two-dimensionally output image signals g(r) and f(r) which are output from the color image inputting device 1 as two-dimensional (I, Q) signals are stored in the color image memories 2a and 2b, respectively. Correlation operations are performed in the proceeding process on these image signals in the same manner as in the seventh or eighth embodiment.

Thus, in the present embodiment, two components representing hue and saturation, such as (I, Q) of (Y, I, Q) signals, are obtained from input three-dimensional signals, such as (R, G, B) signals, whose components are not divided into a component representing brightness and two components representing hue and saturation, and correlation operations are then performed thereon in the same manner as in the seventh or eighth embodiment. The process in which the three-dimensional signals are transformed into two-dimensional signals is expressed using matrix operation as follows: ##EQU12## wherein transformation matrix A may be a previously defined one, such as that used in (R, G, B) to (I, Q) transformation in an NTSC signal, and which is expressed by: ##EQU13##

If only specific types of color image are input, K-L transformation in terms of color such as that described in the first embodiment may be performed on the input image beforehand, and a matrix for transforming the input three-dimensional signal to a secondary or ternary main component signal may be defined as the transformation matrix A.

Thus, in this embodiment, since input three-dimensional image signals such as (R, G, B) which are not divided into a component representing brightness and two components representing hue and saturation, are transformed into two-dimensional signals, and correlation operations are then performed thereon, the quantities of calculations can be decreased, thereby decreasing the size of the apparatus which is capable of conducting correlation operations with a high degree of accuracy.

The tenth embodiment of this invention has a function of transforming input image signals 0 which are color three-dimensional signals such as (R, G, B) signals used in the ninth embodiment into one-dimensional signals defined in a plane perpendicular to the axis representing brightness in color space, the function being provided by a matrix operator incorporated in the color image inputting device 1. FIG. 14 is a block diagram of this embodiment. The color three-dimensional signals 0 are input to the buffer memory 110 in the color image inputting device 1. The three-dimensional signals stored in the buffer memory 110 are concurrently input to a matrix operator 112 comprised by a multiplication/addition operator or by a look up table memory and an adder where linear operation is conducted between the three-dimensional signals so as to obtain new one-dimensional signals. The two one-dimensional image signals g.sub.Z (r) and f.sub.Z (r) which are output from the color image inputting device 1 are input to the color image memories 2a and 2b, respectively. The output signals from the color memories 2a and 2b are input to the correlation operator 9, and the results of the operation .phi.(r) is stored in the image memory 24. The above-described operation is controlled by the controller 25.

Thus, in this embodiment, input three-dimensional signals such as (R, G, B) signals which are not divided into a component representing brightness and two components representing hue and saturation are transformed into one-dimensional signals defined in a plane representing hue and saturation in color space, and a correlation operation is then performed between the one-dimensional images. The one-dimensional signal can be expressed using linear combination between three components as follows:

where transformation coefficients (b.sub.1, b.sub.2, b.sub.3) may be ones which are defined beforehand such as I or Q signal in an NTSC signal, or secondary or ternary main components obtained by performing K-L transformation on limited types of images beforehand. Alternatively, they may be components by lineary combining the I and Q signals or the secondary and ternary main components.

Thus, in this embodiment, since input three-dimensional signals such as (R, G, B) signals which are not divided into a component representing brightness and two components representing hue and saturation are transformed into one-dimensional signals and a correlation operation is then performed thereon, the quantities of calculations can be reduced, thereby reducing the size of the apparatus which is capable of conducting correlation operations with a high degree of accuracy.

The eleventh embodiment of this invention involves correlation operations performed on multi-dimensional images having at least two dimensions using secondary and following main components in the main components obtained by the main component analysis which have small variance. FIG. 15 is a block diagram of this embodiment. A correlation operational apparatus for multi-dimensional images of this embodiment is roughly comprised by an image inputting unit 220, a main component analyzing unit 221, an image transforming unit 222, a correlation operational unit 223, an image memory 224, and a controller 225 for controlling the operations of these components.

Multi-dimensional input image signals 200 are assumed to be image signals having n dimensions (n.gtoreq.2). The multi-dimensional images may be images composed of a plurality of images obtained at a narrow wavelength area by the use of an interference filter, remote sensing images obtained using a plurality of bandwidths, or images composed of a plurality of images obtained under a different illumination condition. Of course, they may be color images.

The multi-dimensional input image signals 200 are converted into multi-dimensional digital image signals having suitable size and gradient by a multi-dimensional image inputting device 201. The two types of image which are output from the multi-dimensional image inputting device 201 are stored in image memories 202a and 202b, respectively. Supposing that the multi-dimensional image stored in the image memory 202b is f(r)={f.sub.1 (r), f.sub.2 (r), . . . f.sub.n (r)}, n component images {f.sub.1, f.sub.2, . . . f.sub.n } are input to a covariance matrix operator 204 comprised by a squarer, a multiplier, an adder, and a multiplication/addition operator, where a covariance matrix expressed by Equation (25) is calculated, and the results are stored in a memory 205. ##EQU14## where: C.sub.ij =<(f.sub.i -<f.sub.i >) (f.sub.j -<f.sub.j >)>

i, j=1, 2, . . . n

< >is an operator for set mean ##EQU15## N is the number of picture elements of an image, and

f=(f.sub.1, f.sub.2, . . . f.sub.n).sup.t.

The covariance matrix element values stored in the memory 205 are input to an eigenvalue operator 206 where n eigenvalues .alpha..sub.l (l=1, 2, . . . n) which are a solution for a proper equation expressed by Equation (26) are calculated

where:

B is a matrix whose column vector is composed of the eigenvector of C, and

.LAMBDA. is a diagonal matrix whose diagonal element is composed of the eigenvalue .alpha..sub.l of C, i.e.,

B=(b.sub.1, b.sub.2, . . . b.sub.n): b.sub.l (l=1, 2, . . . n) is an eigenvector expressed by b.sub.l =(b.sub.l1, b.sub.l2, . . . b.sub.ln).sup.t, and ##EQU16##

In an actual operation, the eigenvalue operator 206 solves the following Equation (27).

where E is a unit matrix.

If n is small, the eigenvalue operator 206 may be comprised by a multiplier, a divider, an adder, and a look up table memory, like the one which is applied to an operation in which n=3 in the first embodiment. However, the calculation process becomes complicated as n becomes large. In that case, the eigenvalue operator 206 may be comprised by a programmable microprocessor.

Subsequently, the covariance matrix elements stored in the memory 205 and the output .alpha..sub.l from the eigenvalue operator 206 are input to an eigenvector operator 207 where an eigenvector .alpha..sub.l for m eigenvalues .alpha..sub.l (m<n) in n eigenvalues .alpha..sub.l is calculated. The eigenvector operator 207 may be comprised by a multiplier, an adder, and a look up table memory. Alternatively, it may be a programmable microprocessor. The eigenvector operator 207 performs calculations which solve Equation (28).

Subsequently, the n-dimensional signals (g.sub.1, g.sub.2, . . . g.sub.n) and (f.sub.1, f.sub.2, . . . f.sub.n) of the multi-dimensional images g(r) and f(r) respectively stored in the image memories 202a and 202b are input to matrix operators 208a and 208b each of which is comprised by a multiplication/addition operator or a look up table memory and an adder, respectively. The output b.sub.l from the eigenvector operator 207 is input to both of the matrix operators 208a and 208b, and the two images g(r) and f(r) are projected in the directions of m eigenvectors so as to obtain the following transformed images: ##EQU17##

The thus-transformed multi-dimensional images g'(r) and f'(r) which are composed of m components are input to correlation operators 209-l to 209-m for each component where correlation operations are performed thereon. The outputs from the correlation operators 209-l to 209-m are added by an adder 210, and the output of the adder 210 is stored in an image memory 224.

Thus, in this embodiment, when correlation operations are to be performed on input images which are multi-dimensional images composed of n-dimensional components, the eigenvector directions of the image in the multi-dimensional space and the variances in those directions, i.e., the eigenvalues of the covariance matrix, are obtained by the main component analysis. In that case, the multi-dimensional space is a vector space whose orthogonal axes represent the individual component values of the multi-dimensional components. Then, m eigenvectors having small variance, i.e., having a small eigenvalue, are selected so as to project each of the multi-dimensional images in those m eigenvector directions and thereby decrease the number of dimensions of each of the multi-dimensional images from n to m. m may be a fixed value. Alternatively, it may be any value up to the number of correlation operators 209 which is set by the controller 225 by comparing the magnitude of the m eigenvalues. Thereafter, correlation operations are performed for each component on the multi-dimensional images transformed to m-dimensional images having small variance. In consequence, the quantities of operations are decreased, and correlation operations with a high degree of accuracy are enabled.

As will be understood from the foregoing description, in this embodiment, since a vector which is perpendicular to a vector having large variance in multi-dimensional space is actually obtained by calculations, an optimal projection transformation can be performed on multi-dimensional images. Further, accurate correlation operations requiring small quantities of calculations are enabled.

A correlation operational apparatus for multi-dimensional images having at least 2 dimensions of this embodiment includes a mean vector operating unit which substitutes for the main component analyzing unit 221 of the eleventh embodiment. FIG. 16 is a block diagram of this embodiment. As in the eleventh embodiment, n component values (f.sub.1, f.sub.2, . . . f.sub.n) of one image f(r) of the two multi-dimensional images f(r) and g(r) which have been input to the image inputting unit 220 are input to adders 211-l to 211-n, respectively, where they are added to the accumulated values of those of the previously input picture elements, and the results are again stored in memories 212-l to 212-n.

After this addition has been performed on all of the picture elements of the multi-dimensional image f(r), the resultant values (S.sub.1, S.sub.2, . . . S.sub.n) stored in the memories 212-l to 212-n are input to a vector operator 213 comprised by a squarer, a multiplier, an adder, and a look up table memory so as to obtain mean vector d=(d.sub.1, d.sub.2, . . . d.sub.n) first, where: ##EQU18##

The vector operator 213 then calculates two vectors (u and v) perpendicular to the mean vector d, where u and v are both n-dimensional unit vectors obtained from the relationships of d . u=d . v=u . v=0, d=u.times.v and .vertline.u.vertline.=.vertline.v.vertline.=1. Although the number of vectors obtained may be any value within (n-1), it is set to 2 in this embodiment to simplify the explanation.

Subsequently, n component values of the two multi-dimensional images f(r) and g(r) stored in the image memories in the image inputting unit 220 as well as the outputs (u and v) from the vector operator 213 are input to the image transforming unit 222 comprised by two matrix operators, where the multi-dimensional images f(r) and g(r) are respectively projected in (u and v) directions so as to transform them into two two-dimensional images f'(r)={f.sub.u (r), f.sub.v (r)} and g'(r)={g.sub.u (r), g.sub.v (r)}. Thereafter, correlation operations are performed between the images f'(r) and g'(r) by the correlation operator 223 comprised by two correlation operators and one adder in the same manner as in the first to third embodiments, and the results are stored in the image memory 224.

Only one unit vector u perpendicular to the mean vector d may be calculated by the vector operator 213, so that the multi-dimensional images f(r) and g(r) are projected in the unit vector u direction by the image transforming unit 222 so as to obtain transformed images f.sub.u (r) and g.sub.u (r). These one-dimensional images f.sub.u (r) and g.sub.u (r) are then correlated by a correlation operating unit 223 comprised by one correlation operator, and the results of the correlation are stored in the image memory 224.

This embodiment utilizes the fact that, when there is correlations between the component images of the multi-dimensional images, the main component (the primary main component) direction having the largest variance which is actually obtained in the eleventh embodiment is very close to the mean vector direction obtained in this embodiment. Therefore, in this embodiment, n-dimensional images can be transformed into images having a maximum of (n-1) dimensions which are effective for correlation operations by virtue of the simple operation of mainly adding the components of the multi-dimensional image. In consequence, the calculations and the apparatus which is capable of performing correlation operations at a high degree of accuracy are both simplified.

A thirteenth embodiment of this invention involves reduction of the number of dimensions of input multi-dimensional images whose types are limited and whose distribution in the multi-dimensional space are known by using the previously set transformation matrix without using the main component analyzing unit or the mean vector operating unit. FIG. 17 is a block diagram of this embodiment. The input multi-dimensional image signals 200 having n-dimensional components are temporarily stored in a buffer memory 214 in the multi-dimensional image inputting device 201, and n image component signals from the buffer memory 214 are input to a matrix operator 215, where the n-dimensional signals are transformed into the m-dimensional (m<n) signals using transformation matrix which has been appropriately set beforehand. The two types of multi-dimensional image g(r) and f(r) output from the multi-dimensional image inputting device 201 are stored in multi-dimensional image memories 202a and 202b, respectively

The output signals from the multi-dimensional image memories 202a and 202b are input for each component to the correlation operators 209-l to 209-m where correlation operations are performed thereon. The outputs from the correlation operators 209-l to 209-m are input to the adder 210 where they are added to obtain the total sum thereof, and the results are stored in the image memory 24. If the n-dimensional signals are transformed into one-dimensional signals by the matrix operator 215, no adder 210 is necessary. The above-described operation is controlled by the controller 225.

This embodiment is applied to a case in which only specific types of multi-dimensional image are input which enables estimation of the largest variance direction in the multi-dimensional space. Transformation matrix used to decrease the number of dimensions of the input multi-dimensional images may be set by performing K-L transformation on an object image group or the image group of the same type as the object image group or by obtaining mean vector. Therefore, the same correlation operations as those in the eleventh or twelfth embodiment can be provided without incorporating a main component analyzing unit or a mean vector operating unit.

Thus, in the present embodiment, multi-dimensional images whose largest variance direction in the multi-dimensional space can be presumed are correlated. Therefore, the quantities of calculations can be greatly decreased, decreasing the size of the apparatus which is capable of performing correlation operations with a high degree of accuracy.

As can be seen from the foregoing description, the present invention can be applied to stereo matching of color images which is conducted to grasp an object three-dimensionally in various fields such as extraction of contours from aerial photographs, analysis of stereographic surface structure of an object using endoscopic or microscopic images, and robot vision by a plurality of color images input from one movable camera or a plurality of cameras.

The present invention can also be adopted to the automatic focusing mechanism of a camera or a microscope so as to enable the accuracy of the known mechanism to be improved. The present invention can also be applied to automatic recognition technique of microscopic images which is used in the fields of cytodiagnosis in medical science, biology, earth science or examinations of metal materials. This means that an accuracy with which objects are determined in any of the above-mentioned fields can be improved because of use of color information in addition to the form thereof This may lead to enlargement of the application fields of the technique. Applications of this invention on an industrial basis includes painting inspection in a production line, a robot vision which recognizes both color and form, and a automatic sorting machine for perishable food In any of these cases, application of this invention enables a color as well as a form to be used as parameters.