Apparatus for Producing a Panoramic Image Using a Plurality of Optical Systems

1. Field of the Invention

This invention relates to an image pickup apparatus using an image pickup element such as a CCD (charge coupled device) or an image pickup tube, and particularly to a double-eye image pickup apparatus using a plurality of image pickup elements and an imaging optical system such as a lens.

2. Related Background Art

In recent years, for the purpose of forming a wide panoramic image or forming a highly minute image, there has been proposed a double-eye image pickup apparatus which has a plurality of image taking systems each comprising an imaging optical system and an image pickup element and picks up the image of a common object to thereby form and output a synthesized image by the use of an image signal input from each of said image pickup elements.

A system for obtaining a panoramic image is such that the plurality of different images of an object area are picked up at a time by a plurality of image taking systems and the same object existing in these images is extracted and the respective images are connected together on the basis of relative position information in the images to thereby obtain a synthesized panoramic image.

On the other hand, a system for obtaining a highly minute image is such that as in the case of a panoramic image, the same object existing in each image is extracted and an interpolating process is carried out on the basis of relative position information in the image to thereby newly obtain a highly minute image. A double-eye image pickup apparatus based on this principle is such that as shown, for example, in FIG. 22 of the accompanying drawings, a left image taking system 1010.sub.L and a right image taking system 1010.sub.R are prepared and the image of an object 1101 is picked up by the left image taking system 1010.sub.L and the right image taking system 1010.sub.R. A left image I.sub.L obtained by the left image taking system 1010.sub.L and a right image I.sub.R obtained by the right image taking system 1010.sub.R are then synthesized by sampling a corresponding point by a CPU 1120 to thereby obtain an output image I.sub.OUT which is highly minute as compared with a case where the image of an object is picked up by a single image taking system.

However, the above-described double-eye image pickup apparatuses have suffered from the following problems. In the apparatus for obtaining a panoramic image, a plurality of images are connected together and therefore are not limited within the maximum angle of field of an image taking system and further, a panoramic image is re-formed from a plurality of images picked up at the same timing and therefore, there can be realized a flexible image pickup apparatus of a construction comprising a combination of a plurality of image taking systems enough to be able to cover a desired object area including a moving object, but images are picked up in juxtaposed relationship with one another except for some connected portions thereof and therefore, rather almost all image regions become objects having no corresponding point. In this case, objects corresponding to each other in the connected portion of the images sometimes appear as a so-called double image in which the objects do not coincide with each other on the formed image but exist doubly, and this has led to a problem that the continuity of the image is lost, thus resulting in the deteriorated dignity of the image. Here, the continuity of the image refers to the fact that in a gazed portion, the shape (outline or the like), density (luminance) ,etc. of an object are connected continuously and smoothly.

Also, in the apparatus for obtaining a highly minute image, highly accurate corresponding point relative position information (one pixel pitch or less) is required and the image interpolating method is also required to be highly accurate. Accordingly, when such accuracy is not sufficiently satisfied, high frequency noise components which originally ought not to exist may be superposed one upon another, thus resulting in rather the deterioration of the image. On the other hand, as regards time resolution, when it is supposed that this double-eye image pickup apparatus is used for television, it is necessary that about 30 frames of images per second be continuously introduced and therefore, the processing of these images must be rapid and is desired to be as simple as possible.

The present invention has been made in view of these problems and a first object thereof is to provide a double-eye image pickup apparatus for forming an image which does not lose continuity from the image formation of an area in which a double image exists and the vicinity of the area, thereby preventing the deterioration of the image.

A second object of the present invention is to provide a double-eye image pickup apparatus which satisfies the real time required by the image pickup apparatus and yet is free of the deterioration of at least the quality of image as compared with an original image before a synthesized image is formed.

To achieve the above objects, the double-eye image pickup apparatus of the present invention has a plurality of imaging optical systems, a plurality of image taking systems each comprising an image pickup element corresponding to each of said imaging optical systems, and an image processing portion for forming and outputting a synthesized image by the use of image signals input from said image pickup elements by image-picking up an object, and is characterized in that pairs of corresponding points corresponding to the same portion of said object are sampled from each of said image signals, and with regard to at least some pairs of corresponding points of said sampled pairs of corresponding points, at least one pair of corresponding points of said some pairs of corresponding points are used as synthesized image forming data and the other pairs of corresponding points are not used as the synthesized image forming data.

Also, said synthesized image may be a panoramic synthesized image comprising said image signals juxtaposed and combined together, and in this case, said image processing portion may be:

one which, with regard to the pairs of corresponding points of said sampled pairs of corresponding points which belong to an image connected to a double image non-generating area image-picked up only by one image taking system, uses are said synthetized image forming data the corresponding point of those pairs of corresponding points which belongs to an image most probably connected to said double image non-generating area;

one which, with regard to the pairs of corresponding points of said sampled pairs of corresponding points which differ in the distance to a double image non-generating area image-picked up only by one image taking system, uses as said synthesized image forming data the corresponding point of those pairs of corresponding points which is nearest to said double image non-generating area; or

one which, with regard to the pairs of corresponding points of said sampled pairs of corresponding points which belong to an image connected to one of respective double image non-generating areas image-picked up only by one image taking system, uses the corresponding point in said one image as said synthesized image forming data, and with regard to the corresponding point belonging to images connected to both of said double image non-generating areas, enlarges the corresponding point in one of said two or more images in the direction of juxtaposition of the images and uses it as said synthesized image forming data.

Further, said synthesized image may be a highly minute image provided by synthesizing said image signals, and in this case, said image processing portion may be:

one which, with regard to the pairs of corresponding points of said sampled pairs of corresponding points in which the spacing between the corresponding points is smaller than a predetermined spacing, uses one corresponding point in said pairs of corresponding points as said synthesized image forming data;

one which, with regard to at least some of said pairs of corresponding points which require the correction of distortion before the sampling of the corresponding points, uses one of said some corresponding points as said synthesized image forming data;

one which, with regard to at least some of said sampled pairs of corresponding points in which the degree of blur of image foreseen from the in-focus object plane of each said image taking system and the numerical aperture of each said imaging optical system exceeds a predetermined threshold value, uses one corresponding point of said some pairs of corresponding points as said synthesized image forming data; or

one which, with regard to at least some of the pairs of corresponding points which, after said pairs of corresponding points are sampled, appear at a predetermined spatial frequency or higher alternately with a pixel point in which corresponding points forming a pair are not found and an area in which said corresponding points forming said pair are not found, uses one corresponding point of said some pairs of corresponding points as said synthesized image forming data.

Also, the double-eye image pickup apparatus has a plurality of imaging optical systems, a plurality of image taking systems each comprising an image pickup element corresponding to each of said imaging optical systems, and an image processing portion for forming and outputting a synthesized image by the use of image signals input from said image pickup elements by image-picking up an object, and is characterized in that one of said image signals is used as a reference image for the formation of the synthesized image and the other image signal is used as an auxiliary image, pairs of corresponding points corresponding to the same portion of said object are sampled from said image signals, and with regard to some pairs of corresponding points of said sampled pairs of corresponding points, the corresponding point in said reference image is used as synthesized image forming data and the corresponding point in said auxiliary image is not used as synthesized image forming data.

In this case, said synthesized image is a panoramic synthesized image comprising said image signals juxtaposed and combined together, and said image processing portion may be one which, with regard to the pairs of corresponding points of said sampled pairs of corresponding points which belong to an image connected to a double image non-generating area image-picked up only by one image taking system, uses as the reference image the image signal most probably connected to said double image non-generating area.

Further, said image taking system having said image pickup element outputting the image signal which provides said reference image may has the optical axes of its imaging optical system disposed toward said object lying in the front direction of the apparatus,

said synthesized image is a highly minute image comprising said image signals combined together, and

said image processing portion may be one which, with regard to the pairs of corresponding points of said sampled pairs of corresponding points in which the spacing between the corresponding point in said reference image and the corresponding point in said auxiliary image is smaller than a predetermined spacing, uses the corresponding point in said reference image as said synthesized image forming data and does not use the corresponding point in said auxiliary image as said synthesized image forming data,

said synthesized image is a highly minute image comprising said image signals combined together, and

said image processing portion may be one which, with regard to at least some of said pairs of corresponding points which require the correction of distortion before the sampling of the corresponding points, uses the corresponding point in said reference image as said synthesized image forming data and does not use the corresponding point in said auxiliary image as said synthesized image forming data,

said synthesized image is a highly minute image comprising said image signals combined together, and

said image processing portion may be one which, with regard to at least some of said sampled pairs of corresponding points in which the degree of blur of image foreseen from the distance from the in-focus object plane of each said image taking system and the numerical aperture of each said imaging optical system exceeds a predetermined threshold value, uses the corresponding point in said reference image as said synthesized image forming data and does not use the corresponding point in said auxiliary image as said synthesized image forming data,

said synthesized image is a highly minute image comprising said image signals combined together, and

said image processing portion may be one which, with regard to at least some of said sampled pairs of corresponding points which appear at a predetermined spatial frequency or higher alternately with a pixel point in which corresponding points forming a pair are not found and an area in which said corresponding points forming said pair are not found, uses the corresponding point in said reference image as said synthesized image forming data and does not use the corresponding point in said auxiliary image as said synthesized image forming data.

The double-eye image pickup apparatus of the present invention constructed as described above has an image processing portion which samples pairs of corresponding points corresponding to the same portion of an object from image signals, and with regard to at least some of the sampled pairs of corresponding points, uses at least one corresponding point of said some pairs of corresponding points as synthesized image forming data and does not use the other corresponding point as the synthesized image forming data, whereby the deterioration of the synthesized image is prevented. More particularly, in the formation of a panoramic synthesized image, a double image is sometimes created near the connected portion of two images, but by at least one corresponding point of said pairs of corresponding points being used as the synthesized image forming data and the other corresponding point being not used as the synthesized image forming data, said double image is removed and also, good connection of the image with a double image non-generating area is kept. Also, in the formation of a highly minute image, when the corresponding points of the pairs of corresponding points exist very proximately to each other, at least one corresponding point of the pairs of corresponding points is used as synthesized image forming data and the other corresponding point is not used as synthesized image forming data, whereby any unnecessary high frequency component created in sensitive reaction to the error of the relative pixel position of the pairs of corresponding points or a noise component superposed on a pixel value is suppressed. Further, in the formation of a highly minute image, the processing in a distorted image portion, a blurred image portion and an occlusion image portion in which pairs of corresponding points are not found is changed as described above, whereby the omission of any unnecessary processing procedure and the suppression of the deterioration of image by a wrong countermeasure are accomplished.

Also, by setting a reference image for the formation of a synthesized image, an image having at least a predetermined level of image quality is always output. Particularly, an image taking system having an image pickup element outputting an image signal which provides a reference image has the optical axis of its imaging optical system disposed toward said object, whereby an image from at least the front of the object can always be output without being subjected to any special processing, and even during close-up photographing, the photographer can effect photographing without receiving any unnatural feeling of operation.

FIG. 1 schematically shows the construction of a first embodiment of the double-eye image pickup apparatus of the present invention.

FIG. 2 is a block diagram of an image processing unit shown in FIG. 1.

FIG. 3 is a view of the essential portions of the optical system of the double-eye image pickup apparatus shown in FIG. 1 as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units.

FIGS. 4A and 4B are for illustrating the positional relation of an object when the image of the object is picked up by the double-eye image pickup apparatus shown in FIG. 1, FIG. 4A showing the position of the object when the object is seen in a direction similar to that in FIG. 3, and FIG. 4B being a view of the effective light receiving portions of right and left CCD sensors as they are seen from the image pickup lens unit side.

FIG. 5 shows left and right images when a composition in which a road sign is seen on the background of an automobile is image-picked up by the double-eye image pickup apparatus shown in FIG. 1.

FIG. 6 is for illustrating the tigonometrical survey method when the area division by a double image area-dividing unit shown in FIG. 2 is effected.

FIG. 7 is a histogram obtained by the double image area-dividing unit shown in FIG. 2 when the composition shown in FIG. 5 is image-picked up.

FIG. 8 shows a synthesized image obtained by synthesizing the left and right projected images shown in FIG. 5 by an image processing portion shown in FIG. 2.

FIG. 9 is a block diagram of an image processing portion in a second embodiment of the double-eye image pickup apparatus of the present invention.

FIG. 10 is an enlarged view of the area of the essential portions of the optical system shown in FIG. 3 which is indicated by hatching, for illustrating the image processing procedure of the second embodiment of the double-eye image pickup apparatus of the present invention.

FIG. 11 is a view of the essential portions of an optical system in a third embodiment of the double-eye image pickup apparatus of the present invention as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units.

FIG. 12 is a view of the effective light receiving portions of left and right CCD sensors shown in FIG. 11 as they are seen from the image pickup lens unit side.

FIG. 13 is a view of the essential portions of an optical system in a fourth embodiment of the double-eye image pickup apparatus of the present invention as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units.

FIG. 14 is a view of the effective light receiving portions of left and right CCD sensors shown in FIG. 13 as they are seen from the image pickup lens unit side.

FIG. 15 is a block diagram of an image processing portion in a fifth embodiment of the double-eye image pickup apparatus of the present invention.

FIGS. 16A and 16B are views when a composition in which a mountain is seen on the background of an automobile is image-picked up, for illustrating the image synthesizing process in a sixth embodiment of the double-eye image pickup apparatus of the present invention, FIG. 16A showing left and right images before image synthesis, and FIG. 16B showing an image after image synthesis.

FIG. 17 is a block diagram of an image processing portion in a seventh embodiment of the double-eye image pickup apparatus of the present invention.

FIG. 18 is a view of the essential portions of an optical system in the seventh embodiment of the double-eye image pickup apparatus of the present invention as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units.

FIG. 19 shows the pixel positions of CCD sensors with respect to portions of left and right images obtained by the optical system shown in FIG. 18.

FIG. 20 is a view of the essential portions of an optical system in an eighth embodiment of the double-eye image pickup apparatus of the present invention as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units.

FIG. 21 is a block diagram of an image processing portion in a ninth embodiment of the double-eye image pickup apparatus of the present invention.

FIG. 22 is a view for illustrating the principle of a double-eye image pickup apparatus for synthesizing two images obtained by picking up the image of a common object by two image taking systems to thereby obtain a highly minute image.

Some embodiments of the present invention will hereinafter be described with reference to the drawings.

FIG. 1 schematically shows the construction of a first embodiment of the double-eye image pickup apparatus of the present invention. The double-eye image pickup apparatus of the present embodiment obtains a panoramic synthesized image by parallel-connecting two images obtained by picking up the image of an object by two image taking systems, i.e., a right image taking system 10.sub.R and a left image taking system 10.sub.L.

The left image taking system 10.sub.L will first be described. The left image taking system 10.sub.L has an image pickup lens unit 11.sub.L as an imaging optical system incorporated in a lens barrel, not shown, a color resolving prism 12.sub.L for resolving light from an object into three primary colors mounted on the image pickup lens unit 11.sub.L, and three CCD sensors 13.sub.L (only one of which is shown) each having a rectangular effective light receiving portion as an image pickup element provided for each of the lights resolved by the color resolving prism 12.sub.L. The image pickup lens unit 11.sub.L is comprised of a plurality of lenses including a focusing lens unit 15.sub.L driven by a focus motor 14.sub.L and a magnification changing lens 17.sub.L driven by a zoom motor 16.sub.L, and the motors 14.sub.L and 16.sub.L are driven in conformity with control signals from an all-system control unit 21 and a focus zoom control unit 22, respectively, in a control unit 20 for controlling the optical system. On the other hand, the right image taking system 10.sub.R is also constructed similarly to the left image taking system 10.sub.L, and the optical axis L.sub.R of the image pickup lens and 11.sub.R of the right image taking system 10.sub.R is on the same plane as the optical axis L.sub.L of the image pickup lens unit 11.sub.L of the left image taking system 10.sub.L.

Also, the lens barrels in which the image pickup lens-units 11.sub.L and 11.sub.R are incorporated are coupled to the rotary shafts of convergence angle motors 18.sub.L and 18.sub.R, respectively, driven in conformity with a control signal from the convergence angle control unit 23 of the control unit 20. The rotary shafts of the convergence angle motors 18.sub.L and 18.sub.R extend in a direction perpendicular to a plane containing the optical axes L.sub.L and L.sub.R of the image pickup lens units 11.sub.L and 11.sub.R, respectively, and by the convergence angle motors 18.sub.L and 18.sub.R being driven, the image pickup lens units 11.sub.L and 11.sub.R are rotated with the color resolving prisms 12.sub.L, 12.sub.R and the CCD sensors 13.sub.L, 13.sub.R, and the angle formed by the optical axes L.sub.L and L.sub.R of the image pickup lens units 11.sub.L and 11.sub.R (the convergence angle) is set. In the respective image taking systems 10.sub.L and 10.sub.R, there are provided focus encoders 24.sub.L and 24.sub.R for detecting the positions of the focusing lens units 15.sub.L and 15.sub.R, respectively, zoom encoders 25.sub.L and 25.sub.R for detecting the positions of the magnification changing lens units 17.sub.L and 17.sub.R, respectively, and convergence angle encoders 26.sub.L and 26.sub.R for detecting the convergence angle. As these, use may be made, for example, of exterior type members like potentiometers, or members which detect the positions or angles thereof by the signal information of a driving system itself such as a pulse motor.

On the other hand, an image output unit 40 is connected to the CCD sensors 13.sub.L and 13.sub.R through an image processing unit 30 which is a feature of the present invention. The image processing unit 30, as shown in FIG. 2, has an image input portion 32 comprising a left image memory 31.sub.L and a right image memory 31.sub.R for holding video signals which are image signals from the CCD sensors 13.sub.L and 13.sub.R (see FIG. 1) of the image taking systems 10.sub.L and 10.sub.R, an image conversion portion 38 for forming a synthesized image on the basis of left and right images obtained from the video signals input to the image input portion 32, and a synthesized image memory 39 for holding the image synthesized by the image conversion portion 38 and outputting it to an image output unit 40. The image conversion portion 38 is divided into a corresponding point sampling unit 33 for sampling the pairs of corresponding points between the two images input to the image input portion 32 with regard to all pixels of a double image generating area which will be described later, a double image area-dividing unit 34 for calculating the three-dimensional position (distance information) of each pair of corresponding points from the result of the sampling of the pairs of corresponding points, and effecting the area division of the double image by that information, a double image mode-discriminating judging unit 35 for determining the image of the double images divided into areas which is more probably connected to a double image non-generating area, double image area in either image simultaneous-removal unit 36 for leaving a double image belonging to the image side which is more probably connected to said double image non-generating area and using it as synthesized image forming data, and removing a double image belonging to the other image side, and an after-removal processing unit 37 for applying post-processing to the image after the double image is removed.

FIG. 3 is a view of the essential portions of the optical system of the double-eye images pickup apparatus shown in FIG. 1 as they are seen in a direction perpendicular to a plane formed by the optical axes L.sub.L and L.sub.R of the image pickup lens units 11.sub.L and 11.sub.R, respectively. However, to simplify the illustration, the color resolving prisms 12.sub.L and 12.sub.R (see FIG. 1) are omitted and each one of the left and right CCD sensors 13.sub.L and 13.sub.R is only shown. As shown in FIG. 3, the image pickup lens unit 11.sub.R and CCD sensor 13.sub.R of the right image taking system 10.sub.R have an in-focus object plane 50.sub.R, and the area which can be image-picked up is limited to the area defined between straight lines 51.sub.R and 52.sub.R, by the effective light receiving portion of the CCD sensor 13.sub.R, and the area from a line of intersection B.sub.R between the in-focus object plane 50.sub.R and end surfaces 51.sub.R and 52.sub.R to a line of intersection A provides an effective object area. Likewise, with regard to the left image taking system 10.sub.L, the area from a line of intersection A on an in-focus object plane 50.sub.L to a line of intersection B.sub.L provides an effective object area. The focus motors 14.sub.L, 14.sub.R (see FIG. 1) and zoom motors 16.sub.L, 16.sub.R (see FIG. 1) of the left and right image taking systems 10.sub.L, 10.sub.R are controlled so that the distances between the in-focus object planes 50.sub.L, 50.sub.R and the CCD sensors 13.sub.L, 13.sub.R and the imaging magnifications may be equal between the left and the right, and the convergence angle motors 18.sub.L, 18.sub.R (see FIG. 1) are controlled so that the end portions of the effective object areas of the image taking systems 10.sub.L, 10.sub.R may coincide with each other on the line of intersection A. The control of the motors 14.sub.L, 14.sub.R, 16.sub.L, 16.sub.R, 18.sub.L and 18.sub.R i.sup.s effected through the control unit 20 (see FIG. 1) which has received the signals from the encoders 24.sub.L, 24.sub.R, 25.sub.L, 25.sub.R, 26.sub.L and 26.sub.R (see FIG. 1). Particularly, the convergence angle motors 18.sub.L and 18.sub.R are controlled in response to the signals of the positions of the in-focus object planes 50.sub.L, 50.sub.R and the position of the end portion of the effective object area which are calculated from the output signals from the focus encoders 24.sub.L, 24.sub.R and zoom encoders 25.sub.L, 25.sub.R.

In the system constructed as described above, one of objects image-picked up by the image taking systems 10.sub.L and 10.sub.R which has the possibility of being included in both images is an object existing in an area 53 indicated by hatching (an area having a boundary sandwiched between end surfaces 51.sub.L and 52.sub.R and farther than the line of intersection A). Here, when with it being taken into account that the area 53 having the possibility of being included in these both images expands to infinity relative to the image taking systems 10.sub.L and 10.sub.R, the area 53 is projected onto the in-focus object planes 50.sub.L and 50.sub.R toward the centers of the object side principal planes of the image pickup lens units 11.sub.L and I 11.sub.R, the area 53, with regard to the right image taking system 10.sub.R, corresponds to the area 55.sub.R between the line of intersection D.sub.R between an arrow C.sub.R indicating a plane passing through the center of the object side principal plane of the image pickup lens unit 11.sub.R and parallel to the end surface 51.sub.L and the in-focus object plane 50.sub.R and the line of intersection A. Likewise, with regard to the left image taking system 10.sub.L, the area 53 corresponds to the area 55.sub.L between the line of intersection D.sub.L between an arrow C.sub.L indicating a plane passing through the center of the object side principal plane of the image pickup lens unit 11.sub.L and parallel to the end surface 52.sub.R and the line of intersection A. From these facts, what is image-picked up apparently as objects on the areas 55.sub.L and 55.sub.R may become a double image, and these areas-55.sub.L and 55.sub.R become a double image generating area. The other area is an area which is image-picked up only by one image taking system and therefore, becomes a double image non-generating area in which no double image is not generated.

The image processing procedure in the double-eye image pickup apparatus of the present embodiment will now be described with reference to FIG. 2.

When an object is image-picked up by the left and right image taking systems 10.sub.L and 10.sub.R, image information is input as a video signal for each of the image taking systems 10.sub.L and 10.sub.R to the image input portion 32 of the image processing portion 30, and is held in the respective image memories 31.sub.L and 31.sub.R. They are FIGS. 4A and 4B that show an example of the positional relation between each image taking system 10.sub.L, 10.sub.R and the object at this time, FIG. 4A showing the position of the object when the object is seen in a direction similar to that in FIG. 3, and FIG. 4B is a view of the effective light receiving portions of the left and right CCD sensors when the object lying at the position shown in FIG. 4A is image-picked up as they are seen from the image pickup lens unit side. As shown in FIG. 4A, an object 60 existing in the area 53 having the possibility of being included in the both images is projected onto the CCD sensors 13.sub.L and 13.sub.R so as to apparently exist in areas 56.sub.L and 56.sub.R on the in-focus object planes 50.sub.L and 50.sub.R of the left and right image taking systems 10.sub.L and 10.sub.R. That is, as shown in FIG. 4B, on the right CCD sensor 13.sub.R, an image 60.sub.R corresponding to the object 60 is projected into the area 55.sub.R between the line of intersection D.sub.R and the line of intersection A, and on the left CCD sensor 13.sub.L, an image 60.sub.L corresponding to the object 60 is projected into the area 55.sub.L between the line of intersection D.sub.L and the line of intersection A. It is to be understood here that the left and right CCD sensors 13.sub.L and 13.sub.R count the pixel positions in horizontal and vertical directions with points L.sub.O and R.sub.O as origins, respectively, and the positions of the images 60.sub.L and 60.sub.R in the respective images are indicated by the coordinates (m.sub.L, n.sub.L) and (m.sub.R, n.sub.R) of representative points such as the centroids thereof.

This will be specifically described with a case where a composition in which a road sign is seen on the background of an automobile is image-picked up taken as an example. FIG. 5 shows left and right projected images when a composition in which a road sign is seen on the background of an automobile has been image-picked up. The automobile 61 is located substantially on the in-focus object planes 50.sub.L and 50.sub.R (see FIGS. 3, 4A and 4B) of the two image taking systems, respectively. In FIG. 5, areas 55.sub.L and 55.sub.R are double image generating areas as described above, and the in-focus object planes 50.sub.L and 50.sub.R of the respective image taking systems are on the automobile 61 and therefore, in the left and right images, the automobile 61 is smoothly connected, while the road sign 62 is behind the automobile 61 and therefore appears doubly in the two images.

Subsequently, the pairs of corresponding points of each image are sampled in the corresponding point sampling unit 33 shown in FIG. 2. As a typical corresponding point sampling method, there is the template matching method. This method is to consider, for example, a template surrounding a certain point in the left image, and determine a corresponding point by the comparison of the similarity of the right image to the image of the template. The correlation method which is a method of comparison of similarity is to take the correlation between the pixel value in the image of the template and the pixel value in a searched-for image, and use the coordinates in which the correlation assumes a maximum value as a corresponding point, and the relational expression thereof is shown below. ##EQU1##

In equation (1), R(m.sub.R, n.sub.R) and L(m.sub.L, n.sub.L) are the pixel values of the right and left images, and .sigma.(m.sub.R, n.sub.R, m.sub.L, n.sub.L) represents the degree of correlation. Also, m.sub.R, n.sub.R, m.sub.L and n.sub.L indicate pixel coordinates. It is because the pixel coordinates axis shown in FIGS. 4A and 4B is defined so as to be bisymmetrical that in the square sum or product sum calculation, the signs before i and j become converse between the right and left images. In the normalized correlation of equation (1) , the maximum value is 1.

In the present embodiment, with regard to the pixel points in the area 55.sub.R of the right image shown in FIG. 4B, the values of equation (1) corresponding to all points in the area 55.sub.L of the left image are first calculated, and the pairs of coordinates of the right and left images which assume a maximum value are calculated. Next, of the calculated pairs of coordinates, a pair of corresponding points which become a double image is adopted with regard only to a case where the value .sigma. of equation (1) is equal to or greater than a predetermined threshold value .sigma.th, that is, .sigma..gtoreq..sigma..sub.th. This is at least necessary to prevent the deterioration of the synthesized image by it being erroneously judged to be a double image, because in the case of the wide image taking systems of the present embodiment, of the area projected onto the in-focus object planes 50.sub.L and 50.sub.R, the double image non-generating area occupies a considerable portion.

After the pairs of corresponding points of the left and right images are sampled, the area division of the double image is effected by the double image area-dividing unit 34 shown in FIG. 2. In the double image area-dividing process, the positions of the pairs of corresponding points in three-dimensional space are found by the trigonometrical survey method shown below.

When as shown in FIG. 6, the center points O.sub.L and O.sub.R of the object side principal planes of the left and right image pickup lens units 11.sub.L and 11.sub.R (see FIG. 3) are disposed line-symmetrically on the X axis with respect to the Z axis and the length of a base line linking the center points O.sub.L and O.sub.R together is a base line length b, the coordinates of the center points O.sub.L and O.sub.R are represented by (-b/2,0,0) and (b/2,0,0), respectively. Also, the projection points on the left and right CCD sensors 13.sub.L and 13.sub.R when a point P in three-dimensional space is projected toward the center points O.sub.L and O.sub.R are P.sub.L and R.sub.R, respectively, and the coordinates of the points P, P.sub.L and P.sub.R are represented by (X, Y, Z) , (X.sub.L, Y.sub.L, Z.sub.L) and (X.sub.R, Y.sub.R, Z.sub.R), respectively. Here, a plane formed by linking the three points P, P.sub.L and P.sub.R in three-dimensional space together is called an epipolar plane, and the line of intersection between the epipolar plane and the sensor surface is called an epipolar line.

At this time, the coordinates (X, Y, Z) of the point P are given by equations (2), (3) and (4) shown below. ##EQU2##

On the other hand, when the angle formed by the optical axes L.sub.L and L.sub.R of the left and right image pickup lens units 11.sub.L and 11.sub.R with respect to a straight line passing through the center points O.sub.L and O.sub.R of the object side principal planes thereof and parallel to the Z axis (this angle is called a convergence angle) is defined as .theta. and the focal length of each image pickup lens unit 11.sub.L, 11.sub.R is f, there are established the relations that

and the coordinates (X, Y, Z) of the point P are calculated from the above equations.

There is then found a histogram in which the point P (X, Y, Z) is the positions of the pairs of corresponding points of the double image and the Z coordinates value thereof (the distance in the direction of depth) is a parameter. From this histogram, area division is effected with an area in which the depth distance becomes discontinuous as the boundary.

For example, in the case of the images shown in FIG. 5, there is obtained a histogram as shown in FIG. 7 wherein the image is divided into two areas .alpha. and .beta.. That is, the area shown in FIG. 7 corresponds to the automobile 61 shown in FIG. 5, and the area .beta. shown in FIG. 7 corresponds to the road sign 62 shown in FIG. 5.

After the area division is effected, the area-divided double images are subdivided into four modes shown below, by the double image mode-discriminating judging unit 35 shown in FIG. 2. That is, in the images shown in FIG. 4B, they are the following four modes:

(Mode 1): a double image including a point on the line of intersection D.sub.R and not including any point on the line of intersection D.sub.L ;

(Mode 2): a double image not including any point on the line of intersection D.sub.R and including a point on the line of intersection D.sub.L ;

(Mode 3): a double image not including any point on the line of intersection D.sub.R and not including any point on the line of intersection D.sub.L ; and

(Mode 4): a double image including a point on the line of intersection D.sub.R and including a point on the line of intersection D.sub.L. With regard to the double image mentioned as Mode 1, the sum total S1 of the area thereof is found and also, with regard to the double image mentioned as Mode 2, the sum total S2 of the area thereof is found. In the present embodiment, the double images mentioned as Mode 3 and Mode 4 are not subjected to any special processing.

Then, if the area S1 found by the double image mode-discriminating judging unit 35 is greater than or equal to the area S2, only the double image corresponding points existing in the right image are left and the corresponding points in the left image are removed, by double image area in either image simultaneous-removal unit 36 shown in FIG. 2. If conversely, the area S1 is smaller than the area S2, only the double image corresponding points existing in the left image are left and the corresponding points in the right image are removed. This is based on the concept that if the area of the double image mentioned as Mode 1 is great, the-double image in the right image is more connected with the area in which no double image is generated, and if the area of the double image mentioned as Mode 2 is great, the double image in the left image is more connected with the area in which no double image is generated.

When the corresponding points in one image are removed, the pixels after the removal are subjected to post-processing such as the substitution of the value from the surrounding pixel, by the after-removal processing unit 37 shown in FIG. 2. Thereafter, the left and right images are connected together and held in the synthesized image memory 39, and are output to the image output unit 40 such as a monitor.

FIG. 8 shows an image obtained by the left and right images shown in FIG. 5 being subjected to the above-described processing and connected together. As shown in FIG. 8, the road sign lying rearwardly of the in-focus object planes 50.sub.L and 50.sub.R (see FIG. 3) is classified as the aforedescribed Mode 2, and the double image 62' in the area 55.sub.R of the left image is removed. On the other hand, the automobile 61 is classified as Mode 4 and therefore is not particularly subjected to processing, but it is substantially on the in-focus object planes 50.sub.L and 50.sub.R and therefore is continuously connected. Thereby, a natural image free of the generation of a double image is obtained.

One double image is removed by the processing as described above, whereby there can be left a double image component of the left and right images in which a component connected to the area in which no double image is generated is greater and therefore, there can be obtained a synthesized image having a large portion smoothly connected to the area in which no double image is generated.

A second embodiment of the double-eye image pickup apparatus of the present invention will now be described. FIG. 9 is a block diagram of an image processing portion in the second embodiment of the double-eye image pickup apparatus of the present invention. The present embodiment is similar in construction to the first embodiment shown in FIGS. 1, 3, 4A and 4B except for the construction of the image processing portion and therefore, portions similar to those of the first embodiment are given similar reference numerals and need not be described. The construction of the image processing portion will hereinafter be described.

As shown in FIG. 9, the image processing portion 130 has an image input portion 32 comprising a left image memory 31.sub.L and a right image memory 31.sub.R for holding video signals from the CCD sensors 13.sub.L and 13.sub.R (see FIG. 1) of image taking systems 10.sub.L and 10.sub.R, respectively, an image conversion portion 138 for forming a synthesized image on the basis of left and right images obtained from the video signals input to the image input portion 32, and a synthesized image memory 39 for holding the image synthesized by the image conversion portion 138 and outputting it to an image output unit 40. The image conversion portion 138 is subdivided into a corresponding point sampling unit 133 for sampling the pairs of corresponding points in the two images input to the image input portion 32 which are between the two images with regard to all pixels of a double image generating area, a double image area-dividing unit 134 for calculating the position of each corresponding point from the result of the sampling of the pairs of corresponding points, and effecting the area division of the double image by that information, an either image of two systems removal unit 136 for leaving at least one double image and using it as synthesized image forming data, on the basis of the areas divided by the double image area-dividing unit 134, and an after-removal processing unit 137 for applying post-processing to the image after the double image is removed.

The image processing procedure of the present embodiment will now be described. In the arrangement as shown in FIGS. 3 and 4A, it is often the case that the main object is located in the central portion of the panoramic synthesized image and moreover in the in-focus portion centering around the line of intersection A. Therefore, when another object exists on the background of the main object, in the right image, the image of said another object appears at the right of the main object, and in the left image, the image of said another object appears at the left of the main object, and said another object becomes a double image. In this case, if as shown in FIG. 10, the area 53 shown in FIG. 3 is bisected into a left area 53L and a right area 53R, the image of said another object in the right image is considered to be continuously or smoothly connected to the image from the right area and likewise, the image of said another object in the left image is considered to be connected to the image from the left of the left area 53L.

On the basis of the above-described consideration, image reconstruction attaching importance to the continuity of the image is effected in the present embodiment. For this purpose, the process of leaving the image of the object existing in the right area 53R in the right image shown in FIG. 4B and leaving the image of the object existing in the left area 53L in the left image shown in FIG. 4B is carried out. Judgment as to whether the object exists in the right area 53R or in the left area 53L can be done by finding the pairs of corresponding points of the double image by a corresponding point sampling method similar to that adopted in the first embodiment, and when the typical point coordinates (m.sub.L, n.sub.L) of the image 60.sub.L shown in FIG. 4B and the typical point coordinates (m.sub.R, n.sub.R) of the image 60.sub.R shown in FIG. 4B are here considered to be a pair of corresponding point coordinates after the corresponding point sampling process, actually the following branching-off processes can be carried out as viewed from the positional relation between the images 60.sub.L and 60.sub.R :

(a) To leave the image of the object in the right image if m.sub.R >m.sub.L ;

(b) To leave the image of the object in the left image of m.sub.R <m.sub.L ; and

(c) To leave the image of the object in the left and right images if m.sub.R =m.sub.L.

With regard to the removed pixel, for example, the post-process of substituting the signal value of the pixel around it for it can be carried out. The left and right images formed by the above-described processing are synthesized so as to join each other on the lines of intersection A and A', whereby there can be obtained a panoramic synthesized image having no discontinuous portion created in the main object area and in which the creation of discontinuity in the background image area, i.e., the generation of a double image, is minimized.

A further effect of the present embodiment is that since the lines of intersection A and A' which are the end portions of the in-focus object planes 50.sub.R and 50.sub.L are controlled so as to always coincide with each other, the image of the main object always becomes continuous and-smooth and also, an image within a predetermined range of angle of field can always be photographed even during the movement of the in-focus object planes 50.sub.L and 50.sub.R resulting from any change in the distance of the main object.

A third embodiment of the double-eye image pickup apparatus of the present invention will now be described. This embodiment differs in the convergence angles of the left and right image taking systems from the second embodiment, and will hereinafter be described with reference to FIG. 11. FIG. 11 is a view of the essential portions of an optical system in the third embodiment of the double-eye image pickup apparatus of the present invention as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units. In FIG. 11, portions similar to those shown in FIG. 3 are given the same reference characters and the schematic construction of the apparatus which is not shown is similar to that of the second embodiment and therefore need not be described.

As shown in FIG. 11, the convergence angles of left and right image taking systems 10.sub.L and 10.sub.R are controlled so that the point of intersection between end surfaces 51.sub.L and 52.sub.R which are the end surfaces of the effective object areas of the image taking systems 10.sub.L and 10.sub.R may coincide with the line of intersection A' between the in-focus object planes 50.sub.L ' and 50.sub.R ' of the image taking systems 10.sub.L and 10.sub.R during close-up photographing. In this case, the in-focus distance is varied by the zooming of image pickup lens units 11.sub.L and 11.sub.R and the end surfaces 51.sub.L and 52.sub.R which are the end surfaces of the effective object areas are varied with the fluctuation of the effective angle of field caused by zooming and therefore, the control of the convergence angles is effected in operative association with zoom control. Focusing and zoom control are effected as in the first embodiment. Also, the positions of the in-focus object planes 50.sub.L and 50.sub.R are calculated from the out signal values of focus encoders 24.sub.L, 24.sub.R (see FIG. 1) and zoom encoders 25.sub.L, 25.sub.R (see FIG. 1).

In the construction of the present embodiment, there is the possibility that an object existing in an area 53' indicated by hatching in FIG. 11 is photographed as a double image. The object existing in this area 53', as in the first embodiment, is projected onto CCD sensors 13.sub.L and 13.sub.R so as to apparently exist in areas 55.sub.L and 55.sub.R on the in-focus object planes 50.sub.L and 50.sub.R of the left and right image taking systems. FIG. 12 is a view of the effective light receiving portions of the left and right CCD sensors as they are seen from the image pickup lens unit side. It is to be understood here that the pixel coordinates values of left and right images are defined by (m.sub.L, n.sub.L) and (m.sub.R, n.sub.R) , respectively, as in the case of FIG. 4B.

The image processing procedure of the present embodiment will now be described with reference to FIGS. 11 and 12. In the present embodiment, the following three processes are carried out to form a panoramic synthesized image from the left and right image s:

1 Sampling of the pairs of corresponding points in the areas 55.sub.L and 55.sub.R ;

2 The removal of one image based on the positional relation between the pairs of corresponding points in the areas 55.sub.L and 55.sub.R ; and

3 The synthesizing process for continuously connecting the image on the line of intersection A between the in-focus object planes 50.sub.L and 50.sub.R.

Of these, the sampling of the pairs of corresponding points mentioned under item 1 above can be realized by a processing method similar to that in the second embodiment.

The removal of one image mentioned under item 2 above will now be described.

The main object, as in the second embodiment, is often located in the central portion of the panoramic synthesized image and moreover near the in-focus portion centering around the line of intersection A between the in-focus object planes 50.sub.L and 50.sub.R of the left and right image taking systems 10.sub.L and 10.sub.R. Accordingly, again in the present embodiment, image reconstruction attaching importance to the continuity of image is effected on the basis of consideration similar to that shown in the second embodiment. So, the area 53', as in the second embodiment, is considered to be bisected into a right area 53.sub.L ' and a left area 53.sub.R ', and the following three branching-off processes are carried out on the basis of the above-described pixel coordinates values.

(a) To leave the image of the object in the right image if m.sub.R >m.sub.L ;

(b) To leave the image of the object in the left image if m.sub.R <m.sub.L ; or

(c) To leave the image of the object in the left and right images if m.sub.R =m.sub.L.

The continuous synthesizing process mentioned under item 3 above will now be described.

First, in FIG. 12, the area in the right image which is at the left of the line of intersection A and the area in the left image which is at the right of the line of intersection A are deleted. Subsequently, the two images after the deletion are made coincident with each other on the line of intersection A and synthesized, whereby there is obtained a panoramic synthesized image having no discontinuous portion created in the main object area and in which the creation of discontinuity in the background image area, i.e., the generation of a double image, is minimized.

A fourth embodiment of the double-eye image pickup apparatus of the present invention will now be described. This embodiment differs in the convergence angles of the left and right image taking systems from the second and third embodiments, and will hereinafter be described with reference to FIG. 13. FIG. 13 is a view of the essential portions of an optical system in the fourth embodiment of the double-eye image pickup apparatus of the present invention as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units. In FIG. 13, portions similar to those shown in FIG. 3 are given the same reference characters and the schematic construction of the apparatus which is not shown is similar to that of the second embodiment and therefore need not be described.

As shown in FIG. 13, the convergence angles of the left and right image taking systems 10.sub.L and 10.sub.R are controlled so that end surfaces 51.sub.R and 52.sub.L which are the end surfaces of the effective object areas of the image taking systems 10.sub.L and 10.sub.R may interest each other at infinity, that is, may be parallel to each other. In this case, it is necessary that the control of the convergence angles be effected in operative association with zoom control, as described in connection with the third embodiment. Focusing and zoom control are effected as in the first embodiment.

In the construction of the present embodiment, there is the possibility that an object existing in an area 53" indicated by hatching in FIG. 13 is photographed as a double image. When this area 53" is projected onto CCD sensors 13.sub.L and 13.sub.R through image pickup lens units 11.sub.L and 11.sub.R, this area 53" spreads over the whole area of the effective light receiving portions of the CCD sensors 13 and 13. FIG. 14 is a view of the effective light receiving portions of the left and right CCD sensors as they are seen from the image pickup lens unit side. It is to be understood here that the pixel coordinates values of left and right images are defined by (m.sub.L, n.sub.L) and (m.sub.R, n.sub.R), respectively, as in the case of FIG. 4B.

The image processing procedure of the present embodiment will now be described with reference to FIGS. 13 and 14. Again in the present embodiment, as in the third embodiment, the following three processes are carried out to form a panoramic synthesized image from the left and right images:

1 The sampling of the pairs of corresponding points in the whole area of the images;

2 The removal of one image based on the positional relation between the pairs of corresponding points in the whole area of the images; and

3 The synthesizing process for continuously connecting the image on the line of intersection A between the in-focus object planes 50.sub.L and 50.sub.R.

The corresponding point sampling process mentioned under item 1 above can be realized by a processing method similar to that in each of the above-described embodiments, but in the present embodiment, it is necessary that this process be carried out over the whole area of the effective light receiving portions.

With regard also to the one image removing process mentioned under item 2 above, importance is attached to the continuity of image on the basis of consideration similar to that in the second embodiment, and with regard to each pair of corresponding point coordinates, the following branching-off processes can be carried out:

(a) To leave the image of the object in the left image if m.sub.R >m.sub.L ;

(b) To leave the image of the object in the right image if m.sub.R <m.sub.L ; or

(c) To leave the image of the object in the left and right images if m.sub.R =m.sub.L. In the case of the present embodiment, this process is converse at left and right from that in the second and third embodiments.

With regard to the synthesizing process of item 3 above, in FIG. 14, the area in the right image which is at the right of the line of intersection A and the area in the left image which is at the left of the line of intersection A are deleted, whereafter the two images after the deletion are disposed with the left and right reversed so as to coincide with each other on the line of intersection A and are synthesized, whereby there is obtained a panoramic synthesized image having no discontinuous portion created in the main object area and in which the creation of discontinuity in the background image area, i.e., the generation of a double image, is minimized.

A fifth embodiment of the double-eye image pickup apparatus of the present invention will now be described. FIG. 15 is a block diagram of the image processing portion in the fifth embodiment of the double-eye image pickup apparatus of the present invention. This embodiment is similar in construction to the first embodiment shown in FIGS. 1, 3, 4A and 4B except for the construction of the image processing portion and therefore, portions similar to those of the first embodiment are given the same reference characters and need not be described. The construction of the image processing portion will hereinafter be described.

As shown in FIG. 15, the image processing portion 230 has an image input portion 32 comprising a left image memory 31.sub.L and a right image memory 31.sub.R for holding video signals from the CCD sensors 13.sub.L and 13.sub.R (see FIG. 1), respectively, of image taking systems 10.sub.L and 10.sub.R, an image conversion portion 238 for forming a synthesized image on the basis of left and right images obtained from the video signals input to the image input portion 32, and a synthesized image memory 39 for holding the image synthesized by the image conversion portion 238 and outputting it to an image output unit 40. The image conversion portion 238 is subdivided into a corresponding point sampling unit 233 for sampling the pairs of corresponding points between two images input to the image input portion 232 with regard to all pixels of the double image generating area, a double image area-dividing unit 234 for calculating the three-dimensional position (distance information) of each pair of corresponding points from the results of the sampling of the pairs of corresponding points and effecting the area-division of the double image by that information, a double image mode-discriminating judging unit 235 for classifying each double image into three kinds of modes, i.e., whether the double image divided into areas is included in only the double image generating areas of the both images (the areas 55.sub.L and 55.sub.R shown in FIG. 4B), or included also in the double image non-generating area of one image (the area except the double image generating area), or included also in the double image non-generating areas of the both images, a common images of two systems' images removal unit 236 for removing one of the double image on the basis of the modes classified by the double image mode-discriminating judging unit 235 except the case where the double image exists also in the double image non-generating areas of the both images, and an after-removal processing unit 237 for carrying out the post-processing of effecting substitution or the like for the removing unit when the double image is removed, and laterally magnification-changing a predetermined portion when the double image is not removed.

The image processing procedure in the double-eye image pickup apparatus of the present embodiment will now be described.

When an object is first image-picked up by the left and right image taking systems 10.sub.L and 10.sub.R, image information is input as a video signal from each of the image taking systems 10.sub.L and 10.sub.R to the image input portion 32 of the image processing portion 230, and is held in the respective image memories 31.sub.L and 31.sub.R.

Subsequently, in the corresponding point sampling unit 233, the sampling of the corresponding points in the double image generating area is effected by a template matching method similar to that described in connection with the first embodiment and further, in the double image area-dividing unit 234, the positions of the pairs of corresponding points of each double image in three-dimensional space are found by the tigonometrical survey method or the line in a manner similar to that described in connection with the first embodiment, and a histogram having the Z coordinates values thereof as parameters is found and the area division of the double image is effected.

After the area division of the double image is effected, each area-divided double image is classified into the following four modes in the double image mode-discriminating judging unit 235. That is, in each image shown in FIG. 4B, the modes are the following four modes:

(Mode 1) a double image including a point on the line of intersection D.sub.R and not including any point on the line of intersection D.sub.L ;

(Mode 2) a double image not including any point on the line of intersection D.sub.R and including a point on the line of intersection D.sub.L ;

(Mode 3) a double image not including any point on the line of intersection D.sub.R and not including any point on the line of intersection D.sub.L ; and

(Mode 4) a double image including a point on the line of-intersection D.sub.R and including a point on the line of intersection D.sub.L.

The following processes differing between the four modes classified by the double image mode-discriminating judging unit 235 are carried out in the double image removing unit 236 and after-removal processing unit 237 shown in FIG. 15.

(Processing of Mode 1) The double image in the right image is left and the double image in the left image is removed. In the left image after the removal, the post-process of for example, substituting the values from the surrounding pixels for the pixels of the removed portion is carried out.

(Processing of Mode 2) Mode 2 corresponds to the road sign 62 in the images shown in FIG. 5, and in this case, processing converse at left and right from the processing of Mode 1 is carried out.

(Processing of Mode 3) One of the left and right double images is left and the other double image is removed. The post-processing is the same as the processing of Mode 1.

(Processing of Mode 4) One of the left and right double images is enlarged to double with respect to the lateral direction thereof and is synthesized so as to contact with the lines of intersection D.sub.L and D.sub.R shown in FIG. 4B. Or in order that it may be smoothly connected with the double image non-generating area, conversion may be done from the coordinates of the pairs of corresponding points of the left and right images by maps shown below, and then the left and right images may be enlarged to double with respect to the lateral direction thereof and synthesized so as to contact with the lines of intersection D.sub.L and D.sub.R. Maps therefor include, for example,

Here, it is to be understood that when left upper points in the areas 55.sub.L and 55.sub.R shown in FIG. 4B are the origins of the areas 55.sub.L and 55.sub.R, respectively, the coordinates of the double image in the right image are represented by (X.sub.R, Y.sub.R), the coordinates of the double image in the left image are represented by (X.sub.L, Y.sub.L) and the coordinates of the synthesized image from the double image are represented by (X.sub.S, Y.sub.S). Also, d.sub.R is the distance between the coordinates (X.sub.R, Y.sub.R) of the double image in the right image and the line of intersection D.sub.R, and d.sub.L is the distance between the coordinates (X.sub.L, Y.sub.L) of the double image in the left image and the line of intersection D.sub.R.

As described above, when the double image in the areas 55.sub.L, 55.sub.R is included in the double image non-generating area of the both images, that double image is enlarged to double in the lateral direction thereof, whereby there is obtained a synthesized image in which the double-image generating area is smoothly connected with the double image non-generating area.

A sixth embodiment of the double-eye image pickup apparatus of the present invention will now be described. This embodiment differs from the fifth embodiment in the processing of Mode 4 classified in the fifth embodiment. In the other points, this embodiment is similar to the fifth embodiment and therefore the characteristic portions thereof will hereinafter be described.

FIGS. 16A and 16B are for illustrating the image synthesizing process in the present embodiment, and are views in which a composition in which a mountain is seen on the background of an automobile which is a main object is image-picked up. In this case, by effecting the area division of the double image in a manner similar to that described in connection with the fifth embodiment, each of the left and right images is vertically bisected into an area including chiefly the automobile which is the main object and an area not including the automobile, as shown in FIG. 6A. Further, in areas 55.sub.L and 55.sub.R which are double image generating areas, lower areas are defined as .alpha. and .alpha.' and an upper area is considered by dividing it into the areas .beta. and .beta.' of the mountain and the background area .gamma. and .gamma.' thereof. Also, in the area except the areas 55.sub.L and 55.sub.R, i.e., the double image non-generating area, upper areas are defined as .delta. and .delta.'. In FIG. 16A, the pair of .beta. and .beta.' and the pair of .gamma. and .gamma.' become double images classified as Mode 4 in the fifth embodiment, and in the present embodiment, one of these double images is not enlarged to double in the lateral direction thereof, but one double image (the portion indicated by hatching in FIG. 6A and having a width l.sub.2) is removed, whereafter in the right image, the right end portion of the lower area (the portion indicated by hatching in FIG. 16A and having a width l.sub.12) is removed and also, in the left image, the left end portion of the lower area (the portion indicated by hatching in FIG. 16A and having a width l.sub.11 ; l.sub.11 =l.sub.12, l.sub.11 +l.sub.12 =l.sub.2) is removed, and the whole image spreading at the left and right of the double image at the same proportion to the two upper and lower areas, i.e., the whole right area .delta. of the areas .beta. and .gamma. and the whole left area .delta.' of the areas .beta.' and .gamma.' are also laterally magnification-changed. By these being enlarged and synthesized, there is obtained a panoramic synthesized image as shown in FIG. 16B wherein the left and right images are smoothly connected together at the central portion of the image. When there is any portion which is not classified as Mode 4, the removing process and post-process shown in the fifth embodiment are carried out.

In the present embodiment, the end portions of the double image non-generating portion in the lower area are removed, but usually the main object is disposed in the central portion of the image and therefore, there will be not so great influence even if the end portions of the image are removed. Also, in the present embodiment, there is shown an example in which the image is divided into two upper and lower areas, equal processing can also be carried out when the image is divided into three or more areas. Further, not only the lateral magnification changing process, but also the vertical magnification change can be effected at the same proportion as the lateral magnification change and the upper and lower end portions of the image are cut to provide a synthesized image, whereby there can also be obtained an image free of distortion in the length-to-width ratio.

A seventh embodiment of the double-eye image pickup apparatus of the present invention will now be described. FIG. 17 is a block diagram of the image processing portion in the seventh embodiment of the double-eye image pickup apparatus of the present invention. This embodiment is for synthesizing two images obtained by image-picking up an object by the use of two image taking systems, i.e., the right image taking system 10.sub.R and the left image taking system 10.sub.L, thereby obtaining a highly minute image, and the construction thereof is similar to that of the first embodiment shown in FIG. 1, except for the construction of the image processing portion and therefore, portions similar to those in the first embodiment are given similar reference characters and need not be described. The construction of the image processing portion will hereinafter be described.

As shown in FIG. 17, the image processing portion 330 has an image input portion 32 comprising a left image memory 31.sub.L and a right image memory 31.sub.R for holding video signals from the CCD sensors 13.sub.L and 13.sub.R (see FIG. 1), respectively, of the image taking systems 10.sub.L and 10.sub.R, an image conversion portion 338 for forming a synthesized image on the basis of left and right images obtained from the video signals input to the image input portion 32, and a synthesized image memory 39 for holding the image synthesized by the image conversion portion 338 and outputting it to an image output unit 40. The image conversion portion 338 is subdivided into a corresponding point sampling unit 333 for sampling the pairs of corresponding points between the two images input to the image input portion 32 with regard to all pixels of the area of the two images which can become the object of the two image taking systems, a nearer corresponding point discriminating unit 334 for judging from the result of the sampling of the pairs of corresponding points whether the position of each corresponding point is far by a predetermined distance relative to the pixel position of a reference image, a nearer corresponding point removal unit 335 for removing, when it is judged that the position of the corresponding point is not far by the predetermined distance relative to the pixel position of the reference image, that corresponding point from data for synthesized image, and an image interpolation unit 336 for effecting an interpolation process to the reference image by using the corresponding points which have not been removed by the nearer corresponding point removal unit 335 as the data for synthesized image.

FIG. 18 is a view of the essential portions of an optical system in the present embodiment as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units. As shown in FIG. 18, the convergence angles of the left and right image taking systems 10.sub.L and 10.sub.R are controlled so that the respective optical axes L.sub.L and L.sub.R thereof may be parallel to each other. Also, the focuses and zoom of the image taking systems 10.sub.L and 10.sub.R are controlled so that the distances between and the imaging magnifications of the respective in-focus object planes 50.sub.L, 50.sub.R and the CCD sensors 13.sub.L, 13.sub.R may be equal at left and right, and the in-focus object planes 50.sub.L and 50.sub.R are formed on the same plane. Motors 14.sub.L, 14.sub.R and 16.sub.L, 16.sub.R (see FIG. 1) for controlling the focuses and zoom are driven through the control unit 20 (see FIG. 1) which has received signals form the encoders 24.sub.L, 24.sub.R and 25.sub.L, 25.sub.R (see FIG. 1) as in the first embodiment. In the present embodiment, an area 58 indicated by hatching is an area which can become the object of the left and right image taking systems 10.sub.L and 10.sub.R, and an object existing in this area 58 is an object which becomes the subject of a highly minute image.

The image processing procedure in the double-eye image pickup apparatus of the present embodiment will now be described with reference to FIGS. 17 and 18.

When an object is first image-picked up by the left and right image taking systems 10.sub.L and 10.sub.R, image information is input as a video signal from each of the image taking systems 10.sub.L and 10.sub.R to the image input portion 32 of the image processing portion 330, and is held in the respective image memories 31.sub.L and 31.sub.R.

Subsequently, in the corresponding point sampling unit 333, the sampling of the pairs of corresponding points in the area 58 is effected by a template matching method similar to that described in connection with the first embodiment. In the present embodiment, however, pixel relative position accuracy of one pixel pitch or less is required and therefore, in order to effect the matching by an image low frequency component having little error, the size of the template can be made large, or before the matching calculation is effected, the process of sampling the low frequency component of the image can be carried out.

On the basis of this sampling of the pairs of corresponding points, the left image by the left image taking system 10.sub.L is combined with the right image by the right image taking system 10.sub.R to thereby obtain a highly minute image, and this processing will now be described with reference to FIG. 19. FIG. 19 represents the pixel positions of the CCD sensors with regard to parts of left and right images obtained by the optical system shown in FIG. 18. In FIG. 19, a point (l, m) indicated by a mark x shows the pixel position in the image by the right image taking system 10.sub.R which provides a reference image for a highly minute synthesized image, and a point indicated by a mark .largecircle. shows an addable pixel position (l, m)' in the highly minute synthesized image. On the other hand, a point Q indicated by a square mark shows the pixel position in the image by the left image taking system 10.sub.L calculated by above-described sampling of the corresponding points.

Here, the distance of the square mark point Q relative to-one of x mark points is first calculated by the nearer corresponding point discriminating unit 334, and whether the distance is 1/10 or less of the pixel pitch P is judged. If as a result, said distance is 1/10 or less of the pixel pitch P, this square mark Q is removed from data for synthesized image by the nearer corresponding point removal unit 335, and is not used as a parameter for calculating the value of the .largecircle. mark point (l, m)'. On the other hand, if said distance exceeds 1/10 of the pixel pitch P, this square mark point Q is intactly used as a parameter for calculating the value of the .largecircle. mark point (l, m)'. Subsequently, the interpolation process of the image by the left image taking system 10.sub.L is carried out to the image by the right image taking system 10.sub.R by the image interpolation unit 336, but with regard to any of the case where the square mark point Q is used and the case where the square are mark point Q is not used, for example, the coordinates conversion method (IEEE Trans, Acoust, 33 (85) 1151) for a two-dimensional unequal interval sampling point can be used as an interpolation method of calculating the value of the .largecircle. mark point. Here, of the image from the left image taking system 10.sub.L which is not the reference image, the pixel point having no corresponding point on the reference image side from the right image taking system 10.sub.R or the image point which is not calculated is of course not used as a parameter for determining the value of the .largecircle. mark point (l, m)'.

The present embodiment described above has the effect that when the relative positions of the pixels in the left and right images are very proximate to each other, any unnecessary high frequency component created in sensitive reaction to the error of the relative pixel positions or a noise component in which pixel values are superposed one upon another can be suppressed. Thereby, such a situation that the synthesized image is more deteriorated in image quality than the reference image can be prevented.

In the present embodiment, there has been shown the case where the right image is the reference image, but converse image synthesis may of course be effected with the left image as the reference image.

An eighth embodiment of the double-eye image pickup apparatus of the present invention will now be described. This embodiment, like the seventh embodiment, is a double-eye image pickup apparatus for obtaining a highly minute image, and differs in the convergence angles of the left and right image taking systems from the seventh embodiment. FIG. 20 is a view of the essential portions of an optical system in the eighth embodiment of the double-eye image pickup apparatus of the present invention as they are seen in a direction perpendicular to a plane formed by the optical axes of image pickup lens units, and portions similar to those in the seventh embodiment are given the same reference characters.

As shown in FIG. 20, the convergence angle of the right image taking system 10.sub.R of the left and right image taking systems 10.sub.L and 10.sub.R is fixed at zero degree, and the right image taking system 10.sub.R is set so as to always face the front of an object (not shown) with the optical axis of the right image taking system 10.sub.R directed toward the object. On the other hand, the convergence angle of the left image taking system 10.sub.L is controlled so that the optical axis L.sub.L of the left image taking system 10.sub.L may intersect the optical axis L.sub.R of the right image taking system 10.sub.R on the point of intersection between the in-focus object planes 50.sub.L and 50.sub.R of the two image taking systems 10.sub.L and 10.sub.R. The focus motor 14.sub.L (see FIG. 1) and zoom motor 16.sub.L (see FIG. 1) for the left image taking system 10.sub.L are controlled so as to keep the positional relations between the in-focus object planes 50.sub.L, 50.sub.R and the optical axes L.sub.L, L.sub.R in operative association with the setting of the focus motor 14.sub.R (see FIG. 1) and zoom motor 16.sub.R (see FIG. 1) for the right image taking system 10.sub.R. The control of the motors 14.sub.L, 14.sub.R, 16.sub.L and 16.sub.R is effected through the control unit 20 (see FIG. 1) which has received signals from the encoders 24.sub.L, 24.sub.R, 25.sub.L and 25.sub.R (see FIG. 1), as in the seventh embodiment.

The present embodiment also is for forming a highly minute image with the input image from the right image taking system 10.sub.R as the reference image, and the image forming method thereof is similar to that of the seventh embodiment and therefore need not be described.

In the present embodiment, the image from the image taking system having its optical axis fixed is used as the reference image and therefore, even if there exists a portion which cannot be highly minutely synthesized, at least the image from the front can always be output without being subjected to any special processing. Also, the image from the front relative to the apparatus is used as the reference image and therefore, even during the close-up photographing by the double-eye image pickup apparatus, the photographer can effect photographing without receiving any unnatural feeling of operation.

A ninth embodiment of the double-eye image pickup apparatus of the present invention will now be described. FIG. 21 is a block diagram of the image processing portion in the ninth embodiment of the double-eye image pickup apparatus of the present invention. This embodiment differs only in the construction of the image processing portion from the seventh and heighth embodiments and is similar in the arrangement of the optical system and the other points to the seventh and eighth embodiments and therefore, portions similar to those in those embodiments are given the same reference characters as those in the first embodiment and need not be described, and only the differences of the present embodiment from the seventh and eighth embodiments will hereinafter be described.

(1) A corresponding point non-sampling area discriminating unit 437 for judging an area in which the sampling of pairs of corresponding points is not effected, on the basis of a distortion amount signal from the control unit 20 is provided before a corresponding point sampling unit 433, and as regards that portion of an image in which distortion is great and needs complicated correction, one image is not used as data for synthesized image.

In the marginal portion of the input image, the distortion of the image resulting from the distortion of the left and right image pickup lens units (not shown) and from the fact that the optical axes of these image pickup lens units are not parallel to each other becomes great and therefore, it is necessary that highly accurate correction of the distortion be effected before the sampling of the pairs of corresponding points is effected. On the other hand, when an ordinary natural image or the like is to be picked up, the resolving power required for the marginal portion of the image is low as compared with that required for the central portion of the image and therefore, there may be a case where the highly minute synthesis in the marginal portion of the image is not very much required. Copying with such a situation, in the marginal portion of the image, an output image is obtained by the use of one input image alone. Even in such case, the correction of distortion becomes necessary, but since the accuracy required in such case is low as compared with that required when synthesis is done, the purpose can be achieved by the use of an easier processing method.

(2) Provision is made of an image bright-sharpness discriminating unit 434 for presuming the degree of blur of the image from the F value (numerical aperture) from the control unit 20 after the sampling of the pairs of corresponding points by the corresponding point sampling unit 433, and as regards that portion of the image which deviates from the in-focus object plane and in which blur is great, one image is not used as data for synthesized image.

In the image bright-sharpness discriminating unit 434, how far the corresponding point on the object is from the in-focus object plane is calculated from the information of the sampling of the pairs of corresponding points effected at each point on the input image, and the degree of blur of the image is presumed from this distance information and the F values (numerical apertures) of the left and right image pickup lens units. With regard to a portion in which the synthesis effect is judged to be small as a result of the comparison of this degree of blur with the pixel density of the CCD sensor (not shown), an output image is obtained by the use of one input image alone. Thereby, a part of the synthesizing process such as interpolation can be omitted.

(3) Provision is made of an occlusion area discriminating unit 435a for judging, after the sampling of the pairs of corresponding points by the corresponding point sampling unit 433, whether an occlusion area exists, and an occlusion generation frequency discriminating unit 435b for calculating an area appearing with a component of a predetermined frequency or higher, from the occlusion area judged by the occlusion area discriminating unit 435a, and with regard to that area, preventing one image from being used as data for synthesized image.

The occlusion area refers to an area which, when at a point in one image, a corresponding point is not found in the other image by the process of sampling the pairs of corresponding points, comprises these points. In an object wherein such occlusion area appears often and complicatedly in the image, erroneous correspondence between corresponding points becomes liable to occur, and this results in the deterioration of the synthesized image. Accordingly, one input image alone is used in a complicated occlusion area. As a specific example of the method, with regard to each pixel point of each input image, the presence or absence of corresponding points in the other image is memorized in a memory, and an area having a component in which the localized spatial frequency of the variation in the presence or absence of corresponding points is a predetermined frequency or higher is calculated. With regard to this area, the outputting process is carried out by the use of one input image alone.

The change of the processing procedure accompanying the blur of image or the occlusion mentioned under items (2) and (3) above can be done with regard to a corresponding area alone, or when the rate which that area occupies in the image exceeds a predetermined limit, selection by which, with regard to the entire image area, an output image is obtained by the use of one input image alone may be effected. In any case, the degree of improvement or deterioration of the quality of the combined output image is foreseen and one or both input images are selected.

Also, in the present embodiment, there have shown three techniques, whereas of course, these techniques need not be effected at a time, but each technique independently has an effect.

In the above-described embodiments except the seventh embodiment in which the left and right image taking systems 10.sub.L and 10.sub.R are disposed so that their optical axes L.sub.L and L.sub.R may be parallel to each other, the in-focus object planes 50.sub.L and 50.sub.R of the left and right image taking systems 10.sub.L and 10.sub.R, respectively, are not on the same plane. Therefore, some geometrical distortion occurs in the connected portion of the images. Further, in the case of the eighth embodiment in which one of the image taking systems is set so as to face the front of the object, some blur occurs in at least one of points in the object which should correspond to each other. In order to prevent this, the light receiving surfaces of the CCD sensors 13.sub.L and 13.sub.R of the left and right image taking systems 10.sub.L and 10.sub.R can also be disposed at a predetermined angle with respect to the respective optical axes L.sub.L and L.sub.R so that the in-focus object planes 50.sub.L and 50.sub.R of the left and right image taking systems 10.sub.L and 10.sub.R may be disposed on the same plane. With regard to the geometrical distortion, the distortion correcting process can be carried out to the input image.

Also, in the first embodiment, etc., the left and right image taking systems 10.sub.L and 10.sub.R are disposed so that the end portions of their in-focus object planes 50.sub.L and 50.sub.R may coincide with each other, but in order to absorb the occurrence of any error of the mechanism control system, the end portions of these in-focus object planes 50.sub.L and 50.sub.R can be disposed partly in superposed relationship with each other, and the correspondence between the images in the superposed portion can be effected and on the basis thereof, the synthesizing process can also be carried out.

Further, the optical arrangement in each of the above-described embodiments, the left and right image taking systems 10.sub.L and 10.sub.R are disposed bisymmetrically and the effective light receiving portions of the CCD sensors 13.sub.L and 13.sub.R are disposed so as to be symmetrical about the optical axes L.sub.L and L.sub.R of the image taking systems 10.sub.L and 10.sub.R, whereas the present invention is not restricted to such disposition. For example, the CCD sensors 13.sub.L and 13.sub.R may be disposed at a predetermined angle with respect to the optical axes L.sub.L and L.sub.R, whereby it is also possible to form the left and right in-focus object planes 50.sub.L and 50.sub.R on a plane. Also, the CCD sensors 13.sub.L and 13.sub.R can assume a disposition deviated in a direction perpendicular to the optical axes.

Also, in each of the above-described embodiments, the double-eye image taking system has been shown as comprising two image taking systems 10.sub.L and 10.sub.R, whereas the number of the image taking systems is not limited two, but may be three or more. The image pickup elements are not restricted to the CCD sensors 13.sub.L and 13.sub.R, but may be ones using, for example, image pickup tubes, or image pickup elements having a color resolving system removed therefrom and having mosaic-like color filters disposed in the light receiving portions thereof.

The present invention, which is constructed as described above, has the following effects.

By having an image processing portion which samples pairs of corresponding points corresponding to the same portion of an object from each image signal, and with regard to at least some of the sampled pairs of corresponding points, uses at least one corresponding point of said some pairs of corresponding points as synthesized image forming data and does not use the other corresponding points as synthesized image forming data, the deterioration of the synthesized image can be prevented.

In the formation of a panoramic synthesized image, a double image generated near the connected portion of two images can be removed and the good connection of the image with the double image non-generating area can be kept.

Also, in the formation of a highly minute image, when adjacent corresponding points of pairs of corresponding points are very proximate to each other, at least one corresponding point of that pair of corresponding points is used as synthesized image forming data and the other corresponding point is not used as synthesized image forming data, whereby any unnecessary high frequency component created in sensitive reaction to the error of the relative pixel position of the pair of corresponding points or a noise component superposed on the pixel value can be suppressed. Further, in the formation of a highly minute image, with regard to at least some of pairs of corresponding points which requires distortion correction before the sampling of corresponding points, or pairs of corresponding points in which the degree of blur of image foreseen from the distance from the in-focus object plane of each image taking system and the numerical aperture of each imaging optical system exceeds a predetermined threshold value, or pairs of corresponding points which appear at a predetermined spatial frequency or higher alternately with a pixel point for which corresponding points forming a pair are not found and an area in which the corresponding points forming said pair are not found, one corresponding point of said some pairs of corresponding points is used as said synthesized image forming data, whereby the little effective and unnecessary processing procedure in a distorted image portion, a blurred image portion and an occlusion image portion can be omitted and also the deterioration of image by wrong correspondence can be suppressed.

Further, one of the image signals is used as a reference image for synthesized image formation and the other image signal is used as an auxiliary image, whereby an image having at least a predetermined level of image quality can always be output.

Particularly, the image taking system having the image pickup element outputting the image signal which provides the reference image is disposed with the optical axis of the imaging optical system thereof looking toward said object, whereby the image from at least the front of the object can always be output without being subjected to any special processing and even during close-up photographing, the photographer can effect photographing without receiving any unnatural feeling of operation.