Subject image extraction device

[0002] The present invention relates to a subject image extraction device for extracting an image of a desired subject from an inputted image, and in particular, to a subject image extraction device which can extract an image of a subject which existed in a designated 3-D space domain at the moment of shooting.

[0003] Various kinds of methods are proposed for extracting an image of a desired subject or object from an inputted image, such as a method utilizing prestored information concerning the subject as disclosed in Japanese Patent Application Laid-Open No.HEI7-220095 (hereafter, referred to as ‘document No.1’), a method utilizing two or more image pickup devices (cameras) as disclosed in Japanese Patent Application Laid-Open No.HEI7-177423 (hereafter, referred to as ‘document No.2’) and Japanese Patent Application No.HEI8-271914 (hereafter, referred to as ‘document No.3’), etc.

[0004] In the method utilizing prestored subject information which is disclosed in the document No.1, information concerning the image pickup device such as the focal length of the image pickup device, the size of a pixel of a sensor of the image pickup device, the number of pixels of the sensor, etc. and information concerning the subject such as the size of the subject, the distances between the center point on the subject and parts of the subject, etc. are preliminarily recorded and stored in a storage unit. Then, the size of the subject in the image which has been picked up by the image pickup device is estimated using the information concerning the image pickup device and the subject. Meanwhile, a plurality of parts of the image which are considered to be candidates of the image of the subject are extracted from the image. Then, it is judged whether each of the candidate parts is the genuine subject image or not, by comparing the size of the candidate part with the estimated size of the subject in the image. And one candidate part whose size is the nearest to the estimated size of the subject in the image is extracted as the image of the subject.

[0005] In the method utilizing two or more image pickup devices which is disclosed in the document No.2, corresponding points in each image which have been picked up by the two or more image pickup devices are searched for, and disparities between the images with regard to the corresponding points are obtained using the difference of coordinates between the corresponding points in each image. Then, part of an image whose disparity between the images are large (i.e. parts which existed in the foreground near to the image pickup devices at the moment of shooting) are extracted as the image of the subject.

[0006] In the method utilizing two or more image pickup devices which is disclosed in the document No.3. The subject is shot in the centers of images by the two or more image pickup devices, and the aforementioned search of point correspondence (hereafter, referred to as ‘point correspondence search’) is executed from the center of the images. Then, outline point judgment (i.e. judgment whether a point in an image is a point on the outline of the subject or not) is executed using 3-D coordinates (corresponding to the point) which are calculated using the corresponding points obtained by the point correspondence search, and the outline points are used for extraction of the subject image. Even when false correspondence occurred in the point correspondence search and parts of the outline points dropped out from the extraction, such parts are restored by means of outline restoration assuming continuity.

[0007] However, in the method disclosed in the document No.1, the aforementioned information concerning the subject such as the size of the subject, the distances between the center point on the subject and parts of the subject, etc. and the information concerning the image pickup device such as the focal length of the image pickup device, the size of a pixel, the number of pixels, the distance between the subject and the image pickup device, etc. are needed to be prepared, and thus images of subjects whose information have not been prepared are impossible to be extracted from the image which has been picked up by the image pickup device. Further, according to the method, either the distance between the center point on the subject and part of the subject or the distance between the principle point of the image and part of the image is necessary as the information. In the case where the former distance is used, expensive devices such as an infrared irradiation unit etc. are necessary for obtaining the distance. And in the case where the latter distance is used, methods such as autofocusing, which does not operate stably under insufficient shooting conditions such as poor lighting etc., have to be employed, and thus stable extraction of the subject image is difficult.

[0008] In the method utilizing two or more image pickup devices and judging the depth using the disparity, the accuracy of subject image extraction is highly dependent on setting of a threshold value (in disparity) which is used to distinguish between the background and the foreground, and the setting of the disparity threshold value for accurate subject image extraction is very difficult for the operator of the subject image extraction device since the disparity is not a value which can be directly measured. The method of the document No.2 tries to resolve the problem by obtaining 3-D coordinates using the disparities. However, the method needs troublesome control or measurement of relative positions of the two or more image pickup devices, angles between the optical axes of the image pickup devices, etc. Further, according to the method, when false correspondence occurred in the point correspondence search, the part where the false correspondence occurred is extracted as noise, or in other words, parts of the subject image drop out.

[0009] In the method disclosed in the document No.3, the aforementioned problem concerning the difficulty of the threshold value determination is resolved by calculating 3-D coordinates without using the disparities, in which parameters which represent relative positions of the image pickup devices and angles between the optical axes of the image pickup devices are calculated using images of a predetermined pattern picked up by the image pickup devices, instead of controlling or measuring the relative positions of the image pickup devices and angles between the optical axes of the image pickup devices. Further, with regard to the aforementioned problem concerning the point correspondence search, error rate in the point correspondence search is reduced in the method of the document No.3, by shooting the subject in the centers of the images by the image pickup devices and executing the point correspondence search starting from the center of the images with high priority. Furthermore, even in the case where the false correspondence occurred in the point correspondence search, parts which have dropped out from the extraction or parts which have been incorrectly extracted are corrected by the outline restoration on the assumption of continuity. However, according to the method, all the image pickup devices have to be correctly controlled to shoot the subject in the centers of images. Further, in the case where the background has a complex scene, error rate in the point correspondence search increases and thus the outline restoration is necessitated to be executed considerably oftener, causing difficulty in the subject image extraction especially when the subject has a complex outline.

[0010] It is therefore the primary object of the present invention to provide a subject image extraction device by which an image of a desired subject can be extracted from an inputted image, without needing preparation of the information concerning the subject or the image pickup device.

[0011] Another object of the present invention is to provide a subject image extraction device by which an image of a desired subject can be extracted from an inputted image, with easy setting of a threshold value for distinguishing between the background and the foreground.

[0012] Another object of the present invention is to provide a subject image extraction device by which an image of a desired subject can be extracted from an inputted image, without needing precise control or measurement of the positions and attitudes of image pickup devices.

[0013] Another object of the present invention is to provide a subject image extraction device by which 3-D coordinates corresponding to a point in an inputted image can be accurately obtained and thereby subject image extraction can be executed with higher accuracy.

[0014] In accordance with the present invention, there is provided a subject image extraction device comprising an image storage means, a feature point extraction means, a parameter calculation means, a 3-D coordinates calculation means, a 3-D coordinates determination means, a judgment means, an outline point extraction means, and a subject extraction means. The image storage means includes main image storage section for storing images which have been picked up by a main image pickup section for picking up a main image from which an image of a subject is extracted, and two or more subsidiary image storage sections for storing images which have been picked up by two or more subsidiary image pickup sections for picking up subsidiary images to be referred to in subject image extraction. The feature point extraction means includes three or more feature point extraction sections for extracting feature point coordinates in the images which have been stored in each of the image storage sections. The parameter calculation means calculates parameters which represent relative positions and attitudes of the main image pickup section and the subsidiary image pickup sections, using the feature point coordinates in the images of a 3-D object on which a predetermined pattern is drawn which have been picked up by the main image pickup section and the subsidiary image pickup sections. The 3-D coordinates calculation means includes two or more 3-D coordinates calculation sections for calculating 3-D candidate coordinates which are considered to correspond to the feature point coordinates of a feature point in the main image, using the feature point coordinates in the main image, the feature point coordinates in the subsidiary images, and the parameters obtained by the parameter calculation means. The 3-D coordinates determination means determines genuine 3-D coordinates which correspond to the feature point in the main image, using the 3-D candidate coordinates calculated by the 3-D coordinates calculation sections in the 3-D coordinates calculation means. The judgment means judges whether or not the genuine 3-D coordinates corresponding to the feature point in the main image is in a designated 3-D space domain, and accepts the feature point if the genuine 3-D coordinates corresponding to the feature point is judged to be in the designated 3-D space domain. The outline point extraction means extracts subject outline points from a plurality of feature points which have been accepted by the judgment means. And the subject extraction means extracts part of the main image which is surrounded by the subject outline points as the subject image.

[0015] Preferably, the feature point extraction sections in the feature point extraction means extract edge-like parts in the images where large changes occur in color intensity or luminance as the feature point coordinates, in the extraction of the feature point coordinates from the main image or the subsidiary images.

[0016] Preferably, the parameter calculation means calculates eleven or more parameters per one image pickup section.

[0017] Preferably, the 3-D coordinates determination means determines the genuine 3-D coordinates by selecting a group of 3-D coordinates from each of the 3-D candidate coordinates calculated by each of the 3-D coordinates calculation sections so that the sum of distances between the 3-D coordinates included in the group will become the smallest and defining the genuine 3-D coordinates by the average of the 3-D coordinates included in the group.

[0018] Preferably, the subject image extraction device further comprises an image pickup means including the main image pickup section and the two or more subsidiary image pickup sections.

[0019] Preferably, the subject image extraction device further comprises a subject display means for displaying the part of the main image which has been extracted by the subject extraction means.

[0020] Preferably, the number of the image pickup sections is three.

[0021] In accordance with another aspect of the present invention, there is provided a subject image extraction device comprising an image storage means, a feature point neighborhood extraction means, a parameter calculation means, a corresponding point determination means, a 3-D coordinates calculation means, a 3-D coordinates determination means, a judgment means, an outline point extraction means, and a subject extraction means. The image storage means includes main image storage section for storing images which have been picked up by a main image pickup section for picking up a main image from which an image of a subject is extracted, and two or more subsidiary image storage sections for storing images which have been picked up by two or more subsidiary image pickup sections for picking up subsidiary images to be referred to in subject image extraction. The feature point neighborhood extraction means includes three or more feature point neighborhood extraction sections for extracting feature point coordinates in the images which have been stored in each of the image storage sections and extracting color/luminance information of pixels in the neighborhood of the feature point coordinates. The parameter calculation means calculates parameters which represent relative positions and attitudes of the main image pickup section and the subsidiary image pickup sections, using the feature point coordinates in the images of a 3-D object on which a predetermined pattern is drawn which have been picked up by the main image pickup section and the subsidiary image pickup sections. The corresponding point determination means includes two or more corresponding point determination sections for selecting a group of candidate corresponding points in the subsidiary image which are considered to correspond to the feature point in the main image, from the feature points in the subsidiary images, using the feature point coordinates in the main image and the subsidiary image and the color/luminance information of pixels in the neighborhood of the feature point coordinates which have been extracted by the feature point neighborhood extraction means.

[0022] The 3-D coordinates calculation means includes two or more 3-D coordinates calculation sections for calculating 3-D candidate coordinates which are considered to correspond to a feature point in the main image, using the feature point coordinates in the main image, the group of candidate corresponding points, and the parameters obtained by the parameter calculation means. The 3-D coordinates determination means determines genuine 3-D coordinates which correspond to the feature point in the main image, using the 3-D candidate coordinates calculated by the 3-D coordinates calculation sections in the 3-D coordinates calculation means.

[0023] The judgment means judges whether or not the genuine 3-D coordinates corresponding to the feature point in the main image is in a designated 3-D space domain, and accepts the feature point if the genuine 3-D coordinates corresponding to the feature point is judged to be in the designated 3-D space domain. The outline point extraction means extracts subject outline points from a plurality of feature points which have been accepted by the judgment means. And the subject extraction means extracts part of the main image which is surrounded by the subject outline points as the subject image.

[0024] Preferably, the feature point neighborhood extraction sections in the feature point extraction means extract edge-like parts in the images where sudden changes occur in color intensity or luminance as the feature point coordinates, in the extraction of the feature point coordinates from the main image or the subsidiary images.

[0025] Preferably, the parameter calculation means calculates eleven or more parameters per one image pickup section.

[0026] Preferably, the 3-D coordinates determination means determines the genuine 3-D coordinates by selecting a group of 3-D coordinates from each of the 3-D candidate coordinates calculated by each of the 3-D coordinates calculation sections so that the sum of distances between the 3-D coordinates included in the group will become the smallest and defining the genuine 3-D coordinates by the average of the 3-D coordinates included in the group.

[0027] Preferably, the subject image extraction device further comprises an image pickup means including the main image pickup section and the two or more subsidiary image pickup sections.

[0028] Preferably, the subject image extraction device further comprises a subject display means for displaying the part of the main image which has been extracted by the subject extraction means.

[0029] Preferably, the number of the image pickup sections is three.

[0030] The objects and features of the present invention will become more apparent from the consideration of the following detailed description taken in conjunction with the accompanying drawings, in which:

[0031] FIG. 1 is a block diagram showing a subject image extraction device according to an embodiment of the present invention;

[0032] FIG. 2 is a schematic diagram showing preferred arrangement of image pickup sections 1a,1band 1cof the subject image extraction device of FIG. 1;

[0033] FIG. 3 is a perspective view showing an example of a 3-D object on which a predetermined pattern is drawn which is used in the embodiment of FIG. 1;

[0034] FIG. 4A is a schematic diagram showing a main image used in the embodiment of FIG. 1;

[0035] FIG. 4B is a schematic diagram showing a subsidiary image used in the embodiment of FIG. 1;

[0036] FIG. 5 is a schematic diagram showing the principle of triangular surveying;

[0037] FIG. 6 is a schematic diagram showing a 3-D coordinates determination process of the subject image extraction device of FIG. 1;

[0038] FIG. 7 is a schematic diagram explaining ‘block matching’ used in the embodiment of FIG. 8; and

[0039] FIG. 8 is a block diagram showing a subject image extraction device according to another embodiment of the present invention.

[0040] Referring now to the drawings, a description will be given in detail of preferred embodiments in accordance with the present invention.

[0041] FIG. 1 is a block diagram showing a subject image extraction device according to an embodiment of the present invention.

[0042] The subject image extraction device of FIG. 1 comprises an image pickup unit 1, an image storage unit 2, a feature point extraction unit 3, a parameter calculation unit 4, a 3-D coordinates calculation unit 5, a 3-D coordinates determination unit 6, an outline point extraction unit 7, and a subject display unit 8.

[0043] The image pickup unit 1 is composed of an image pickup section 1afor picking up a main image to be mainly used in a subject image extraction process which will be described below, and image pickup sections 1band 1cfor picking up subsidiary images to be referred to in the subject image extraction process. The image pickup sections 1a,1band 1care preliminarily used to shoot a 3-D object 21 on which a predetermined pattern is drawn in order to obtain parameters 400 which will be described below, and are used to shoot a subject 22 whose image will be extracted using the parameters 400 etc.

[0044] The image storage unit 2 is composed of image storage sections 2a,2band 2c.The image storage sections 2a,2band 2cstore images 10a,10band 10cof the 3-D object 21 and images 20a,20band 20cof the subject 22 which have been picked up by the image pickup sections 1a,1band 1c,respectively.

[0045] The feature point extraction unit 3 is composed of feature point extraction sections 3a,3band 3c.The feature point extraction sections 3a,3band 3cextract coordinates 30a,30band 30cof a plurality of feature points from the images 10a,10band 10cwhich have been stored in the image storage sections 2a,2band 2crespectively, and extract coordinates 50a,50band 50cof a plurality of feature points from the images 20a,20band 20cwhich have been stored in the image storage sections 2a,2band 2crespectively.

[0046] The parameter calculation unit 4 calculates and stores the parameters 400 which represent relative positions and attitudes of the main image pickup section 1aand the subsidiary image pickup sections 1band 1c,using the feature point coordinates 30a,30band 30cwhich have been extracted by the feature point extraction sections 3a,3band 3crespectively.

[0047] The 3-D coordinates calculation unit 5 is composed of 3-D coordinates calculation sections 5band 5c.The 3-D coordinates calculation section 5bcalculates 3-D candidate coordinates 60busing the parameters 400 and the feature point coordinates 50aand 50bwhich have been extracted by the feature point extraction sections 3aand 3b.The 3-D coordinates calculation section 5ccalculates 3-D candidate coordinates 60cusing the parameters 400 and the feature point coordinates 50aand 50cwhich have been extracted by the feature point extraction sections 3aand 3c.Incidentally, the 3-D candidate coordinates 60bis a group of 3-D coordinates (i.e. a group of 3-D vectors each of which is representing 3-D coordinates) which are considered to be candidates for genuine 3-D coordinates which correspond to the feature point (2-D) coordinates 50ain the main image 20awhich have been extracted by the feature point extraction section 3a.Similarly, the 3-D candidate coordinates 60cis a group of 3-D coordinates (i.e. a group of 3-D vectors each of which is representing 3-D coordinates) which are considered to be candidates for genuine 3-D coordinates which correspond to the feature point (2-D) coordinates 50ain the main image 20awhich have been extracted by the feature point extraction section 3a.

[0048] The 3-D coordinates determination unit 6 determines genuine 3-D coordinates 700 which correspond to the feature point coordinates 50ain the main image 20a,using the 3-D candidate coordinates 60band the 3-D candidate coordinates 60c.

[0049] The outline point extraction unit 7 judges whether the feature point 50ais a point in the subject 22 or not, using the 3-D coordinates 700, and extracts subject outline point coordinates 800. Here, the subject outline point coordinates 800 is a group of 2-D coordinates (i.e. a group of 2-D vectors each of which is representing 2-D coordinates) which correspond to a plurality of subject outline points (i.e. feature points 50awhich exist on the outline of the subject 22).

[0050] The subject display unit 8 displays part of the image 20awhich are surrounded by the subject outline points, using the subject outline point coordinates 800 extracted by the outline point extraction unit 7.

[0051] The feature point extraction unit 3, the parameter calculation unit 4, the 3-D coordinates calculation unit 5, the 3-D coordinates determination unit 6, and the outline point extraction unit 7 are realized, for example, by a microprocessor unit which is composed of a CPU, ROM (Read Only Memory), RAM (Random Access Memory), etc., and necessary software. The image storage unit 2 is realized by one or more storage devices such as an HDD (Hard Disk Drive), an MO (Magneto-Optical disk), etc. The subject display unit 8 is realized, for example, by a display unit such as an LCD (Liquid Crystal Display), etc. and a microprocessor unit.

[0052] In the following, the operation of the subject image extraction device of FIG. 1 will be described, in which both a point itself and coordinates of the point are expressed by the same word ‘point’ for brevity.

[0053] In this embodiment, the parameters 400 which represent relative positions and attitudes of the image pickup sections 1a,1band 1care needed to be calculated by the parameter calculation unit 4 first of all, using the coordinates 30a,30band 30cof the feature points in the images 10a,10band 10cof the 3-D object 21.

[0054] For the calculation of the parameters 400, the image pickup sections 1a,1band 1crespectively pick up images 10a,10band 10cof the 3-D object 21 on which a predetermined pattern (which is shown in FIG. 3, for example) is drawn.

[0055] The optical axes of the image pickup sections 1a,1band 1cmay be parallel, or may have convergence angles between them. Referring to FIG. 2, the main image pickup section 1a,for picking up the main image 10ais placed in the middle of the three image pickup sections 1a,1band 1c.Such arrangement is advantageous since occlusion can be made smaller.

[0056] The images 10a,10band 10cof the 3-D object 21 on which the predetermined pattern is drawn picked up by the image pickup sections 1a,1band 1care stored in the image storage sections 2a,2band 2crespectively, and then supplied to the feature point extraction sections 3a,3band 3cin the feature point extraction unit 3, respectively.

[0057] After the images 10a,10band 10care supplied to the feature point extraction unit 3, six or more feature points P0, P1, P2, P3, P4, P5, . . . are chosen so that all the feature points will not be in one plane of the 3-D object 21, more concretely, so that a 3-D coordinate system can be at least generated using four points out of the six or more feature points P0, P1, P2, P3, P4, P5, . . . Subsequently, coordinates 30a,30band 30cof the feature points P0, P1, P2, P3, P4, P5, . . . in the images 10a,10band 10care extracted by the feature point extraction sections 3a,3band 3c,respectively. Here, each of the coordinates 30a,30band 30cis a group of 2-D coordinates (corresponding to the feature points P0, P1, P2, P3, P4, P5, . . . ) in the image 10a,10b,10c,respectively. The choice of the feature points can be done by various kinds of methods, for example, by instruction of the operator of the device who is watching the images 10a,10band 10c,or by automatic extraction by means of image processing such as pattern matching etc.

[0058] Then, the parameter calculation unit 4 calculates the parameters 400 using the coordinates 30a,30band 30cof the feature points P0, P1, P2, P3, P4, P5, . . . , and the calculated parameters 400 are stored in the parameter calculation unit 4.

[0059] The calculation of the parameters 400 can be executed based on a method which is disclosed in Japanese Patent Application No.HEI7-206932. In the following, the method for the calculation of the parameters 400 will be described.

[0060] First, a 3-D coordinate system is defined so that coordinates of the feature points P0, P1, P2 and P3 (shown in FIG. 3) in the 3-D coordinate system will be (0, 0, 0), (1, 0, 0), (0, 1, 0) and (0, 0, 1) respectively. Here, the 3-D coordinate system is not needed to be a rectangular coordinate system with rectangular axes as is defined by the P0, P1, P2 and P3 which are shown in FIG. 3, and generally, the 3-D coordinate system may be an oblique coordinate system which is defined by feature points P0, P1, P2 and P3 which exist on oblique axes. Hereafter, coordinates of the feature points P4 and P5 in the 3-D coordinate system will be expressed as (X₄, Y₄, Z₄) and (X₅, Y₅, Z₅) respectively.

[0061] Incidentally, 2-D coordinates of the feature points P0, P1, P2, P3, P4 and P5 in the image 10awill be hereafter expressed as

[0062] (u_i^c,v_i^c)(i=0, . . . ,5),

[0063] and coordinates of the feature points P0, P1, P2, P3, P4 and P5 in the image 10bwill be hereafter expressed as

[0064] (u_i^l, v_i^l)(i=0, . . . ,5),

[0065] and coordinates of the feature points P0, P1, P2, P3 P4 and P5 in the image 10cwill be hereafter expressed as

[0066] (u_i^r, v_i^r)(i=0, . . . ,5),

[0067] Here, using the coordinates:

[0068] (u_i^c, v_i^c),

[0069] of the feature points in the image 10aand the 3-D coordinates (X_k, Y_k, Z_k) (k=4, 5) of the feature points P4 and P5, and using the following parameters:

[0070] c_j^c(j=1, . . . ,3),

[0071] which represent the attitude of the main image pickup section 1a,the following equations (1), (2), (3) and (4) are derived.

[0072] {X4(u₄^c−u₁^c)}c₁^c+{Y4(u₄^c−u₂^c)}c₂^c+{Z4(u₄^c−u₃^c)}c₃^c=(1−X4−Y4−Z4)(u₀^c−u₄^c)  (1)

[0073] {X4(v₄^c−v₁^c)}c₁^c+{Y4(v₄^c−v₂^c)}c₂^c+{Z4(v₄^c−v₃^c)}c₃^c=(1−X4−Y4−Z4)(v₀^c−v₄^c)  (2)

[0074] {X5(u₅^c−u₁^c)}c₁^c+{Y5(u₅^c−u₂^c)}c₂^c+{Z5(u₅^c−u₃^c)}c₃^c=(1−X5−Y5−Z5)(u₀^c−u₅^c)  (3)

[0075] {X5(v₅^c−v₁^c)}c₁^c+{Y5(v₅^c−v₂^c)}c₂^c+{Z5(v₅^c−v₃^c)}c₃^c=(1−X5−Y5−Z5)(v₀^c−v₅^c)  (4)

[0076] The above parameters:

[0077] c_j^c(j=1, . . . ,3),

[0078] are elaborated on in A. Marugame et al., “Structure Recovery from Scaled Orthographic and Perspective Views,” Proceedings IEEE ICIP-96, vol.2, pp.851-854 (1996), and are representing the strain (extension/contraction) of three axes of the 3-D coordinate system due to projection of the 3-D coordinate system onto 2-D plane by the image pickup section 1a.The parameters can be obtained from the equations (1), (2), (3) and (4), using, for example, method of least squares, which is described for example in a document: K. Kanatani, “Image understanding”, Morikita Shuppan, Japan (1990).

[0079] Parameters:

[0080] c_j^l(j=1, . . . ,3),

[0081] which represent the attitude of the subsidiary image pickup section 1band parameters:

[0082] c_j^r(j=1, . . . ,3),

[0083] which represent the attitude of the subsidiary image pickup section 1ccan be obtained by the same method as the case of the main image pickup section 1a.

[0084] The parameters:

[0085] c_j^c, c_j^l, c_j^r(j=1, . . . ,3),

[0086] which have been obtained by the above calculation and the following coordinates:

[0087] (u_i^c,v_i^c), (u_i^l,v_i^l), (u_i^r,v_i^r)(i=0, . . . ,3),

[0088] are stored in the parameter calculation unit 4 as the parameters 400. Here, the number of parameters included in the parameters 400 per one image pickup section (1a,1bor 1c) is 11. It is generally known that 11 parameters per one image pickup device are enough for expressing relative positions and attitudes of a plurality of image pickup devices.

[0089] Now that the parameters 400 representing relative positions and attitudes of the image pickup sections 1a,1band 1chave been obtained and stored in the parameter calculation unit 4, extraction of desired subject in the main image 20acan be executed. In the following, the subject image extraction process will be described in detail.

[0090] The subject 22 is shot by the image pickup sections 1a,1band 1cof the image pickup unit 1 and images 20a,20band 20care obtained respectively.

[0091] The images 20a,20band 20care stored in the image storage sections 2a,2band 2cof the image storage unit 2, respectively.

[0092] From the images 20a,20band 20cwhich have been stored in the image storage sections 2a,2band 2crespectively, feature point coordinates 50a,50band 50care respectively extracted by the feature point extraction sections 3a,3band 3cof the feature point extraction unit 3.

[0093] The extraction of the feature points may be done by various kinds of methods, for example, by instruction of the operator of the device who is watching the images 20a,20band 20c,or by automatic extraction using differential filters such as Sobel filter, Laplacian filter, etc. The extraction of feature points 50aare usually executed in units of pixels in the main image 20a.In the case where differential filters are utilized for the aforementioned automatic extraction of the feature points 50a,edge-like parts in the main image 20awhere large changes occur in color intensity (in cases of color image) or luminance (in cases of black and white image) are extracted as the feature points 50a,and thereby a few of all the pixels in the main image 20aare extracted as the feature points 50aby the feature point extraction section 3a,for example. The same goes for the extraction of feature points 50band 50cfrom the images 20band 20c.

[0094] A plurality of feature points 50aare extracted one after another by the feature point extraction section 3ain the feature point extraction unit 3, and each feature point 50awill be adopted as being a point on the outline of the subject 22 (i.e. adopted as a ‘subject outline point’) or rejected as not being a subject outline point, by the processes which will be described below. Finally, a group of subject outline points which are obtained as above will be used for extraction of the subject image.

[0095] The 3-D coordinates calculation section 5bin the 3-D coordinates calculation unit 5 calculates the 3-D candidate coordinates 60busing the parameters 400 and the feature points 50aand 50b,and the 3-D coordinates calculation section 5ccalculates the 3-D candidate coordinates 60cusing the parameters 400 and the feature points 50aand 50c.As mentioned above, the 3-D candidate coordinates 60bare a group of 3-D coordinates (i.e. a group of 3-D vectors each of which is representing 3-D coordinates) which are considered to be candidates for genuine 3-D coordinates which correspond to the feature point (2-D) coordinates 50ain the main image 20awhich have been extracted by the feature point extraction section 3a,and the 3-D candidate coordinates 60cis a group of 3-D coordinates (i.e. a group of 3-D vectors each of which is representing 3-D coordinates) which are considered to be candidates for genuine 3-D coordinates which correspond to the feature point (2-D) coordinates 50ain the main image 20awhich have been extracted by the feature point extraction section 3a.

[0096] In the following, the operation of the 3-D coordinates calculation section 5bwill be described in detail.

[0097] The 3-D coordinates calculation section 5bchooses one of the feature points 50aone after another, and the following processes are executed to each of the feature points 50a.When 2-D coordinates of the feature point 50ain the image 20ais expressed as (u^c, v^c) as shown in FIG. 4A, the position of a feature point 501 in the image 20bwhich corresponds to the feature point 50ain the image 20ais restricted on a line 502 shown in FIG. 4B (which is called an epipolar line). The epipolar line 502 is determined according to the relation between the positions of the image pickup sections 1aand 1b.Therefore, the 3-D coordinates calculation section 5bcalculates the epipolar line 502 using the following parameters which are included in the parameters 400.

[0098] i c_j^c,c_j^l(j=1, . . . ,3), (u_i^c,v_i^c), (u_i^l,v_i^l)(i=0, . . . ,3),

[0099] The calculation of the epipolar line 502 can be executed according to the aforementioned document No.3 or a document: The proceedings of the 1st Image Media Processing Symposium (1996), pages 15-16, as follows.

[0100] First, using the parameters:

[0101] (u_i^c,v_i^c)(i=0, . . . ,3), c_j^c(j=1, . . . ,3), (u^c,v^c),

[0102] the following primary intermediate parameters t₁₁, t₁₂, t₁₃, t₂₁, t₂₂, t₂₃, d₁and d₂are obtained.

[0103] t11={c₁^c(u^c−u₁^c)−(u^c−u₀^c)}  (5)

[0104] t12={c₂^c(u^c−u₂^c)−(u^c−u₀^c)}  (6)

[0105] t13={c₃^c(u^c−u₃^c)−(u^c−u₀^c)}  (7)

[0106] t21={c₁^c(v^c−v₁^c)−(v^c−v₀^c)}  (8)

[0107] t22={c₂^c(v^c−v₂^c)−(v^c−v₀^c)}  (9)

[0108] t23={c₃^c(v^c−v₃^c)−(v^c−v₀^c)}  (10)

[0109] d1=u₀^c−u^c  (11)

[0110] d2=v₀^c−v^c  (12)

[0111] Subsequently, using the primary intermediate parameters t₁₁, t₁₂, t₁₃, t₂₁, t₂₂, t₂₃, d₁and d₂and the following parameters:

[0112] (U_i^l, V_i^l) (i=0, . . . ,3), C_j^l(j=1, . . . ,3),

[0113] the following secondary intermediate parameters U₀, U₁, V₀, V₁, S₀and S₁are obtained.

[0114] U₀=u₀^l(t₁₁t₂₂−t₁₂t₂₁)+(c₁^lu₁^l−u₀^l) (t₂₂d₁−t₁₂d₂)+(c₂^lu₂^l−u₀^l)(t₁₁d₂−t₂₁d₁)  (13)

[0115] U₁=(c₁^lu₁^l−u₀^l)(t₁₂t₂₃−t₁₃t₂₂)+(c₂^lu₂^l−u₀^l)(t₁₃t₂₁−t₁₁t₂₃)+(c₃^lu₃^l−u₀^l)(t₁₁t₂₂−t₁₂t₂₁)  (14)

[0116] V₀=U₀^l(t₁₁t₂₂−t₁₂t₂₁)+(c₁^lV₁^l−V₀^l) (t₂₂d₁−t₁₂d₂)+(c₂^lv₂^l−v₀^l)(t₁₁d₂−t₂₁d₁)  (15)

[0117] V₁=(c₁^lv₁^l−v₀^l)(t₁₂t₂₃−t₁₃t₂₂)+(c₂^lv₂^l−v₀^l)(t₁₃t₂₁−t₁₁t₂₃)+(c₃^lv₃^l−v₀^l)(t₁₁t₂₂−t₁₂t₂₁)  (16)

[0118] S₀=(t₁₁t₂₂−t₁₂t₂₁)+(c₁^l−1)(t₂₂d₁−t₁₂d₂)+(c₂^l−1)(t₁₁d₂−t₂₁d₁)  (17)

[0119] S₁=(c₁^l−1)(t₁₂t₂₃−t₁₃t₂₂)+(c₂^l−1)(t₁₃t₂₁−t₁₁t₂₃)+(c₃^l−1)(t₁₁t₂₂−t₁₂t₂₁)  (18)

[0120] Subsequently, using the secondary intermediate parameters U₀, U₁, V₀, V₁, S₀and S₁, the parameters A, B and C of the epipolar line 502: Au+Bv+C=0 (Here, the (u, v) are 2-D coordinates in the image 20b.) are obtained by the following equations (19)-(21). As mentioned above, the epipolar line 502 is a line in the image 20bwhich restricts the position of the feature point 501 in the image 20bwhich corresponds to the feature point 50ain the image 20a.

[0121] A=S₁V₀−S₀V₁  (19)

[0122] B=−(S₁U₀−S₀U₁)  (20)

[0123] C=U₀V₁−U₁V₀  (21)

[0124] Now that the parameters A, B and C of the epipolar line 502 have been obtained, feature points Q_m: (u_m, v_m) (m=1, . . . ) which exist in the ε₁-neighborhood of the epipolar line 502 in the image 20bcan be expressed by the following inequality. $\begin{array}{cc}\frac{\left|A{u}_m+B{v}_m+C\right|}{\sqrt{A^2+B^2}}<{ϵ}_1& \left(22\right)\end{array}$

[0125] Incidentally, the number of the feature points Q_min the ε₁-neighborhood of the epipolar line 502 is finite (for example, of the order of 10), since feature points Q_mare usually defined in units of pixels in the image 20b.The above feature points Q_m: (u_m, v_m) (m=1, . . . ) are candidate corresponding points in the image 20bwhich are considered to correspond to the feature point 50a:(u_c, v_c) in the image 20a.

[0126] After the feature points Q_m: (u_m, v_m) (m=1, . . . ) are obtained, the 3-D candidate coordinates 60b:

[0127] (X_m^l,Y_m^l,Z_m^l) (m=1, . . . ),

[0128] are calculated for each pair of the feature point 50aand the feature point Q_m(m=1, . . . ).

[0129] The calculation of the 3-D candidate coordinates 60b:

[0130] (X_m^l,Y_m^l,Z_m^l) (m=1, . . . ),

[0131] can be executed based on the aforementioned Japanese Patent Application No.HEI7-206932 as follows, using the feature point 50a:(u^C, v^c), the feature points Q_m: (u_m, v_m) (m=1, . . . ), and the following parameters.

[0132] (u_i^c,v_i^c), (u_i^l,v_i^l)(i=0, . . . ,3), c_j^c,c_j^l(j=1, . . . ,3),

[0133] First, referring to FIG. 5, a set of simultaneous equations which represent a back projection line 503 of the feature point 50a(u^c, v^c):

[0134] {c₁^c(u^c−u₁^c)−(u^c−u₀^c)}X_m^l+{c₂^c(u^c−u₂^c)−(u^c−u₀^c)}Y_m^l+{c₃^c(u^c−u₃^c)−(u^c−u₀^c)}Z_m^l=u₀^c−u^c  (23)

[0135] {c₁^c(v^c−v₁^c)−(v^c−v₀^c)}X_m^l+{c₂^c(v^c−v₂^c)−(v^c−v₀^c)}Y_m^l+{c₃^c(v^c−v₃^c)−(v^c−v₀^c)}Z_m^l=v₀^c−v^c  (24)

[0136] and a set of simultaneous equations which represent a back projection line 504 of the candidate corresponding point 51b(Q_m: (u_m, v_m)):

[0137] {c₁^l(u_m−u₁^l)−(u_m−u₀^l)}X_m^l+{c₂^l(u_m−u₂^l)−(u_m−u₀^l)}Y_m^l+{c₃^l)(u_m−u₃^l)−(u_m−u₀^l)}Z_m^l=u₀^l−u^l  (25)

[0138] {c₁^l(v_m−v₁^l)−(v_m−v₀^l)}X_m^l+{c₂^l(v_m−v₂^l)−(v_m−v₀^l)}Y_m^l+{c₃^l(v_m−v₃^l)−(v_m−v₀^l)}Z_m^l=v₀^l−v^l  (26)

[0139] are derived.

[0140] According to the principle of triangular surveying, the 3-D candidate coordinates 60b:

[0141] (X_m^l,Y_m^l,Z_m^l)(m=1, . . . ),

[0142] can be obtained by solving the simultaneous equations (23). (24), (25) and (26). The solution:

[0143] (X_m^l,Y_m^l,Z_m^l) (m=1, . . . ),

[0144] of the simultaneous equations (23), (24), (25) and (26) can be obtained by means of method of least squares etc.

[0145] The 3-D candidate coordinates 60b:

[0146] (X_m^l,Y_m^l,Z_m^l) (m=1, . . . ),

[0147] obtained as above are sent to the 3-D coordinates determination unit 6, with the feature point coordinates 50a(U^c, V^c) attached thereto as a label.

[0148] Meanwhile, the 3-D coordinates calculation section 5cobtains the 3-D candidate coordinates 60cby the same method as the 3-D coordinates calculation section 5b,using the following parameters which are included in the parameters 400.

[0149] (u_i^c,v_i^c), (u_i^r,v_ir)(i=0, . . . ,3),c_j^c,C_j^r(j=1, . . . ,3),

[0150] The obtained 3-D candidate coordinates 60c:

[0151] (X_n^r,Y_n^r,Z_n^r) (n=1, . . . . ),

[0152] are sent to the 3-D coordinates determination unit 6, with the feature point coordinates 50a(u^c, v^c) attached thereto as a label.

[0153] The 3-D coordinates determination unit 6, which has been supplied with the 3-D candidate coordinates 60band 60cfrom the 3-D coordinates calculation sections 5band 5c,determines the genuine 3-D coordinates 700 using the 3-D candidate coordinates 60band 60cwhich correspond to the same feature point 50a:(U^c, V^c).

[0154] In the following, the operation of the 3-D coordinates determination unit 6 will be described in detail. Referring to FIG. 6, the feature point 50ain the image 20ais in a one-to-one correspondence with a corresponding point 61 on the subject 22 in the 3-D coordinate system. Thus, a 3-D candidate point 60bwhich corresponds to a (2-D) candidate corresponding point 51b(Q_m: (u_m, v_m)) in the image 20b(Here, the candidate corresponding point 51bin the image 20bis a point which is considered to correspond to the feature point 50ain the image 20a.) and a 3-D candidate point 60cwhich corresponds to a (2-D) candidate corresponding point 51cin the image 20c(Here, the candidate corresponding point 51cin the image 20cis a point which is considered to correspond to the feature point 50ain the image 20a.) have to be the same point (i.e. the aforementioned corresponding point 61), and thus have to have almost the same 3-D coordinates. Therefore, for all the combinations of the 3-D candidate points 60b:

[0155] (X_m^l,Y_m^l,Z_m^l) (m=1, . . . ),

[0156] and the 3-D candidate points 60c:

[0157] (X_n^r,Y_n^r,Z_n^r)(n=1, . . . ),

[0158] , the following absolute norm d_mn:

[0159] d_mn=(X_m^l−X_n^r)²+(Y_m^l−Y_n^r)₂+(Z_m^l−Z_n^r)²  (27)

[0160] or the following L²norm d_mn:

[0161] d_mn=|X_m^l−X_n^r|+|Y_m^l−Y_n^r|+|Z_m^l−Z_n^r|  (28 )

[0162] is calculated. Here, the norm d_mnrepresents the distance between the 3-D candidate point 60band the 3-D candidate point 60c.Then, if the minimun value d=d_ij(where d_ij=min(d_mn)) is not larger than a threshold value ε₂, the feature point 50a:(u^c, v^c) is accepted and the 3-D coordinates (X, Y, Z) of the point 61 are determined as follows.

[0163] X=(X₁^l+X_j^r)/2  (29)

[0164] Y=(Y_i^l+Y_j^r)/2  (30)

[0165] Z=(Z_i^l+Z_j^r)/2  (31)

[0166] And if the minimum value d=d_ijis larger than the threshold value ε₂, the feature point 50a:(u^c, v^c) is rejected or discarded as a point out of the subject. The above (preliminary) judgment using the threshold value ε₂is executed in order to reject feature points 50awhich have not been picked up by all of the image pickup sections 1a,1band 1c.

[0167] After the determination by the 3-D coordinates determination unit 6, the genuine 3-D coordinates 700: (X, Y, Z) of the point 61 is outputted to the outline point extraction unit 7, with the feature point coordinates 50a(u^c, v^c) in the image 20aattached thereto as a label.

[0168] The outline point extraction unit 7 first executes judgment whether the feature point 50ais a point in the subject 22 or not, in which the outline point extraction unit 7 accepts the feature point 50a:(u^c, v^c) (which has been attached to the 3-D coordinates 700: (X, Y, Z) of the point 61) if the 3-D coordinates 700: (X, Y, Z) is within the confines of a designated 3-D space domain which has been preliminarily determined or which is designated on the spot by the operator. A plurality of feature points 50a:(u^c, v^c) and their 3-D coordinates 700: (X, Y, Z) are supplied to the outline point extraction unit 7 successively, and the outline point extraction unit 7 executes the above judgment (whether the feature point 50a:(u^c, v^c) can be accepted or not) successively. Then, from a plurality of feature points 50awhich have been accepted by the outline point extraction unit 7 and which exist on a horizontal line (i.e. a plurality of accepted feature points 50a:(u^c, v^c) which have the same Y coordinate v^c), the outline point extraction unit 7 selects a left-hand feature point (a feature point 50aon the horizontal line whose X coordinate u^cis the smallest) and a right-hand feature point (a feature point 50aon the horizontal line whose X coordinate u^cis the largest), and sends the left-hand feature points and the right-hand feature points corresponding to each Y coordinate v^cinto the subject display unit 8 as the subject outline point coordinates 800.

[0169] Then, the subject display unit 8 displays part of the main image 20awhich is surrounded by the subject outline point coordinates 800 (i.e. part of the image 20abetween the left-hand feature points and the right-hand feature points corresponding to each Y coordinate V^c), thereby the image of the subject 22 is extracted from the main image 20aand displayed on the subject display unit 8.

[0170] As described above, in the first embodiment, the main image picking section 1aand the subsidiary image picking sections 1band 1care used to shoot the subject, and images 10a,10band 10cof the 3-D object 21 on which a predetermined pattern is drawn are also picked up by the image picking sections 1a,1band 1cin order to obtain parameters 400 which represent relative positions and attitudes of the image pickup sections 1a,1band 1c.In the embodiment, 11 parameters per one image pickup section (generally known to be enough for expressing relative positions and attitudes of a plurality of image pickup sections) are obtained by the parameter calculation unit 4. From the images 20a,20band 20cin which the subject 22 has been picked up, the feature points 50a,50band 50care extracted by the feature point extraction unit 3, and the genuine 3-D coordinates 700 corresponding to each feature point 50ain the main image 20aare obtained by the 3-D coordinates calculation unit 5 and the 3-D coordinates determination unit 6 using the parameters 400 and the feature points 50band 50c.In the outline point extraction unit 7, judgment whether each feature point 50ais a point in the subject 22 or not is executed using the 3-D coordinates 700 corresponding to the feature point 50a,in which a feature point 50ais accepted if the 3-D coordinates 700 corresponding to the feature point 50ais within the confines of a designated 3-D space domain which has been preliminarily determined or which is designated on the spot by the operator, and otherwise, the feature point 50ais rejected as not a point in the subject 22, and the subject outline point coordinates 800 are extracted from a plurality of feature points 50awhich have been accepted. Then, part of the main image 20awhich is surrounded by the subject outline point coordinates 800 is extracted and displayed by the subject display unit 8, thereby the subject image extraction from the main image 20ais completed.

[0171] Therefore, according to the first embodiment, an image of a desired subject can be extracted from the main image 20a,using the 3-D coordinates calculated for each extracted feature point 50aand executing the judgment with regard to the designated 3-D space domain which has been preliminarily determined or which is designated on the spot by the operator, without needing preparation of the information concerning the subject or the image pickup devices.

[0172] Further, according to the first embodiment, subject image extraction can be executed with easy setting of a threshold value for distinguishing between the background and the foreground, since 3-D coordinates corresponding to a feature point 50ain the image 20aare automatically calculated using the parameters 400 etc. and the operator of the device can set the threshold value by measurable value such as distance or by designating the 3-D space domain.

[0173] Further, according to the first embodiment, subject image extraction can be executed without needing precise control or measurement of the positions and attitudes of the image pickup devices, since the parameters 400 representing relative positions and attitudes of the image pickup sections 1a,1band 1ccan be automatically obtained using the images 10a,10band 10cof the 3-D object 21 on which a predetermined pattern is drawn.

[0174] Further, according to the first embodiment, accuracy of subject image extraction can be remarkably improved, since the 3-D coordinates 700 which correspond to each extracted feature point 50ain the main image 20acan be accurately obtained using the parameters 400 calculated from the images 10a,10b,10cpicked up by the image pickup sections 1a.1band 1c,and the feature points 50band 50cextracted from the subsidiary images 20band 20c.The accuracy of subject image extraction is increased with a synergistic effect by the extraction of the feature points 50a,50band 50cfrom the aforementioned edge-like parts of the images 20a,20band 20cand by the accurate calculation of the genuine 3-D coordinates 700 and by the designation of the designated 3-D space domain.

[0175] Incidentally, although three image pickup sections (i.e. one main image pickup section 1aand two subsidiary image pickup sections 1band 1c) were used in the image pickup unit 1 in the above embodiment, it is also possible to use four or more image pickup sections (i.e. three or more subsidiary image pickup sections). In the case where N pieces of subsidiary image pickup sections are used, N+1 pieces of image storage sections, N+1 pieces of feature point extraction sections, and N pieces of 3-D coordinates calculation sections are used. In this case, (N+1)×11 parameters (i.e. 11 parameters per image pickup sections) are calculated by the parameter calculation unit 4 and N groups of 3-D candidate coordinates (60b,60c,. . . ) are obtained by the 3-D coordinates calculation unit 5, and in the 3-D coordinates determination unit 6, the sum of norms d_mn(with regard to every pair of subsidiary images) are used instead of the norm d_mnof the equation (27) or (28), and the averages in the equations (29) through (31) are replaced by averages divided by N (not by 2).

[0176] FIG. 8 is a block diagram showing a subject image extraction device according to the second embodiment of the present invention.

[0177] The subject image extraction device of FIG. 8 comprises an image pickup unit 1, an image storage unit 2, a feature point neighborhood extraction unit 33, a parameter calculation unit 4, a corresponding point determination unit 9, a 3-D coordinates calculation unit 55, a 3-D coordinates determination unit 6, an outline point extraction unit 7, and a subject display unit 8.

[0178] The image pickup unit 1 is composed of an image pickup section 1afor picking up the main image to be mainly used in the subject image extraction process, and image pickup sections 1band 1cfor picking up subsidiary images to be referred to in the subject image extraction process. The image pickup sections 1a,1band 1care preliminarily used to shoot a 3-D object 21 on which a predetermined pattern is drawn in order to obtain the parameters 400, and are used to shoot a subject 22 whose image will be extracted using the parameters 400 etc.

[0179] The image storage unit 2 is composed of image storage sections 2a,2band 2c.The image storage sections 2a,2band 2cstore images 10a,10band 10cof the 3-D object 21 and images 20a,20band 20cof the subject 22 which have been picked up by the image pickup sections 1a,1band 1c,respectively.

[0180] The feature point neighborhood extraction unit 33 is composed of feature point neighborhood extraction sections 33a,33band 33c.The feature point neighborhood extraction sections 33a,33band 33cextract coordinates 30a,30band 30cof a plurality of feature points from the images 10a,10band 10cwhich have been stored in the image storage sections 2a,2band 2crespectively, and extract coordinates 50a,50band 50cof a plurality of feature points from the images 20a,20band 20cwhich have been stored in the image storage sections 2a,2band 2crespectively, and further extract color/luminance information 52a,52band 52cof pixels in the neighborhood of the feature points 50a,50band 50crespectively.

[0181] The parameter calculation unit 4 calculates and stores the parameters 400 which represent relative positions and attitudes of the main image pickup section 1aand the subsidiary image pickup sections 1band 1c,using the feature point coordinates 30a,30band 30cwhich have been extracted by the feature point extraction sections 3a,3band 3crespectively.

[0182] The corresponding point determination unit 9 is composed of corresponding point determination sections 9band 9c.The corresponding point determination section 9bselects a group of candidate corresponding points 90bin the image 20b(points in the image 20bwhich are considered to correspond to the feature point 50ain the image 20a) from the feature points 50b,using the feature points 50aand 50band the color/luminance information 52aand 52bof pixels in the neighborhood of the feature points 50aand 50bwhich have been extracted by the feature point neighborhood extraction sections 33aand 33band using the parameters 400. The corresponding point determination section 9cselects a group of candidate corresponding points 90cin the image 20c(points in the image 20cwhich are considered to correspond to the feature point 50ain the image 20a) from the feature points 50c,using the feature points 50aand 50cand the color/luminance information 52aand 52cof pixels in the neighborhood of the feature points 50aand 50cwhich have been extracted by the feature point neighborhood extraction sections 33aand 33cand using the parameters 400.

[0183] The 3-D coordinates calculation unit 55 is composed of 3-D coordinates calculation sections 55band 55c.The 3-D coordinates calculation section 55bcalculates 3-D candidate coordinates 60busing the feature point 50awhich has been extracted by the feature point neighborhood extraction section 33a,the candidate corresponding points 90bwhich have been selected by the corresponding point determination section 9b, and the parameters 400. The 3-D coordinates calculation section 55ccalculates 3-D candidate coordinates 60cusing the feature point 50awhich has been extracted by the feature point neighborhood extraction section 33a,the candidate corresponding points 90cwhich have been selected by the corresponding point determination section 9c,and the parameters 400.

[0184] The 3-D coordinates determination unit 6 determines genuine 3-D coordinates 700 which correspond to the feature point coordinates 50ain the main image 20a,using the 3-D candidate coordinates 60band the 3-D candidate coordinates 60c.

[0185] The outline point extraction unit 7 judges whether the feature point 50ais a point in the subject 22 or not, using the 3-D coordinates 700, and extracts subject outline point coordinates 800.

[0186] The subject display unit 8 displays part of the image 20awhich are surrounded by the subject outline points, using the subject outline point coordinates 800 extracted by the outline point extraction unit 7.

[0187] In the following, the operations of the corresponding point determination unit 9 (which has been newly introduced into the second embodiment) and the feature point neighborhood extraction unit 33 and the 3-D coordinates calculation unit 55 (whose operations have been modified in the second embodiment) will be described in detail. The operations of the other blocks in FIG. 8 is the same as those of the first embodiment.

[0188] First, the operation of the feature point neighborhood extraction unit 33 will be described. When the images 10a,10band 10cof the 3-D object 21 on which the predetermined pattern is drawn are supplied to the feature point neighborhood extraction unit 33, the feature point neighborhood extraction sections 33a,33band 33coperate in the same way as the feature point extraction sections 3a,3band 3cof the first embodiment and extract the coordinates 30a,30band 30cof a plurality of feature points from the images 10a,10band 10cand output the feature point coordinates 30a,30band 30cto the parameter calculation unit 4, respectively. When the images 20a,20band 20cof the subject 22 are supplied to the feature point neighborhood extraction unit 33, the feature point neighborhood extraction sections 33a,33band 33cextract the feature points 50a,50band 50crespectively in the same way as the feature point extraction sections 3a,3band 3cof the first embodiment, and further extract color/luminance information (color intensity information in the case of color images or luminance information in the case of black and white images) 52a,52band 52cof pixels in the neighborhood of the feature points 50a,50band 50crespectively. The neighborhood can be defined in various ways, however, a rectangle whose center is on the feature point is generally used for the neighborhood.

[0189] Next, the operation of the corresponding point determination unit 9 will be described. The corresponding point determination section 9bin the corresponding point determination unit 9 extracts candidate corresponding points 5b(i.e. the Q_m: (u_m,v_m) (m=1, . . . ) which are obtained by the aforementioned inequality (22)) in the image 20bwhich corresponds to the feature point 50aby calculating the epipolar line in the same way as the 3-D coordinates calculation section 5bof the first embodiment. Subsequently, the corresponding point determination section 9bexecutes block matching, which is known as an image processing technique, between the color/luminance information 52aof the pixels in the neighborhood of the feature point 50aand the color/luminance information 52bof the pixels in the neighborhood of a candidate corresponding point 51b.FIG. 7 is a schematic diagram explaining the block matching. In the block matching between the color/luminance information 52aand 52b,two blocks of the same size respectively including the feature point 50aand the candidate corresponding point 51bare formed first, and comparison of the color/luminance between two corresponding points in the two blocks is executed. Then, the difference of the color/luminance is added up with respect to each point in the blocks (The sum is referred to as a matching error.). If the matching error is larger than a predetermined threshold value, the candidate corresponding point 51bis rejected as an inadequate point (a point not corresponding to the feature point 50a). After the rejection by the block matching, a group of candidate corresponding points 90bfrom which the inadequate points have been removed is sent to the 3-D coordinates calculation section 55bin the 3-D coordinates calculation unit 55. At this point, the candidate corresponding points 90bhave been screened by the block matching, and the number of the candidate corresponding points 90bhave been considerably reduced due to the block matching. Meanwhile, the corresponding point determination section 9cdetermines a group of candidate corresponding points 90cin the same way as the corresponding point determination section 9b,and sends the group of candidate corresponding points 90cto the 3-D coordinates calculation section 55cin the 3-D coordinates calculation unit 55. Incidentally, although the above corresponding point determination unit 9 used the parameters 400 for calculating the epipolar line and obtaining the candidate corresponding points, it is also possible to obtain candidate corresponding points without using the parameters 400.

[0190] Lastly, the operation of the 3-D coordinates calculation unit 55 will be described. The 3-D coordinates calculation section 55bin the 3-D coordinates calculation unit 55 obtains almost all 3-D candidate coordinates 60bby the same operation as the 3-D coordinates calculation section 5bof the first embodiment. However, in the second embodiment, the candidate corresponding points 90bhave been already obtained by the corresponding point determination section 9b,therefore the 3-D coordinates calculation section 55bdoes not calculate the candidate corresponding points Q_m: (u_m, v_m) (m=1, . . . ) and uses the candidate corresponding points 90bsupplied by the corresponding point determination section 9b.

[0191] As described above, according to the second embodiment, as well as the same effects as those of the first embodiment, screening of corresponding points is preliminarily executed by the corresponding point determination unit 9 by means of the block matching, and thus accuracy of calculation and determination of the 3-D coordinates which correspond to the feature point 50acan be further improved, and calculation time for the 3-D coordinates (which was the major portion of calculation time of the subject image extraction device) can be considerably shortened.

[0192] As set forth hereinbefore by the subject image extraction device according to the present invention, an image of a desired subject can be extracted from an inputted image with high accuracy, without needing preparation of the information concerning the subject or the image pickup devices, without needing precise control or measurement of the positions and attitudes of the image pickup devices, and with easy setting of a threshold value for distinguishing between the background and the foreground.

[0193] The subject image extraction devices which have been described referring to FIG. 1 and FIG. 8 can be applied to reduction of the amount of data which is transmitted between video phone etc. In the case of video phone, a speaker talking near the video phone (i.e. in the designated 3-D space domain) can be extracted as the subject from an image, and the image of the speaker only can be encoded and transmitted, thereby the amount of data can be considerably reduced, performance of video phone can be improved, and privacy of speakers can be protected.

[0194] Incidentally, although the subject image extraction devices of FIG. 1 and FIG. 8 included the image pickup unit 1, subject image extraction can of course be executed by a subject image extraction devices which is not provided with the image pickup unit 1 as long as images of a 3-D object and images including a subject which have been picked up by three or more image pickup sections (devices) are supplied. Therefore, subject image extraction devices according to the present invention without image pickup units are also possible. Similarly, subject image extraction devices according to the present invention for subject image extraction only (i.e. not for displaying the extracted subject image) are also possible. Description of such subject image extraction devices is omitted for brevity.

[0195] While the present invention has been described with reference to the particular illustrative embodiments, it is not to be restricted by those embodiments but only by the appended claims. It is to be appreciated that those skilled in the art can change or modify the embodiments without departing from the scope and spirit of the present invention.