Methods and system for measuring three dimensional spatial coordinates and for external camera calibration necessary for that measurement

[0002] The present invention refers, in general, to methods and systems for videogrammetry and, in particular, to methods and a system for measuring three-dimensional spatial coordinates and for external camera calibration required for that measurement.

[0003] There is a known method in photogrammetry for analyzing two-dimensional images, captured on photographic film, to produce three-dimensional coordinate measurements. At least two images of an overlapping surface portion are required. Modern photogrammetry has incorporated computers to carry out many of its functions, but the process has not yet been entirely automated. Videogrammetry has added two major improvements to the art of photogrammetry. Firstly, rather than recording images on film it records images directly on opto-electronic imaging devices, such as photosensitive diodes or other imaging surfaces. And secondly, since the images are immediately digitized and stored in a computer, they are readily accessible to manipulation and analysis by computational methods.

[0004] Attempts have been made to develop better systems and methods for measuring three-dimensional spatial coordinates. For example, U.S. Pat. No. 5,589,942 dated Dec. 31, 1996, granted to Gordon, for a “Real time three dimensional sensing system” discloses a system which utilizes two flexibly located cameras for receiving and recording visual information with respect to a sensed object illuminated by a series of light planes. Each pixel of each image is converted to a digital word and the words are grouped into stripes, each stripe comprising contiguous pixels. One pixel of each stripe in one image is selected and an epi-polar line of that point is drawn in the other image. The three dimensional coordinate of each selected point is determined by establishing the point on said epi-polar line which also lies on a stripe in the second image and which is closest to a known light plane. This system uses a complex pattern of projected stripes, which creates difficulties in finding homologous points in the pair of images, and leads to false positives and false negatives in homologous point identification. The system requires the light plane to be known.

[0005] U.S. Pat. No. 4,627,734, dated Dec. 9, 1986, granted to Rioux, for a “Three dimensional imaging device and method” discloses a three-dimensional imaging system operating in accordance with the known active triangulation method, and employing a laser beam that is projected onto an area of the surface to be examined. The solution is characterized by synchronized scanning of projected and deflected beams. The main disadvantage of the system consists in the fact that the position and orientation of the projected light beam must be known in order to calculate three dimensional coordinates.

[0006] U.S. Pat. No. 5,440,392 dated Aug. 8, 1995, granted to Pettersen et al. for a “Method and system for point by point measurement of spatial coordinates” describes a technical solution for point by point measurement of spatial coordinates. A touch probe, including three point-sized light sources at known local coordinates, touches the point to be measured. Using a camera to acquire an image of the three light sources, the system calculates the three dimensional coordinates of the point touched. It is obvious that this solution requires physically touching the object to be measured and is time-consuming.

[0007] There is accordingly a need for a method and system which overcome, or at least alleviate, the disadvantages of the prior art. It is therefore desirable to offer a technical solution which is faster, non-contact, and calibrated quickly and easily, without the use of special equipment.

[0008] Broadly, the method of measuring three-dimensional spatial coordinates starts by synchronizing at least two cameras so that they can capture simultaneous images. Then the cameras are set up with the object to be measured in their fields of view. A simple pattern is then projected onto the surface of the object, and simultaneous images which include the projected pattern are captured. The images are processed by a computer, which uses conventional software algorithms to find the pattern images, calculate centroids from the pattern images, group the centroids into homologous pairs, and calculate three-dimensional spatial coordinates.

[0009] In one aspect of the above method, wherein the projected simple pattern is a spot, images are discarded and recaptured, if necessary, until there is just one centroid in each camera image. Therefore, grouping these centroids into a homologous pair is trivial.

[0010] In another aspect of the above method, wherein the projected simple pattern is a stripe, it is necessary to segment the stripe images into sets of continuous stripe images. In one camera image, each continous stripe image is sampled at a known interval, and a centroid is calculated across the stripe for each sample, yielding a series of two dimensional points in the camera image. For each point in the series, the known internal and external calibrations of both cameras are used to determine a corresponding two-dimensional point on a stripe within the other camera image. Each such pair of corresponding points are grouped as a homologous pair.

[0011] Generally, the method of external camera calibration starts by synchronizing at least two cameras, so that they capture simultaneous images. Then, the cameras are placed so that their fields of view include a surface that will be used for calibration, and a simple pattern is projected onto that surface. A pair of simultaneous images of the simple pattern are captured. The images are processed by a computer, which uses conventional software algorithms to find the pattern images, to discard any images in which the pattern is not found, to calculate centroids of the pattern images, and to group the centroids into homologous pairs. Then, the pattern is projected onto a different portion of the surface, and new images are captured and processed, until the required number of homologous pairs have been found. The computer uses software for the methods of resection and ray bundle adjustment, known in photogrammetry, to compute an external calibration, with relative scale, of the positions and orientations of the cameras. Then, the system is used to acquire three-dimensional spatial coordinates from the surface of a target object of known dimensions. From this, a ratio is calculated and applied to the external calibration, in order to transform the relative scale into a usable absolute scale.

[0012] Broadly, the system used to carry out the above methods comprises a computer provided with software to execute the methods, at least two cameras containing image planes divided into a multiplicity of photosensitive sites, and a light projector capable of projecting a simple pattern onto a surface. Simultaneously exposed images of the simple pattern are captured and stored in computer memory.

[0013] Alternatively, the system can use charge couple devices, charge injection devices, or complementary metal-oxide semiconductors for image planes.

[0014] Alternatively, the system can use a light projector that is hand-held or coupled to a computer-controlled electro-mechanical beam positioner.

[0015] Alternatively, the system can use a light projector that is a laser device, a light-emitting diode (LED) device, or a light projector using focussing optics to project an image onto an external surface.

[0016] Alternatively, the system can use a light projector that uses collimating or focussing optics.

[0017] FIG. 1 depicts the system for carrying out the present invention, and

[0018] FIG. 2 depicts the use of a projected stripe for external camera calibration.

[0019] Referring now to the accompanying FIG. 1, there is depicted a system for measuring three-dimensional spatial coordinates and for external camera calibration 10, which comprises a computer 12 with image capture electronics boards 14 to which are connected two cameras 16 and 18. Alternatively, more cameras may be added, if required, for more complete coverage of the surface. A light projector 20 is used to project a simple pattern 22, such as a small spot of which the image approximates a point light source, onto a surface 24. Alternatively, simple pattern 22 may be a larger spot, of which the image is a disk, or it may be a stripe. Simultaneously-exposed images of projected simple pattern 22 are produced by cameras 16 and 18, then captured by image capture electronics boards 14, and finally stored in the memory of computer 12. Computer 12 is provided with software to control cameras 16 and 18 and to process and analyze the images. Cameras 16 and 18 contain imaging surfaces, which are charge-couple devices (CCD). The imaging surfaces are divided into a multiplicity of photo-sensitive sites.

[0020] Alternatively, cameras 16 and 18 may contain imaging surfaces which are charge injection devices (CID) or complementary metal-oxide semiconductor (CMOS) devices.

[0021] Light projector 20 is hand-held. Alternatively, light projector 20 is coupled to a computer-controlled electro-mechanical beam positioner capable of altitude/azimuth movement.

[0022] Light projector 20 is a laser device. Alternatively, light projector 20 is a light-emitting diode (LED) device or a slide projector.

[0023] Light projector 20 uses a coherent light source. Alternatively, light projector 20 can use an incoherent light source.

[0024] Light projector 20 uses collimating optics. Alternatively, light projector 20 can use focussing optics.

[0025] In the case of a laser or LED device, beam forming optics that expand the light in the shape of a flat fan can be used.

[0026] In the case of a slide projector, the slide comprises an opaque mask surrounding a transparent pattern which is one of the above-mentioned three simple patterns: a small spot, a larger spot, or a stripe.

[0027] The components of the system mentioned in the above description are of conventional type and are readily available.

[0028] The system described above is used to measure three-dimensional spatial coordinates as follows: Cameras 16 and 18 are synchronized in order to capture simultaneous images. Then, cameras 16 and 18, kept in a fixed relationship, are directed towards the surface of an object to be measured. Then, a simple pattern is projected onto a portion of the surface of the object, which portion is in the field of view of both cameras 16 and 18. Image capture electronics boards 14 are used to capture simultaneous images of the projected simple pattern in cameras 16 and 18. Computer 12 is provided with software, using conventional algorithms, to perform the following operations:

[0029] images of the projected pattern are found within the captured images;

[0030] if images of the pattern are not found within both camera images, the camera images are discarded, the pattern is moved, and new images are captured;

[0031] centroids of the projected pattern on the image planes of cameras 16 and 18 are calculated;

[0032] the above-calculated centroids are grouped into homologous pairs (a pair of centroids is homologous if they are centroids of two images of the same portion of the same simple pattern), and

[0033] three-dimensional coordinates are calculated from each homologous pair of centroids, internal and external camera calibrations being known.

[0034] Some of the above operations differ depending on the type of projected simple pattern.

[0035] If the projected simple pattern is a spot, the above operations, except the first and the last, are replaced by the following:

[0036] the camera images are discarded and new ones captured, if images of the pattern are not found within both camera images, or if multiple images of the pattern are found within a single camera image;

[0037] two-dimensional centroids are calculated, for the images of the spot, on the image planes of cameras 16 and 18, and

[0038] the resulting centroids are grouped as a homologous pair.

[0039] If the projected simple pattern is a stripe, the above operations, except the first and the last, are replaced by the following:

[0040] the stripe image within each camera image is segmented into a set of one or more continuous stripe images;

[0041] in one camera image, one-dimensional centroids are calculated across each continous stripe image at a known sampling interval, which yields a series of two-dimensional points on the image plane;

[0042] for each two-dimensional point in the series, a corresponding two-dimensional point on a stripe image in the other camera image is found, using the known internal and external camera calibrations, and

[0043] each such pair of corresponding two-dimensional points is grouped as a homologous pair.

[0044] An external camera calibration determines the positions and orientations of the cameras. The external camera calibration, along with the already-established internal camera calibration and the homologous pairs, mentioned above, is necessary for using the system to calculate three-dimensional spatial coordinates, as described above. To perform external camera calibration, the system, in the case where the simple pattern is a spot, is used as follows:

[0045] 1. Cameras 16 and 18 are synchronized in order to capture simultaneous images and are directed independently towards a surface used for calibration.

[0046] 2. A simple spot pattern is projected onto a portion of the said surface used for calibration, which portion is located in the field of view of both cameras 16 and 18.

[0047] 3. A pair of images of the projected simple pattern is captured simultaneously.

[0048] 4. Computer 12 is provided with software, using conventional algorithms, to perform the following operations:

[0049] images of the projected pattern within the captured images are found; if no images of the pattern are found, or multiple images of the pattern are found within a camera image, the camera images are discarded, the projected pattern is moved, and new camera images are captured;

[0050] centroids of the projected pattern, on the image planes of cameras 16 and 18, are calculated;

[0051] the above-calculated centroids are grouped into homologous pairs;

[0052] 5. Operations 2 to 4 are repeated, so as to obtain at least three homologous pairs of centroids derived from projecting the simple pattern onto different portions of the surface used for calibration.

[0053] 6. Computer 12, provided with software, applies the methods of resection and ray bundle adjustment, known in photogrammetry, to calculate a partial external calibration, with an unknown absolute scale, of the positions and orientations of cameras 16 and 18.

[0054] 7. An absolute scale is determined by using the system to measure a target of known dimensions, then establishing the ratio between measured dimensions and known dimensions, and applying that ratio to the already-calculated partial external calibration.

[0055] In the case where the simple pattern is a stripe, the system is used as follows (FIG. 2):

[0056] 1. Cameras 16 and 18 are synchronized in order to capture simultaneous images and are directed independently towards a surface used for calibration 24′.

[0057] 2. Surface used for calibration 24′ comprises planes that intersect so that their lines of intersection are in the fields of view of cameras 16 and 18.

[0058] 3. A simple pattern 22′, which in this case is a stripe, is projected onto a portion of surface used for calibration 24′, which portion is located in the field of view of both cameras 16 and 18, so that the stripe crosses the said lines of intersection.

[0059] 4. A pair of camera images 26 containing projected simple pattern 22′ is captured simultaneously.

[0060] 5. Computer 12 is provided with software, using conventional algorithms, to perform the following operations:

[0061] a pair of stripe images 28 of projected simple pattern 22′, within pair of camera images 26, already captured, are found; if no images of projected simple pattern 22′ are found, or multiple images of projected simple pattern 22′ are found within either one of pair of camera images 26, that pair of camera images 26 is discarded, then projected simple pattern 22′ is moved, and a new pair of camera images 26 is captured;

[0062] in camera images 26, one-dimensional centroids are calculated across each image of the pair of stripe images 28, at a known sampling interval, which yields a series of two-dimensional points on the image plane;

[0063] each stripe image of the pair of stripe images 28 is divided into straight-line segments;

[0064] the intersection points of adjoining straight-line segments are found in both of the pair of stripe images 28;

[0065] the found intersection points are grouped into homologous pairs, in order, from top to bottom of each image;

[0066] 6. If the necessary minimum number of homologous pairs of intersection points is not obtained, projected simple pattern 22′ is moved to a new position on surface used for calibration 24′ and operations 3 to 5 are repeated.

[0067] 7. Computer 12, provided with software, applies the methods of resection and ray bundle adjustment, known in photogrammetry, to calculate a partial external calibration, with an unknown absolute scale, of the positions and orientations of cameras 16 and 18.

[0068] 8. An absolute scale is determined by using the system to measure a target of known dimensions, then establishing the ratio between measured dimensions and known dimensions, and applying that ratio to the already-calculated partial external calibration.

[0069] At this stage the external calibration is complete, and the system can be used to measure three-dimensional spatial coordinates.

[0070] The system and its uses, described above, are based on the same inventive concept, which consists in projecting a simple pattern to generate homologous points for use with standard photogrammetric techniques.