Techniques for photogrammetric systems

[0001] The application claims priority of U.S. provisional application 60/233,471, filed Sep. 19, 2000 and entitled “A New General Purpose Photogrammetric 3D Measure System”.

[0002] The present invention relates to the measurement of any visible objects by measurement of points or similar structures. More particularly it relates to a method and apparatus for the measurement of 3D coordinates of points by analyzing images of same.

[0003] 3D Measurements are well known in the art and are widely used in the industry. The purpose is establishing the 3 coordinates of any desired point with respect to a reference point and coordinate system. As known in the prior art, these measurements can be accomplished with coordinate measuring machines (CMMs), teodolites, photogrammetry, laser triangulation methods, interferometry, and other contact and non-contact measurements. However all tend to be complex and expensive to implement in an industrial setting.

[0004] U.S. Pat. No. 4,431,290: “Portable photogrammetry system”, issued Feb. 14, 1984, provides a short-range hand-held photogrammetry system consisting of two cameras in rigidly fixed positions at the remote ends if a hollow bar. The system provides for simultaneous image capture.

[0005] A number of useful methods also exist as exemplified by U.S. Pat. No. 4,802,757: “System for determining the attitude of an airborne platform or the alike”, issued Feb. 7, 1989. That patent describes a system that employs digital image grayscale correlation in order to relatively position the moving image sensor at the times of the successive image captures.

[0006] However, applications of these systems and methods tend to be limited. Some are physically too large to be moved and easily applied, others require a lot of human intervention. Most require a relatively long data acquisition time where an object has to stand still. Furthermore, they are optimized for a specific object size. Thus, what is needed is a flexible, easily implemented system that can measure in a wide variety of industrial settings. A system performing measurements at sites radically varying by size and complexity, for example measurements in the construction industry as well as in continuous manufacturing processes, is needed.

[0007] Accordingly, an object of the present invention is to provide a widely applicable and efficient techniques for systems based on photogrammetric principles, which can measure three-dimensional coordinates.

[0008] It is another object of the present invention for the techniques to apply to either stationary or hand-held cameras. The techniques can use different means of image acquisition, such as photo camera, video camera, and x-ray, and the described methods are equally applicable to all modalities of image capture.

[0009] According to a first broad aspect of the present invention, there is provided a method for performing camera position approximation. A method is taught for finding the position pf an object in front of a camera. The starting point is an image taken by the camera in question for which the position is unknown. Three points are selected on the image and are projected into an image space by Intersecting them with the known focal point from the image. The relationship between these three points is known. A first point, point A, is selected arbitrarily on a first of the three rays produced. Point A is preferably selected close to the image plane. A second point, point: B, is found on a second of the three rays by respecting the known distance between two of the points selected on the image plane. In order to determine the location of point C, a circle is drawn, its axis on a path AB land its radius such that known distance AC and BC are respected. Point C is then identified as the point on the circle that is closest to the third ray, wherein point A, point B, and point C define the device position and define its x, y, z coordinates; and wherein orientation angles Ω, Φ, and κ can be calculated from the x, v, z coordinates of the points A, B, C. thereby defining the camera position.

[0010] Preferably, the process is repeated by choosing different positions for point A in order to find the point C closest to the third ray.

[0011] According to a second broad aspect of the present invention, there is provided a method for reconstructing planar objects wherein the planar objects are captured in a plurality of two-dimensional images taken from different angles. The method comprises reconstructing a set of points in an image plane in three dimensions using standard photogrammetric techniques. The set of points should contain at least three points. A three-dimensional position for a selected point must then be calculated on the planar object by identifying the point on one of the two-dimensional images. The actual calculation comprises intersecting a ray that passes from a focal point of a camera through the selected point on the image plane and a plane that is calculated to fit well with the set of points.

[0012] According to a third broad aspect of the present invention, there is provided a method for precision estimation of a measured three-dimensional coordinate based on Monte-Carlo simulation. The first step is to choose a three-dimensional point in an imaged space and calculate the point's theoretical position in the image planes. This point becomes a simulation point. Reference points are then chosen in the image planes and their positions are calculated. Tolerances for internal and external parameters of the system (or camera) are then acquired from a system database. A suitable amount of error to positions and parameters are then added to the simulation, the error being determined according to the acquired tolerances. The simulation point is then reconstructed based on its position in images obtained through the simulation and a three-dimensional coordinate is produced. The last step is to compare the original three-dimensional point with the three-dimensional coordinate that was produced by the reconstruction to obtain a precision estimation. Repeating the precision estimation with different amounts of error and averaging out the results can produce a better estimation.

[0016] For the photogrammetric reconstruction of points in three dimensions, two or more photos of a scene are needed. Further, it is necessary to know the camera parameters and position for every photo taken. This requirement is in contradiction with the requirement for flexibility and mobility. Therefore, when camera positions are not known, camera position approximation is applied: along with the measured object, a sufficient number of reference points need to be present in the photos and they are used to find the positions of mobile camera(s) after the images are taken. Note that reference points are only needed if the camera positions and parameters are not already precisely known.

[0017] In general, calibration of the camera position and parameters is performed by using known reference points that make part of the scene. First, these points are identified in the photos. Their known positions in the scene (3D) and in the photos (2D) make it possible to first find the approximation of the camera position for every photo. They are also used to adjust the internal camera parameters, which may vary from one photo to another, like the focus and principal point in the compensation phase. Further they are also used to adjust implementation specific parameters of the imaging device such as lens distortion correction parameters. The parameters are the radial, tangential and centering error correction parameters.

[0018] The Camera Position Approximation software determines the camera position from information about reference points in each of the images. The reference points can be identified in the image by pattern recognition methods and the system determines the camera position without any operator assistance. If the number of machine recognizable reference points is not sufficient in one or more of the images then the user is required to identify the reference points.

[0019] FIG. 1 demonstrates the process of finding the position of an object in front of a camera. The starting point is an image taken by the camera in question for which the position is unknown. Three points are selected on the image and are projected into an image space by intersecting them with the known focal point from the image. The relationship between these three points is known. A first point, point A, is selected arbitrarily on a first of the three rays produced. Point A is preferably selected close to the image plane. A second point, point B, is found on a second of the three rays by respecting the known distance between two of the points selected on the image plane. In order to determine the location of point C, a circle is drawn, its axis on a path AB and its radius such that known distance AC and BC are respected. Point C is then identified as the point on the circle that is closest to the third ray. The process can be repeated by choosing different positions for point A in order to find the point C closest to the third ray. Please refer to Table 1 for the algorithm in pseudo-code.

[0020]

TABLE 1
Algorithm In Pseudo-Code
Program FindCameraPosition(image, referenceElementModel):
ReferencePointsArray =
	= correlation(image, referenceElementModel, reference_point_pattern)
imagePointA = Select3ReferencePointsFromArray(ReferencePointsArray)
imagePointB = Select3ReferencePointsFromArray(ReferencePointsArray)
imagePointC = Select3ReferencePointsFromArray(ReferencePointsArray)
rayA = new Ray(focus, imagePointA)
rayB = new Ray(focus, imagePointB)
rayC = new Ray(focus, imagePointC)
for (all pointA on rayA)
{
	sphereB = new Sphere (radius=d(A,B), center=pointA)
	pointB = cut_path_and_sphere_3D(rayB, SphereB)
	axisAB = new Path( through pointA and pointB)
	circleC = new Circle (radius=d(C,AxisAB), center=projection of C on axisAB)
	error = distance_path_to_circle(rayC, circleC)
	if (detect_local_minimum(error))
	{
	position = find_rot_transl(pointA, pointB, pointC)
	error_all = find_error_for_all_points(all points)
	if (detect_global_minimum(error_all))
	store(position)
	}
}

[0021] Compensation is the process of calculating the internal camera parameters: the focal length—F, and the principal point offsets—Xpp and Ypp. At the same time the estimated camera positions are fine adjusted. During compensation parameters for all images are optimized in the same process and a common best fit is found.

[0022] There are two ways to calibrate a block of photos: either only by calibration of the block of photos (for camera parameters and positions) or by both pre-calibration of the camera and calibration of a block of photos.

[0023] Photo block calibration alone: Calibration of a block of photos is performed to obtain the internal camera parameters (Xpp, Ypp, F) and the external parameters (the positions from where the photos were taken—X,Y,Z,K,P,O). Reference points are identified in every photo. A data-fitting algorithm using the maximum likelihood method adjusts the said parameters and positions in order to harmonize the mathematical model with the measured data.

[0024] PRE-Calibration and block calibration: Although the information in the block of photos is self sufficient for the calibration to be performed, sometimes it is advantageous to perform the Pre-calibration of the camera to improve further on the precision of the measurement. Pre-calibration of the camera parameters (Xpp, Ypp, F) can only be meaningful if the parameters are not changing. Auto-focus cameras, in general, cannot use pre-calibration. Pre-calibration is performed best with a large number of precise points that are well distributed in the field of view of the camera. A good set of reference points is the “calibration cube”. The calculated camera parameters (Xpp, Ypp, F) are permanently stored in the camera properties. These parameters are reused every time that the camera is used. They are considered as very precise.

[0025] The pre-calibrated parameters will enter the algorithm of the photo block calibration (camera parameters and position) with a very small standard deviation value. This means the camera parameters will strongly “stick” to the pre-calibrated values during the calibration of a block. At the same time, predominantly the camera position will be adjusted to fit the measured data best.

[0026] The “Internal Orientation Start Value” option leaves the choice to the user to choose between the pre-calibrated (“Calibrated per camera”) and the approximated value for the internal parameters. The pre-calibrated option will be the default if pre-calibration was performed for the active camera. If so, the proper parameter values that are stored in the project database will be applied. It should be used if the camera has been carefully pre-calibrated and remained untouched since. Otherwise, the “approximated” values for the internal parameters are to be used.

[0027] In a second embodiment the camera positions and parameters are fixed and known and no reference elements are needed in the images and the camera position approximation and compensation are not applied.

[0028] In the reconstruction phase a point needs to be selected in at least two images taken from different angles in order to be restituted. Any triangulation-based algorithm can be applied to find the points' position in object space based on known camera position parameters and the point's position in multiple images. For point identification, images are displayed side by side. The identification of points in the multiple images can be performed in a number of modalities. In the manual modality the user is able to select any matching points on multiple images by clicking them with the mouse.

[0029] In the first automatic modality the system searches for points with a user specified image pattern in user specified areas of the images. Image patterns like points, crosses or circles can be specified as targets for the search algorithm. The system applies a pattern recognition method, like for example the normalized grayscale correlation. For increased precision subpixel methods are applied. These algorithms are precise up to a portion of a picture element—pixel. In the second automatic modality the user points out any object point in any of the images and the system identifies the same object point, if visible, in other images. In any of the modalities, the system allows the user to perform a manual correction of the point positions. Information about the relative orientation between the two images can be used to harmonize the orientation of the left and right image by rotation. This is to facilitate the pattern recognition task.

[0030] The algorithm used to reconstruct planar objects begins by reconstructing a set of points in an image plane in three dimensions using standard photogrammetric techniques. The set of points should contain at least three points. A three-dimensional position for a selected point must then be calculated on the planar object by identifying the point on one of the two-dimensional images. The actual calculation comprises intersecting a ray that passes from a focal point of a camera through the selected point on the image plane and a plane that is calculated to fit well with the set of points.

[0031] In another embodiment, the system performs a precision estimate of the reconstructed points. The system performs for each of the points of interest a simulation of the physical process of image capture, calibration and reconstruction. The simulation is based on the given tolerances for the system parameters—e.g. reference points.

[0032] The method used for precision estimation of a measured three-dimensional coordinate is based on Monte-Carlo simulation. The first step is to choose a three-dimensional point in an imaged space and calculate the point's theoretical position in the image planes. This point becomes a simulation point. Reference points are then chosen in the image planes and their positions are calculated. Tolerances for internal and external parameters of the system (or camera) are then acquired from a system database. A suitable amount of error to positions and parameters are then added to the simulation, the error being determined according to the acquired tolerances. The simulation point is then reconstructed based on its position in images obtained through the simulation and a three-dimensional coordinate is produced. The last step is to compare the original three-dimensional point with the three-dimensional coordinate that was produced by the reconstruction to obtain a precision estimation. Repeating the precision estimation with different amounts of error and averaging out the results can produce a better estimation.

[0033] In one variation of the invention, a structured light source is provided to project a multiplicity of target patterns. This is to facilitate the correlation of corresponding points in a multiplicity of captured images.

[0034] As the reconstruction phase is done, the system allows the user to perform different tasks of data editing and analyzing such as: modification of the data origin and orientation, re-sampling, comparison with other model, export in different formats etc.

[0035] It will be understood that numerous modifications thereto will appear to those skilled in the art. Accordingly, the above description and accompanying drawings should be taken as illustrative of the invention and not in a limiting sense. It will further be understood that it is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains and as may be applied to the essential features herein before set forth, and as follows in the scope of the appended claims.