US 6,542,249 B1Grant
Three-dimensional measurement method and apparatus
Issue Date:2003-04-01
•21 Claims
•16 Drawing Sheets
Abstract
The apparatus and method measure the three-dimensional surface shape of a surface without contact with the surface, and without any physical constraint on the device during measurement. The device is a range-sensor or scanner, and in one embodiment is a laser-camera sensor, which has a portable camera and multi-line light projector encased in a compact sensor head, and a computer. The apparatus provides three-dimensional coordinates in a single reference frame of points on the surface. The sensor head does not have to be physically attached to any mechanical positioning device such as a mechanical arm, rail, or translation or rotation stage, and its position in three-dimensional space does not have to be measured by any position-tracking sensor. This allows unrestricted motion of the sensor head during scanning, and therefore provides much greater access to surfaces which are immovable, or which have large dimensions or complex shape, and which are in confined spaces such as interior surfaces. It also permits measurement of a surface to be made by a continuous sweeping motion rather than in stages, and thus greatly simplifies the process of measurement. The apparatus can be hand-held, mounted on any moving device whose motion is unknown or not accurately known, or airborne. The apparatus and method also permit unknown and unmeasured movement of the object whose surface is to be measured, which may be simultaneous with the movement of the range-sensor head.
Metadata
Assignee
- The University of Western Ontario
Inventors
- Jonathan D. Kofman
- George K. Knopf
Application Information
Application Number:US 09/357,368
Filing Date:1999-07-20
Priority Date:1999-07-20
Art Unit:7
Classifications
IPC:
G01B 1124
Patent Drawings (16 sheets)
Description
Background of the Invention
[0002] 1. Field of the Invention
[0003] This invention relates to the measurement of the shape of an object or scene, and more particularly, to a method and apparatus to generate three-dimensional (3-D) coordinates of points of a surface, as well as a 3-D model of an object or scene, by a freely moving sensor whose position is not measured by any position tracking device, and which performs the measurement without contacting the object. The object and sensor are permitted free movement relative to one another, and therefore either the object or the sensor or both may be moving freely simultaneously.
[0004] 2. Description of the Prior Art
[0005] Range sensors or scanners are commonly used for non-contacting measurement of the surface geometry of an object or scene (multiple objects). Typically the range-sensor consists of a range-sensor head and a mechanical positioning device, as shown schematically in FIG. 1 (prior art), for the particular case where the range-sensor head consists of a laser source connected to a camera. Usually, the range-sensor head acquires profiles of 3-D coordinates of the surface being measured. In order to acquire sufficient data to describe an object surface, the range-sensor head must be translated or rotated relative to the object surface to acquire numerous profiles, which will later be integrated into a single object model. The object may also be translated or rotated relative to a stationary or moving range-sensor head. In all cases, the movement of the moving body or bodies, the range-sensor head, the object, or both, is carried out by mechanical positioning devices which move the range-sensor head or object, either by predefined increments or to unknown positions which are measured by a position sensor (not shown) on the positioning device, during the data acquisition. Accurate information of the incremental movement of the moving bodies is required in order to integrate all profiles collected from the same position of the range-sensor apparatus.
[0006] However, for large objects, such as car-body panels, engines, machines, or statues, whose dimensions exceed the working dimensions of the rotation or translation stages, or for immovable objects, or objects which are difficult or not desirable to manipulate, such as human bodies and archaeological artifacts, the method of moving the object relative to a stationary range-sensor head cannot be used. Furthermore, the scanning of such objects by moving the range-sensor head, cannot be carried out with the mechanical positioning device at a single position. The mechanical positioning devices are limited in their range of movement, and cannot move the range-sensor head completely around the object.
[0007] One approach to measure such objects is to first take one series of scans across the object (usually moving along only one translation axis), as shown in FIG. 2a(prior art) and then reposition the entire scanning apparatus (the mechanical positioning device which holds the range-sensor head) several times to new positions or viewpoints to acquire a series of profiles or a range-image at each viewpoint, as shown in FIG. 2b(prior art). The different range images or views, which consist of many profiles each and have large movement between viewpoints, can then be integrated into a single reference frame if there is sufficient overlap between range images, as shown in FIG. 3a-3c(prior art). However, one problem with this method is that estimates of the movement between viewpoints must be obtained before or during the data acquisition process, as these estimates are necessary to carry out the integration of the range-views. Another problem with this method is that very complex planning is needed to measure the full object surface and ensure that some overlap between views is obtained. Furthermore, for many environments, it is not possible to move translation or rotation devices to the site of the object, or to several different viewpoints around the object. The use of mechanical positioning devices is also not possible in measuring confined interior surfaces of complex shaped objects.
[0008] There have been several approaches to measure the surface geometry of objects without mechanical translation and rotary stages. These employ a hand-held range-sensor head, which would be continuously swept over the object surface to perform a scanning operation.
[0009] One device sold by 3D Scanners Ltd., of London, England, employs a range-sensor head attached to mechanical linkages or arms which are instrumented with rotary position sensors. The range-sensor head is hand-held and its motion is controlled by the user. The device allows a more continuous scanning and a more free access to the surface of an object than with translation and rotation stages by permitting simultaneous translations and rotations of the range-sensor head in all six degrees of freedom (three translations, three rotations). However, the device is highly limited by the lengths and possible orientations of the mechanical arms, and can therefore only accommodate small objects, or the apparatus requires repositioning and complex planning for large objects. The mechanical linkages and position sensors also add to the cost.
[0010] Another apparatus employs a non-mechanical magnetic-field tracking device to determine the position of the sensor-head, (see Fisher, R. B., Fitzgibbon, A., Gionis, A., Wright, M. and Eggert, D. “A Hand-Held Optical Surface Scanner for Environmental Modeling and Virtual Reality”, Virtual Reality World '96 Conference,pp. 1-16, Stuttgart, Germany, 1996; and New Zealand Patent 293713 (Polhemus)), and if necessary, the target object (see U.S. Pat. No. 5,886,775, “Noncontact digitizing imaging system”) during scanning. The device has eliminated the more bulky mechanical positioning and position sensing devices, and permits continuous sensor-head movement by hand to facilitate the range data acquisition process. However, because of the use of magnetic-field sensors, the system is restricted to non-ferrous environments. It is also costly to include the position tracking devices. Furthermore, in the case where the object is moving, contact must be made with the object to fix a position sensor to it. The magnetic-field position tracking system also has a limited range.
[0011] A third apparatus uses a non-mechanical optical position sensor to track the position of the range-sensing head during scanning (see U.S. Pat. No. Re. 035,816, “Method and apparatus for three-dimensional non-contact shape sensing”, and U.S. Pat. No. 5,198,877 of the same title). Three point light sources (referred to as pilot lights) located on the range-sensing head emit light one at a time, using strobing. The emitted light is received by three multiplexed light sensors (operated one at a time) to determine the position of the range-sensor head. A major drawback of the apparatus is that it requires unobstructed lines of sight between the pilot lights located on the range-sensor head, and the pilot-light sensors. This would be restrictive or prohibitive, in manipulation of the range-sensor head when scanning completely around objects, or inside confined cavities. Another drawback of this apparatus is the use of components used to generate and detect light points on a surface for the purposes of sensor-head position tracking only. These components are separate from the range-sensing light source and receiver and add considerable cost and complexity to the system.
[0012] A fourth device and method is the only one known to the inventors to involve a fully unconstrained sensor without positioning devices [see Hebert P., and Rioux, M. “Toward a hand-held laser range scanner: integrating observation-based motion compensation”, Proc Three-Dimensional Image Capture and Applications: SPIE-3313, pp. 2-13, 1998; and international patent application no. PCT/CA98/00324, publication no. WO 98/45817). The method requires two separate scans of the same region by the range-sensor head, to acquire two sets of profiles having an angle with respect to one another, and thus to obtain a collection of crossing profiles. Each single profile is considered to be acquired from a different unknown viewpoint. An algorithm adjusts each profile position and orientation separately to minimize the spacing between intersecting profiles. While the method eliminates the use of position sensors, the requirement of two separate scans at an angle to one another requires complex planning of the scanning path, and an awkward scanning process by a user when the device is hand-held. Alternatively, the two sets of profiles are acquired simultaneously by the range-sensor head, using two rotating mirrors which reflect two projected planes of light from two light sources. This adds considerable cost and complexity to the system.
Summary of the Invention
[0013] The invention has the following objects, although not all embodiments of the invention may achieve all objects of the invention.
[0014] An object of the present invention is to allow non-contact measurement of the surface information of an object or scene using a physically unconstrained range-sensor head, without any mechanical positioning devices or mechanical position-measurement devices. This would allow the device to be compact and would permit measurement of large, immovable objects of complex shape, interior surfaces and surfaces in confined spaces, which do not allow access of a range-sensor equipped with mechanical positioning devices.
[0015] Another object of the present invention is to provide a method of non-contact scanning an object to measure surface information without the need to fix the range-sensor head position at accurately known incremental locations, to measure the range-sensor head position at incremental locations, or to fix the entire scanning apparatus at planned positions. This would facilitate the acquisition of range data by reducing the planning of the scanning.
[0016] Another object of the present invention is to provide an apparatus and method to measure surface information of an object or scene without contacting the object, by a range-sensor without having to know the position of the range-sensor head or measure its position in 3-D space during the acquisition of surface information. This suggests another object of the invention, namely to provide an apparatus and method to measure surface information of an object or scene without any position tracking or measuring devices, whether physical, magnetic, optical based or otherwise, to track the position of the range-sensor head, which may be freely moving in space.
[0017] Another object of the present invention is to provide an apparatus and method to measure surface information using continuous motion rather than in stages, to eliminate complex planning of the operation by the user.
[0018] Scanning with the present invention could be accomplished by a hand-held range-sensor head, where the user would continuously sweep the range-sensor head over the object surface. Alternatively, a physically unconstrained range-sensor head could be airborne. An unconstrained range-sensor which would eliminate the problem of requiring knowledge of the relative range-sensor head movement with respect to the object, would also permit measurement of continuously moving objects with unknown positions.
[0019] A further object of the present invention is to provide an apparatus and method to measure the surface information of an object by a range-sensor head, where the object or range-sensor head may be freely moving in 3-D space or both the object and sensor head may be freely moving simultaneously in 3-D space at any time during the surface measurement, where surface measurement is made without contacting the surface by the range-sensor, and where the positions in 3-D space of the range-sensor head and surface are not known and not measured during the acquisition of surface information, except for the range measurement itself of the relative distance of the surface to the range sensor head by the range-sensor head.
[0020] Another object of the present invention is to provide an apparatus and method to measure surface information of an object or scene without contacting the object, using a simple system, having no moving parts, few or no signals to measure other than those provided by the optical sensor which measures range (the distance from the object to the range sensor), which in one embodiment is a laser-camera unit. In certain embodiments of the present invention, this suggests that no analog/digital (A/D) converter would be necessary to measure such signals.
[0021] Another object of the present invention is to provide an apparatus and method to measure surface information of an object or scene using a simple system having low cost.
[0022] Another object of the present invention is to provide an apparatus and method to measure surface information of an object or scene, where the entire apparatus is highly portable.
[0023] Another object of the present invention is to provide an apparatus and method to measure surface information of an object or scene, where there are no restrictions on the materials in the environment surrounding the sensor, or on the ambient lighting.
[0024] An optional object of the invention is to acquire other information about the surface of an object, such as color intensity, gray-level intensity, temperature, etc. In the invention, any sensor-head which acquires surface information may therefore be used, i.e. not just a sensor which measures only range.
[0025] In a preferred embodiment of the invention, the range-sensor head has a projector which simultaneously projects a few multiple lines, and a charged-couple-device (CCD) camera, coupled to a computer using a framegrabber. The computer, framegrabber and computer programs containing the processing algorithms are the only other primary elements of the preferred apparatus. Some other less preferable embodiments may omit the framegrabber. A computer monitor is an optional component used to interface with the user, and to display the object geometry in real time and after processing.
[0026] A few profiles of three-dimensional coordinates of points of the object surface are acquired at a single range-sensor head viewpoint. In a preferred embodiment of the invention, this is achieved by capturing in a single camera image the light contours or profiles produced by projecting the lines of light simultaneously onto the object surface. A mathematical geometric surface can be fit to the few profiles of a view, and interpolation on the surface between the profiles can be performed. This is done to permit matching of adjacent surfaces fit to points of adjacent views.
[0027] The apparatus and method provide an integration of all range data acquired by the range sensor head at different unknown and unmeasured viewpoints, into a single set of points in a single reference frame and into a single geometric model, by the matching of adjacent surfaces fit to points of adjacent views, and by applying the transformations determined during the matching process. The method of the invention applies equally to matching any subset of the acquired views, where the subset is a sequence of overlapping views which are not necessarily adjacent in the originally acquired sequence of views.
[0028] The method of this invention applies equally to acquisition of points in a more random fashion than by profiles. In this case, a geometric surface can still be fit to the points although they do not belong to one of a few profiles.
Brief Description of the Drawings
[0029] The invention will now be described in greater detail, with reference to the accompanying drawings, listed below:
[0030] FIG. 1: Schematic representation of a laser-camera range-sensor apparatus (prior art). A signal range profile is obtained for each position of the range-sensor head.
[0031] FIG. 2a:Range image comprised of a series of profiles acquired by translation of the range-sensor head (prior art). The mechanical positioning device remains at a single viewpoint (position and orientation) during this acquisition.
[0032] FIG. 2b:Conventional acquisition of range images from five different viewpoints to describe complete object surface geometry (prior art). The entire range-sensor apparatus, which includes the mechanical positioning device and range sensor head, is positioned at each of the five viewpoints. At each viewpoint, the range-sensor head is translated across one region of the object surface.
[0033] FIG. 3:
[0034] a) Matching of overlapping range-image views.
[0035] b) Alternative configuration of overlapping range views
[0036] c) Integration of multiple views into a single reference frame and model.
[0037] FIG. 4: Schematic representation of the unconstrained range-sensor.
[0038] FIG. 5: Pattern variation from the basic three-line projected line pattern for the unconstrained range-sensor: a) parallel line sets, b) square line sets, c-e) L-shaped line sets, f-h) cross pattern.
[0039] FIG. 6: Schematic representation of unconstrained hand-held range-sensor.
[0040] FIG. 7:
[0041] Range views (prior art):
[0042] a) Single profile per camera view,
[0043] b) Overlap of single profiles at only a single point in an adjacent-view pair,
[0044] c) No overlap of single profiles in an adjacent-view pair.
[0045] Range views (present invention):
[0046] d) 3-profile view.
[0047] e) Approximately 90% overlap in an adjacent-view pair using 3 profiles per view in translation.
[0048] f) Approximately 90% overlap in an adjacent-view pair using 3 profiles per view in rotation. The surface region between profiles in e) and f) is obtained by interpolation.
[0049] g) Continuous sequence of overlapping views. For clarity only about 50% overlap is shown.
[0050] h) Integration of a sequence of views into a single reference frame and model.
[0051] FIG. 8:
[0052] a) A range view containing three profiles, each having distinct left and right edges
[0053] b) Extraction of six profiles from the six edges of the three profiles for increased information and resolution of data
[0054] c) Approximately 90% surface overlap using 6 profiles
[0055] FIG. 9: Flowchart of processes for operation of the unconstrained range sensor.
[0056] FIG. 10: Calibration of multi-line range-sensor head.
[0057] a) acquisition of 3-D calibration object coordinates.
[0058] b) acquisition of the corresponding 2-D image coordinates.
[0059] FIG. 11: Mapping of a set of 3-D calibration object coordinates, acquired in a single plane, to a set of corresponding 2-D image coordinates. This is shown for one light plane edge or center, but is carried out for each light plane edge or center. Each column of points, in each set, is acquired at a different position of the calibration bar.
[0060] FIG. 12: A pair of successive range views showing V1profiles . . . within profile separation distance d from corresponding V0profiles—for a) translation and b) rotation.
[0061] FIG. 13: Surface patches, Sik, approximated to 6×6 grids in a pair of successive range views.
[0062] FIG. 14: Surface patches, Sik, approximated to 3×3 grids in a pair of successive range views.
[0063] FIG. 15: Construction of a spin image for point q:
[0064] a) sampling of q′ on a surface patch Sik.
[0065] b) computing the components of distance q′−q, α and β.
[0066] c) mapping α and β into a spin-image bin for each point q′.
[0067] FIG. 16: Flowchart of the iterative-closest-point algorithm adapted for small sensor displacement.
Detailed Description of the Invention
[0068] Description of the Apparatus
[0069] Referring to FIG. 4, the preferred embodiment of the invention uses an unmounted and unconstrained range-sensor head 1, having a multiple light-line projector 2 and a camera 3 mounted for example on a bracket 4 to capture images of the light lines projected onto an object 5. The camera is connected to a computer 6 which applies various algorithms to obtain the desired surface information from the images, as will be explained in detail later below.
[0070] The multiple light-line projector 2 projects several lines, for example two to four or more, but three are preferred for faster acquisition. The planes of light cast through space are ideally parallel, with an approximate separation of 2 mm (or a few mm), although a low tolerance on parallelism may be used to permit several degrees of divergence about any axis between planes. Other separation distances may be used in the invention. The use of highly parallel planes without significant divergence simplifies the calibration of the range-sensor head, but divergence can be accommodated for in the calibration process, as discussed later. The three-line projector is used as an example and is the preferred embodiment, but the invention is not limited to this number of lines. The three-line light projector may be three light sources mounted in a bracket to cast three approximately parallel planes of light, a single light source with a three-line optic lens, or another known projection method such as a single light spot projected using beam-splitters, prisms and rotating mirrors. The preferred device uses diode laser light projectors with visible light of 635-670 nm wavelength, although other types of projectors, white light and other wavelengths may also be used. When light of limited bandwidth is used, a bandpass optical filter is optionally mounted on the camera (preferably in the standard filter mount of the camera or using an adapter to fit it) to permit the camera to view only or mainly the laser light even in the presence of ambient white light. This allows use of the device without imposing restrictions on the environment lighting.
[0071] It is simpler and preferred to project the few lines of light simultaneously, and to capture them with a single camera image. Alternatively, the lines may be projected separately one at a time, and captured in separate camera images, which are later combined and treated as one image of a few profiles. This could have an advantage in distinguishing between lines during the acquisition, although it is not necessary.
[0072] Other means to project and acquire profiles of an object surface may be used. Instead of projecting a line, a single spot can be projected and swept along a line by rotating a mirror or prism at a fast rate to effectively trace out a line. The line can be captured by a single camera image. The method and device described herein applies to this “flying spot” method. It also applies to other grid projection methods such as those which project light through a grid to cast lines of light or shadow. Coding profiles by projecting lines of different width or projecting light through a grid to produce lines of different thickness may be used to aid in distinguishing lines.
[0073] The camera is preferably a CCD type. It has an acute angle with the approximately parallel planes of light, with a preferred angle range of about 20° to 60°. The separation distance between the camera and light projector depends on the angle, the depth of field, and field of view of the camera, but is such that the profiles can be viewed for the given angle. Optionally, a second camera may be used on the other side of the light projector to detect any part of the profile which may be occluded from the first camera. Variations to the proposed light projection pattern which may contribute to increased accuracy of the system include the three-line sets in a parallel, square, L-shape, or cross pattern (see FIG. 5).
[0074] The scanner head of the invention may use an ergonomically designed and cosmetically appealing handle as the camera-projector bracket, in the form of a telephone receiver (see FIG. 6), or pistol-like grip (not shown). These both can include encasement of the camera, optical components and light sources (see FIG. 6). The scanner head may be made without these ergonomic features using a more simple bracket, where ergonomics is not required and for use where the scanner head is attached to a stationary or moving device such as a robot or vehicle, which may be common in an industrial environment.
[0075] The scanner head may include a switch in the form of a finger or hand trigger, or button which will signal to the host computer when to start and stop acquiring data. An A/D (analog/digital) board in the host computer could be used in the case that a switch is used. However, alternatively this signaling to the host computer could be done by switches connected to the computer peripheral ports, or the signaling could be done directly by the computer peripherals, such as keyboard or mouse, and therefore eliminate the need for the A/D board.
[0076] An alternative means for starting and stopping the acquisition of point data without the use of an A/D board and separate signal detection, is to use a mechanical switch in the above physical forms to provide power to the light projector or projectors. With the camera continually sampling new images without saving them, and the host computer continually interpreting the images, the host computer can then begin and continue to acquire point data only when the image of the projected light is detected. As long as no image of the projected light is detected in a sampled camera image, the computer would only continue to check new images for the presence of the projected light image but not save images and not attempt to acquire point data.
[0077] During scanning, images of the few profiles are acquired simultaneously in a single camera image, or as stated above, they are acquired in separate images which are combined into one image.
[0078] The light projector and camera may obtain electric power through a cable and the camera may send images to the host computer or framegrabber via a cable. Alternatively, the light projector and camera may obtain electric power by one or more batteries attached to, or encased in the range sensor head. The camera may alternatively send images to the host computer or framegrabber by radio frequency transmission. When both of these cableless alternatives are used, the range sensor head would be fully detached from other devices, that is, there would not even be attachment to cables.
[0079] In one embodiment of the invention, the computer 6 is a laptop or notebook computer, to make the entire system highly portable.
[0080] The apparatus optionally includes a computer monitor as a user interface and to display the computed 3-D points and 3-D geometric model.
[0081] The method of using the apparatus to capture and process the images will now be described in detail.
[0082] Description of the Method
[0083] The approach employed by the invention is to capture a few profiles of 3-D coordinates of object surface points simultaneously (in one embodiment two to four profiles for example, but three are preferred) in each camera image or camera view and thus at each range-sensor head viewpoint, as shown in FIGS. 7d-7g,rather than capturing only a single profile per viewpoint as shown in FIGS. 7a-7c(prior art). The use of a few profiles per camera view, rather than one, permits fitting of a geometric surface to the points of the profiles of a view, and it subsequently permits interpolation of data between profiles within a view, thus providing additional information per view than the acquired points themselves, as well as the additional information in the additional profiles themselves compared to only one profile. A sequence of such range views is acquired during the scanning process, where each view contains several profiles, and each view has a surface fit to it. The different range views acquired at different viewpoints are then integrated into a single object model by matching adjacent overlapping geometric surfaces of the adjacent views as shown in FIGS. 7g-7h.This is done using the following two processes. The registration of a pair of range images or range views is the determination of the transformation between one view to bring it into the reference frame of the other view. The integration of the views, is the process of repeatedly applying the appropriate transformations between view pairs determined by the registration process, to ultimately obtain all range data of all views in a single reference frame and model. Algorithms to perform these processes are described later.
[0084] The use of a few profiles per camera view generates overlap of approximately 90% of the surface region of a view, as opposed to a single point or no overlap when using a single profile per camera view (compare FIGS. 7d-7gto FIGS. 7a-7c(prior art)). This overlap is essential to perform the matching of views for accurate registration and integration of adjacent range-image views. This is an advantage over other methods which produce a single profile per camera view.
[0085] An optional feature of the method is to extract the left and right edges of each profile of a view to double the amount of data provided by the three profiles (compare FIGS. 8ato 8b) and therefore increase the accuracy of the surface fitting and interpolation. While methods may exist to acquire 100 or more profiles in a single camera view by projecting a multi-line grid onto a surface, the acquisition of only a few profiles per camera view permits the fast processing necessary for a continuously moving camera, and minimization of redundant data. The key concept of the method is therefore in the acquisition of only a few (e.g. three) profiles per camera view, which is the appropriate trade-off to increase surface information per view, and increase overlap between view pairs (compared to conventional single-line scanning), while minimizing redundant data, data storage, and real-time processing during acquisition.
[0086] The process of performing a surface shape measurement can be accomplished by carrying out any relative displacement or motion between the range-sensor head and the surface to be measured, where essentially the range sensor sweeps over the surface to be measured. The relative sweeping motion between the surface and range-sensor head can be carried out, for example, by a user holding a hand-held sensor head, or using any other device to which the sensor head is attached, which in an industrial environment may be a robotic arm or vehicle, or even an airborne device. It should again be noted that any one of the surface or sensor head may be moving at any time, or both the surface and sensor head may move simultaneously at any time, during the acquisition of range data by the sensor-head. The movement may be made either purposely to carry out the scanning motion, or inadvertently during the scanning process. In all cases, the position and movement of the range-sensor head, and the position and movement of the surface, do not need to be measured, tracked or known during the acquisition of the surface information. The movement of the surface and range-sensor should generally be continuous, but may include pauses, reversing and changing direction. Movement in any combination of six degrees of freedom (three translations and three rotations) is permitted, although to effectively scan the surface, views should be acquired such that they cover the surface area of interest.
[0087] The computer 6 is programmed to apply various algorithms as follows, described in detail later below:
[0088] 1) calibrate the range-sensor head to determine the 3-D coordinate for any 2-D image point which represents a point on a profile (Algorithm 1);
[0089] 2) acquire the 2D images of the profiles and save the 2-D data of the profiles in minimal form for all edges of the profiles, and for all viewpoints during scanning (Algorithm 2);
[0090] 3) from the 2-D data, reconstruct the 3-D coordinates in the local frame of the range-sensor head of all points which comprise the profiles (Algorithm 3);
[0091] 4) register all views by computing the transformations between adjacent views (Algorithm 4); and
[0092] 5) integrate all views into a single reference frame (Algorithm 5).
[0093] FIG. 9 is a flowchart which summarizes the processes for operation of the unconstrained range sensor, and which indicates the function of the various algorithms other than Algorithm 1. The function of Algorithm 1 is not illustrated in FIG. 9, since it is not performed each time a surface measurement is made. It is intended to be only performed for calibration when manufacturing the device, or when recalibrating the system for any other purpose, such as changing the camera or light line parameters.
[0094] The complete set of algorithms, excluding the calibration which is intended to be factory-used only (although it could be done by the user), is run either automatically as one computer program without user intervention, or as separate algorithms, except for algorithms 4 and 5 which would normally be run as one program. The possibility of automating the complete surface measurement process provides an advantage over current systems which require user intervention to acquire, register and integrate views from different viewpoints.
[0095] A final stage in the process, i.e. once all views have been integrated into a single reference frame and saved as points in a single reference frame, is to display all points as a point cloud, as profiles, or as surfaces, which can be printed out, displayed on a computer monitor, or otherwise used, or generate a geometric model of the measured surface (the last box of FIG. 9), which can be printed out, displayed on a computer monitor, or otherwise used, e.g. for input to other display, manipulation or model modification software, or rapid-prototyping, machining, etc. This final step is done by commercially available software, as described at the end of the algorithms section below.
[0096] Detailed Description of the Algorithms
[0097] Algorithm 1
[0098] The calibration of the range-sensor head, for the preferred embodiment of a light-projector-camera head, is normally performed at the time of manufacturing, although it could be done at any time between measurement sessions, to recalibrate the sensor, as described above. The calibration essentially determines a mapping between a set of 2-D image coordinates corresponding to a set of known 3-D object coordinates. It is used to determine the relationship or mapping of any 2-D image coordinate to a 3-D object coordinate, which will ultimately be used during the 3-D reconstruction in Algorithm 3 described below. The process involves acquisition of 2-D image coordinates corresponding to known points in 3-D space that lie in the planes of the profiles or profile edges (see (1) Knopf, G. and Kofman, J. (1996). Neural Network Mapping of Image-to-Object Coordinates for 3D Shape Reconstruction, in Intelligent Robots and Computer Vision XV: Algorithms, Techniques, Active Vision and Materials Handling, D. P. Casasent, (Ed.), Proc. Soc. Photo-Opt. Instr. Eng. (SPIE) 2904, pp. 129-137; and (2) Range Sensor Calibration Using a Neural Network. Artificial Neural Networks in Engineering (ANNIE '98)—Smart Engineering System Design: Neural Networks, Fuzzy Logic, Evolutionary Programming, Data Mining and Rough Sets. ASME, pp.141-146). Interpolation to relate all image pixels (coordinates) to 3-D coordinates, is done using one of two methods: 1a) surface fitting through all calibration 3-D control points and separately through all 2-D image points, or 1b) neural network mapping. Other calibration methods which relate an image coordinate to a 3-D coordinate, either using the known geometry and/or optical parameters of the light plane and camera, or by any method which maps a set of 2-D image coordinates to a corresponding set of known 3-D object coordinates, may be used in the invention. Algorithm 1 and its associated process of calibration would be done typically by the factory and not by the end user, although it could be done by the user.
[0099] This calibration is performed for each light plane, or each edge of each light plane if edges are used. The mathematical mapping of 2-D to 3-D coordinates is performed separately for each edge or line to be calibrated. The acquisition of 2-D data points for the calibration is performed simultaneously for all edges or lines. An example of a method to acquire sets of 2-D image points corresponding to 3-D points of a planar grid in space, to calibrate the range-sensor head follows (see FIGS. 10 and 11).
[0100] A calibration object, a moveable vertical calibration bar with several horizontal lines scored on the surface at known fixed vertical intervals of, for example, a few millimeters, is translated horizontally by known increments in a plane parallel to the first plane of light of the multi-line projector, toward the light source, in the +X direction, (as shown in FIGS. 10aand 11) or away from the light sources, in the −X direction. The intersections of the projected light lines, which cast planes of light through space, and each horizontal marking form a bright spot on the calibration bar, and a bright blob in the image. These blobs are used to determine the 2-D image calibration points corresponding to the known 3-D calibration-object coordinates, as described below. Each horizontal marking will produce six 2-D image calibration points for all six 6 light plane edges or three 2-D image calibration points if the acquisition is being done simultaneously for three profiles.
[0101] The Y and X 3-D calibration-object coordinates are known from the known positions of the scored marks on the calibration bar, and the known increment of the bar in 3-D space, respectively. The Z coordinate is taken as zero for all 3-D points of the first light plane edge or center, if centers are used. For all 3-D points of the other edges (or centers), the distance from the edge (or center) of the light plane to the edge (or center) of the first light plane at the same score mark is physically measured, for example by micrometer, or by another camera system where the distance between pixels is related to the distance between edges (see FIG. 10a). This physically measured distance is the Z coordinate. In the case where the planes of light are highly parallel, the calibration is greatly simplified, as the known distance between planes and the width of the light planes can be used to determine the distance of any edge (or center) of a plane to the edge (or center) of the first plane. This distance is the Z coordinate.
[0102] At each incremental movement of the calibration bar, the image coordinates are acquired by the computer. The following is an example of how this is performed for each image coordinate, although other methods may be used. FIG. 10bshows a typical 2-D camera image for one location of the calibration bar. A blob is determined as a region where a contiguous group of pixels has a significantly higher gray-level or color intensity than points outside of this region, as shown in FIG. 10b.This blob region is found by sampling points across and down the image and checking the intensity level of each pixel sampled. The position, in 2-D image coordinates, of the centroid of the entire image blob is the image coordinate of the point represented by the blob, if calibrating the center of a light plane (three profiles from three planes of light). The centroid of the left and right edges of the blob are used as the image coordinates if calibrating for edges of light planes (six profiles from three planes of light). This approach provides for subpixel resolution during the calibration procedure.
[0103] A complete set of 2-D image calibration points corresponding to known 3-D object points is obtained by repeating the above process of incrementing the calibration bar through space, and measuring and recording 3-D coordinates while computing and saving the image coordinates at each incremental position. FIG. 11 shows a set of 3-D calibration object points acquired for a single plane or plane edge, and the corresponding image points. The set of image points shown in FIG. 11, is acquired from multiple positions of the calibration bar and the process of acquiring 2-D image coordinates from blobs, as described above. The set of points shown is therefore, not from a single camera image. Three such pairs of sets are actually acquired when using light plane centers, or six such pairs when using light plane edges; however, these other planes and their corresponding image points are not shown for clarity. Once a set of 2-D image calibration points corresponding to known 3-D object coordinates is collected for each light edge (or center), the mapping to relate the two sets is carried out for each pair of sets, (i.e. one mapping of a pair of sets for each light plane edge or center) as follows, using either Algorithm 1(a) or Algorithm 1(b).
[0104] Algorithm 1(a)
[0105] A low order surface is approximated to the image coordinates. This surface may be of any type but parametric surfaces, and in particular, Bezier and B-Spline are most common and most useful. A similar surface is approximated to the 3-D known coordinates, using standard surface approximation methods. Both surfaces will be planes as only planar coordinates were used for each set. The points of the 3-D coordinate set will lie in a rectangular grid formed from the known fixed vertical intervals on the calibration bar, and the known horizontal incremental displacements of the calibration bar. However, the points of the image coordinates, although coplanar, will not lie in a rectangular grid due to the perspective and camera distortion from the camera lens. The parametric coordinates of the two planar surfaces will be used to directly map the points of one set with the other. Interpolation between these calibration points at any parametric coordinate and therefore for any 2-D image point within the calibrated region, can be done during the reconstruction of 3-D coordinates from 2-D image coordinates by Algorithm 3.
[0106] Algorithm 1(b)
[0107] The mapping method based on artificial neural networks is described in detail in Knopf, G. and Kofinan, J. “Neural Network Mapping of Image-to-Object Coordinates for 3D Shape Reconstruction”, SPIE 2904, pp. 129-137, 1996; and in Knopf, G. K. and Kofman J. “Range Sensor Calibration Using a Neural Network”, ANNIE '98—Smart Engineering System Design: Neural Networks, Fuzzy Logic, Evolutionary Programming, Data Mining and Rough Sets. ASME, pp.141-146, 1998, but is summarized here. One of several types of artificial neural networks, can be used to perform the mapping of 2-D image calibration points to the known 3-D object coordinates. System input-output pairs for training the network are obtained from these two corresponding sets of data. A transformation between the two coordinate systems can be determined by the neural network. In particular, a Bernstein Basis Function Network has been used. Basis function networks are two-layer computing structures whose outputs are a linear combination of the basis (or kernel) functions computed by the neural units in the first layer. In terms of the BBF network, the individual neurons in the first layer transform the parametric inputs (uxyz, wxyz), into nonlinear outputs using Bernstein polynomials of degree n and m given by:
[0108] respectively, where the parametric inputs (uxyz, wxyz) are derived from the image coordinates (i, j), by any normalization procedure such as, for example,
[0109] where A and B are the number of pixel columns and rows in the 2-D image, respectively.
[0110] The weighted summation of the Bernstein polynomials generates a nonlinear mapping of the observed inputs-to-outputs and is mathematically given by:
[0111] where λxyz(uxyz, wxyz) are the 3-D coordinate outputs, 0≦uxyz≦1, 0≦wxyz≦1, and vector ψij=[ψxij, ψyij, ψzij]Tis the (ith, jth) control point.
[0112] The control points for the nonlinear mapping correspond to the weight vectors for the neurons in the output layer. The weights are trained using the following algorithm:
[0113] Step 0. Set the degree of the Bernstein polynomials for the functional mapping; that is, select n and m.
[0114] Step 1. Initialize the control points, or weights, ψij to small positive and negative random values. Set the learning rate parameter α to a small value; i.e. 0.1≦α≦1.0.
[0115] Step 2. While the stopping condition is false, do Steps 3-8.
[0116] Step 3. For each randomly selected training pair (Uxyz, Xuw), where:
[0117] Uxyz=[uxyz, wxyz]T, and Xuw=[xuw, yuw, zuw]T, do Steps 4-7.
[0118] Step 4. Compute the output of the basis function neurons in layer 1 using Equation 1a and 1b.
[0119] Step 5. Determine the response of the neurons in the output layer using Equation 2.
[0120] Step 6. Calculate the error for each output neuron in layer 2 by
[0121] e=Xuw−λxyz(uxyz, wxyz) (3a)
[0122] ex=xuw−x(uxyz, wxyz) (3b)
[0123] ey=yuw−y(uxyz, wxyz) (3c)
[0124] ez=zuw−z(uxyz, wxyz) (3d)
[0125] where the error vector is given by e=[ex, ey, ez]T.
[0126] Step 7. Update the weights according to:
[0127] ψij(κ+1)=ψij(κ)+α(eλxyz(uxyz, wxyz)+(ψij(κ)−ψij(κ−1))) (4)
[0128] where κ is discrete time.
[0129] Step 8. Test for stopping condition:
[0130] If the largest error that occurred in Step 3 is smaller than a specified tolerance, then STOP; otherwise continue.
[0131] After convergence, the weights of the neural network correspond to the control points of a polygon net that will generate a bi-parametric Bezier function that best approximates the input-output mapping of the training data set.
[0132] Once the relationship between the image and object coordinates has been established, the mapping function or transformation which relates the two sets is used in Algorithm 3 to compute a 3-D coordinate for any pixel or 2-D image coordinate sampled during a real object measurement. The generalization capability of the neural network enables intermediate points not present in the set of calibration points to be accurately interpolated.
[0133] Algorithm 2
[0134] Algorithm 2 involves the acquisition of surface profiles by detecting the profiles in the 2-D images in real-time during the scan and storing the necessary useful information. In Algorithm 2, the extraction of coordinates from two edges of a profile to be used directly as separate surface data is not known to have been done previously. Edges have been previously extracted and averaged to determine the centerline of a profile. Algorithm 2 detects the edges of all profiles, by scanning across the camera image and detecting abrupt changes in pixel intensity. Only the 2-D image coordinates which correspond to edge points are saved. No information of the approximate remaining 500×500 or 250,000 pixels is saved, thus reducing the stored image information considerably. In scanning across the camera image, the algorithm counts the profiles being detected to identify and label each measured point with the proper profile.
[0135] Alternatively, if the edges are not used to double the available profile information as discussed above, the coordinates of the left and right edges at each point along a profiles are averaged to obtain the center of the profile. Alternatively for each position along the profile, the coordinate position corresponding to the peak intensity across the width of the profile may be used instead of the center. Other methods to determine the appropriate 2-D coordinate from points across the width of a profile may be used in the invention.
[0136] An optional function may be included in the algorithm to smooth the raw profile data if it is has noise. A simple average of neighboring points, or spline or polynomial piecewise curve fitting may be used. This may be done on the 2-D image profile, or on the 3-D curve once reconstructed.
[0137] In the preferred embodiment, gray-level (monochrome) intensities are used. Alternatively, color information, that is the red, green and blue intensities, may be obtained for each pixel of a profile point, using a color camera. Furthermore, the gray-level intensity or color intensities may be obtained without the projected light being on by turning the light on and off, and sampling the images for both cases. In this way, the gray-level or color intensity information of the object being measured can be acquired and stored without the effect of the projected light, and therefore with the original appearance of the object in the ambient lighting. This light intensity information can be acquired to correspond directly to the 2-D image coordinate of each acquired point, and furthermore, to the 3-D coordinate which is computed, for each acquired point. Furthermore, the gray-level or color intensity information can be acquired at image pixel locations between the projected-light profile images, to be saved and used in the final generation of the 3-D surface of the measured object. One option in acquiring surface information, is to use infrared cameras to will give intensity information indicative of surface temperature.
[0138] Algorithm 3
[0139] Algorithm 3 reconstructs the 3-D coordinate data from the 2-D image points of the profiles, using the functions obtained as a result of the calibration process of Algorithm 1. The process involves either using Algorithm 3a or 3b.
[0140] Algorithm 3a
[0141] Given X(u,w)=F1(u, w), where F1is the function describing the surface, S1, fit to the 3-D calibration points X=[x y z]Tand given J(u,w)=F2(u, w), where F2is the function describing the surface, S2, fit to the 2-D image points J=[i j]Tboth determined during the calibration Algorithm 1a. The 3-D coordinates of any point on S1which correspond to any new measured 2-D image point on S2are determined by first searching and locating on S2the coordinates J=[i j]T, and thus determining the parametric coordinates (u,w) which corresponding to the 2-D image coordinate J. From the parametric coordinates (u,w), the coordinates, x, y, z are computed using X(u,w)=F1(u, w) above.
[0142] Algorithm 3b
[0143] The 3-D coordinates of any point Xxyz(uxyz, wxyz) are obtained for any image coordinates (i, j) using Equation 2 in Algorithm 1b above, where the control point weights, ψij, are those determined at the completion of the training of the neural network, and where the image coordinates (i,j) are normalized as described in Algorithm 1b, to obtain the parametric coordinates (uxyz, wxyz) as input.
[0144] Algorithm 4
[0145] Algorithm 4 involves one of three techniques developed to register two adjacent range-image views. All three techniques take advantage of the small sensor-head movement between views, involve surface fitting to the points of the profiles of each view, and use local searches around points to match points of the two views.
[0146] Algorithm 4a matches points of one view with the closest points interpolated on the surface of the other view, and iteratively minimizes the distance between matched points. Algorithm 4b finds the best match of curvature features at points of one view with curvature features at points interpolated on the surface of the other view, without iteration. Algorithm 4c finds the best match of spin-image point features at points of one view with spin-image point features at points interpolated on the surface of the other view, without iteration. These will be described in more detail below.
[0147] The invention should not be limited to the view registration and integration methods of Algorithm 4 described, but should include any known technique for registration of conventional range images, adapted for a few profiles per view and small camera motion between viewpoints. Some of the known techniques known in the art include those based on determining point-to-point correspondences using other features and minimization of point-to-surface distances [see Chen Y. and Medioni G., “Object modeling by registration of multiple range images”, CVGIP Image and Vision Computing,10, pp. 145-155, 1992].
[0148] The following pertains to all three algorithms, Algorithms 4a, 4b and 4c. The algorithms are carried out to register two adjacent views, which essentially involves determination of the transformation which aligns one view with the other. The following describes the matching of two adjacent views composed of 6 profiles each, regardless of how the profiles were actually obtained. It is important to note that this is an example only. As stated earlier, the present invention applies to views composed of any number of profiles two or greater, but three or greater is preferred, and as few as possible are preferred. It is also important to note that the method of the invention and algorithms apply to views where all profiles are acquired directly as the profiles formed on the object surface when light is used, using the center of the profile, the peak intensity of the profile or similar method, to obtain for example, 3 profiles acquired from 3 light profiles on the surface; or using profiles acquired from the 2 edges of each profile on the surface, to obtain for example, 6 profiles from the edges of 3 profiles on the surface, or even 2 profiles acquired from the left and right edges of a single projected line, where a surface will be fit to the 2 profiles; or using a combination of the edges and center or the edges and peak intensity, to obtain for example, 3 profiles acquired from the left and right edges and the center of a single projected line, or from the left and right edges and the peak intensity of a single projected line.
[0149] FIG. 12 shows a pair of successive view comprised of 6 profiles each. Let V0and V1be a reference and a new view, respectively, of a pair of successive overlapping views. Let Pjk, j=0, 1, 2, 3 . . . 5 be the jthprofile of the kthview Vk, k=0, 1 (FIG. 12). It is assumed that the spatio-temporal density between views V0and V1is sufficiently high that the inner four profiles P11, P21, P31, and P41, of V1are contained in the surface described by the points of V0, and all points of each of these inner profiles Pj1, j=1 to 4, lie within ±d of Pj0, where d is the spacing between profiles within an image. The distance which defines the expected maximum movement between two successive views could alternatively be any distance which still allows sufficient overlap between views. Although an additional fifth profile of V1may also overlap V0(P01for the case shown in FIG. 12a), it is assumed that no information of the direction of translation from one viewpoint to the other is available a priori to determine which additional profile overlaps V0. Furthermore, only part of the profile (P01) may overlap as shown in FIG. 12b.From these assumptions, point-to-point correspondence or matching between V1and V0is therefore based on matching only points of the inner four profiles of V1with V0. For the case of more profiles, again one boundary profile on each side of the view V1would be omitted from the matching process, or alternatively two profiles on each side of the view V1could be omitted. If information of the direction of scanning is more certain in some application, then the omitted profile or profiles could be from one side only, on the leading side in the direction of scanning.
[0150] Surface Fitting
[0151] Due to the small movement between successive views and the small separation between profiles within a view, each profile of a view is considered to be parametrically matched to a corresponding profile of the previous view, and to have the same number and indexing of points. To enable this, extra points at the ends of some profiles can be omitted from the correspondence analysis but retained for the final building of the model. As well, uniform axial or parametric point resampling may be used.
[0152] The fitting of a surface to the points of the acquired profiles can be done in numerous ways known in the art. Parametric surfaces such as Bezier and B-Spline surfaces are most common although others may be used. A single surface patch may be fit to all points of a view, or alternatively, separate surface patches may be used fit to any number of a subset of points of the view. Separate surface patches may be fit to have common boundaries or they may be fit to overlap. Surfaces or surface patches may be fit to interpolate all profile points by using a control-point net having the same number of points as the grid of points being fit, or the surface may approximate the points by using a control-point net which has fewer number of control points than the grid of points being fit. Furthermore, points may be skipped when forming a surface patch. For example, every second point in a profile could be used to fit a surface patch. The following is therefore an example only and does not restrict the method of surface fitting which may be employed in the present invention. The example uses separate surface patches, which overlap, although they are fit and used one at a time as described below. As well, various matching techniques may be employed in the present algorithm and invention, and the following matching technique is only an example.
[0153] FIG. 13 shows the fitting of surface patches, Sik, approximated to 6×6 grids in a pair of successive range views. Intersecting views are shown in local frames not intersecting for clarity.
[0154] Let qijkbe the ithpoint of profile Pjk, i=0, 1, . . . n−1 for n points per profile. In FIG. 13, nine points are shown to be used for each profile, although typically the number of points used would be as high as 100 or 500 points, and furthermore, no restriction is made on the number of points used per profile. For both views, V0and V1, and at each point index i, where 2≦i≦n−4, a bicubic Bezier surface patch, Sik, is approximated to the 36 points on a 6×6 grid defined by corner grid points qi−2 0k, qi+3 0k, qi+3 5k, and qi−2 5k. The surface patches are fit using a 4×4 control-point polygonal net and any standard parameterization method such as chord length or centripetal parameterization, (see (1) D. F. Rogers and N. G. Fog, “Constrained B-spline curve and surface fitting”, Computer-Aided Design21, pp. 641-648, 1989; and (2) B. Sarkar and C. H. Menq, “Parameter optimization in approximating curves and surfaces to measurement data”, Computer-Aided Geometric Design8, pp. 267-290, 1991), or any variation of these methods such as using polynomial parameterization. Any other parameterization method may be used in the invention. As only the four inner profiles of V1are used for matching, for each surface patch Si1of V1, a match is performed only for the four points qij1, 1≦j≦4, belonging to the four inner profiles. No points other than the four described are used for matching. The next row of points qi+1 j1, 1≦j≦4 are matched to V0using the next surface patch Si+1 kof each view, which is fit to the grid bounded by corner points qi−1 0k, qi+4 0k, qi+4 5k, and qi−1 5k. In this manner, points selected for matching are consistently used from the same nominal parametric position in a patch. This assists in making the local regions surrounding points more identical in shape when comparing point features from V1and V0. The restriction to match only 4 points for each patch is not necessary for this algorithm and invention. Alternatively, more points of V1corresponding to the original acquired range points could be used for each patch. The computed result of any matching process which matches points of V1with points of view V0could be averaged using the computations based on more than one patch. Furthermore, any number of points sampled on the V1between original grid points could be used.
[0155] FIG. 14 shows an alternative method, as an example, using a pair of successive range views of three profiles per view. Intersecting views are again shown in local frames not intersecting for clarity. The method uses surface patches, Sik, fit to a 3×3 grid, and only a single point per patch, qi 1k, is used for matching.
[0156] The following relates to Algorithms 4b and 4c which involve determining point-to-point correspondences between V1and V0based on matching point features. Algorithm 4a, which is based on an iterative technique, is described later.
[0157] Alorithms 4b and 4c
[0158] View Matching by Point-to-point Correspondence (Matching)
[0159] To perform a point match between views, a point feature (not shown) in the form of a 2-D curvature array is first computed at point qij1of V1. For Algorithm 4b this is a surface curvature feature, and for Algorithm 4c, this is a localized spin-image feature. The computation of these features is described later. As stated earlier, other features could be used. A local search for a matching point feature in the corresponding surface patch of V0is carried out by computing features at points sampled in a region surrounding the parametrically similar point, having the same profile and point indices, qi j0of V0. This is shown in FIG. 13 for the matching of qi21to a point in the region surrounding qi20. Correspondence is based on the closest match between point features. This correspondence process is performed for the four points (qij1, 1≦j≦4) of a patch and is repeated for all patches. As described above, alternatively, this could be performed for more points than the four shown. Also, the use of a 6×6 grid requires that the first two and last three end points of a profile are excluded from the correspondence process although these points are included in the final registration.
[0160] The complete correspondence algorithm incorporating the point features is summarized below:
[0161] For all i, 2≦i≦n−4:
[0162] 1. Fit a surface patch Sikto the 36 points bounded by qi−2 0k, qi+3 0k, qi+3 5k, and qi−2 5k, for k=1, 2 (V0and V1).
[0163] 2. For all j, 1≦j≦4:
[0164] a) Compute a point feature Ωij1at qij1on surface patch Si1.
[0165] b) Compute point features at points sampled on surface patch Si0in the region surrounding qij0as follows:
[0166] For all u, u(qij0)−Δu≦u≦u(qij0)+Δu (increment by δu):
[0167] For all w, w(qij0)−Δw≦w≦w(qij0)+Δw (increment by δw):
[0168] Compute a point feature Ω(u, w) at q(u, w) on surface patch Si0.
[0169] where ε is a predefined threshold for the match of Ω, and
[0170] where ξbkis the value in bin b of a 2-D curvature array (explained later) and in the 2-D spin image array (explained later), for Algorithms 4b and 4c, respectively, for an array computed in view Vk, where k refers to the view, and B is the number of bins or array coordinates. For example, an n×m array will have B=n×m bins or array coordinates. RMSD thus represents the root mean square of the residual differences between bin scores between the two 2-D arrays, one in each of the two different views, where the bins having the same 2-D array coordinates are used to compute the differences, and the evaluation is over all bins.
[0171] Then: The point at which the minimum (closest match) occurred, q(u, w) or q(x, y, z), is the match point q0ij1(u, w) or q0ij1(x, y, z) corresponding to point q1ij1, where the superscript f in qfif1has been added to denote the frame in which the point is determined.
[0172] Else: Label point q1ij1as having no (reliable) match in V0.
[0173] Once corresponding point sets are determined between V1and V0, determination of the transformation between the two sets of corresponding points in two coordinate frames, the absolute orientation problem, can be solved by one of several existing methods which have been reviewed by Sabata and Aggarwal, [see B. Sabata and J. K. Aggarwal, “Estimation of motion from a pair of range images: a review”, CVGIP Image Understanding, 54, pp. 309-324, 1991]. The method based on quaternions, presented in detail by Horn, [see B. K. P. Horn, “Closed-form solution of absolute orientation using unit quaternions”, J. Opt. Soc. Am. A-4, pp. 629-642, 1987], is a preferred method, although other variations common in the art to remove outliers, for example those based on least median squares, may be used. A summary related to the computation of transformations between any two data sets, and the application of a transformation to a data set, is given at the end of the algorithms section.
[0174] The transformation from V1to V0is determined only using point pairs for which a successful point correspondence is established. Points labeled as having no reliable match are omitted.
[0175] Let Q11={qij11(x,y,z)}, the original point set of range image V1; Q1C1⊂Q11, the subset of points for which a reliable correspondence has been obtained; and Q1C0⊂{qij10, (x, y, z)}, the set of points belonging to V1, computed in the V0frame, and corresponding to Q1C1as determined by the matching process above. While the correspondence between Q1C0and Q1C1, as determined by the above technique, may not be exact, the transformation, T, which best registers the corresponding point sets, Q1C0and Q1C1, in a least squared sense, can be determined by solving the absolute orientation problem. As stated, the method based on quaternions, presented by Horn [see above reference], is a preferred method. Once the transformation is determined, the entire original point set of range view V1, Q11, that includes profiles P01and P51and any profile endpoints which were excluded from the matching process, can be transformed to the reference frame V0, by applying the transformation, T, as follows: Q10=T(Q11), where Q10is the new transformed set of points of V1transformed to the reference frame V0.
[0176] Point Features for Algorithms 4b and 4c
[0177] Algorithm 4b uses a point feature which is itself a 2-D array of curvature parameters computed at many points uniformly and parametrically sampled in the local neighborhood of the point being analyzed, and centered on the point. The parameters are derived from known surface curvature parameters presented in detail in P. J. Besl and R. C. Jain, “Invariant surface characteristics for 3D object recognition in range images”, Comput. Vision Graphics Image Process. CVGIP-33, pp. 33-80, 1986; and in J. Kofman, and G. K. Knopf, “Registration and Integration of Narrow and Spatiotemporally-Dense Range Views”, in Vision Geometry VII, R. A. Melter et al. (Eds.), SPIE 3454, pp. 99-109, 1998. Any other curvature parameters may be used in this algorithm.
[0178] Algorithm 4c uses a spin-image point feature which is a 2-D array called a spin image, computed at a point of a surface. Local spin maps, adapted from a method which uses global spin maps [see Johnson, E. and Hebert, M. “Surface registration by matching oriented points”, Proc IEEE Int Conf Recent Advances3D Digital Imaging and Modeling,pp. 121-128, 1997] are used. Details of the creation of the local spin maps are described in Kofman, J. and Knopf, G. K. “Point correspondences between successive range views using localized spin images”, SPIE 3837, 1999 (to be published)], and are summarized below:
[0179] Points q′ are sampled over the entire patch and the distance components perpendicular and parallel to the surface normal at q, α and β, respectively, are computed for each q′ sampled
[0180] β={circumflex over (n)}·(q′−q) (6a)
[0181] α={square root over (∥q′−q∥2−β2)}, (6b)
[0182] where {circumflex over (n)} is the unit normal at q, as shown in FIG. 15.
[0183] The distance α and the signed distance β, for each q′ sampled, are mapped into the spin image at q by incrementing the count at the bin coordinate (α,β) for each q′ having these coordinates in the q frame. Thus points at the same β and radius α from q will increment the value in the same bin. In the localized spin image adaptation, points q′ are sampled only locally over the surface patch Sik. To use as much of the spin image as possible without leaving many unfilled bins, α and β are scaled and shifted based on their minimum and maximum values over the range view. Alternatively, this scaling is made more adaptable by using a different scale factor for each of the four points to be matched within a patch. Non-integral values of α and β are interpolated to allow bin sharing using the nearest four bins.
[0184] Algorithm 4a
[0185] Algorithm 4a uses the surface or surface patch fitting described above as applicable below. FIG. 16 summarizes the algorithm in a flowchart. The algorithm is also summarized below. Essentially, the algorithm computes a transformation between two adjacent views that will minimize the distances between points on one surface, and the closest points to them found in the interpolated region on the other view surface used as a reference. The points of the first surface may be points sampled on the surface at the location of the profile points or grid points; points sampled between grid points or profile points by interpolation; a combination of profile points and interpolated points; or the original profile points without using the surface fit. The points of the second surface are interpolated on the surface.
[0186] This algorithm is based on the iterative closest point (ICP) method [see Besl P. J. and. McKay N. D., “A method for registration of 3-D shapes”, IEEE Trans. Pattern Anal. Machine Intell. PAMI-14, pp. 239-256, 1992.], but adapted specially here using local rather than global searches for corresponding close points on the second surface, because of the small range-sensor head movement between views. In the algorithm below, the computation of the transformation between two sets of corresponding points in two coordinate frames, is computed, as described in Algorithms 4a and 4b, by any one of several existing methods known in the art [see B. Sabata and J. K. Aggarwal, “Estimation of motion from a pair of range images: a review”, CVGIP Image Understanding, 54, pp. 309-324, 1991]. The method based on quaternions, presented in detail by Horn, [see B. K. P. Horn, “Closed-form solution of absolute orientation using unit quaternions”, J. Opt. Soc. Am. A-4, pp. 629-642, 1987], is a preferred method, although other variations common in the art to remove outliers, for example, those based on least median squares, may be used. See the summary related to the computation of transformations between any two data sets, and the application of a transformation to a data set, given at the end of the algorithms section.
[0187] The registration technique of Algorithm 4a is now summarized. Details of the local search for a closest point are described afterwards:
[0188] 1. Create an initial point set X1from points sampled from:
[0189] i) the V1surface at the location of the grid points (profile points); or
[0190] ii) points sampled from the V1surface at the grid points and between grid points; or
[0191] iii) the profile points of V1
[0192] The above three methods represent variations of Step 1.
[0193] 2. Apply an initial transformation to X1to get a first set of transformed X1: X10The initial transformation matrix is the identity matrix I (no transformation). In this way, no assumption or estimate of the scanner head movement is necessary. Alternatively, a transformation representing a small movement expected during the scanning procedure may be used, but this is not necessary.
[0194] 3. Compute the closest points X0on S0to the currently transformed points X10
[0195] 4. Compute the distances, Di, between the current closest points, X0and the current transformed points, X10, for all points: Di=∥x1i0−x0i∥, where x1i0∈X10and X0i∈X0, for i=1,2 . . . N; and N is the number of points in X0and in X10.
[0196] 5. Compute the mean square of these distances, Di, over all points of X0and X10:
[0197] 6.
[0198] If: (e<ε, where ε is a predefined threshold, OR |e−eprevious|<Δ, where Δ is a predefined threshold):
[0199] Then:
[0200] a) The current transformation is the final transformation to transform V1to V0.
[0201] b) Stop algorithm.
[0202] Else:
[0203] i) Compute the transformation that best registers X1to the current closest point set X0.
[0204] ii) Apply the new transformation computed in (i) to X1to get a new set of currently transformed points, X10.
[0205] iii) Repeat from Step 3.
[0206] In the above algorithm, the sampling of surfaces S0and S1to obtain point sets X0and X1may use any of the possible methods discussed in Algorithms 4b and 4c, and furthermore is not be restricted to these methods. For example, four points of the interior profiles could be sampled for each patch, where the next four points of a profile are sampled from the next patch. Alternatively, for example, more points could be sampled from any one patch. Furthermore, any point may obtain its coordinates by sampling more than one patch (at the same profile and point index) and averaging their coordinate values.
[0207] Search for a Closest Point
[0208] The search for a closest point in V0to a point in V1is now described. Because of the small movement between views, the search uses sampling of a very small region surrounding a starting point on the V0surface, which is parametrically matched (of the same point and profile indices) to the point in V1for which a close point is sought. This is similar to the method of search for correspondences in Algorithms 4b and 4c. Furthermore a coarse-to-fine and narrowing search is done to compute closest points. For example, to perform a search for a closest point in V0to a point in V1, a small grid of points which define a search region surrounding the starting point in V0is first sampled on V0. Nine points have been used effectively, although the method is not restricted to this size of a grid. The closest point at this step is determined by the minimum Euclidean distance over all sampled points of the grid, between the sampled point and the point of V1. Where the current closest point to the point in V1is found, another set of points surrounding that close point are sampled, this time with the spacing between the sampled points reduced, and a new closest point is found. This is repeated until the region of search is below a predefined size. The final closest point in V0for the point in V1is then determined as the last close point found. The preceding description summarizes the method to determine the closest point in V0to a single point in V1, which in step 4 above is referred to as x1i0. In step 4, the entire procedure just described would be carried out for all points x1i0εX10i=1,2 . . . N. The preceding procedure is an example of several possible variations which may be used in the invention, to search for a close point, where the search is performed locally around a starting point and where the amount of computation is to be minimal.
[0209] Alternatives to Algorithm 4a, in addition to those already discussed, may be used in the invention. For example, for the stopping criterion in Step 5, e may be based on the root mean square sum of differences or the sum of squares differences, rather than the mean square sum of differences.
[0210] Algorithms 4a, 4b and 4c described above, are methods to register two adjacent views. In the method of the invention, the Algorithm 4 which is used, (4a, 4b or 4c) is repeated for all adjacent view pairs of a sequence of views to obtain a transformation between each pair of views, to register each view with the previous one. Algorithm 5 integrates all views into a single view by applying the appropriate transformation to all views as discussed below.
[0211] One alternative to Algorithm 4a involves the registration of all views simultaneously. For Algorithm 4a, the registration transformation of all pairs of adjacent views are computed simultaneously. Given V0and V1, the reference view and new view, respectively, of any pair of views; at each iteration of the Algorithm 4a, the reference view V0of pair l, is transformed or updated by applying the current transformation of that view by considering view V0of pair l to be the new view V1of the previous pair l−1. The process is carried out using all view pair registrations, and for each iteration.
[0212] Other alternatives may include computation of the registration transformation of non-adjacent pairs, for example, Vkwith Vk−2, which may overlap.
[0213] It should be noted that the method of Algorithm 4, and the invention in general, differ greatly from the methods employed by Rioux and Hebert, [see international patent application no. PCT/CA98/00324, publication no. WO 98/45817]. In the present invention, the views which are sets of profiles do not have to cross one another, and in fact, preferably should not cross as shown in FIG. 12a.On the other hand, the present invention does accommodate rotation of views which will result in crossing profiles, if rotation does happen to occur by a less than ideal scan. In the invention noted above by Rioux and Hebert, crossing of profiles is required. Furthermore, in their invention the position and orientation of each profile are modified individually. In the present invention, all profiles within a view or set are treated as a rigid entity, and transformations in space are applied to entire views, and thus to several profiles identically. Other differences not noted here exist.
[0214] Algorithm 5
[0215] Algorithm 5 integrates a sequence of M views Vk, where k=0, 1, . . . M−1 into a single reference frame by applying transformations to all view pairs sequentially to ultimately bring all views into a single reference frame. Firstly, the transformation TM−2which brings the last view VM−1into the reference frame of VM−2is applied to VM−1. As described previously, this transformation is applied to all profiles and all points of the view, even if they had been omitted earlier in the correspondence process, either because the points were among the few profile endpoints, or if the points did not produce a reliable correspondence between adjacent views. The augmented V−2, V′M−2, includes the original points of VM−2and those from TM−2(VM−1). This process is repeated for each new augmented view in reverse sequence until all views are transformed into an augmented V0, V′0which includes all profile points of all views.
[0216] Summary re Algorithms
[0217] Optional additional algorithms include a refinement of the integration of all views by repeating the registration and integration in the reverse direction and averaging coordinate values of the forward and reverse processes; and smoothing point data by averaging coordinates of neighboring points.
[0218] As stated earlier, a final geometric model can be constructed from the final augmented view which includes all points from all views, using commercially available software, such as, for example, Geometric Wrap™ produced by Raindrop Geomagic, Inc., P.O. Box 12216, Research Triangle Park, N.C. 27709, USA, (presently at http://www.geomagic.com/index.html on the Internet). This software automatically converts point cloud data into highly accurate 3-D computer models. The model can be displayed on the host computer monitor, printed, or otherwise used, such as for example: input to other display, manipulation or model modification software, or for machining or rapid-prototyping. Alternatively, the 3-D points may be saved in VRML format and displayed, for example, as a cloud of points, as profiles, or as surfaces, using Cosmo™ produced by Silicon Graphics, Inc., Mountain View, Calif., U.S.A. The complete set of end-user algorithms, excluding only the factory-used calibration, for profile acquisition, 3-D reconstruction, registration and view sequence integration (Algorithms 2-5), would be run automatically as one computer program. This represents an advantage over current systems which require user intervention to acquire, register and integrate views taken from different apparatus viewpoints, as discussed earlier. Furthermore, this would permit automation of the complete surface measurement process, and use of the surface data in an automated application.
[0219] Transformations Between Two Data Sets
[0220] The following discussion applies to a set of points or a single point. To simplify point set matrices, a single point is used below.
[0221] A point (or as set of points) can be transformed to another coordinate frame as follows:
[0222] r0=Rr1+τ (8)
[0223] where
[0224] r0is the new transformed point, r1=[x y z]Tis the point before transformation;
[0225] R is the rotation matrix:
[0226] where c and s, are abbreviations for cosine and sine, respectively; λ, β, α, are the rotation angles about the x, y and z axes respectively;
[0227] and
[0228] τ=[txtytz]T (10)
[0229] is the translation matrix, where the elements of the matrix, tx, ty, and tzare the translation components in x, y, and z directions, respectively.
[0230] Alternatively, a homogeneous transformation matrix, H, which incorporates the rotation and translation matrices into a single matrix, can be used to apply a transformation to r1to transform it to the coordinate frame of r0.
[0231] r0=Hr1 (11)
[0232]
[0233] which can be simplified as:
[0234] where r1=[x y z 1]T.
[0235] The method of Horn [see above references] to compute the transformation between two corresponding data sets r1and r0is summarized as follows:
[0236] 1. Compute centroids, {overscore (r)}0and {overscore (r)}1, of r0and r1.
[0237] 2. Subtract centroids {overscore (r)}0and {overscore (r)}1from each point of r0and r1, to get sets r′0and r′1, respectively.
[0238] 3. Compute matrix
[0239] where n is the number of points in each set, and r′0xiis the x component of the ithpoint of r′0, etc.
[0240] 4. Compute matrix N as follows:
[0241] 5. Compute the maximum positive eigenvalue λmof N.
[0242] 6. Compute the matrix of cofactors {overscore (N)}λm, of (N−λmI)
[0243] 7. Compute the eigenvector, em, corresponding to eigenvalue λmusing a row or sum of rows of {overscore (N)}λm:
[0244] em=[e0exeyez]T (16)
[0245] 8. Compute the unit quaternion, {dot over ({circumflex over (q)})}, by normalizing the quaternmion {dot over (e)}m, which has its elements from the normalized computed eigenvector, em:
[0246] where
[0247] {dot over (e)}m=e0+iex+jey+kez (17b)
[0248] 9. Compute the rotation matrix R from the unit quaternion:
[0249] 10. Compute the translation vector, τ
[0250] τ={overscore (r)}0−R{overscore (r)}1 (19)
[0251] 11. The transformation to bring a set r1into the frame of r0can then be computed using Equation 8, or using Equation 11, where H is computed from Equation 12b.
Claims
What is claimed is:
1. A method of obtaining three-dimensional data for a surface, said method comprising the steps of:
a. obtaining three-dimensional information of various points of the surface in a plurality of at least partially overlapping views by a sensor head, where the positions and relative positions of said surface and said sensor head are not known and are derived from said information of said surface obtained by the sensor head;
b. registering a plurality of said views, by computing the transformations between overlapping views; and
c. integrating said views into a common reference frame by applying appropriate transformations to points of each registered view.
2. A method as recited in claim 1, wherein said three-dimensional information is in the form of points along several spaced-apart surface profiles.
3. A method as recited in claim 2, wherein said profiles are obtained by using images from a camera positioned to view the intersection of the light of several generally parallel planes of light intersecting with a surface by a projector at an angle from the camera, for a plurality of overlapping views of the surface.
4. A method as recited in claim 3, wherein said three-dimensional information of points along said profiles is obtained by processing two-dimensional data from said images from said camera, to reconstruct three-dimensional coordinates in the local frame of a sensor head comprising said camera and said projector, of a plurality of points which comprise the profiles.
5. A method as recited in claim 4, wherein said processing to reconstruct three-dimensional coordinates is based on previously-obtained calibration mathematical function for said sensor head, said calibration mathematical function having been obtained from calibration via execution of a first algorithm to determine a mathematical mapping relationship between any two-dimensional image point and its corresponding three-dimensional coordinate which represents a point of a profile.
6. A method as recited in claim 4, wherein said processing of two-dimensional information is carried out by execution of a second algorithm on a computer connected to receive said images from said camera, wherein said second algorithm:
a. detects the profiles in the two-dimensional images from the camera in real-time during a scan by said sensor head;
b. extracts the coordinates from two edges of a profile;
c. saves the two-dimensional image coordinates which correspond to edge points;
d. in scanning across the camera image, counts the profiles being detected to identify and label each measured point with the proper profile; and
e. stores information from the preceding steps of this claim.
7. A method as recited in claim 6, wherein said steps of extracting and saving coordinates includes smoothing raw profile data by a simple average of neighboring points or by spline or polynomial piecewise curve fitting, on the two-dimensional image profile, or on the three-dimensional curve once reconstructed.
8. A method as recited in claim 4, wherein said acquisition of two-dimensional images is carried out by execution of a second algorithm on a computer connected to receive said images from said camera, where said second algorithm:
a. detects the profiles in the two-dimensional images in real-time during the scan;
b. extracts the coordinates from two edges of a profile,. by scanning across the camera image and detecting abrupt changes in pixel intensity;
c. computes the two-dimensional image coordinates which correspond to the center of the profile and saves such information;
d. in scanning across the camera image, counts the profiles being detected to identify and label each measured point with the proper profile; and
e. stores the necessary useful information from the preceding steps.
9. A method as recited in claim 8, wherein said step of computing the center of the profile comprises averaging coordinates of left and right edges at each point along a profile to obtain the center of the profile.
10. A method as recited in claim 8, wherein said step of computing the center of the profile comprises computing the two-dimensional image coordinates which correspond to the peak light intensity across the width of the profile.
11. A method as recited in claim 8, wherein said steps of extracting and saving coordinates includes smoothing raw profile data by a simple average of neighboring points or by spline or polynomial piecewise curve fitting, on the two-dimensional image profile, or on the three-dimensional curve once reconstructed.
12. A method as recited in claim 5, wherein said processing to reconstruct three-dimensional coordinates includes reconstrucing three-dimensional coordinates using the mathematical mapping relationship.
13. A method as recited in claim 1, wherein said step of registering said views includes searching locally around the points to match points of said plurality of views.
14. A method as recited in claim 1, wherein said step of integrating said views includes applying view-pair transformations sequentially to bring the registered views into a single reference frame.
15. A method of obtaining three-dimensional data for a surface, said method comprising the steps of:
a. obtaining three-dimensional data of various points of the surface in a plurality of at least patially overlapping views by use of a range-sensor head, where the positions and relative positions of the surface and the range-sensor head are not tracked or measured by any other sensor or device, and are otherwise unknown, the three-dimensional data being in the form of points along several spaced apart surface profiles;
b. registering a plurality of said views, by computing the transformations between overlapping views; and
c. integrating said views into a common reference frame by applying appropriate transformations to points of each registered view.
16. A method as recited in claim 15, wherein said profiles are obtained by using images from a camera positioned to view the intersection of the light of several generally parallel planes of light intersecting with said surface at an angle from the camera, for a plurality of overlapping views of the surface.
17. Apparatus for obtaining three-dimensional data of a surface, comprising:
wherein the positions and relative positions of said surface and said sensor head are not tracked or measured by any other sensor or device.
a sensor head comprising a light projector and a camera, said light projector projecting at least several generally parallel lines on the surface, said camera being positioned for capturing an image of said lines projected onto said surface at an angle from said light projector and said lines on said surface; and
a computer for analyzing said image, wherein said computer is programmed to:
a. compute three-dimensional data of various points of the surface in a plurality of at least partially overlapping views from the camera;
b. register a plurality of said views, by computing the transformations between overlapping views; and
c. integrate said views into a common reference frame by applying appropriate transformations to points of each registered view;
18. Apparatus as recited in claim 17, wherein said light projector comprises a plurality of light sources mounted to project a corresponding number of approximately parallel planes of light.
19. Apparatus as recited in claim 17, wherein said light projector comprises a single light source with a several-line optic lens mounted to project several approximately parallel planes of light.
20. Apparatus as recited in claim 17, wherein said light projector comprises a single light spot projected using beam-splitters, prisms and rotating mirrors, to define said generally parallel lines.
21. The method as recited in claim 13, further including surface fitting the points of each view.
Patent Citations (15)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4705401(A) | 1987-11-01 | Addleman et al. | Applicant |
| US4737032(A) | 1988-04-01 | Addleman et al. | Applicant |
| US4821200(A) | 1989-04-01 | Oberg | Applicant |
| US5027281(A) | 1991-06-01 | Rekow et al. | Applicant |
| US5198877(A) | 1993-03-01 | Schulz | Applicant |
| US5218427(A) | 1993-06-01 | Koch | Applicant |
| US5561526(A) | 1996-10-01 | Huber et al. | Applicant |
| US5568384(A) | 1996-10-01 | Robb et al. | Applicant |
| US5612905(A) | 1997-03-01 | Maillart et al. | Applicant |
| US5621807(A) | 1997-04-01 | Eibert et al. | Applicant |
| US5753931(A) | 1998-05-01 | Borchers et al. | Applicant |
| US5870220(A) | 1999-02-01 | Migdal et al. | Applicant |
| US5886775(A) | 1999-03-01 | Houser et al. | Applicant |
| WOPCT/US96/10485 | 1997-01-01 | Applicant | |
| WOPCT/CA98/00324 | 1998-10-01 | Applicant |
Non-Patent Literature (20)
- Chen Y. and Medioni G., “Object modeling by registration of multiple range images”, CVGIP Image and Vision Computing, 10, pp. 145-155, 1992.Applicant
- Fisher, R.B., Fitzgibbon, A., Gionis, A., Wright, M. and Eggert, D. “A Hand-Held Optical Surface Scanner for Environmental Modeling and Virtual Reality”, Virtual Reality World '96 Conference.Applicant
- Hebert P., and Rioux, M. “Toward a hand-held laser range scanner: integrating observation-based motion compensation”, Proc Three-Dimensional Image Capture and Applications: SPIE-3313, pp. 2-13, 1998;.Applicant
- Hebert, P. and Rioux, M. National Research Council of Canada, Ottawa, Canada, http://www.vit.iit.nrc.ca/Pages_Html/English/hand_held.html.Applicant
- El-Hakim, S.F. “A System for Indoor Virtual Environment Modeling”, http://www.vit.iit.nrc.ca/elhakim/Mapping.html.Applicant
- Besl P.J. and. McKay N. D., “A method for registration of 3-D shapes”, IEEE Trans. Pattern Anal. Machine Intell. PAMI-14, pp. 239-256, 1992.Applicant
- Knopf, G. and Kofman, J. “Neural Network Mapping of Image-to-Object Coordinates for 3D Shape Reconstruction”,, in Intelligent Robots and Computer Vision XV: Algorithms, Techniques, Active Vision and Materials Handling, D.P. Casasent, (Ed.), Proc. Soc. Photo-Opt. Instr. Eng. (SPIE) 2904, pp. 129-137, 1996.Applicant
- Knopf, G. K. and Kofman J. “Range Sensor Calibration Using a Neural Network”, ANNIE '98—Smart Engineering System Design: Neural Networks, Fuzzy Logic, Evolutionary Programming, Data Mining and Rough Sets. ASME, pp.41-146, 1998.Applicant
- J. Kofman, and G.K. Knopf, “Registration and Integration of Narrow and Spatiotemporally-Dense Range Views”, in Vision Geometry VII, R.A. Melter et al. (Eds.),.), Proc. Soc. Photo-Opt. Instr. Eng. (SPIE) 3454, pp. 99-109, 1998.Applicant
- Kofman, J. and Knopf, G.K. (1998). Range Sensing by an Unconstrained and Continuously Moving Laser-Scanner Without Sensor-Head Pose Measurement. 2nd International Symposium on Intelligent Manufacturing Systems—IMS '98, Aug. 6-7, 1988, Sakarya, Turkey, 783-793.Applicant
- Kofman, J. and Knopf, G.K. (1999). “Point correspodences between successive range views using localized spin images”, in Intelligent Robots and Computer Vision XVIII: Algorithms, Techniques, and Active Vision, D.P. Casasent, (ED.), Proc. Soc. Photo-Opt. Instr. Eng. (SPIE) 3837.Applicant
- 3D Scanners. http://www.3dscanners.com.Applicant
- CAPOD systems AB. http://www.capod.se/ ;http://www.capod.se/prod/hard/freescan.html.Applicant
- Cyberware. http://www.cyberware.com http://www.cyberware.com/pressReleases/michelangelo.html.Applicant
- Hymarc http://hymarc.com.Applicant
- InnovMetric. Polyworks software. http://www.innovmetric.com/Anglais/page_modpol.html.Applicant
- Kreon Industries Inc. “A New Liberty in Reverse Engineering”, Commercial literature.Applicant
- Polhemus. http://www.polhemus.com/hlsl.htm.Applicant
- Vitana Corporation. http://www.vitana.com, http://www.shapegrabber.com.Applicant
- Vorum Research Corp. scanGogh http://www.oandp.com/publicat/busworld/summer98/tech.htm.Applicant