Three-dimensional object measurement process and device

[0002] The invention concerns a process for the three-dimensional measurement of objects through optical exposures, projected patterns, and calculations of triangulation, in which the devices for the projection of the pattern and for the image capture are constructed separately from one another and can, for the measurement, be positioned independently of one another.

[0003] Furthermore, the invention concerns a device for such a process.

[0004] The use of processes that work on an optical basis offer a multiplicity of advantages for the purpose of measuring. A measurement can be carried out quickly and without contact. Electronic image converters, such as CCD arrays, for example, the output signals of which can be stored or evaluated after digitalization, already belong to the state of the art.

[0005] Processes and devices for the representation and three-dimensional optical measurement of spatial surfaces are already known. These are based upon triangulation processes in which patterns of points, lines, or any other chosen patterns are projected, at a certain angle, onto the surface under consideration, and the projected patterns are captured, under another viewing angle, by means of an optical device and an image converter. The known geometry between the direction of projection and the direction of capture permits the three-dimensional calculation of support points of the surface.

[0006] If a uniform pattern of lines is projected onto an object with spatial extensions, for example, then, because of the surface shape of the object, a distorted line pattern is brought about in a direction of view which is different from the direction of projection. If the object is represented from this direction of consideration on a CCD-array by means of a suitable optical device, and the image signals are digitalized and made available to a data processing unit, then it is possible to identify the lines at the various points of the image and, given a knowledge of the optical path of the rays, and under consideration to the geometrical configuration of the projected pattern of lines, to compute the 3D coordinates by means of triangulation.

[0007] In the capture of several individual images with different directions of capture, it is already state of the art to combine the partial information obtained about three-dimensional surfaces by means of the 3D coordinates of corresponding surface segments into a total information by means of so-called “registration-algorithms”.

[0008] It is now desirable to do the following for endoscopic investigations, for example:

[0009] a) To use a separate unit for the projection of the pattern and for the capture of the image, and to use this with an orientation that is freely selectable in relation to the surface to be measured. In order to obtain relevant data for the calculation of the triangulation, a minimum angle between the orientation of both of the units should not be fallen short of in this connection;

[0010] b) Insofar as a series of individual images is captured during the capture process, to change the position and orientation of both of the units independently of one another in relation to the object, such as for the simplification of the handling, for example, in order to be able to measure additional surface areas by means of additional image captures.

[0011] It is thus the task of the invention to create the process and the device for the three-dimensional measurement of objects of the type just described, in which the position and orientation of the image capture unit and of the projection unit can be adjusted to one another and, in the event that more than one capture is useful for this purpose, to follow the independent movement of the two units towards one another that has taken place during the course of the capture process. The geometrical connection between the image capture and the pattern projection determined for the specific individual image must then be taken into consideration for the calculation of triangulation.

[0012] This task is solved, in accordance with the invention, through the fact that:

[0013] a) Through the use of an electronic surface image converter, a capture process is carried out;

[0014] b) Through the use of a source of beams and suitable optical devices, a pattern is projected onto the object at an angle which is different from the direction of capture that is to be measured;

[0015] c) Through the use of a suitable unit, the output signals of the image converter are digitalized, and these data are made available to a data processing apparatus;

[0016] d) Through the use of a data processing apparatus, 3D coordinates of the points of the surface of the object to be measured are determined, by means of suitable circuits and/or algorithms, through calculations of triangulation from the image data of the individual images;

[0017] while additional devices are provided in such a manner, and/or the process coming for use and/or the devices are configured in such a manner, that the position and the orientation of the image capture unit and of the projection unit to one another can be determined. If the position and the orientation of both units to one another is known, then, for the capture of an individual image, 3D coordinates can be determined by means of a triangulation calculation, in the known manner and method, for a number of points of the surface.

[0018] It is, in accordance with the invention, advantageous to use additional devices for this purpose in order to measure the position and the orientation of the image capture unit and of the projection unit to one another.

[0019] In accordance with the invention, this can be carried out, in an advantageous manner, both directly as well as indirectly, both by means of scanning or in connection with measuring means, as well as without contact, such as by optical means, for example.

[0020] If information on the object which is additional to that information acquired from an individual exposure is now to be determined, then the image capture unit must be able to record a sequence of individual images. For the task in accordance with the invention, namely, that the projection unit and the image capture unit are moved differently during the course of the image capture process, the devices and/or processes in accordance with the invention must, in an advantageous manner, be configured in such a manner that the geometrical configuration of the image capture unit and of projection unit can be adjusted to one another for the sequential images.

[0021] This can, in accordance with the invention, be advantageously guaranteed through the fact that the above-stated measuring devices are also evaluated at the point in time of the exposures that are additionally carried out, and the measuring results are taken into consideration, in the way and manner described above, in the calculation of triangulation.

[0022] For the event that, in the sequence of the individual images, the object is recorded in such a manner that the 3D coordinates determined from the individual images describe surface segments that are in agreement to a considerable degree, then—as has been described above—the specific 3D information can be combined by means of “registration algorithms” corresponding to the state of the art.

[0023] If the capture of the individual images takes place in rapid sequence, then, with a corresponding handling of the units, it can be assumed that—first of all—the portion of corresponding surface segments is sufficient and—secondly—that the geometrical correspondence of the image capture unit and of the projection unit to one another has only been altered in extent in relation to the previous capture.

[0024] If, therefore, the coordination of both units in the first capture was known, for example, and if a considerable number of 3D coordinates describing the surface could be computed, then the information contained in the next capture differs in the following components:

[0025] a) The 3D coordinates of the corresponding surface segments can be represented displaced linearly in the three spatial directions;

[0026] b) The 3D coordinates of the corresponding surface segments can be represented displaced in a rotational manner in relation to the three spatial directions;

[0027] c) The relative position of the image capture unit and of the projection unit to one another can be changed linearly in the three spatial directions;

[0028] d) The relative position of the image capture unit and of the projection unit to one another can be changed in a rotational manner in relation to the three spatial directions.

[0029] For the purpose of the generalization of expressions, these possibilities of change will also be termed in the following as “degrees of freedom”.

[0030] Matching algorithms—which form, from the different data records of 3D coordinates, the intersections of that data which are to be assigned to corresponding surface segments, and which vary and optimize the possible changes (a) and (b) listed above in such a manner that the intersection of the specific 3D data of both data records fit optimally with one another in regard to the corresponding displacements and rotations—already belong to the state of the art in the technical area of graphic data processing. One presupposition for this is that the changes (c) and (d) listed above are known.

[0031] Algorithms which systematically vary a finite number of degrees of freedom and optimize these in accordance with one or more criteria of quality are already known from numerical mathematics. In a systematic variation, however, the computational expense is raised to a higher power with every additional level of freedom.

[0032] Recursively gradated processes which, through a stochastic variation of the degree of freedom and a step-wise selection of each specifically best variant, under the assumption that the quantities of information that are to be evaluated are essentially similar, reach an optimal solution surprisingly quickly, are already known, in connection with the phrase “evolution strategies”, from the field of bionics.

[0033] In this sense, it is, in accordance with the invention, advantageous for the solution of the task in accordance with the invention to use expanded matching algorithms in such a manner that:

[0034] First of all—In a recursive numerical process, multiple so-called “successor” variants are first formed through the stochastic variation of the above-stated degrees of freedom (c) and (d);

[0035] Secondly—For every successor variant, the 3D data records are, in the known manner, computed from image information by means of triangulation, in accordance with the assumptions made for the degrees of freedom (c) and (d);

[0036] Thirdly—For every successor variant, the intersections of those data which are to be assigned to the corresponding surface segments are formed, in the known way and manner, in a matching computation from the different data records of 3D coordinates, and the above-listed degrees of freedom (a) and (b) are varied and optimized in such a manner that the intersections formed from the specific 3D data of both data records can, under consideration of the specifically-selected geometrical change, be optimally transformed into one another in relation to the level of freedom (a) and (b);

[0037] Fourthly—Out of all of the successor variants, that one which has the maximum value achieved in regard to the agreement of the data from both data records describing the same surface segment is selected in the sense of an evolution;

[0038] Fifthly—Corresponding to the evolution strategy described, and proceeding from the assumptions that were made for the degrees of freedom (c) and (d) of the previously-selected best successor variants with reduced boundaries of the degrees of freedom (c) and (d), new successors of the next generation are again stochastically varied in different variants and, in a manner corresponding to the above-stated five points—first to fifth—it is then proceeded in a recursive manner for long enough until the boundaries of variation of the degrees of freedom (c) and (d), and the differences between the generations in regard to the maximum value achieved in every passage, are sufficiently small.

[0039] As the result, all information required is known with sufficient precision.

[0040] It is, in accordance with the invention, advantageous to construct the devices for the projection of the pattern and for the capture of the image in such a manner that these are connected rigidly and in an unambiguous position and orientation with one another during the capture of the first individual image. During the further course of the capture, the connection is detached and a movement independent of one another is thereby made possible. The additional three-dimensional measurement values of the object can then be computed, for every new image capture, with the expanded registration algorithm described above and, as described above, combined into a total information.

[0041] In an alternative manner, it is, in accordance with the invention, advantageous to measure the geometrical relation of the projection unit and of the image capture unit to one another through the fact that the pattern used is projected both onto the object as well as onto the image capture unit, and the pattern resulting on a defined surface rigidly connected with the image capture unit is recorded by one area of the image converter. In this sense, the image converter can record the object with one part of its areal field of view, and record the pattern projected onto the image capture unit with the other part.

[0042] In the evaluation of the image information, the geometrical position of the two units to one another is, in accordance with the invention, first of all advantageously computed from the distortion of the sample projected onto the surface of the image capture unit, which is, for example, flat and constructed as ground glass disk, by means of triangulation, and the object is, in addition, spatially measured, in relation to the assignment of both of the units to one another, in the way and manner as described above.

[0043] A process in which, in order to prevent multiplicities in the calculations of triangulation, coded patterns are projected onto the object, is already known from the German patent application “A process for increasing the significance of the three-dimensional measurement of objects” of the date Sep. 12, 1996 by the same applicant.

[0044] It is, in accordance with the invention, particularly advantageous to use coded patterns for the determination of the coordination of the projection unit and of the image capture unit to one another in such a manner.

[0045] A process and a device in which beam paths can be effectively or ineffectively switched in an alternating manner by means of optical means placed synchronously with the alternating image current of the image converter are already known from the German patent application number 196 36 354.3 by the same applicant.

[0046] It is, in accordance with the invention, particularly advantageous to synchronously switch the direction of view of the image capture unit, such as by means of two surface-active LCD-units—which are positioned on two optical axes of a beam splitting devices, for example—with the image alternating frequency of the image converter in such a manner that they are alternately optically permeable and non-permeable. This has the advantage that, for the purpose of determining the geometrical coordination of the projection unit with the image capture unit, the field of view of the image converter, and thereby the resolution in the representation of the object, does not have to be reduced.

[0047] It is, in accordance with the invention, advantageous to additionally provide further optical devices for the projection of a pattern which is different, for example, onto the image capture unit on the projection unit.

[0048] In an alternative manner to the surface rigidly connected with the image capture unit that has been described above, the pattern can, in accordance with the invention, be advantageously projected onto a separately-positioned surface with a known spatial extension, in order to determine the coordination of the image capture unit and of the projection unit with one another, and can be represented on the image converter of the image capture unit.

[0049] A second image capture unit for the purpose of determining the coordination of the image capture unit and of the projection unit with one another can advantageously be provided in accordance with the invention.

[0050] All of the configurations of the process and of the device that are described in this invention can, in a particularly advantageous manner, be combined with one another in multiple ways and manners in accordance with the invention.

[0051] The device for the three-dimensional measurement of objects of the type described above solves the task in accordance with the invention through the characteristics that:

[0052] a) At least one electronic surface image converter which uses an optical device suitable for the capture of images is used;

[0053] b) At least one device which carries out the digitalization of the output signals of the at least one image converter and makes these data available to a data processing apparatus is used;

[0054] c) At least one device which permits the projection of a pattern is used;

[0055] d) The devices for the projection of the pattern and for the capture of the image are constructed independently of one another;

[0056] e) At least one data processing apparatus which uses suitable circuits and/or algorithms is suited for the calculation of 3D coordinates in accordance with the triangulation process; and:

[0057] f) At least one device is suited for the determination of the geometrical coordination between the projection unit and the image capture unit.

[0058] In one configuration of the device in accordance with the invention which is particularly advantageous, the image capture unit is suitable for the capture of at least two successively following images.

[0059] One configuration of the device in accordance with the invention which is particularly advantageous provides devices which can store the data of at least two individual images.

[0060] One configuration of the device in accordance with the invention which is particularly advantageous provides devices which process the data of at least two individual images, and are able to combine these by means of suitable circuits and/or algorithms.

[0061] Further advantages, characteristics, and possibilities of application of the present invention result from the following description of preferred examples of implementation in connection with the attached diagrams, FIGS. 1, 2, 3 and 4. These depict the following:

[0062] FIG. 1: In schematic form, one form of implementation of the image capture device with the projection [unit] and image capture unit separate from one another, which are here connected together, with the help of positioning rods, for the purpose of the capture of a first image under defined geometrical conditions in relation to the direction of projection and capture.

[0063] FIG. 2: The same device as in FIG. 1, but decoupled in order to carry out further captures.

[0064] FIG. 3: In schematic form, one further form of implementation of the image capture device with the projection [unit] and image capture unit separated from one another, characterized by the additional projection of the same pattern by way of a second beam path of the projection unit directly onto a ground glass disk of the image capture unit, with this being for the purpose of the continuous calculation of their geometrical situation to one another.

[0065] FIG. 4: By way of example, one form of implementation of a coded pattern.

[0066] In the forms of implementation in accordance with FIGS. 1, 2 and 3, the projection unit and the image capture unit are, for the purpose of simplifying the description, only depicted schematically without known, obvious device parts. For the simplification of the description, it was additionally dispensed with describing forms of implementation of device parts that belong to the state of the art, such as specific forms of implementation of connections or of part-fixing devices, in full detail.

[0067] In the form of implementation of FIG. 1, the source of beams (2.7), the optical devices (2.4 and 2.6), and the pattern support (2.5) are rigidly connected with the support mount (2.3) of the projection unit. The optical devices in the form of implementation depicted here involve a source of beams (2.7) by means of which the object can be illumined on its surface, a condenser (2.6), as well as a pattern support on which the pattern (whether etched in or in the form of a slide transparency) is positioned. The support (2.2) is, for hygienic reasons (for the purpose of disinfection), connected with the support mount (2.3) in a detachable manner. The connection of the support (2.2) with the support mount (2.3) is configured in such a manner that, after a change of the support (2.2), its set geometrical positions are reproduced precisely (without adjustment). The pattern placed on the pattern support (2.5) is projected onto the object that is to be captured (1.1) by means of a beam deflection device (2.1). In order to obtain relevant information for the 3D calculation, a projected line pattern is applied transversely to the level spread on the triangulation angle.

[0068] The following are rigidly connected with the support mount (3.3) of the image capture unit: the optical device (3.4), the electronic image converter (3.5), as well as the positioning rods (3.10). The support (3.2) is, for hygienic reasons (for the purpose of disinfection), connected with the support mount (3.3) in a detachable manner. The connection of the support (3.2) with the support mount (3.3) is configured in such a manner that, after a change of the supports (3.2), its set geometrical position is reproduced precisely (without adjustment). The electronic image converter (3.5) is configured as a CCD array and connected, by way of the electrical connections (3.9), with the control unit (3.6), the A/D converter (3.7), and the data processing unit (3.8). The beams reflected from the object (1.1) are imaged on the image converter (3.5) by means of the optical devices (3.4), in this case an objective lens.

[0069] For the carrying out of the first capture, the projection unit and the image capture unit are, with the help of two positioning rods (3.10), brought into a defined position in relation to one another. The triangulation angle which is consequently known permits the calculation of 3D coordinates of the captured image.

[0070] FIG. 2 depicts the same projection unit and image capture unit, but in the decoupled condition after the carrying out of the first capture in a series of captures of the same object, however. Further images of additional areas of the object and/or from a changed direction of view can now be captured. Because of the high frequency of capture, such as of 50 Hz in the configuration depicted, for example, a high level of overlapping of the images, which is in every case sufficient, can be assumed, which makes it possible, by means of suitable registration algorithms and numerical processes, to compute the positions of the projection unit and the image capture unit relative to one another as well to the object, which positions have changed in relation to the specifically preceding capture, and to thereby also compute the triangulation angle which is specifically relevant, for example. This makes possible the determination of the 3D coordinates of an object, which could not be determined with a single capture.

[0071] FIG. 3 depicts one form of implementation in which, by means of the projection of the same pattern by way of a second beam path—that is, in addition to the (invasive) parts of the device that are used—which is guided from the projection unit to the image capture unit, the relative positions of the projection unit and of the image capture image to one another are known at the point in time of every capture. In this form of implementation, the pattern is diverted by 90 degrees by means of a beam splitting device (2.8) and projected, through a lens (2.9) with a divergent beam path positioned on the projection unit, onto a ground glass disk (3.12) positioned on the image capture unit. The image of the portion of a pattern which thereby results is, by means of a beam deflection device (3.11), diverted by 90 degrees into the beam path of the support of the image capture unit in such a manner that it is imaged on one half of the two-dimensional field of view of the electronic image converter (3.5). Through the evaluation of the significant pattern segments of the image in the data processing unit (3.8) by means of suitable numerical processes, the position of the projection unit relative to the image capture unit is known for every individual capture. With this form of implementation, the 3D coordinates of which can only be computed through the evaluation of a multiplicity of captures, the attainable precision of the 3D coordinates can be significantly increased in relation to the one described above, particularly in the case of complex objects.

[0072] FIG. 4 depicts, by way of example, one form of implementation of the pattern, through which, because of the different pattern of each of the individual concentric elements (coding), differences in the calculation of triangulation can be avoided.

[0073] One particularly advantageous use of the invention is to be seen in the area of invasive medicine. For the three-dimensional optical imaging of an organ inside the human body, for example, the forward part of the support (2.3) as well as of the support (3.2), for example, are inserted through the abdominal wall, the capture process is started by means of an activation element which is not depicted in the diagrams but which is present in the framework of the data processing unit (3.8), and, during the course of the capturing process, the projection unit and the image capture unit are moved in such a manner that all relevant surface areas of the object (1.1) are gradually represented, both on the image converter (3.5) as well as by the projection of the pattern of lines. The capture process is ended by an additional activation of the activation element stated in the preceding.

[0074] As the result of the individual image information captured, the images of the pattern of lines distorted by the surface form of the object (1.1) are now presented. Given a knowledge of the optical beam paths, and with consideration to the geometrical configuration of the projected pattern of lines for the corresponding individual exposures, a multiplicity of support points of 3D coordinates can be computed from the distortion of the pattern of lines. As the result of the 3D coordinates assigned to the individual images, furthermore, the 3D coordinates can be combined in the way and manner described above so that, as the result, the 3D coordinates of the entire surface investigated are present despite the limitation of the projection surface and of the field of view of the image capture unit. Gaps in information for reason of undercuts are thereby no longer present. The user can additionally optimize the guiding of the capture device by means of an online calculation and indication of the specific interim findings.

[0075] The forms of implementation which have been depicted here are distinguished, among other points, by the fact that:

[0076] a) No movable parts are used in the projection unit and image capture unit;

[0077] b) The optical structure is configured in a simple manner;

[0078] d) Apart from the support units (2.2 and 3.2), the support mounts (2.3 and 3.3), and the pattern support for the projection of the pattern (2.5), hardware components of the commercially available type can be used.

[0079] A process and a device in which optical devices synchronized with the image alternation frequency of the image converter, in such a manner that a pattern is alternately projected onto the surface to be measured and the surface considered is completely illuminated, is already known from the German patent application number 196 35 354.3 of the same applicant.

[0080] In one particularly advantageous configuration of the image capture device in accordance with the invention, additional devices in the previous sense are positioned and placed in such a manner that both 3D information as well as colored views of the surface scanned, for example, can be obtained from a series of individual images.

[0081] 1.1 Object with three-dimensional extension

[0082] 2. Projection Unit

[0083] 2.1 Beam deflection device (mirror)

[0084] 2.2 Support (constructed here as a body of optical glass)

[0085] 2.3 Support mount

[0086] 2.4 Optical device (objective lens)

[0087] 2.5 Pattern support (with pattern etched in or a slide transparency, for example)

[0088] 2.6 Optical device (condenser)

[0089] 2.7 Source of beams

[0090] 2.8 Beam splitting device

[0091] 2.9 Optical device (lens)

[0092] 3. Image Capture Unit

[0093] 3.1 Beam deflecting device (mirror)

[0094] 3.2 Support (constructed here as a body of optical glass)

[0095] 3.3 Support mount

[0096] 3.4 Optical device (objective lens)

[0097] 3.5 Electronic image converter (CCD Array)

[0098] 3.6 Control unit

[0099] 3.7 A/D converter-frame Grabber

[0100] 3.8 Data processing unit

[0101] 3.9 Electrical connections

[0102] 3.10 Positioning rods

[0103] 3.11 Beam deflecting device (deflecting prism)

[0104] 3.12 Ground glass disk