Method and assembly for the photogrammetric detection of the 3-d shape of an object

[0001] The invention relates to methods and arrangements for the photogrammetric detection of the 3D shape of an object.

[0002] Contactless detection of the 3D shape of bodies and objects such as motor vehicle bodies, human body parts and the like is an important measuring method in quality control, in reverse engineering, in the generation of numerical 3D models of physically existing bodies, etc.

[0003] A number of different methods are known in literature and on the market, which are based either on stripe projection, laser line projection, laser transit time measurement, or photogrammetry, and some are based on a combination of different methods.

[0004] The so-called passive methods of short-range photogrammetry are especially cost-effective since merely calibrated cameras or digital cameras are needed here, but no expensive projection units or precisely calibrated mechanical structures are required.

[0005] One typical representative of this group is the PhotoModeler software of the EOS company (see the Internet website www.photomodeler.com). This software allows to develop the numerical 3D model of an object from different images of the object which overlap as far as the captured parts of the object are concerned. In doing so, however, the positions of markers stuck onto the body to be measured or of distinct features such as corners or lines of such body need to be manually marked on the screen on the different 2D photographs by the user using a mouse and need to be referenced in pairs, that is, corresponding markers need to be associated with each other manually in the different photographs. Where a dense sequence of XYZ coordinate values is required, several hundred to thousand such assignment processes need to be carried out manually. This is not only extremely tedious, but also prone to errors, since it is almost impossible to assign without errors many markers that look the same.

[0006] It is further known from photogrammetry to use markers with encoded point numbers (encoded point markers) which bear a pattern with an encoded point number. For this purpose, point encodings are used that are distributed on a line or ring or over an area and that are arranged around the measuring point proper. The identity of the encoded point markers can be recognized automatically by an image processing unit, whereupon the points in the images can then be associated in pairs automatically.

[0007] These encoded point markers, however, need to be fairly large in order to accommodate a code such as, e.g., a radial bar code, a two-dimensional point code or the like. As a rule, each marker requires an individual code. This in turn prevents this method from being applied in the case of small bodies requiring many markers since due to the restricted spatial conditions, these point markers can no longer be applied in sufficient numbers.

[0008] Further known are photogrammetric methods which can do without encoded markers and iteratively determine the association of markers that look the same. But these methods require accurately calibrated cameras, are very lengthy, and demand of the user good knowledge of the photogrammetric boundary conditions, in particular the appropriate imaging positions and the necessary amount of overlap, for the iterative solution methods to converge.

[0009] Patent specification EP 0 760 622, Sensing Process and Arrangement for the Three-Dimensional Shape in Space of Bodies or Body Parts, Inventor and Applicant: Robert Massen, describes a method and an arrangement by means of which body parts to be digitized can be marked in a simple and cost-effective way by covering the body part with a tight-fitting elastic envelope featuring markers adapted to be evaluated photogrammetrically. As an example, this patent specification discusses the marking of a human foot to generate 3D models for the production of prostheses or made-to-measure shoes. This method allows, in a simple and cost-effective way, to apply a large number of markers and thereby to provide the basis for obtaining dense XYZ coordinates by photogrammetry. This, however, does not yet solve the problem of simple automatic registration of markers from the overlapping regions of the different images.

[0010] The object of the present invention is to provide methods and arrangements for the photogrammetric detection of the 3D shape of an object, which are suitable for automatic referencing of photogrammetric markers and for high-resolution digitizing of objects, bodies or body parts, and which more particularly overcome the above-mentioned drawbacks occurring in photogrammetric methods that use encoded point markers.

[0011] This object is attained by a method of detecting the 3D shape of an object by photogrammetry, comprising the steps of providing point markers and one or more area markers on the surface of the object, each area marker comprising a plurality of point markers, forming a background of the point markers, and having a surface area having a characteristic optical configuration; taking a plurality of photogrammetric images of the object from respective different views; performing an image processing of the images, in which first the area markers respectively corresponding in the images are associated with one another using their characteristic optical configuration, and then the point markers comprised by the area markers and respectively corresponding in the images are associated with one another with the aid of the area marker association; and, using the point marker association, determining the 3D shape of the object by means of a photogrammetric evaluation process.

[0012] Furthermore, this object is attained by a method of detecting the 3D shape of an object by photogrammetry, comprising the steps of providing point markers on the surface of the object in such a manner that groups of a plurality of point markers are formed, the point markers in a group being arranged in relation to one another such that a characteristic point marker arrangement is produced; taking a plurality of photogrammetric images of the object from respective different views; performing an image processing of the images, in which first the characteristic point marker arrangements respectively corresponding in the images are associated with one another and then, with the aid of such association, the point markers comprised by the characteristic point marker arrangements and respectively corresponding in the images are associated with one another; and, using the point marker association, determining the 3D shape of the object by means of a photogrammetric evaluation process.

[0013] In addition, the object of the invention is attained by an arrangement for detecting the 3D shape of an object by photogrammetry, comprising an imaging system for obtaining photogrammetric images of different views of the object and a system for measuring and evaluating the images and for determining the 3D shape of the object, which is characterized in that the arrangement further includes point markers and one or more area markers provided on the surface of the object, each area marker comprising a plurality of point markers, forming a background of the point markers, and having a surface area having a characteristic optical configuration.

[0014] Finally, the object is attained in accordance with the invention by an arrangement for detecting the 3D shape of an object by photogrammetry, comprising an imaging system for obtaining photogrammetric images of different views of the object and a system for measuring and evaluating the images and for determining the 3D shape of the object, which is characterized in that the arrangement further includes point markers which are provided on the surface of the object in such a manner that groups of a plurality of point markers are formed, the point markers in a group being arranged in relation to one another such that a characteristic point marker arrangement is produced.

[0015] The methods and arrangements in accordance with the invention result in the following advantageous effect:

[0016] The inclusion of characteristically configured background areas behind the point markers for referencing, and the combination of noncoded point markers to form characteristic superordinate point marker arrangements allows a convenient and precise automated referencing of points even when the object whose 3D shape is to be determined by photogrammetry is small and/or requires a multitude of point markers for a sufficient determination of the 3D shape by means of photogrammetric methods.

[0017] Advantageous further developments of the invention are characterized in the dependent claims.

[0018] Further features and advantages of the invention will be apparent from the following description of an embodiment with reference to the drawings, in which:

[0019] FIG. 1 shows two different encoded point markers that are part of the prior art;

[0020] FIG. 2 illustrates an embodiment of the invention in the case of a human foot that is covered by an envelope provided with point markers and colored area markers;

[0021] FIGS. 3, 4 and 5 are diagrammatic illustrations showing how the photogrammetric evaluation is effected with the aid of the point and area markers illustrated in FIG. 2;

[0022] FIG. 6 shows a further embodiment of the invention, in which the point markers are combined in groups having characteristic point marker arrangements in the form of a constellation;

[0023] FIG. 7 is a diagrammatic view of point markers, as are used in a preferred embodiment of the invention.

[0024] FIG. 1 shows two possible embodiments of the encoded point markers mentioned in the introduction to the specification, that are part of the prior art.

[0025] The invention will now be explained by way of example, but not in a limiting sense, with reference to the employment of optical detection of the 3D shape of a human foot for obtaining the 3D data required for an automated selection of shapes of shoes that fit or suitable shoe lasts.

[0026] It is assumed here, likewise by way of example, but not in a limiting sense, that the foot is clad in a tight-fitting, elastic envelope according to the above-mentioned patent EP 0 760 622, which has on its surface noncoded point markers that look the same, in the form of small crosses.

[0027] As an example, the idea underlying a preferred embodiment of the invention will be discussed, that is, the idea of background marking involving a coloration, carried out region by region, of the background of the point markers. Since it is not possible to print color pictures in patent specifications, the different coloration is represented in the Figures by different black-and-white textures.

[0028] FIG. 2 shows a human foot 1, which is clad in a tight-fitting, elastic envelope 2. The envelope 2 has on its surface a large number of identically looking photogrammetric point markers 3 in the form of small crosses.

[0029] The intersection point of the two lines of the point markers represents an XY coordinate in the particular image observed, that can be determined using methods of two-dimensional image processing and can also be exactly determined as a two-dimensional coordinate in relation to the coordinate system of the particular image even in the case of different imaging angles, perspective distortions or the like. When digital cameras are used for image taking, for example, the coordinate system on which each image is based is dictated by the line and column structure of the sensor array and by the piercing point of the optical axis.

[0030] As an example, only three regions, R1, R2 and R3 with the different background colors RED, GREEN and YELLOW are reproduced in FIG. 1. It is known to a person of ordinary skill in the art of image processing that color cameras can be used for automatic segmenting of such color regions with the aid of the colors occurring, which the system has been previously trained to recognize, i.e. such color regions can be automatically recognized and distinguished from each other. Methods of this type are described, for example, in German Patent DE 36 39 636 C2, Automatic Inspection of Lengths of Textile Fabric, Inventor and Applicant: Robert Massen.

[0031] It is further known that such color segmenting can be made to be especially robust and independent of the viewing angle if color classifiers are employed that are independent of the brightness (which is dependent on the viewing angle). This can be done by, e.g., classification of each pixel using the color tone following conversion of the RGB color signals to the HSI color range (hue, saturation, intensity).

[0032] As is shown clearly in FIG. 3, as an example it is assumed that images are taken of the surface of the body to be digitized as defined by the three regions R1, R2 and R3 from three different imaging positions, A1, A2 and A3, using a digital camera 4. The respective three two-dimensional images, B1, B2 and B3, each show overlapping sectors of the body surface.

[0033] By a pixel-by-pixel color classification of all three images into the colors RED, GREEN and YELLOW, each of the three images, B1, B2 and B3, can be automatically decomposed into three so-called color class images 5 (in FIG. 4),

[0034] B1-> B1RED,B1GREEN,B1YELLOW

[0035] B2-> B2RED,B2GREEN,B2YELLOW

[0036] B3-> B3RED,B3GREEN,B3YELLOW,

[0037] as is shown in FIG. 4 using the region B2 as an example. Accordingly, an automatic association of all regions that match in pairs has now already occurred, that is, while the photogrammetric markers have not yet been referenced, the background regions, R1, R2, and R3 of these markers have already been referenced.

[0038] The background regions, each of which comprises a plurality of point markers, will also be referred to as areas markers below because of their planar configuration, by analogy with the known point markers.

[0039] According to the invention, the point markers within two referenced regions (area markers) are automatically associated with each other by a comparison of their positions in relation to the region boundaries and/or by an evaluation of the distinguishable figures formed by a plurality of point markers and/or by minor alterations of their shapes.

[0040] FIG. 5 shows, by way of example, how the photogrammetric point markers situated directly at the region boundaries can be associated with each other based on the information contained in the color class images about these boundaries and the corners formed by regions.

[0041] As an example, one side of the triangular red region R1 is assumed to adjoin the likewise triangular yellow region R3 and the green region R2 having the form of a parallelogram.

[0042] Forming along the lateral edge 6 observed is furthermore a boundary between three countries, i.e. a distinct point 7 where three area markers (regions) marked with three different colors meet. This point may be easily found by comparing all color class images with a neighborhood operator in the two overlapping images B1 and B2. One of ordinary skill in the art of color image processing is familiar with such methods. Once the coordinate of this point in the two images B1 and B2 observed has been established, the noncoded photogrammetric point markers, shown here by the small black-and-white crosses 8 and 9 as an example, can easily be associated with one another using classical search methods, beginning at this starting point, parallel to the color region boundary and within the corresponding regions.

[0043] A further advantage of the color-coded regions is the possibility to check automatically after each image is taken whether the degree of overlap of two images taken is sufficiently large. The degree of overlap results from the number of pixels with the identical color in each image.

[0044] In addition to the area marking of the background regions by color, texture or similar features, the shape of the background region may also be made to bear information that facilitates automated referencing. For instance, owing to their pointed corners, triangular regions are suited to be easily recognized and associated by means of methods of 2D image processing.

[0045] Aside from the automated referencing of the photogrammetric point markers based on their position in relation to the region boundaries, other features of these noncoded point markers may also be used. As shown in FIG. 6, according to the invention the photogrammetric point markers are for example grouped to form characteristic point marker arrangements similar to constellations, i.e. the relative position of several noncoded point markers within a region is selected such that automatically identifiable geometric shapes are formed, such as triangles, parallelepipeds, oblong structures, etc.. In particular, those structures are of advantage which are invariant with respect to affine image transformations such as translation, rotation, elongation, extension, perspective distortion. Such image distortions are always produced when an image sector is taken from two different imaging positions. Such group arrangements are always possible since the position of the photogrammetric markers may be freely selected within limits, in particular in the case of bodies having continuously curved surfaces.

[0046] It is further possible to make the photogrammetric point markers within one region identifiable by slightly modifying their shapes. This may be done, for example, by varying the length or appearance of the bars, as shown in FIG. 7. This type of encoding differs substantially from the known methods of encoding by means of bar codes or the like. In the method according to the invention, only the point markers within one region are marked differently; it is not necessary for all point markers covering the entire body to be identifiable. The coding may therefore be very simple and only needs to distinguish between few marker patterns. For this reason, small markings may be used.

[0047] By combining the individual methods of referencing the photogrammetric point markers within an area marker (region), the markers contained in identical regions from two or more overlapping images may be easily associated with one another using methods of two-dimensional image processing and pattern recognition, even in the case of larger color regions.

[0048] Accordingly, the two method steps of

[0049] (a) referencing the background regions associated with one another on the basis of planar features, such as color, texture, degree of polarization, NIR reflection, degree of polarization, and the like,

[0050] (b) referencing the noncoded photogrammetric point markers within one region on the basis of their positions in relation to the regions, their grouping structures, and simple features of shape

[0051] allow to perform a complete automatic photogrammetric evaluation of a body or body part marked in accordance with the invention.

[0052] It is, of course, also possible according to the invention to combine the method as described with conventionally encoded photogrammetric point markers, as are illustrated, e.g., in FIG. 1.

[0053] In the embodiment of the invention using the area markers, various modifications are conceivable. What is always important is that the area markers exhibit a particular optical configuration, so that different area markers can be distinguished from one another in image processing. This optical configuration may consist in, e.g., a characteristic shape, a characteristic color, a characteristic gray level, a characteristic texture, i.e. a characteristic pattern or a characteristic structure, a characteristic gloss, a surface coating applied to the surface of the area markers and having a characteristic degree of polarization, or a characteristic combination of the above-listed optical configuration features. The area markers may furthermore be characterized in their optical configuration in that they emit radiation either in the visible or in the invisible part of the spectrum.

[0054] Also, the area markers need not necessarily be applied to an envelope. They can just as well be provided directly on the object. Moreover, self-adhesive films may, for example, be attached to the object, the films having the point and area markers applied thereon. Finally, it is also conceivable that the point and area markers are projected onto the surface of the object. The same applies of course to the point markers in the case of the embodiment involving the constellation-type point marker arrangements.

[0055] The point markers may also have different shapes, and they do not necessarily need to consist of crosses. Instead, they may, e.g., consist of the intersection points of a net.