Process and device for the automatic rectification of single-channel or multi-channel images

[0001] 1. Field of the Invention

[0002] The invention pertains to a process and to a device for the automatic rectification of single-channel or multi-channel images, where an image is rectified onto a reference image without precise knowledge of the mapping function.

[0003] 2. Description of the Related Art

[0004] Imprecise and/or insufficient knowledge of the mapping function is present especially when images are recorded by moving recording systems and/or moving image fields and the orientation of the recording system with respect to the image field cannot be determined clearly or can be determined only with extraordinary effort. For example, the mapping functions for rectifying satellite images for land surveying are not known or are known only within certain limits. It is also conceivable that the distortion of the reference image is not known or is known only within certain limits.

[0005] The images are recorded by a recording device and are then available for further processing as single-channel images, such as gray-scale images, or as multi-channel images, such as color images.

[0006] Images for recording and mapping the surface of the Earth are recorded by moving recording systems. The recording system can be located on, for example, a satellite or an aircraft. Known methods for recording and mapping the surface of the Earth are so-called photogrammetric processes, where the recorded images are rectified before they are transferred to the map to be prepared. The necessary transformation of a recorded image into the desired form of appearance is called the mapping process. So that rectification can be performed, the mapping process is modeled as a collinear equation. There are already several basic methods for georeferencing remote-acquired data, which are based either on the principle of calculating positions by means of a model of the mapping process (mapping model) or on the basis of the “control point principle” by means of a reference image. It is also possible to combine the two methods.

[0007] For rectification with the use of a mapping model, the mapping process is described as a collinear equation or is approximated by means of simpler models. If all the parameters of the mapping process are known, it is possible to generate very precise images. In practice, however, not all parameters which play a role in the formation of the image, such as the location and orientation of the satellite, are known with sufficient accuracy. The reasons for this are that there is a limit on the measurement accuracy and the fact that not all parameters are stable over time. This instability over time is the result, for example, of variations in the orbit of the satellite. In addition, the orbit of the satellite is measured discretely, and the interpolation between these points in time results in additional error.

[0008] It is also known that, in the case of incomplete or imprecise mapping parameters, rectification can be carried out with the help of control points. This process is used especially when the image to be rectified, a satellite image, for example, is mapped not onto a defined geometry such as map but rather onto a reference image with unknown distortion. Control points are high-contrast image structures, which are stable over time and which make it possible to determine locations by means of correlations. The interpolation between the control points is usually accomplished with two-dimensional polynomial functions. A source of error in control point rectification is the dissimilarity of the control point structures in the different images, caused by different recording geometries, by different conditions of the objects that form the control point structure, or by the averaging of different objects or parts of objects. Another source of error is the polynomial function used to perform the interpolation, since this function usually does make it possible to model the mapping process with sufficient accuracy. This is especially true in the case of large areas. The greater distance of a point from a control point, the greater the effects attributable to inaccuracies in the polynomial function. This effect is evident especially in the case of higher-order polynomials.

[0009] It is also known that the two methods, i.e., parametric rectification and rectification by means of polynomial functions, can be combined.

[0010] A disadvantage of the known methods is that they are difficult to automate, especially when the knowledge of the mapping process is very limited. It is usually a very complicated matter to adjust all the mapping parameters to an actual situation; this difficulty often leads to the use of pure polynomial rectification. The necessary control points are simply taken from a database. The previously known methods thus have residual errors in the position determination of the control points of the data set to be processed. This residual error cannot be described analytically at all or only with great effort, which means that it is impossible to adjust the parameters appropriately by correcting the mapping model.

[0011] The invention is therefore based on the technical problem of creating a process and a device for the automatic rectification of images by means of which the residual error can be reduced even in the absence of precise knowledge of the mapping model.

[0012] The problem is solved by a process for the automatic rectification of images, wherein at least one image is rectified by a mapping function onto a reference image, and at least some parameters of the mapping function are unknown. The process includes steps of extracting at least three objects from the image; determining at least three control points in the image, such that characteristic object points of the extracted objects are determined as control points; assigning the objects to objects in the reference image, such that assignment is made according to similarity between corresponding objects in the two images and/or on the basis of a vector arid formed by connections between the characteristic object points; and selecting a suitable mapping function and/or adjusting parameters of the mapping function, whereby the mapping function is changed by changing the parameters in such a way that cumulative error in positional differences between control points and corresponding points in the reference image is minimized.

[0013] The problem is further solved by a device for performing the aforesaid steps. The device includes an extraction module for extracting at least three objects from the image; a control point determination module for determining at least three control points in the image, wherein characteristic points of the extracted objects are determined as control points; an object assignment module for assigning the objects to objects in the reference image, such that a correspondence between the objects in the two images is established according to at least one of a similarity between objects, a vector grid formed by connecting characteristic object points, and a selection module. The correspondence is established for at least one of selecting a suitable mapping function and adjusting parameters of the mapping function, whereby the mapping function is changed by changing the parameters in such a way that cumulative error in positional differences between control points and corresponding points in the reference image is minimized.

[0014] Additional advantageous embodiments of the invention are apparent from other preferred embodiments of the invention, described herein below.

[0015] For the automatic rectification of an image onto a reference image, where at least some of the parameters of the mapping function for the rectification process are unknown, at least three objects are extracted from the image and at least three control points are determined. Characteristic points of the extracted objects are then determined as control points. The extracted objects are assigned to objects in the reference image, the objects of one image being assigned to those of the other on the basis of the similarity between the objects and/or on the basis of a vector grid, the vector grid being formed by connecting the characteristic object points. The parameters of the mapping function are adjusted and/or a suitable known mapping function is determined on the basis of the control points and the corresponding points in the reference image. The adjustment and/or determination is carried out in such a way that a minimum cumulative error is achieved for the positional differences between projected control points and the corresponding control points in the reference image. Mapping functions with fixed parameter sets are filed, for example, in a database. An adjustment of this type is of interest when many mapping process of the same type are present. So that suitable objects can be chosen, it is advantageous for the objects to have conspicuous and unique forms, so that gross errors can be excluded. The reference image can be mapped in a known map projection, or it can have an unknown mapping base. If the mapping base is known, the mapping function is taken from the mapping model. If the mapping base of the reference image is unknown, the mapping function will be an affine transformation. By adjusting the parameters to obtain the minimum cumulative error, the mapping function can be adjusted without precise knowledge of the mapping model and/or, if the mapping base is absent, without a mapping model at all.

[0016] In a preferred embodiment, the control points used to adjust the mapping function are weighted in another processing step. For this purpose, areas, i.e., so-called control point structures, are first formed around a possible control point of the image to be rectified and/or of the reference image. The areas are usually rectangular with N*N pixels, where N is any desired natural number. The control point structures are projected into the other image by means of the mapping function, the quality of the control point structure being described at least in terms of its variability, directional contrast, and/or similarity. The variability is derived from the previously determined class membership. Classes with low variability, such as water, generate high quality. Objects with high variability such as vegetation, however, have low quality. The values used for this type of quality evaluation can be determined in advance and filed in tables. Another criterion for the quality of a control point is directional contrast. For an exact determination of the positions of the control points, high contrast within the control point structure is required. Structures with high contrasts in the horizontal (X) and vertical (Y) directions of the image plane thus have high quality. The contrast is calculated on the basis of the maximum gray-scale differences in the rows or columns. Gray-scale values are recorded on one or more channels, e.g., on three channels, as so-called “color values”. In an analysis of the gray-scale differences on a channel, either the contrast found in the image alone or preferably the contrast found in both the image and the associated reference image can enter into the quality index. In addition, similarity is a criterion for the quality index. The degree of similarity is found by comparing the form of the gray-scale curve in the X and Y directions within the control point structures of the gray-scale image with the form found for the reference image. Whereas the variability and the directional contrast in the image to be rectified and/or in the reference image are fixed, the similarity can be improved by making certain adjustments. The individual quality indices are linked to form an overall quality evaluation. There are many different methods that can be used for this type of linkage, and the individual quality indices can be weighted in different ways, depending on the task at hand. On the basis of the overall quality evaluation, a weighting is assigned to each control point, and the control points weighted in this way are then used to adjust the parameters for determining the mapping function. Control points that have a very low quality index should be excluded from any further processing.

[0017] In another embodiment, the positions of the control points obtained in the image to be rectified are determined more precisely by a comparison of the control point structures. It would be possible either to project parts of the control point structure of the image onto the reference image or to do the reverse. Image resolutions are preferably calculated for both image parts, and the less-resolved image part is projected onto the more highly resolved one. First, the gray-scale distribution or gray-scale distributions of the projected image structure are adjusted to the associated control point structure in the image to be rectified and/or in the reference image. It is thus possible to rectify the differences that result, for example, from different lighting conditions during the recording of the image in question. Then the gray-scale values of the individual pixels in the control point structure or image structure are determined, and the differences between adjacent pixels are found. These differences are compared with the differences between the corresponding pixels in the other structure, from which an error value is derived. Then the control point structure in the image to be rectified is shifted in the X or Y direction, and another error value is found. By the stepwise displacement of the control point structure in the X and Y directions, an error matrix is generated. The size of the individual steps is smaller than the dimension of the control point structure. The position of the control point structure that has the smallest error is identified as the best-adjusted position. Positions that have been determined in this way have high positional accuracy. As a result, the adjustment of the imaging parameters can be improved.

[0018] In a preferred embodiment, the parameters of the mapping function are adjusted, the mapping function being changed by changing the parameters in such a way that the cumulative error for the positional differences between the control points and the points of the associated projected image is minimized under consideration of the weighting of the control points. By means of a two-stage adjustment of the mapping parameters, a coarse rectification of the image is performed first, and this is then refined in a second stage. It is also conceivable, however, that the control points are already known within certain limits, which means that the first adjustment step can be omitted.

[0019] In another preferred embodiment, a rectification calculation is carried out which further reduces the residual error in the positional determinations. Here at least one vertical and/or one horizontal correction value is determined for each control point; these correction values embody the deviation of the value of the mapping function from the value of the compensating function at the site of the associated pixel for a projection using the best-adjusted mapping function. The correction values thus obtained can be stored in a correction table and used for further processing. The best-adjusted mapping function is next determined. In many cases, however, a residual error of the best-adjusted mapping function in comparison with the function on which the mapping process is actually based is unavoidable and/or avoidable only with extreme effort. With the help of the correction function taken from the correction table, it is then possible to rectify an image with high quality even if the best-adjusted mapping function still contains residual error.

[0020] In another embodiment, the corner coordinates of projected image elements are determined. An image position of the corner coordinates is obtained from a projection using the determined mapping function and the correction values. By interpolation of the correction values of the control points, it is possible to set up a correction function. Thus the correction values can be determined at the corner coordinates.

[0021] In another embodiment, resampling is then carried out in such a way that the corner coordinates mark out a polygon, preferably a rectangle, and the gray-scale values of all the image elements lying within the polygon, weighted according to their size, are averaged.

[0022] In another preferred embodiment, objects are extracted by a classification process and/or a geometric structure analysis. In the classification process, the properties of the image are analyzed, and objects and/or areas with similar properties are assigned to the same class. Image segments whose elements (pixels) all belong to one class are called objects and/or areas with the same class membership. Classification is based on image properties. These properties can pertain to individual pixels. One property that pertains to a pixel is, for example, a gray-scale value in a channel, which is determined by the reflective behavior of the surface corresponding to the pixel in a defined range of wavelengths. In addition, properties can pertain to several pixels, for which the spatial arrangement of individual gray-scale values plays a role, such as in the case of texture. Another property is obtained from the reflective behavior of the surface in various wavelength ranges (spectral property). On the basis of certain properties, objects can be divided into classes, such as into land use classes such as agriculture, coniferous forest, etc. In particular, clouds and areas with stable reflective behavior such as water or coniferous forests can be recognized in this way. By means of a mask, the image objects which have been recognized as clouds can be labeled. Cloud areas are then excluded from further processing or, if necessary, can be subjected to processing separately. If multispectral remotely-acquired sensing data are available, objects are preferably detected and identified according to a classification method described in German patent DE 199 39 732 C2.

[0023] The objects classified on the basis of their properties can then be classified more accurately by a geometric structure analysis. The geometric structure analysis comprises, for example, an analysis of the objects themselves, including consideration of their size, shape, and/or homogeneity. Preferably information on adjacent objects is also included. For example, image elements in transition regions can be assigned to an object.

[0024] For this purpose, the edge contours of suitable, classified areas are determined in the first step; in addition, a numerical characterization of these objects is done by means of a suitable structure index. The objects which are adjacent to clouds can be labeled separately. These objects are then correlated with corresponding objects of the reference image, so that, as a result, the objects are pre-extracted and identified.

[0025] The positions of the various derived characteristic object points, preferably object center points, are then calculated from the extracted objects. If desired, a search is made for characteristic edge contours in cases where objects are partially covered by clouds.

[0026] The images are generated by any desired recording device, which can produce either single-channel or multi-channel images. A suitable recording device is, for example, a pushbroom camera. The recording device is attached to any desired object that moves relative to the surface to be recorded. It is also conceivable that the surface could be moving relative to the recording device.

[0027] The device according to the invention comprises individual modules. The individual modules can be integrated either into a common computing unit or into separate ones. A suitable computing unit could be, for example, a PC, a workstation, and/or a computer cluster. Some of the steps of the process can be processed in parallel.

[0036] Referring now to FIG. 1, which shows a schematic diagram of the sequence of events that occurs during the automatic rectification of a single-channel or multi-channel image onto a reference image. The method is built in modular fashion by the use of the modules 1-10. All of the information necessary for the rectification process is extractable from the data of the image or of the reference image. The rectification onto the reference image is thus possible without knowledge of the mapping function. If additional information is available, it can also be used in the process. For rectification, it is sufficient that the image and the reference image be present as single-channel, gray-scale images. In modules 1-4, a rough correspondence between image and reference image is achieved with the help of classified objects and their relationships to each other. On the basis of this rough correspondence, a roughly adjusted virtual map is produced in module 5. In modules 6-8, the adjustment of the virtual map is refined. The result obtained from modules 6-8 is a final mapping function with a correction table. Gray-scale percentage values are assigned to surfaces (pixels) in steps 9 and 10.

[0037] The process is illustrated on the basis of the example of the rectification of a satellite image S. The satellite image S is recorded, for example, by a pushbroom camera. The pushbroom camera consists of a row of sensors. By the forward movement of the satellite or of some other moving object such as an aircraft, the satellite image is formed by putting the individual rows together.

[0038] In module 1, the satellite image S is first studied to determine the presence of certain image properties. Relevant properties are, for example, the albedo value, that is, the ratio of reflected radiation to incident radiation; color or gray-scale values; and/or textural properties of a recorded surface. On the basis of these properties, the image objects can be divided into classes, such as into land use classes such as agriculture, coniferous forest, etc. Fractional parts of the image are thus classified in the next step according to their properties, and areas of the same class membership are formed. For the classification process, a suitable classification algorithm is used, by means of which, for example, areas with stable reflective behavior such as the surfaces of bodies of water can be identified.

[0039] Objects are identified in module 2; the edges of the objects are the contours of the areas they form; and a structure analysis model, for example, can be used to give an object a numerical characterization on the basis of its structure index. The edge contours of the objects make it possible to describe the shapes of the individual objects. A description of the various object shapes is a prerequisite for being able to determine the same objects in both the reference image and in the satellite image.

[0040] FIG. 2 illustrates in schematic fashion how, by means of classification and geometric structure analysis, an image O is obtained, from which the edges of the individual objects can be derived.

[0041] In module 3, the center points of the objects are calculated on the basis of the object edges which were previously determined; these center points indicate the positions of the objects. The satellite image S is rectified by a mapping function, either unknown or only partially known, onto a reference image. The object centers and other significant points of the object geometry are used as control points for adjustment of the mapping function.

[0042] In module 4, each of the significant extracted objects according to image O is assigned to an object of the reference image R, and/or extracted objects are identified. It must be remembered here that the objects are shown distorted, and therefore an assignment on the basis of object area shapes is possible to only a limited extent. The job of assigning identifications is therefore accomplished according to the invention with the additional help of vector grids, which can be used to link objects within image O and within the reference image R. To generate a vector grid, at least three significant objects are required. Objects can be spatially assigned and/or identified by an analysis of the geometric similarity of the vector grids.

[0043] FIG. 3 shows in schematic fashion part of an image O and part of a reference image R. In each of the two images, i.e., image O and reference image R, three significant objects O1, O2, O3 and O1′, O2′, O3′ have been discovered, and their center points have been identified as significant object points and selected as control points. The control points are connected by a vector grid. The objects O1, O2, O3 and the vector grid of the image O are distorted in comparison with the reference image. The objects O1, O2, O3 therefore cannot be assigned directly to the objects O1′, O2′, O3′ and thus identified. By means of an analysis of the geometric similarity of the vector grids, however, objects O1, O2, O3 can be assigned spatially to the objects O1′, O2′, O3′ and thus identified.

[0044] In module 5, the unknown or only partially known mapping function is first adjusted to the control points determined in modules 1-4. The original parameter set is calculated from the geometric relationships of the objects to each other. The adjustment is made by changing the parameters of the mapping function.

[0045] FIG. 4 illustrates the determination of a mapping function on the basis of geometric relationships. Images from the Earth's surface 22 are recorded by a satellite 20. A search is made for a mapping function by means of which the geometric relationships of objects O1-O4 of the Earth's surface to each other can be retained as precisely as possible.

[0046] In module 6, additional control points for the adjustment of the mapping function are generated. In principle, any point which can be uniquely identified in both images is suitable as a control point. Potential control points, however, are to be evaluated differently with respect to their suitability. For this purpose, the quality characteristics of the control point structures are determined in module 6. Control point structures are areas with N*N pixels, which lie around a potential control point. The size of the control point structures is a compromise between the ideal, dimensionless control “point” and an image structure which can be recognized again. Suitable control point structures are, for example, 3×3 or 5×5 pixels in size. The quality characteristics are based on variability, directional contrast, and the similarity. The variability of a control point structure is derived from its class membership in correspondence with module 1. A high level of variability has a negative effect on the quality evaluation of the control point. So that the position of the control point can be determined in a following step of the process with a high degree of accuracy, high contrast within the control point structure is advantageous. A high quality index is therefore assigned to a structure with high directional contrasts. The similarity between two structures is very important for comparisons of the projected image with the reference image, because, if the similarity is insufficient, the accuracy of the positioning drops significantly. The quality index is therefore proportional to the similarity. A weighting is assigned to each control point in correspondence with its quality characteristic. Control point structures and the associated quality characteristics are filed in a control point table.

[0047] A new adjustment of the parameters of the mapping function is carried out in module 7 with the larger number of control points and their weighting. This improves the accuracy of the mapping function.

[0048] FIG. 5 illustrates how the adjustment of the parameters improves the determination of the mapping function. The mapping function is determined by a large number of factors in play at the time the image is recorded. For example, the orientation of the satellite 20 has an effect on an image recorded of the Earth's surface 22. The orientation of the satellite 20 and other factors, however, are usually known only within certain limits. The mapping function is adjusted by varying at least one parameter in such a way that the positional error in the image is minimized. The adjustment is done without knowledge of the orientation of the satellite 20 and/or of other factors present at the time the image was recorded.

[0049] Residual positional errors which were not completely avoided by the improved mapping function are dealt with by a compensating calculation in module 8. It is assumed here that the residual positional error will not change sharply over short distances. The compensating calculation therefore includes the near surroundings. As a result of the compensating calculation, an error correction table is constructed in module 8. By interpolation of the error correction table, an error correction function can be found.

[0050] FIG. 6 illustrates the correction table. The correction values are indicated by arrows. Different correction values are assigned to the individual objects O6-O8, as illustrated by the different lengths of the arrows.

[0051] In module 9, corner coordinates of the projected image elements are calculated. The calculation is done with the use of the previously obtained mapping function and the error correction function. Finally, resampling is carried out in module 10, using the previously calculated projected positions.

[0052] FIG. 7 illustrates the rectification of an image obtained on the basis of the mapping function found, with error correction, where an image structure B1 is projected onto a reference structure R1.