Auto-calibration of multi-projector systems

[0001] The invention relates to a method for the automatic calibration of a multi-projector system, which allows to superimpose any number of images produced by different projectors with exact congruence of the pixels in order to increase the total luminosity or to facilitate a stereoscopic representation or to represent an image of several projection images overlapping accurately to the pixel on a (big) screen.

[0002] In order to increase the total luminosity of an image or to produce a passive stereoscopic image, it is necessary that several projectors illuminate the same area of the projection surface. With this image representation, there is the problem that the single images of the projection units normally are not superimposed. Presently, this calibration is made by hand and involves great efforts and is too imprecise.

[0003] For a long time, there has moreover been the desire to create a possibility to increase the pixel resolution of a projection beside the geometry rectification and the increase in luminosity. Since projectors with high resolutions (>1024×768) are very expensive, the obvious solution was to achieve this increase via several projectors. Typically, the associated problem was that it is impossible to position the projectors adjacent to each other without noticeable transition. Mostly, the projectors are oriented obliquely with respect to each other and have a brighter overlapping area.

[0004] These effects have to be compensated by means of a system calibration. The present state of the art is predominated by inflexible expensive manual calibration. There are, however, first attempts at an automatic calibration which, however, are less robust.

[0005] It is aimed at implementing an auto-calibration with exact congruence of the pixels which makes the troublesome manual fine adjustment superfluous. By this calibration, the projection surfaces are oriented to each other, distortions by the geometry of the projection surface are compensated for and the individual projectors are blended into each other such that the overlapping area is no longer perceptible and a projection consisting of n×m projectors appears as a single-projector display to the user.

[0006] U.S. Pat. No. 6,222,593 describes a multi-projector system where the projection image consists of several overlapping partial images each of which is projected by a different projector, the total projection image being picked up by a camera. The control of the projectors is effected in dependence on the respective distance and the respective orientation (rotation about three angles in space) of the projectors relative to the projection surface. Because of this merely global image transformation, the known system cannot be used with arbitrarily shaped projection surfaces.

[0007] Further multi-projection systems are described, for example, in WO-A-00/18139 and WO-A-00/07376 as well. Here, a calibration of the system is effected as well, but it is always assumed that the projection images are distortion-free, i.e., that the projection surface is plane and particularly vertical to the optical axes of the projectors and a camera.

[0008] It is an object of the invention to implement an auto-calibration with exact congruence of the pixels, which makes the troublesome manual fine adjustment superfluous. Additionally, this calibration also compensates for image distortions (with respect to the camera) caused by the geometry of the projection surface and the projector (Keystone), respectively.

[0009] In order to solve the afore-mentioned object, the invention suggests a method for the automatic calibration of a multi-projector system provided with at least two projectors for projecting images onto a projection surface, a digital camera for picking up the projection surface, and a control unit for controlling the projectors and the camera.

[0010] The auto-calibration is performed according to the following steps:

• • [0011] projecting patterns of individual single strips of intersecting lines parallel in groups onto the projection surface by each of the projectors, the positions of the line intersection points of the single-strip patterns in the image to be projected being known,
  • [0012] capturing the projected single-strip patterns by means of the camera,
  • [0013] filtering the captured single-strip patterns for compensating for background noise, ambient light and/or optical distortions of the single-strip patterns caused by the camera and for detecting the line intersection points of the captured single-strip patterns,
  • [0014] detecting the largest projection surface possible on the basis of the captured single-strip patterns,
  • [0015] comparing the positions of the line intersection points within the captured single-strip patterns with the known position of the line intersection points of the single-strip patterns to be projected by the projectors in order to detect correction data for correcting distortions occurring as a consequence of unevennesses of the projection surface,
  • [0016] calculating warp fields and image warping corresponding to the correction data.

[0017] In case of applying the invention to a projection system where the projection images of two (or more) projectors are superimposed as extensively as possible (e.g., for increasing the total luminosity of the projected image or for the stereoscopic representation), the largest projection surface possible is chosen as the largest common rectangular surface possible within the overlapping area of the projection surfaces of all the projectors. On this occasion, the projectors project the same image information onto the projection surface showing the superimposed projection images. Thereby, the luminosity on the projection surface is increased.

[0018] If, however, a projection system is concerned where the individual areas of a total projection image are produced by projectors projecting partial images, the individual partial projection images overlap each other. In this case, the maximum projection surface is that rectangular surface comprising all partial projection surfaces.

[0019] In the invention, standard PC technology, i.e., no expensive special hardware, is used. The projectors used (also referred to as beamers) are computer-controlled and available on the market.

[0020] The filtering of the strip patterns captured by the camera or by a common optical capturing device serves the image processing and compensates for distortions or other optical mistakes/shortcomings of the optical capturing device. When the lines of the strip patters have a width of, e.g., one pixel, that does unfortunately not mean that the lines captured by the capturing device are only one pixel wide; these lines rather have a width of several pixels. But in order to be able to compare the position of the intersection points of the projected lines with those of the intersection points of the captured projected lines, the intersection points (x/y coordinate of a pixel) have to be detected from the Aintersection fields@ by averaging. Besides, it has to be considered that the edges of the lines are serrated in the photograph when the projected lines do not exactly correspond with the row and column alignment of the pixels of the camera, which will be the normal case.

[0021] Hereinafter, the partial steps of the auto-calibration shall be explained with respect to the Figures.

[0040] Hereinafter, a first embodiment of the invention will be explained with reference to FIGS. 1 to 12, wherein the projection images of two projectors are superimposed nearly completely to produce stereoscopic images or projection images of increased total luminosity.

Producing, Capturing and Image Filtering of the Strip Patterns

[0041] To be able to perform a successful geometry rectification by means of image warping, the basic geometry of the projection surfaces has to be detected first. This is done by capturing projected vertical and horizontal strip patterns. These strips have a constant distance which has to be selected in dependence on the complexity of the geometry to be rectified. In the embodiment described herein, a strip distance of 20 pixels has been selected since the wall had many irregularities. Different strip distances and arrangements are possible. From the courses of the lines (curved, straight, oblique) and the line interspaces, the geometry of the projection surface can be inferred. For all projectors, the projected strips are captured by a digital camera (in our case, two projectors meaning two vertical and two horizontal strip images). Additionally, an image is captured where both projectors are switched to black to be able to remove the partly disturbing background and low noise, respectively, from the camera image. Before the image filtering continues, this background image is subtracted from all captured images. The captured images can be seen in FIGS. 2 to 6.

[0042] Due to the use of a lens camera, the captured images have typical radial distortions which have to be removed before the further processing. In case of a distortion-free capturing, this step can be omitted.

[0043] The actual filtering has been effected by standard image processing filter operations. After the image subtraction of image to be filtered and background image, the resulting image is transformed into shades of gray so that each of the three color channels has the same color value. Subsequently, this image is transformed into a black/white image in dependence on a threshold value, the strips being white and the background being black. A dilation and erosion is performed to remove holes in the strips. Subsequently, the lines still have to be filtered such that they are only one pixel wide. This is effected by means of a vertical and horizontal offset filter, respectively. This offset filter is performed in two directions (left and right and up and down, respectively). The average of the results of these two filtering operations yields the center of the strip. Now, the filtered strips are represented by their centers.

[0044] All filtering operations have been performed by means of pixel shaders known per se, with hardware assistance.

Calculation of the Largest Projection Surface Possible

[0045] To guarantee a convergent perception, the largest common projection surface of all projectors has to be determined in the original image ratio (4:3 in this embodiment). To this end, a grid network is produced from the filtered strip patterns first. By means of an optimizing algorithm, the largest common projection surface is looked for within this network which, in our case, consists of two partial grid networks. The grid of the left and right projectors as well as the target projection surface are illustrated in FIG. 7. From these data, the parameters for the warping can be determined subsequently.

Calculation of the Warp Fields and Image Warping

[0046] By means of the afore-calculated grid, the image can be split into warping patches. These patches consist of a dest patch (the image area into which an image section is to be warped) and a src patch (the image area warped into the dest patch).

[0047] The calculation of the corner points of the dest patches is trivial. Thus, the position of all the grid points in the target is known as well due to the given distance of the horizontal and vertical lines.

[0048] The src patches are calculated linearly in dependence on the projection rectangle (in the photo) and the position of the dest patches (in the photo). The calculation for the left lower corner of a src patch in dependence on the left lower corner of a dest patch is indicated:
patch.x1=(pLB.X−projectionRect.left)/(projectionRect.right−projectionRect.left);
patch.y1=(pLB.Y−projectionRect.bottom)/(projectionRect.top−projectionRect. bottom).

[0049] The src patch is transformed into the dest patch by means of a warping process including integrated perspective correction.

[0050] If the distance of the horizontal and vertical lines is selected to be relatively small, it is also possible to realize the compensation of curvatures. It has already been possible to compensate for curvatures of the wall with the line distances of twenty pixels selected in the experiments so that the projected image on the curved projection surface appears to the viewer (from the camera) as being projected onto a plane surface.

[0051] In case of a calibration of two projectors, two warp arrays have to be calculated, in case of more than two projectors correspondingly more. The warp arrays are always determined in dependence on the corresponding grid points, with the restriction of the common largest projection surface. The warped images calculated for a two-channel projection can be seen in FIGS. 8 and 9. The result at the projection surface for one projector can be seen in FIG. 10 and for two projectors in FIG. 11. The comparison of the luminosity is illustrated in FIG. 12.

[0052] To ensure an effective processing, the auto-calibration is made only once (per session). For this reason, the data calculated for the geometry rectification and multi-projector calibration are stored in warp fields which can be used by the X-Rooms™ system (K. Isakovic, T. Dudziak, K. Köchy. X-Rooms. A PC-based immersive visualization environment, Web 3D Conference, Tempe, Ariz., 2002). The rectification by means of the warp fields has been done by hardware assistance again so that a rectification in real time is possible.

Application Fields

[0053] There are many application fields for a multi-projector calibration. It can be used, for example, for a cost-effective luminosity increase or for stereoscopic calibration. Furthermore, a rectification necessitated by the geometry and the projector can be performed without any great problems. Experiments in practice have shown that the auto-calibration of a two-projector system by means of the method according to the invention takes only about 35 seconds including the image capturing (approximately 20 seconds). Thus, the invention represents a significant increase with respect to the manual calibration.

[0054] In the following text, an extension of the afore-described method for the auto-calibration of multi-projector systems is presented with respect to FIGS. 13 to 17. While the possibility of superposing any number of images with exact congruence of the pixels has been described in detail above, the ability of the system of automatically calibrating projector clusters present as n×m projector matrix will be described in the present text.

[0055] For a long time, there has existed the desire of an increase in the pixel resolution of a projection beside the geometry rectification and the increase in luminosity. Since projectors with high resolutions (>1024×768) are very expensive, the obvious solution was to achieve this increase via several projectors. Typically, the associated problem was that it is impossible to position the projectors adjacent to each other without noticeable transition. Mostly, the projectors are oriented obliquely with respect to each other and have a brighter overlapping area (see FIG. 13).

[0056] These effects have to be compensated for by a system calibration. The present state of the art is predominated by inflexible expensive manual calibration. There are, however, first attempts at an automatic calibration which, however, are less robust.

[0057] It is aimed at implementing an auto-calibration with exact congruence of the pixels which makes the troublesome manual fine adjustment superfluous. By this calibration, the projection surfaces are oriented to each other, distortions by the geometry of the projection surface are compensated for and the individual projectors are blended into each other such that the overlapping area is no longer perceptible and a projection consisting of n×m projectors appears as a single-projector-display to the user.

Producing, Capturing and Image-Filtering of the Strip Patterns

[0058] This part of the method has already been described above. Therefore, it is not intended to go into further details here.

Calculation of the Largest Projection Surface Possible

[0059] To guarantee a convergent perception, the largest common projection surface of all projectors has to be determined in the original image ratio. This image ratio is determined from the number of projectors and the arrangement thereof. A cluster with 3 projectors in horizontal direction, for example, has an image ratio of 12:3, the 2×2 system presented in detail herein a ratio of 4:3.

[0060] From the strip patterns filtered before, a grid network is produced again. The largest overlapping projection surface found automatically by the optimization algorithm as well as the grid network on which it is based are illustrated in FIG. 14.

Calculation of the Warp Fields and Image Warping

[0061] The matrices used for the warping of the partial images are calculated according to the above-described method again. The rectified total image (still without blending of the overlapping areas) is shown in FIG. 15.

Blending of the Overlapping Projectors

[0062] The total image produced by the image warping appears already geometrically homogeneous, but has a greater luminosity at the overlapping areas. Therefore, it can still be identified as a composed image very easily. It is the aim of the blending to change this overlapping area such that the image appears homogeneous as to color as well.

[0063] During the pre-processing, a blended image in the color white is produced for each projector. This image has the total resolution (i.e., 2048×1536 pixels in case of 2×2 projectors) and is warped into an image with projector resolution (1024×768) after the blending values have been calculated, and subsequently, it is used as a masking texture (multiplication of the already warped image in current graphic hardware per multi-texturing).

[0064] In the first step for the calculation of the blending values, the grid data of the individual projectors are arranged in a n×m matrix so that they are sorted according to their positions. In the next step, n projectors (2 in our case) are blended with each other horizontally. To this end, a lower and an upper limit is determined across the projection rectangle as well as all the projector grid data (per horizontal strip). These limits lie within the projection surface and within the n projection surfaces to be blended and further, they are parallel to the projection surface. This limit simplifies the blending to be a simple scan line process.

[0065] The horizontal scan line blending is always effected between 2 overlapping images. The right edge is represented by the right edge of the left partial image, the left edge by the left edge of the right partial image. Per line, a linear color value decrease is effected from the starting to the final point. The color values already listed in the blending image—so that a multiple blending is possible as well—and a multiplication factor determined per pixel of the scan line (over the entire scan line of 1.0 60.0) serve as a basis for this color value decrease. Subsequently, the linear blending values determined in this manner are provided with a gamma correction (linBlend^1.0/gamma). This gamma value depends on the projector and can be indicated for each color channel (RGB). The definitely calculated blending value is stored both in the left and the right blending image—depending on the starting and final points of the scan line. Due to the different start and final values (the starting point in the left partial image corresponds to the final value in the right partial image), a so-called cross fade with integrated gamma correction is created.

[0066] Subsequently, all values of the blending image below the lower limit and above the upper limit, respectively, are blackened.

[0067] After the horizontal blending of all strips (2 in our case), a vertical blending still has to be performed. This is also done by the scan line process, here, however, the starting value and final value per vertical scan line are defined by the upper limit of an already horizontally blended strip and the lower limit of the strip arranged above and already blended horizontally.

[0068] The blending images in total projection size now available are transformed into n×m blending images in projector resolution by means of image warping and the already available warp arrays. Now, these blending images can be used in real time for the blending of multi-projector systems by means of multi-texturing.

[0069] The result of the blending of a 2×2 projector cluster can be seen in FIG. 16. By way of example, FIG. 17 shows a surview of all operational steps at a partial image:

• • [0070] distorted partial image on the projection surface (upper line, left picture),
  • [0071] pre-warped partial image (middle line, left picture),
  • [0072] associated blending image (upper line, right picture)
  • [0073] pre-warped partial image multiplied with the blending image (middle line, left picture),
  • [0074] warped, non-blended partial image on the projection surface (lower line, left picture),
  • [0075] warped blended partial image on the projection surface (lower line, right picture).
    Application Fields

[0076] The automatic multi-projector calibration presented herein can be used anyplace where large-scaled projections of high quality are to be produced at low costs and without consuming much time (fairs, showrooms, HD TV, digital cinema). With the calibration presented herein, a geometry rectification as well as an image stitching and an image blending are performed. With the method presented herein, the auto-calibration of a 2×2 cluster takes about 1 minute including the image capturing (about 30 seconds) with standard PC components. Thus, it represents a significant increase over the manual calibration.