Multi-projection image display device

[0002] This invention relates to an image display device which combines two or more images for display. More particularly it relates to a multi-projection image display device which combines images projected from a plurality of projection type display units to form one picture on a screen, and concerns a device or method which is capable of smoothing the seams between images.

[0003] In combining several images, for example, using a multi-projector device composed of a plurality of projectors, it is ideal when the seams between images projected from neighboring projectors can be so smoothly joined as to look invisible, forming one picture (or pictures) on a screen.

[0004] Conventionally, the method as disclosed in Japanese Patent Prepublication No. 94974/96 has been employed to realize this objective. For example, to prevent the projection range of each projector on the screen from overlapping that of an adjacent projector, a shielding plate is installed in the border between neighboring image projection ranges perpendicularly to the screen in such a way that light from one projector does not enter the projection range of the light of an adjacent projector. In addition, transmission type screens with different polarization characteristics are provided between neighboring projection ranges so that light from adjacent projectors cannot reach the viewer physically.

[0005] In the above mentioned optical method for combining neighboring images, however, it is necessary to physically define the image projection range of each projector beforehand. This necessitates adjustment of projectors so that the projected image precisely coincides with the defined projection range. This adjustment includes adjusting the projector angle and the internal optical system of each projector, or in case of cathode-ray tube type projectors, adjusting deflecting voltage waveforms to reshape the projected images.

[0006] However, in case of liquid crystal projectors, the reshaping of projected images is not so easy as in the case of cathode-ray tube type projectors and only optical adjustments are used to change the image projection position and shape, though the freedom of adjustment is limited.

[0007] These adjustment methods are not only troublesome, but they also need frequent readjustments because optimum adjustment levels easily vary with variation in the temperature and the magnetic field in the room in which the adjusted projectors are installed. Therefore, when the conventional methods are used, it is very difficult to display on a screen a whole picture (or pictures) composed of several images with smooth seams.

[0008] Also, installation of shielding plates on the screen might unfavorably affect the uniformity of the screen itself, which might be reflected in the projected images, so that a smoothly joined picture can not be obtained.

[0009] The main object of the this invention is to solve the above-mentioned in a way that problem. Eliminates the need for a special screen structure for prevention of penetration of light from adjacent projectors, as well as precise optical adjustment of projectors, while realizing smoothly joined images in which the seams are not conspicuous. According to the invention, even image projection by projectors can be made by image signal processing.

[0010] To achieve the above stated object, the invention has the following structure.

[0011] According to the invention, adjustment is made on overlapping parts (signals) of several images on a projection surface (for example, a screen). Specifically, in terms of image signals supplied to projectors which project images from the back or front of the screen, a partial image area to be dealt with by each projector is cut out to generate a corresponding image signal, which is processed to perform reshaping or geometric transformation of the projected image and local color correction. This process may be implemented using either hardware or software.

[0012] In addition, the invention is characterized in that, as an image display method employed in a multi-projection image display device provided with a plurality of image output units, on a display screen which displays images projected from said image output units, the image projection area of one of said image output units overlaps that of another image output unit, and said image output unit and said other image output unit deliver mutually correlated. images to the image overlap area.

[0013] Here, a control parameter to be used for the geometric transformation and color correction may be generated as follows: the image signal for a desired image pattern is supplied to each projector, the state of the projected image on the screen is read and the control parameter is calculated from the image state data thus read.

[0014] For each projector, which is in charge of an image projection range adjacent to another image projection range, it is desirable to set its position, angle and optical system so that its maximum image projection range on the screen overlaps that of the adjacent maximum image projection range by a few percent in the border. The areas where neighboring maximum image projection ranges overlap each other are areas where the images projected from the respective neighboring projectors in charge of these projection ranges are optically added or integrated. The state of the image projection on the screen as a result of this optical image addition is read and the read image data is used to calculate control parameters to make an optical image addition model, and this model is used to obtain necessary control parameters for geometric transformation and local color correction of the supplied images (signals) so that smooth connection of neighboring images can be made as a result of optical image addition. According to the obtained geometric transformation and local color correction parameters, image geometric transformation and local color correction are carried out and the processed image signals are supplied to the projectors.

[0015] The invention constitutes a multi-projection image display device to form one or more pictures on a screen, characterized in that image projectors which project images on said screen and a control unit which, when the image projection display range on said screen of one of the projectors has an area which overlaps that of another projector, enables said projector and said other projector to output mutually correlated images to said overlap area on said screen.

[0016] The invention also includes a computer readable storage medium which stores a program to ensure that, on a screen where image output units display images, the image display range of one of said image output units and that of another image output unit partially overlap, and that said image output unit and said other image output unit output mutually correlated images. The invention also constitutes an image signal regenerator which comprises storage media to store image data in divided form for the respective image output units, so that said image output unit and said other image output unit can output mutually correlated images to the area where images from the image output units overlap; and image signal supply units which supply image signals based on image data read from said storage media.

[0017] The invention eliminates the need for readjustments of components, such as the screen and projectors, because it only needs adjustments relating to image overlap areas. Specifically, it involves making an optical image addition model, depending on the conditions of the screen, projectors and other components, and using the model to obtain a solution; as a result, components such as the screen and projectors need not be readjusted as far as an optical image addition model which has a solution can be obtained.

[0018] FIG. 1 is a diagram which shows the arrangement of projectors, a screen, and a screen state monitor camera representing an embodiment of this invention.

[0019] FIG. 2 is a functional block diagram of an image signal control unit.

[0020] FIG. 3 is a diagram which shows positional relationships on the screen between the maximum image projection ranges of projectors and the projection ranges of images processed by the image signal control unit.

[0021] FIG. 4 is a diagram which shows the positional relationship between the maximum image projection range of a single projector and the projection range of an image processed by the image signal control unit.

[0022] FIG. 5 is a functional block diagram of an image corrector.

[0023] FIG. 6 is a more detailed functional block diagram of the image corrector shown in FIG. 5.

[0024] FIG. 7 is a flow chart of the operational sequence for a video signal digitizer.

[0025] FIG. 8 is a flow chart of a frame buffer writing or reading sequence.

[0026] FIG. 9 is a flow chart of the sequence for geometric transformation of images.

[0027] FIG. 10 is a diagram which shows a geometric transformation vector table data structure.

[0028] FIG. 11 is a flow chart of a color transformation sequence.

[0029] FIG. 12 is a diagram which shows a color transformation parameter table data structure.

[0030] FIG. 13 is a diagram which shows an example of an image pattern for correction.

[0031] FIG. 14 is a diagram which shows the image projection ranges before and after geometric transformation.

[0032] FIG. 15 is a diagram which shows four types of overlap of two neighboring maximum image projection ranges.

[0033] FIG. 16 is a flow chart of the sequence for geometric transformation control data generation.

[0034] FIG. 17 is a flow chart of the sequence for geometric transformation vector generation.

[0035] FIG. 18 is a flow chart of sequence for pixel data conversion parameter generation.

[0036] FIG. 19 is a diagram which shows an example of a picture composed of four mutually overlapping quarter images.

[0037] FIG. 20 is a diagram which shows the relationship of overlap areas of neighboring partial images.

[0038] FIG. 21 is a graph showing a weight function used for brightness modulation of image overlap areas.

[0039] FIG. 22 is a block diagram showing projectors with image correctors built in and an image signal control unit.

[0040] FIG. 23 is a diagram which shows an example of a multi-projection image display device that employs a curved surface screen.

[0041] FIG. 24 is a block diagram showing a set of image reproduction units that reproduce partial image signals in parallel and an image signal control unit that has no image correctors.

[0042] FIG. 25 is a block diagram of an image reproduction device that incorporates an image corrector and a temporary image storage unit.

[0043] FIG. 1 shows a system having an array of projectors according to this invention. Four projectors 0121, 0122, 0123, 0124 are arranged in such a way that their maximum image projection ranges 0151, 0152, 0153, 0154 overlap a bit one on another on a rear projection screen 0140. Image signals are supplied from an image signal control unit 0110 to the four projectors 0121, 0122, 0123, 0124. The image signal control unit 0110 operates to process image signals supplied from an external image input 0180 and supply such processed signals to the four projectors 0121, 0122, 0123, 0124 and performs an image correction function to determine how the images are processed. The way the image signal control unit 0110 corrects images is determined according to image signals from a screen state monitor camera 0130 that reads the state of images projected on the screen 0140.

[0044] FIG. 2 is a block diagram showing the functional structure of the image signal control unit 0110. The image signal control unit has the following input and output terminals: a whole image input 0210, partial image inputs 0221, 0222, 0223, 0224, a screen state monitor camera image input 0230, and partial image outputs 0241, 0242, 0243, 0244.

[0045] The image signal supplied from the whole image input 0210 is split by an image splitter 0250 into partial image signals that correspond to the projectors connected to the partial image outputs 0241, 0242, 0243, 0244, respectively; and, these partial image signals are each sent to an image signal selector 0260.

[0046] The image signal selector 0260 selects one of the following image signal sets: the image signal set as input from the image splitter 0250, the image signal set as input from a test pattern generator 0270, and the set of partial image inputs 0221, 0222, 0223, 0224. Then, it supplies the selected set of image signals to the corresponding image correctors 0281, 0282, 0283, 0284, respectively.

[0047] An image correction parameter generator 0290 controls the test pattern generator 0270 to select geometric pattern image signals by means of the image signal selector 0260 and send them through the image correctors 0281, 0282, 0283, 0284 to the projectors 0121, 0122, 0123, 0124 connected to the partial image outputs 0241, 0242, 0243, 0244; and, it loads image signals from the screen state monitor camera 0130 which monitors, through a screen state monitor camera image input 0230, the projection of that geometric pattern on the screen. Based on the image obtained through the screen state monitor camera image input 0230, the image correction parameter generator 0290 works out image correction parameters that enable the images projected from the projectors 0121, 0122, 0123, 0124 to be smoothly connected with least conspicuous seams on the screen with smooth brightness distribution. The worked-out parameters are sent through image correction control lines 0295 and loaded on the image correctors 0281, 0282, 0283, 0284.

[0048] FIG. 3 shows the relationship between the maximum image projection ranges 0151, 0152, 0153, 0154 as shown in FIG. 1 and the projected images as processed by the image signal control unit 0110. The maximum image projection ranges 0301, 0302, 0303, 0304 represent the areas of the maximum images that the projectors 0121, 0122, 0123, and 0124 can project on the screen 0140, respectively. The projectors 0121, 0122, 0123, 0124 are positioned so that their maximum image projection ranges 0301, 0302, 0303, 0304 overlap one another, with overlap areas being disposed between neighboring projection ranges.

[0049] The optimized image projection ranges 0311, 0312, 0313, 0314, which are inside the corresponding maximum image projection ranges 0301, 0302, 0303, 0304, represent image display areas which enable the images projected from neighboring projectors to look smoothly joined.

[0050] FIG. 4 is an enlarged view of one of the maximum image projection ranges shown in FIG. 1. A maximum image projection range border 0401 indicates the range within which a projector can display an image. An optimized image projection range border 0402 shows the area to which the image supplied to the projector is sized down so that images projected from neighboring projectors look smoothly joined. The image signal control unit 0110 processes the input image signal so that the zone 0403 disposed between the maximum image projection range border 0401 and optimized image projection range border 0402 has a pixel value which represents the darkest brightness. This prevents more than one projected image from being displayed in this zone because it is included in the area where the neighboring projector can project an image.

[0051] FIG. 5 is a functional block diagram of an image corrector. An analog image signal is inputted through an image signal input terminal 0501 and converted by a video signal digitizer 0510 into digital data to be processed within the image corrector. The image data in the digitized form for each image frame is carried through an image data transmission line 0561 to a frame buffer block 0520 where it is stored.

[0052] Geometric transformation and color correction parameters for the image are inputted through a control parameter input terminal 0502, carried through a control parameter transmission line 0567 and loaded into a geometric transformation block 0530 and a color transformation block 0540, respectively. In order to modify the output image shape according to the loaded parameter data, the geometric transformation block 0530 loads, through an image data write address transmission line 0563, the address of a pixel to be outputted from an image signal output terminal 0503, calculates the address for the input image pixel to be outputted to that address, and sends the calculation result onto an image data read address transmission line 0565; as a result the corresponding input image pixel data stored in the frame buffer block 0520 is read out.

[0053] The pixel data thus read is color-transformed by the color transformation block 0540 according to the preset parameter data and transferred to a video signal generator 0550, which then outputs it as a video signal. The algorithm of operation of each functional block shown in FIG. 5 will be explained, with reference to FIG. 6 that shows the structure of the implemented functional blocks in FIG. 5, as well as a PAD diagram. In FIG. 6, the video signal digitizer 0610 incorporates a signal splitter 0611 as a means for splitting the input signal from an image signal input terminal 0601 an image color signal a synchronizing signal; an A/D converter 0612 as a means for quantization an image color signal voltage; and an address generator 0613 as a means for generating information on the position of a currently quantized pixel within the image.

[0054] The frame buffer block 0620 consists of two frame memories A and B (0621 and 0622), a write address selector 0625, a read address selector 0626 and a read data selector 0627. The frame memories A and B each store image data for one frame and the write address selector 0625 selects which frame memory is to be used for writing input pixel data and sends the write address and write timing to the selected frame memory. The read address selector 0626 selects, as a read memory, the frame memory where data is not written, and sends the address of a pixel to be read to the selected frame memory, and the read data selector 0627 selects the pixel data sent from the read frame memory and outputs it. The geometric transformation block 0630 contains an address generator 0631 and an address displacement memory 0632. The address generator 0631 calculates which address in the frame memory chosen as a read memory should be used for output of the stored pixel data when the image corrector is going to output the pixel into a position within the image. The address displacement memory 0632 stores all necessary parameters for the calculation for each output pixel position. The color transformation block 0640 has a pixel data converter 0641 and a pixel data conversion parameter memory 0642, where the pixel data converter calculates pixel data to be outputted by the image corrector according to the pixel data read from the frame buffer block 0620 and the pixel data conversion parameter memory 0642 memorizes how pixel data conversion should take place for each output pixel position.

[0055] The video signal generator 0650 has a D/A converter 0651 and a video signal synthesizer, where the D/A converter converts output pixel data into a image color signal and the video signal synthesizer generates a synchronizing signal based on pixel output position data and combines it with the image color signal for conversion into a video signal. FIG. 7 shows the operational sequence for the video signal digitizer 0610. In step 0700, an image signal is split into an image color signal a synchronizing signal. In step 0701, the image color signal is quantized. In step 0702, a pixel address (pixel column and row data) is generated according to the synchronizing signal.

[0056] The numeric image data thus quantized is first stored in the frame buffer block 0620. For the stored numeric image data, the geometric transformation block 0630 generates a read address for geometric transformation and the pixel data read undergoes color correction by the color transformation block 0640.

[0057] FIG. 8 shows the operational sequence for the frame buffer block 0620. Steps 0800 and 0801 are performed to choose one of the two frame buffer memories for reading and the other for writing. In step 0802, a waiting time is taken until data to be written at the write starting position in the top left corner of the screen is supplied to the frame buffer block 0620. In step 0805, pixel data is written in the write frame buffer memory. In step 0806, pixel data is read from the read frame buffer memory according to the address sent from the geometric transformation block 0630, and the pixel data is transferred through the read data selector 0627 to the color transformation block 0640. Steps 0805 and 0806 are repeated until all pixels for a whole image frame are written, and then step 0807 is performed to switch the functions of the write and read frame buffer memories. Then, the steps from 0805 are repeated.

[0058] FIG. 9 shows the operational sequence for the geometric transformation block 0630. In step 0901, a geometric transformation vector table as shown in FIG. 10 is loaded into the address displacement memory 0632.

[0059] The geometric transformation vector table is formed of two-dimensional data composed of two-dimensional vectors which correspond to pixels in one frame of the image on the basis of one vector for one pixel. In step 0903, the row and column data (x[r, c], y[r, c]) of the geometric transformation vector table stored in the address displacement memory 0632 is read for each of the addresses (r, c) supplied in sequence from the video signal digitizer 0610. In step 0904, (r, c) +(x[r, c], y[r, c]) is calculated and the result is outputted as the read address for the frame buffer block 0620. Steps 0903 and 0904 are repeated as long as re-initialization of the address displacement memory 0632 is not requested from the control parameter input terminal 0602. If such re-initialization is requested, the steps from step 0901 are repeated again.

[0060] FIG. 11 shows the operational sequence for the color transformation block 0640. In step 1101, a color transformation parameter table as shown in FIG. 12 is loaded into the pixel data conversion parameter memory 0642 and initialized. The color transformation parameter table is formed of two-dimensional data for three planes which consist of parameters as a result of approximation as N piecewise linear functions of the respective transformation functions for the three color components (red, blue, green) of each pixel in one frame of the image. In step 1103, regarding each of the addresses (r, c) supplied in sequence from the video signal digitizer 0610, color transformation parameters are read for each of the red, blue and green components at the address (row and column) of the color transformation parameter table in the pixel data conversion parameter memory 0642. If the intensity of each color component of the pixel outputted from the frame buffer block 0620 is expressed as “z”, the intensities of the color components are transformed using a transformation equation (equation 1) and transferred to the video signal generator 0650 (step 1104). $\begin{array}{cc}f\left(z\right)=\left(q_{r,c}^{i+1}-q_{r,c}^i\right)\frac{z-p_{r,c}^i}{p_{r,c}^{i+1}-p_{r,c}^i}+q_{r,c}^i\mathrm{if}z\in \left[p_{r,c}^i,p_{r,c}^{i+1}\right]& \left[\mathrm{eq}.1\right]\end{array}$

[0061] Steps 1103 and 1104 are repeated as long as the control parameter input terminal 0602 does not request re-initialization of the pixel data conversion parameter a memory 0642. If such request is made, the steps from step 1101 are repeated.

[0062] The pixel data supplied from the color transformation block 0640 is converted into an analog video signal by the video signal generator 0650 and the pixel data is outputted through the image signal output terminal 0603. Parameter data to be loaded into the geometric transformation block 0630 and color transformation block 0640 as shown in FIGS. 10 and 12 is created by the image correction parameter generator 0290 according to the state of the image on the screen 0140 as monitored by the screen state monitor camera 0130. The geometric transformation parameter as shown in FIG. 10 is calculated as follows.

[0063] A pattern for correction as shown in FIG. 13 is projected from the projectors through the image correctors 0281, 0282, 0283, 0284 onto the screen and the projected image is photographed by the screen state monitor camera 0130 and an apex 1301 or similar point is chosen as a characteristic point in the image. Assuming that, as for a characteristic point like this, position coordinates q in the camera coordinate system can be correlated with coordinates p in the frame memory of the pixel corresponding to that point, position coordinates p in the coordinate system of the frame memory corresponding to an arbitrary point q in the camera coordinate system for an area where no such pattern is displayed can be calculated as follows. Regarding q_1, q_2 and q_3 as three points near q, the points that correspond to coordinates p_1, p_2 and p 3 in the frame memory coordinate system are calculated. Then, if the relation between q, and q_1, q_2, q_3 is expressed by equation 2 which uses appropriate real numbers a and b, p can be represented by equation 3.

[0064] q=q_c+a(q_a−q_c)+b(q_b−q_c)  [eq. 2]

[0065] p=p_c+a(p_a−p_c)+b(p_b−p_c)  [eq. 3]

[0066] As shown in FIG. 14, if the maximum image projection ranges 1401, 1402, 1403, 1404 of the four projectors overlap one another, the coordinates of the apex 1421 (of the maximum image projection range 1402 of an adjacent projector B) which exists within the maximum image projection range 1401 of projector A are read as coordinates in the frame memory coordinate system of the image corrector connected to projector A, using the above-mentioned coordinates transformation method. In other words, it is possible to define an isomorphism Ψ_{AB} which correlates coordinate value x_B of an arbitrary point on the screen, in the frame memory coordinate system of the image corrector connected to projector B, with coordinate value x_A in the frame memory coordinate system of the image corrector connected to projector A.

[0067] Therefore, the coordinate values of an arbitrary position within the projected image discussed below are assumed to be coordinate values in the frame memory coordinate system of the image corrector connected to one of the projectors. Here, the maximum image projection ranges of two neighboring projectors which border on each other horizontally or vertically are represented by S_A and S_B, respectively and the partial border lines in this adjacent area which should border on each other are represented by B_{AB} and B_{BA} respectively.

[0068] In FIG. 14, these partial border lines correspond to, for example, the partial border line 1413 of the maximum image projection range 1401 with apexes 1411 and 1412 as end points and the partial border line 1423 of the maximum image projection range 1402 with apexes 1421 and 1422 as end points. Partial border lines B_{AB} and B_{BA} are expressed as point-sets by equations 4 and 5.

[0069] {b_AB(t)|tε[0,1]}  [eq. 4]

[0070] {b_BA(t)|tε[0,1]}  [eq. 5]

[0071] In border lines B_{AB} and B_{BA}, the segments which are included in or border on the adjacent maximum image projection range are called effective partial border lines. Here, the respective border line segments are defined by equations 6 and 7 as follows.

[0072] {b′_AB(t)=b_AB((1−t)fi_AB+tfÀ_AB)|tε[0,1]}  [eq. 6]

[0073] {b′_BA(t)=b_BA((1−t)fi_BA+tfÀ_BA)|tε[0,1]}  [eq. 7]

[0074] where α_{AB} and β_{AB} in equation 6 are defined by equation 8 for the four cases as shown in Table 1.

[0075] $\begin{array}{cc}\left({\alpha }_{\mathrm{AB}},{\beta }_{\mathrm{AB}}\right)=\{\begin{array}{c}\left(0,1\right)\text{:}\mathrm{case1}\\ \left(0,c\right)\text{:}\mathrm{case2}\\ \left(c,1\right)\text{:}\mathrm{case3}\\ \left(c_0,c_1\right)\text{:}\mathrm{case4}\end{array}& \left[\mathrm{eq}.8\right]\end{array}$

[0076] α_{BA} and β_{BA} are defined in the same way as above.

[0077] The four cases for equation 8 correspond to the four types of overlap of two neighboring maximum image projection ranges as shown in FIG. 15, where the upper partial border line and lower one which join the partial border line B_{BA} of range B, are represented by B_{AB−} and B_{AB+}, respectively.

[0078] Including the possibility that the adjacent maximum image projection range S_B is empty, there are four possible states of a partial border line end point b_{AB} (a) (aε{0, 1} as shown in Table 2.

[0079] Here, the destination point, e_{AB} (a), to which the border line end point b_{AB} (a) moves, is defined by equation 9. $\begin{array}{cc}{e}_{\mathrm{AB}}\left(a\right)=\{\begin{array}{cc}\mathrm{barycenter}\mathrm{of}{S}_A\bigcap S_B\bigcap S_C\bigcap S_D& \mathrm{case}\alpha \\ \frac{b_{\mathrm{AB}}^{\prime }\left(a\right)+b_{\mathrm{BA}}^{\prime }\left(1-a\right)}{2}& \mathrm{case}\beta \\ e_{\mathrm{AC}}\left(1-a\right)& \mathrm{case}\gamma \\ b_{\mathrm{AB}}\left(a\right)+{\Delta }_1{b}_{\mathrm{AB}}\left(1-a\right)+{\Delta }_2{b}_{\mathrm{AB}}\left(a\right)& \mathrm{case}\delta \end{array}& \left[\mathrm{eq}.9\right]\end{array}$

[0080] In equation 9, b_{AB} (1−a) corresponds to Δ_1 b_{AB} (1−a)=0 in case δ, while it is as defined by equations 10 and 11 in the other cases.

[0081] Δ₁b_AB(1−a)=P(e_AB(1−a)−b_AB(1−a),b_AC(1−a)−b_AC(a))  [eq. 10]

[0082] $\begin{array}{cc}P\left(r,s\right)=\frac{r*s}{{\left|s\right|}^2}s& \left[\mathrm{eq}.11\right]\end{array}$

[0083] b_{AC} (a) corresponds to Δ_2 b_{AB} (a)=0 in case δ, while it is as defined by equation 12 in the other cases.

[0084] Δ₂b_AB(a)=P(e_AC(a)−b_AC(a),b_AB(1−a)−b_AB(a))  [eq. 12]

[0085] The effective partial border line end point movement vectors are defined by equations 13 and 14.

[0086] d_AB(a)e_AB(a)−b′_AB(a)  [eq. 13]

[0087] d_BA(a)e_BA(a)−b′_BA(a)  [eq. 14]

[0088] where a=0 or a=1. The movement direction vectors of the entire effective partial border line are defined by equations 15, 16 and 17.

[0089] d_AB(t)=I(d_AB(0),d_AB(1);t)  [eq. 15]

[0090] d_BA(t)=I(d_BA(0),d_BA(1);t)  [eq. 16]

[0091] I(p,q;t)=(1−t)p+tq  [eq. 17]

[0092] Subsequently, using the solution of the vector equation with regard to A(t) and B(1−t) in equation 18, shared border lines G_{AB} and G_{BA} corresponding to partial border lines B_{AB} and B_{BA} are defined by g_{AB} (t) and g_{BA} (t) in equations 19 and 20.

[0093] b′_AB(t)+A(t)d_AB(t)=b′_BA(1−t)+B(1−t)d_BA(1−t)  [eq. 18]

[0094] g_AB(t)=b′_AB(t)+A(t)d_AB(t)  [eq. 19]

[0095] g_BA(t)=b′_BA(t)+B(t)d_BA(t)  [eq. 20]

[0096] In the example shown in FIG. 14, 1430 represents the shared border line with regard to the partial border lines 1413 and 423 of the maximum image projection ranges of projectors A and B. A range enclosed by shared border lines as calculated in this way is called an optimized image projection range; for example 1441, 1442, 1443, 1444 in FIG. 14 are optimized image projection ranges.

[0097] Here, the partial border lines of maximum image projection range S_A which should coincide with the partial border lines of the four projection ranges S_B, S_C, S_D and S_E adjacent to S_A are represented by B_{AB}, B_{AC}, B_{AD} and B_{AE}, respectively.

[0098] The shared partial border line with regard to B_{AX} (Xε{B, C, D, E}) is expressed as G_{AX} and defined by equation 21.

[0099] G_AX={g_AX(t)|tε[0,1]}  [eq. 21]

[0100] The range enclosed by border line=$G_{AX}$ is called an optimized image projection range for the maximum image projection range S_A, and is expressed as Q_A.

[0101] T_A: S_A→Q_A, which denotes image transformation from maximum image projection range S_A to optimized image projection range Q_A, is defined as follows.

[0102] First, an isomorphism from S_A=[0, 640]×[0, 480] to D_2=[0, 1]×[0, 1] is expressed as π_A and is defined by equation 22.

[0103] π_A(x,y)=(x/640,y/320)  [eq. 22]

[0104] Next, an isomorphism from Q_A to D_2, expressed as ξ_A, is defined by equations 23, 24, 25 and 26 with respect to ∂ Q_A.

[0105] ξ_A(g_AB(t))=(t,0)  [eq. 23]

[0106] ξ_A(g_AC(t))=(1,t)  [eq. 24]

[0107] ξ_A(g_AD(t))=(1−t,1)  [eq. 25]

[0108] ξ_A(g_AE(t))=(0,1−t)  [eq. 26]

[0109] Here, continuous transformation φ: D_2→E_2 with respect to ∂ D_2 is defined by equation 27, where E_2 represents two-dimensional vector space.

[0110] φ_A(u,v)=ξ_A¹(u,v)−π⁻¹(u,v)  [eq. 27]

[0111] Then, φ (u, v) within D_2 is defined by equation 28. $\begin{array}{cc}{\phi }_A\left(u,v\right)=I\left(I\left({\phi }_A\left(u,0\right),{\phi }_A\left(u,1\right);v\right)\right),I\left({\phi }_A\left(0,v\right),{\phi }_A\left(1,v\right);u\right);\frac{\rho \left(v\right)}{\rho \left(u\right)+\rho \left(v\right)}& \left[\mathrm{eq}.28\right]\end{array}$

[0112] where ρ (x) denotes a continuous function, which is zero if x=0 or x=1, and defined, for example, by equation 29.

[0113] ρ(x)=x(1−x)  [eq. 29]

[0114] Here, T_A: S_A→Q_A for geometric transformation is defined by equation 30.

[0115] T_A(x)=φ_A(π(x))+x  [eq. 30]

[0116] Furthermore, the elements of the geometric transformation vector table in FIG. 10 are generated by transformation vector function U_A as defined by equation 31. $\begin{array}{cc}{U}_A\left(x\right)=\{\begin{array}{cc}{T}_A^{-1}\left(x\right)-x& \text{:}x\in Q_A\\ 0& \text{:}\mathrm{otherswise}\end{array}& \left[\mathrm{eq}.31\right]\end{array}$

[0117] FIG. 16 shows the control data generating sequence for geometric transformation in the image correction parameter generator 0290.

[0118] In step 1600, geometric transformation parameters for image correctors 0281, 0282, 0283, 0284 are set to zero. For each projector, the steps, 1602, 1603, 1604, 1605 and 1606, are carried out. In step 1602, an image of the pattern as shown in FIG. 13 is projected at the maximum size on the screen. In step 1603, the projected image in step 1602 is photographed by the screen state monitor camera 0130. In step 1604, the points corresponding to the apexes of the checkered pattern in FIG. 13 are read with respect to the camera coordinate system from the shot taken in step 1604. In step 1605, the characteristic point position coordinates are transformed to match the coordinate system of the image corrector frame memories. In step 1606, the geometric transformation vector generating sequence as shown in FIG. 17 is executed.

[0119] FIG. 17 shows the geometric transformation vector generating sequence. In step 1700, the point set on the border lines of the maximum image projection ranges defined by equations 4 and 5 is produced. In steps 1701 and 1702, effective partial border lines are calculated. In step 1703, the points to which the effective partial border line end points move are worked out. In step 1704, the end point movement vectors for the effective partial border lines are calculated.

[0120] In step 1705, the movement direction vectors for the effective border lines are worked out. In step 1706, the border lines of optimized image projection ranges are calculated. In step 1707, geometric transformation control parameters for the entire maximum image projection ranges are calculated.

[0121] The color reproduction quality of projected images varies from one projector to another and also depending on which position on the screen the image is projected on. The function of the color transformation block 0540 of each image corrector is to suppress these two types of variation and control the color reproduction quality so that the projected images from two neighboring projectors join end to end smoothly. The sequence for generating pixel data conversion parameters to be loaded into the color transformation block 0540, as shown in FIG. 12, will be explained next.

[0122] Here, on the above-said shared border lines included in the overlap area of the maximum image projection ranges of neighboring projectors $A$ and $B$, it is assumed that if z represents the color component value in displayed image data corresponding to the position where both the images should coincide ideally, f_X(z) represents the color component value as measured by the screen state monitor, camera 0130 with respect to z, where X denotes A or B.

[0123] Hence, optimized color component function $g(z) $ is defined by equations 32 and 33.

[0124] g(z)=(y_H−y_L)L(h(z);h(z₀),h(z_N))+y_L  [eq. 32]

[0125] $\begin{array}{cc}L\left(x;a,b\right)=\frac{x-b}{a-b}& \left[\mathrm{eq}.33\right]\end{array}$

[0126] Here, y_H, y_L and h(z) are defined by equations 34, 35 and 36, respectively.

[0127] y_H=min(f_A(z_H),f_B(z_H))  [eq. 34]

[0128] y_L=min(f_A(z_L),f_B(z_L))  [eq. 35]

[0129] $\begin{array}{cc}h\left(z\right)=\frac{f_A\left(z\right),f_B\left(z\right)}{2}& \left[\mathrm{eq}.36\right]\end{array}$

[0130] The functions ξ_X(z) for color junction on the border line is defined by equation 37.

[0131] g(z)=f_x(ξ_x(z))  [eq. 37]

[0132] This color junction functions ξ_X(z) is approximated as an N piecewise linear function expressed by equation 1. In this case, the parameter for the i-th segment of the N piecewise linear function is given by equation 38.

[0133] (z_i,f_x⁻¹(g(z_i)))  [eq. 38]

[0134] The color junction function for each point on the shared border line is defined in the above-mentioned way. Supposing that the color transformation function for each point (u, v) within D_2 is represented by ξ_{X[u, v]} (z), the parameter data on the border line as defined by equation 38 is used to define the parameter data for the entire D_2 area using equation 39 in the same way as equation 28. $\begin{array}{cc}{\zeta }_{X\left[u,v\right]}\left(z\right)=I(I\left({\zeta }_{X\left[u,0\right]}\left(z\right),{\zeta }_{X\left[u,1\right]}\left(z\right);v\right),I\left({\zeta }_{X\left[0,v\right]}\left(z\right),{\zeta }_{X\left[1,v\right]}\left(z\right);u\right);\frac{\rho \left(v\right)}{\rho \left(u\right)+\rho \left(v\right)}& \left[\mathrm{eq}.39\right]\end{array}$

[0135] Then, the color junction function η_{X[x, y]} (z) for a position (x, y) in the maximum image projection range is defined by equation 40. $\begin{array}{cc}{\eta }_{X\left[x,y\right]}\left(z\right)=\{\begin{array}{cc}{\zeta }_{X\left[{\xi }_x\left(x,y\right)\right]}\left(z\right)& \text{:}\left(x,y\right)\in Q_X\\ 0& \text{:}\mathrm{otherwise}\end{array}& \left[\mathrm{eq}.40\right]\end{array}$

[0136] The piecewise linear approximate parameter for the defined η_{X[x, y]} (z) is considered as the pixel data conversion parameter for a position (x, y) within the maximum image projection range.

[0137] FIG. 18 shows the pixel data conversion parameter generating sequence in the image correction parameter generator 0290. In step 1800, parameters appropriate to identity transformation are loaded as defaults. The steps 1802 through 1807 are performed for each projector. Step 1802 is a step to control the repetition of steps 1803 through 1806 for each sample point on the borderline of the optimized image projection range. In step 1803, the brightness of each color component is measured. In step 1804, an N piecewise linear approximate function for the measured color component, f_X, is calculated. In step 1805, the optimized color component function is calculated. In step 1806, a pixel data conversion parameter on the border line is calculated. In step 1807, a pixel data conversion parameter for the entire optimized image projection range is calculated.

[0138] This embodiment produces an effect that neighboring projected images join end to end precisely without any gaps oroverlaps, and color discontinuity at the joint of the two images can be eliminated. Further, the required amount of image reshaping and the required pixel color transformation function are automatically calculated, which relieves the operator from the drudgery of manual repositioning of projectors and color adjustments.

[0139] As described next, a second embodiment of this invention has the same structure as the first embodiment except that the functionality of the image splitter 0250 as shown in FIG. 2 and the methods for calculating geometric transformation control parameters and pixel data conversion parameters are different.

[0140] As shown in FIG. 19, the image splitter 0250 splits the image 1901 from the input image signal into partial images 1911, 1912, 1913, 1914 that each have a given amount of overlap with the other adjacent partial images, and generates image signals corresponding to these partial images.

[0141] The ratio of overlap width (W) to image length (L), W/L, is set to a ratio smaller than that of neighboring maximum image projection ranges. Conversely speaking, when this ratio W/L is so set, the projectors 0121, 0122, 0123, 0124 are positioned and angled so that the ratio of overlap of maximum image projection ranges 0151, 0152, 0153, 0154 is larger than the ratio of overlap of partial images W/L.

[0142] The area in each partial image which is enclosed by dividing centerlines 1921 and 1922 is called a major divisional area. In FIG. 19, the hatched areas 1931, 1932, 1933, 1934 are major divisional areas.

[0143] In this embodiment, the border lines of the major divisional areas 1931, 1932, 1933, 1934 are substituted for the partial border lines defined by equations 4 and 5 in the geometric transformation control parameter calculation as used in the first embodiment. Since geometric transformation T for points in the major divisional areas can be calculated using equation 30 as in the first embodiment, geometric transformation control parameters are calculated using equation 31. Calculation of geometric transformation control parameters for areas outside the major divisional areas is performed as follows.

[0144] In neighboring partial images A and B as shown in FIG. 20, it is supposed that the areas which overlap each other are represented by C_{AB} and C_{BA}, and the non-overlap areas by F_{AB} and F_{BA}. The C_{AB} and C_{BA} are each divided by the dividing centerlines 2001 and 2002 into two parts, which are expressed by equations 41 and 42, respectively.

[0145] C_AB=D_AB∪E_AB  [eq. 41]

[0146] C_BA=D_BA∪E_BA  [eq. 42]

[0147] Here, E_{AB} and E_{BA} each represent the sides of the overlap areas C_{AB} and C_{BA} (divided by the dividing centerlines 2001 and 2002) that border on F_{AB} and F_{BA}, respectively. Hence, the major divisional areas of the partial images A and B are expressed as E_{AB} ∪ F_{AB} and E_{BA} ∪ F_{BA}, respectively.

[0148] As indicated in the explanation of the first embodiment, Ψ_{AB}, a function for transforming coordinates in the coordinate system of the partial image B into those of the partial image A, and its inverse function Ψ_{BA} can be defined. Hence, the composite function V_{AB} to be defined on D_{AB} is defined by equation 43 using the geometric transformation function T_B for the partial image B as defined on E_{BA} ∪ F_{BA} by equation 30.

[0149] V_A(x)=Ψ_AB(T_B(Ψ_BA(x)))  [eq. 43]

[0150] A transformation vector function equivalent to equation 31, U_A, is defined by equation 44. $\begin{array}{cc}{U}_A\left(x\right)=\{\begin{array}{cc}{T}_A^{-1}\left(x\right)-x& \text{:}x\in F_A\bigcup E_{\mathrm{AB}}\\ V_{\mathrm{AX}}^{-1}\left(x\right)-x& \text{:}x\in D_{\mathrm{AX}}\\ 0& \text{:}\mathrm{otherwise}\end{array}& \left[\mathrm{eq}.44\right]\end{array}$

[0151] Next, pixel data conversion parameters are calculated with the following procedure. First, for each point in the overlap area C_{AB}, the piecewise linear function defined by the parameter indicated in equation 38 is used to define the pixel data conversion function, as in the first embodiment.

[0152] F_A∩_XF_AX  [eq. 45]

[0153] Then, regarding all neighboring partial images X, in the inside of F_A, an area which does not overlap any adjacent partial image, as expressed by equation 45, the pixel data transformation function is defined by interpolation with equation 39 using the pixel data conversion function as defined on the border as mentioned above. Thus, pixel data conversion function η_{A [x, y]} (z) can be defined by equation 46.

[0154] η_A[x,y](z)=κ_A(x,y)ξ_A[ξ(x,y)](z)  [eq. 46]

[0155] Here, weight function κ_{A} (x, y) is defined by equation 47 and is the product of σ_{A} (x, y) with regard to all partial images X adjacent to the partial image A. $\begin{array}{cc}\kappa \left(x,y\right)=\{\prod _X^1{\sigma }_{\mathrm{AX}}\left(x,y\right)\begin{array}{c}:\left(x,y\right)\in F_A\\ :\left(x,y\right)\in \exists C_{\mathrm{AX}}\end{array}& \left[\mathrm{eq}.47\right]\end{array}$

[0156] σ_{A} (x, y) is a function defined on C_{AB} that has a value as shown in FIG. 21 at point x(t) on line 2010 vertical to the dividing centerline 2001 as shown in FIG. 20 and a constant value in the direction parallel to the dividing centerline 2001.

[0157] As discussed above, the second embodiment is different from the first embodiment in the functionality of the image splitter 0250 as shown in FIG. 2 and the methods for calculating geometric transformation control parameters and pixel data conversion parameters.

[0158] According to the second embodiment, the overlap area of neighboring partial images is transformed into an image which gradually disappears from the inside toward the outside and their overlap areas with this same image characteristic are optically laid one upon another on the screen, so that the influence of partial image alignment error is exhibited as a blur in the joint between them.

[0159] On the other hand, in case of the first embodiment, the influence of partial image alignment error appears as a discontinuous or non-gradual change in the image joint. Since, due to the characteristics of the human sight, human beings are more sensitive to discontinuous image variation than to continuous image variation like blurs, the influence of image alignment error in the second embodiment is less perceivable to the human eyes than that in the first embodiment.

[0160] A third embodiment of this invention has the same structure as the first embodiment except that the internal structure of the image signal control unit 0110 as shown in FIG. 2 and the functionality of the projectors connected to it are different. Referring to FIG. 22, a third embodiment will be is explained next.

[0161] In the third embodiment, each of projectors 2201, 2202, 2203, 2204 incorporates an image corrector with the same functionality as described for the first embodiment. In addition, the projectors are connected with two types of input signal lines: image signal lines 2241, 2242, 2243, 2244 and control signal lines 2231, 2232, 2233, 2234.

[0162] Like the first embodiment, the third embodiment also produces such an effect that neighboring projected images can join end to end without any gaps or overlaps.

[0163] A fourth embodiment of this invention will be described next, referring to FIG. 23. The fourth embodiment employs a non-planar screen.

[0164] A curved surface screen 2300 is used to enable projected images to efficiently cover the visual field of the viewer. Usually, if images are projected on a curved screen surface from projectors designed for projection on a planar screen, projected images will be rectangles whose sum of vertex angles is larger than 360 degrees, as illustrated here by maximum image projection ranges 2321 and 2322. Therefore, normally it is impossible to let the neighboring maximum image projection ranges 2321 and 2322 join end to end without any gaps or overlaps.

[0165] Projectors 2301 and 2302 are positioned so that the adjacent side of one of the two neighboring maximum image projection ranges 2321 and 2322 is included in the other's projection range as shown in FIG. 19, and image signals to the projectors 2301 and 2302 are controlled using the image signal control unit 2310 having the same functionality as explained for the first embodiment so that the neighboring projected images fit the optimized image projection ranges 2331 and 2332.

[0166] A is a fifth embodiment of this invention which uses an image signal control unit 0110 similar to the one as described for the first embodiment, which, however, does not have image correctors 0281, 0282, 0283, 0284 as used in the first embodiment will be explained. The functional block diagram of the image signal control unit of the fifth embodiment is shown in FIG. 24. The image signal control unit has, as input/output terminals, partial image input terminals 2421, 2422, 2423, 2424, an input terminal 2430 for the screen state monitor camera, and partial image output terminals 2441, 2442, 2443, 2444. The image signal selector 2460 selects either image signals from the partial image input terminals or image signals from the test pattern generator 2470 and supplies the selected signals to the partial image output terminals. The respective partial image signals are supplied by image reproduction units 2451, 2452, 2453, 2454 (connected to the partial image input terminals) that reproduce images from visual media such as DVD-ROMs, 2491, 2492, 2493, 2494, which store image data to be dealt with by the respective projectors.

[0167] The image signal control unit as shown in FIG. 24 requires no image correctors. As mentioned earlier, due to such factors as the configuration of the screen, the positional relationship between projectors and the screen and the display characteristics of the projectors, image geometric transformation and image color characteristic transformation are necessary in order to join images project ed from the projectors smoothly. The first embodiment is designed to perform such image correction in real-time by means of hardware. In the fifth embodiment, however, pre-corrected image data for partial images to be dealt with by the respective projectors are stored in visual media like DVD-ROMs. The image reproduction units 2451, 2452, 2453, 2454 realize smooth image connection simply by providing pre-corrected image data to the projectors. Thus, image correctors are unnecessary according to the fifth embodiment of the invention.

[0168] Next, a sixth embodiment of the invention will be explained. The sixth embodiment uses image reproduction processors with image correction functionality as shown in FIG. 25 instead of the image reproduction units 2451, 2452, 2453, 25454 in the fifth embodiment.

[0169] As shown in FIG. 25, the sixth embodiment uses a different projector array system from that of the fifth embodiment. In the sixth embodiment, each image reproduction unit consists of the following: a pickup 2521 which loads image data from an image data storage medium 2511; an image reproduction circuit 2531 which decodes the signal detected by the pickup into image data; an image correction circuit 2532 which processes the image reproduction data according to image correction data inputted from an image correction data input terminal 2551; a medium 2541 which stores the processed image data; and an image storage/reproduction circuit 2533 which reads the processed image data from the temporary storage medium 2541 and sends it to an image output terminal 2552.

[0170] While the fifth embodiment cannot cope with a case in which the amount of image correction required for smooth image connection varies among individual projector array systems, the sixth embodiment solves the problem by enabling optimal image correction for each projector array system using the following sequence: the image correction circuit 2532 pre-corrects partial image data for each projector offline and the corrected image data is first stored in the temporary storage medium 2541; the stored corrected image data is read out from the temporary storage medium 2541 to realize smooth image connection on the screen.

[0171] According to the sixth embodiment, it is possible to join images end to end smoothly on the screen even if an image correction circuit which is too slow to perform real-time image correction, or software-based image processing emulation is used, while the first embodiment requires a processor for real-time image correction to realize smooth image connection. In addition, if the required amount of image correction differs among individual projector array systems, while the fifth embodiment is unable to join images on the screen properly, the sixth embodiment solves the problem by using the optimal amount of image correction for each projector array system.

[0172] According to these embodiments, images projected on a curved-surface screen can be smoothly joined.

[0173] Also, for maximum image projection ranges of neighboring projectors on said screen to overlap each other at least by a given amount, input image data which has been subjected to adequate geometric and color transformation is supplied to the projectors. This makes it possible to join projected images from several projectors smoothly with least conspicuous seams and to form a single whole picture on the screen, making precise positional and optical adjustments of projectors unnecessary.

[0174] Since means for performing real-time geometric and color transformation of image data depending on projector and screen characteristics are provided, smooth connection of projected images from several projectors to form a single picture can be realized just by controlling the image transformation means. To cope with variation in projector positions and optical system adjustments, this invention only carries out signal processing, thereby eliminating the need for optical and mechanical control means which would be more costly and more difficult in improving control accuracy than means for signal processing. As a result, the invention offers the advantage that a less costly, highly reliable multi-projection image display device can be realized.

[0175] Another advantage of the invention is to ensure cost reduction and improved accuracy in image adjustment work in comparison with conventional human visual inspection and manual adjustments, since the invention provides means to automatically control image transformation means by automatic calculation of required amounts of image geometric and color transformation for smooth connection of projected images from several projectors to form a single picture.

[0176] In addition, the theme of joining projected images from several projectors smoothly to form a single image is achieved by optical addition of rays of light from neighboring projectors so as to connect the overlap and non-overlap areas of neighboring projected images smoothly. This means that the influence of projector positioning errors or image geometric transformation means control errors appear as image blurs. Therefore, an advantage of the invention is to be able to make image defects as caused by such errors less conspicuous to the human eyes due to the fact that the human eyes are less sensitive to continuous image variation such as blurs than to discontinuous image variation because of the characteristics of the human sight.

[0177] If a projector incorporates an image data converter capable of performing at least geometric and color characteristic transformation of digitized image signal data based on input image signals, and an arithmetic control unit for controlling said image converter, it is possible to offer not only the same advantages as the multi-projection image display device as mentioned above, but also a further advantage: when each of said projectors is used as a separate unit, geometric transformation and brightness distribution alteration can be made on a projected image and even if the optical axis of the projector is not vertical to the screen, an image with an accurate image shape and precise brightness distribution can be reproduced on the screen. This lightens restrictions on the positional relationship between the screen and projector.

[0178] According to this invention, several images (data) can be joined to form one picture (or pictures) by the use of a relatively simple image display device, and joints of neighboring images can be made less conspicuous.

[0179] As discussed so far, the invention enables projected images from a plurality of image projectors to be joined and displayed as one picture (or pictures).