Image synthesis apparatus and image synthesis method

[0002] 1. Field of the Invention

[0003] The present invention relates to an image synthesis apparatus and an image synthesis method for synthesizing a plurality of images, obtained by an electronic camera, that have at least some overlapping segments.

[0004] 2. Related Background Art

[0005] Conventionally, a well known image synthesis apparatus of this type generates a panoramic image by synthesizing a plurality of images that partially overlap each other. FIG. 1 is a diagram illustrating a plurality of images that are partially overlapped. Some areas of images 301, 302 and 303 are overlapped. FIG. 2 is a diagram showing synthesis processing for the images in FIG. 1. An image 401 is obtained by synthesizing images 301 and 302.

[0006] Conventionally, to provide a panoramic image, a synthesized image is generated as is shown in FIG. 2, and the succeeding image is sequentially added to this image to perform image synthesizing. Therefore, to synthesize the synthesized image 401 and image 303, an overlapping area 401ain the synthesized image 401, for which rotation or a magnification rate is converted, is aligned with an overlapping area 303ain image 303.

[0007] However, since compared with an original image, deterioration of a synthesized image for which conversion, such as rotation, has been performed occurs because of changes in the shape of an object or of the transformation of coordinates, and a positioning error may occur that affects the alignment of the image with the next image and prevents a desired panoramic image from being generated.

[0008] Since this problem becomes more significant as the number of images to be synthesized is increased, only a very limited number of images can be synthesized when the conventional method is used.

[0009] In addition, since synthesized image data must continue to be stored in a memory so that synthesized images can be sequentially generated, a large memory capacity is required.

[0010] There is one well known conventional image synthesis method for synthesizing a plurality of images that are obtained by photographing segments of one part of a subject and then generating a panoramic or elaborate image, whereby when synthesizing the images rotation, correction of magnification rates and coordinate translation are performed for each of the images, and the joints of adjacent images are matched.

[0011] This image synthesis method will be specifically described while referring to FIGS. 3 to 7. FIGS. 3 to 7 are diagrams for explaining the image synthesis processing performed when using the conventional image synthesis method.

[0012] As is shown in FIG. 3, to synthesize three images A, B and C, first, image A and image B are synthesized to obtain a synthesized image Sa, which is then synthesized with image C to obtain a synthesized image Sb, the final result of the synthesis of images A, B and C. In other words, images A, B and C are connected one by one in order to obtain a single synthesized image.

[0013] Furthermore, to synthesize images A and B, first, as is shown in FIG. 4, images A, B and C are displayed in the arranged order. Then, as is shown in FIG. 5, images A and B are synthesized and the synthesized image Sa is displayed. The overlapping portions of images A and B are so displayed that the portions of the two images at the overlap can be identified. As is shown in FIG. 6, image C is thereafter synthesized with the obtained image Sa and the synthesized image Sb is displayed. Similarly, the portions of images B and C that overlap are so displayed that the two images at the overlapping portion are separately discernible.

[0014] And in addition, when as is shown in FIG. 7 images A, B and C overlap each other, an overlapping portion 2201 of the three images is so displayed that again the portions of images A, B and C that overlap can be separately discerned.

[0015] According to the above conventional method, however, since images are synthesized in order when repeating the process employed for linking the images, a synthesized image obtained during at each processing step must be stored in memory, and thus, until such time as a final synthesized image is acquired, a large memory capacity is required.

[0016] In addition, when the synthesis result (synthesized image Sa) obtained for images A and B does not provide a satisfactory image when synthesization of the three images has finally been completed synthesized (synthesized image Sb), the synthesis process for images A and B must be repeated. And then, since the newly obtained synthesized image will differ from the previously obtained synthesized image Sa, image C must be synthesized with the newly synthesized image Sa. Thus, a great amount of labor is required when the processing is repeated to obtain a desired synthesized image.

[0017] Furthermore, as is described above, when images A, B and C overlap each other, the overlapping portion 2201 of the three images is so displayed that the portions of images A, B and C at the overlap are separately discernible. Therefore, it is hard to accurately evaluate a portion at which three or more images overlap, and it is difficult to determine whether a desired synthesis result can be obtained.

[0018] It is, therefore, one objective of the present invention to provide an image synthesis apparatus and an image synthesis method for resolving the above described problems.

[0019] It is another objective of the present invention to provide an image synthesis apparatus and an image synthesis method for precisely performing the positioning of images, even when there are many images to be synthesized.

[0020] To achieve the above objectives, according to one aspect of the present invention, an image synthesis apparatus, for synthesizing a plurality of images in order to generate a synthesized image, comprises:

[0021] a storage unit for selecting from a plurality of images, as an image synthesis unit, two or more images, and for storing image information for the selected images;

[0022] a coordinate transformation parameter generation unit for employing the image information stored in the storage unit to generate coordinate transformation parameters that are used to establish a positional relationship for the selected images before the selected images are synthesized to obtain a single image;

[0023] an image synthesis information generation unit for changing the coordinate transformation parameters, which are generated by the coordinate transformation parameter generation unit, by using as a reference the position of an arbitrary image of the plurality of images, and for furnishing the resultant coordinate transformation parameters as image synthesis information; and

[0024] an image synthesis processing unit for synthesizing the plurality of images in accordance with the image synthesis information generated by the image synthesis information generation unit.

[0025] It is an additional objective of the present invention to provide an image synthesis apparatus and an image synthesis method for determining the characteristic of a synthesized image by employing as a reference the characteristic of an arbitrary image selected from a plurality of images to be synthesized.

[0026] To achieve this additional objective, according to another aspect of the present invention, an image synthesis apparatus, for synthesizing a plurality of images to generate a synthesized image, comprises:

[0027] a storage unit for selecting from a plurality of images, as an image synthesis unit, two or more images and for storing image information for the selected images;

[0028] an adjustment information generation unit for employing the image information stored in the storage unit to adjust the characteristics of the selected images before synthesizing the selected images and obtaining a single image;

[0029] an image synthesis information generation unit for changing the adjustment information, which is generated by the adjustment information generation unit, by using as a reference the characteristic of an arbitrary image selected from the plurality of images, and for providing the resultant adjustment information as image synthesis information; and

[0030] an image synthesis processing unit for synthesizing the plurality of images in accordance with the image synthesis information generated by the image synthesis information generation unit.

[0031] It is a further objective of the present invention to provide a simple and inexpensive image synthesis apparatus that does not need to constantly store image data to be synthesized that occupies a large memory area and that does not require a large memory capacity to store the image data to be synthesized, and an image synthesis method therefore.

[0032] To achieve this further objective, according to an additional aspect of the present invention, an image synthesis apparatus, for generating a synthesized image by synthesizing a plurality of images, each of which share an overlapping portion with at least one other of those images, comprises:

[0033] an arrangement destination unit for designating an arrangement for the plurality of images;

[0034] a parameter generation unit for employing the arrangement for the plurality of images designated by the arrangement designation unit to select from the plurality of images, as an image synthesis unit, two or more images lying adjacent to each other, and for generating parameters that are used to establish, for each of the selected images, a positional relationship for matching the overlapping portions of the selected images; and

[0035] an image synthesis processing unit for synthesizing the plurality of images in accordance with the parameters generated by the parameter generation unit.

[0036] Other objectives and features of the present invention will become apparent during the course of the detailed explanation of the preferred embodiments that is given while referring to the following drawings.

[0037] FIG. 1 is a diagram showing a plurality of images that are partially overlapped;

[0038] FIG. 2 is a diagram showing the processing used to synthesize the images in FIG. 1;

[0039] FIG. 3 is a diagram for explaining the image synthesis processing using a conventional image synthesis method;

[0040] FIG. 4 is a diagram for explaining the image synthesis processing using the conventional image synthesis method;

[0041] FIG. 5 is a diagram for explaining the image synthesis processing using the conventional image synthesis method;

[0042] FIG. 6 is a diagram for explaining the image synthesis processing using the conventional image synthesis method;

[0043] FIG. 7 is a diagram for explaining the image synthesis processing using the conventional image synthesis method;

[0044] FIG. 8 is a block diagram illustrating the arrangement of an image synthesis apparatus according to a first embodiment of the present invention;

[0045] FIGS. 9A and 9B are diagrams showing position alignment processing;

[0046] FIG. 10 is a block diagram illustrating the arrangement of an image processing unit in FIG. 8;

[0047] FIG. 11 is a diagram showing an arrangement window for a plurality of images;

[0048] FIG. 12 is a diagram showing the procedures for synthesizing a plurality of images arranged in the arrangement window;

[0049] FIG. 13 is a diagram showing the processing used for detecting corresponding point information;

[0050] FIG. 14 is a diagram showing gradation correction processing;

[0051] FIG. 15 is a graph showing a correction coefficient;

[0052] FIG. 16 is a diagram showing a gradation correction table;

[0053] FIG. 17 is a diagram showing synthesis processing;

[0054] FIG. 18 is a diagram showing coordinate transformation parameter synthesis processing;

[0055] FIG. 19 is a diagram showing a drawing area for a synthesized image G;

[0056] FIG. 20 is a block diagram illustrating the arrangement of an image processing unit for an image synthesis apparatus according to a second embodiment of the present invention;

[0057] FIG. 21 is a diagram showing the procedures used for positioning two images;

[0058] FIG. 22 is a diagram showing a processing list;

[0059] FIG. 23 is a graph showing a correction coefficient;

[0060] FIG. 24 is a block diagram illustrating the arrangement of an image synthesis unit for an image synthesis apparatus according to a third embodiment of the present invention;

[0061] FIG. 25 is a diagram showing the state where a plurality of images have been arranged;

[0062] FIG. 26 is a diagram for describing the image synthesis processing;

[0063] FIG. 27 is a block diagram illustrating the arrangement of an image synthesis apparatus according to a fourth embodiment of the present invention;

[0064] FIG. 28 is a diagram depicting the external appearance of a personal computer that constitutes the image synthesis apparatus in FIG. 27;

[0065] FIG. 29 is a flowchart showing image synthesis processing performed by the image synthesis apparatus in FIG. 27;

[0066] FIG. 30 is a diagram showing an example image arrangement designation window at step S102 in FIG. 29;

[0067] FIG. 31 is a diagram showing another example image arrangement designation window at step S102 in FIG. 29;

[0068] FIG. 32 is a diagram showing an additional example image arrangement designation window at step S102 in FIG. 29;

[0069] FIGS. 33A, 33B, 33C and 33D are diagrams showing example combinations of two adjacent images at step S103 in FIG. 29;

[0070] FIG. 34 is a diagram showing an example two image synthesis window at step S103 in FIG. 29;

[0071] FIG. 35 is a diagram showing another example two image synthesis window at step S103 in FIG. 29;

[0072] FIG. 36 is a diagram showing an additional example two image synthesis window at step S103 in FIG. 29;

[0073] FIG. 37 is a diagram showing a further example two image synthesis window at step S103 in FIG. 29;

[0074] FIG. 38 is a diagram showing an example synthesis window for all images at step S104 in FIG. 29;

[0075] FIG. 39 is a flowchart showing image synthesis retrying processing performed by the image synthesis apparatus in FIG. 27;

[0076] FIGS. 40A and 40B are diagrams illustrating state shifting performed in a window for explaining the batch synthesis function of the image synthesis apparatus in FIG. 27;

[0077] FIGS. 41A and 41B are diagrams illustrating the state shifting performed in the window for explaining the batch synthesis function of the image synthesis apparatus in FIG. 27;

[0078] FIGS. 42A and 42B are diagrams illustrating the state shifting performed in the window for explaining the batch synthesis function of the image synthesis apparatus in FIG. 27; and

[0079] FIGS. 43A and 43B are diagrams illustrating the state shifting performed in the window for explaining the batch synthesis function of the image synthesis apparatus in FIG. 27.

[0080] The preferred embodiments of an image synthesis apparatus and an image synthesis method according to the present invention will now be described.

[0081] FIG. 8 is a block diagram illustrating the arrangement of an image synthesis apparatus according to a first embodiment of the present invention. In FIG. 8, a plurality of images that partially overlap each other are recorded on an external recording medium 101.

[0082] A CPU 102 reads the images from the external recording medium 101 and performs image synthesis processing which will be described later. A temporary memory unit 103 is used to temporarily store the images that are read. An image processing unit 104 synthesizes the images stored in the temporary memory unit 103 to generate a panoramic image.

[0083] A display unit 105 displays a synthesized image, a graphic user interface (GUI), etc. An operating unit 106 includes a button and a mouse. These individual sections are connected to a CPU bus along which they can exchange data and control signals.

[0084] The image synthesis apparatus reads images from the external recording medium 101, and stores in the temporary memory unit 103 only two images for which positions are to be aligned.

[0085] FIGS. 9A and 9B are diagrams showing the position alignment processing. To synthesize a plurality of images 301, 302 and 303, the original image is employed during a corresponding point detection process performed for the positioning.

[0086] In FIG. 9A the corresponding point detection process is performed using overlapping portions 301aand 302ain order to align the positions of images 301 and 302. Similarly in FIG. 9B the corresponding point detection process is performed using overlapping portions 302band 303ain order to align the positions of images 302 and 303. Since in either case the original image is employed instead of a synthesized image, the corresponding point detection process can be precisely performed.

[0087] FIG. 10 is a block diagram illustrating the arrangement of an image processing unit 104. In FIG. 10, an image setting unit 201 sets two images for a plurality of images for which the corresponding point detection process is to be performed. A corresponding point detecting unit 202 extracts the locations of corresponding points in the area in which the two images overlap, as is shown in FIGS. 9A and 9B.

[0088] A coordinate transformation parameter generating unit 203 generates coordinate transformation parameters for the two images. A gradation correction processing unit 204 generates a correction coefficient to match the gradations of the two images, and prepares a gradation correction table.

[0089] A synthesis processing unit 205 synthesizes the two images that are set, and outputs the synthesis result to the display unit 105. A coordinate transformation parameter synthesis processing unit 206 sets a reference image for a plurality of images, and converts coordinate transformation parameters for the other images based on the reference image. A gradation correction table renewal processing unit 207 updates a gradation correction table based on the reference image.

[0090] A drawing area generating unit 208 sets a drawing area for the finally obtained synthesized image. An image synthesis information memory unit 209 is used to store information required for generating a final synthesized image. An all image synthesis processing unit 210 generates a final synthesized image using all the images employed for synthesis and all the image synthesis information. All of these individual sections are connected to an internal bus along which they exchange data and control signals.

[0091] An explanation will now be given for the image synthesis processing performed by the thus arranged image synthesis apparatus. The image setting unit 201 employs a plurality of images stored in the temporary memory unit 103 to determine an image arrangement and two images for which the corresponding point detection unit 202 detects corresponding points.

[0092] FIG. 11 is a diagram showing an arrangement window for a plurality of images. When the arrangement window is displayed on the display unit 105 and the positional information is employed for the images arranged in the window, the positional relationship for the images can be set. It should be noted that amount of memory that is required can be reduced by using thumb-nail images in the arrangement window.

[0093] FIG. 12 is a diagram showing the synthesis processing for a plurality of images arranged in an arrangement window. The image setting unit 201 initiates the processing by employing as a reference an image at the upper left corner of the arrangement window (image 1 in FIG. 12).

[0094] First, whether there is an image to the right of image 1 to use as a reference is determined (arrow path (1)). When there is an image on the right side, an image 2 is selected. When the synthesis processing for images 1 and 2 has been completed, a search is made to determine whether to the right of image 2 there is an image to use as a reference (arrow path (2)), and an image 3 is selected.

[0095] Continuing, a search is performed for an image to the right of image 3. In FIG. 12, however, since there is no image to the right of image 3 (an arrow indicated by a broken line), the search returns to the image at the left end of the same row in which is image 3, i.e., to image 1. A search is made below image 1 for an image to use as a reference (arrow path (4)), and an image 4 is selected. The same process is repeated and two images are set for of a plurality of images.

[0096] The image setting unit 201 loads to the temporary memory unit 103 only the two images selected from the external recording medium 101, and performs the following processing. When image synthesis information is stored in the image synthesis information memory unit 209, the two images are abandoned by the temporary memory unit 103.

[0097] When the corresponding point detecting unit 202 employs small images to perform the corresponding point detection process, the image setting unit 201 reduces the sizes of the two images that have been set. Furthermore, when of the RGB color information only a G signal is employed to perform the corresponding point detection process, the image setting unit 201 extracts a G signal from a color image. In addition, when an edge image is employed for the corresponding point detection process, the image setting unit 201 extracts edge information from an image.

[0098] In this manner, the two images set by the image setting unit 201 are transmitted to the corresponding point detecting unit 202, which thereafter detects the corresponding point information for the two images.

[0099] FIG. 13 is a diagram showing the processing for detecting corresponding point information. Images 301 and 302 are the two that were set by the image setting unit 201.

[0100] In this embodiment, images 301 and 302 are employed; however, images that vertically and horizontally constitute half of images 301 and 302 may be employed. In this case, the corresponding point detection process can be performed at high speed. And the speed and the precision of the detection of corresponding points can be further increased by using edge images.

[0101] A template matching method is employed as the method for detecting corresponding point information. According to the template matching method, as is shown in FIG. 13 one image segment is cut off image 301 to serve as a template 601. While the template 601 is being shifted inside a search area in the other image 302, at each position an absolute value is calculated for a difference between a pixel in the template 601 and a corresponding pixel in the search area 602, and the sum of the absolute values is evaluated.

[0102] Corresponding points in the template 601 are acquired as positions at which the minimum sum for the absolute values of the differences is obtained. The corresponding point detecting unit 202 employs at least two of these templates to detect corresponding point information in the images.

[0103] To synthesize two images, The coordinate transformation parameter generating unit 203 calculates coordinate transformation parameters. In this embodiment, affine transformation is employed as one example coordinate transformation. Assuming coordinates (x, y) are located on image 301 and coordinates (x′, y′) are located on image 302, using parameters A, B, C and D, the affine transformation performed for images 301 and 302 can be represented by equations (1) and (2):

[0104] x′=Ax+By+C  (1)

[0105] y′=Bx+Ay+D  (2).

[0106] The coordinate transformation parameter generating unit 203 acquires the parameters A, B, C and D based on the corresponding point information that is obtained by the corresponding point detecting unit 202. Since there are four parameters, corresponding point information for two points are substituted into equations (1) and (2) to resolve the simultaneous equations, so that the parameters A, B, C and D can be obtained. When corresponding point information for more than three points is acquired, the parameters can be obtained using the least square method.

[0107] The gradation correction processing unit 204 performs gradation correction processing to perform gradation matching for the two images. FIG. 14 is a diagram showing the gradation correction processing.

[0108] During the gradation correction processing, the corresponding points in images 301 and 302 are calculated using equations (1) and (2), and sample data are acquired. That is, a point that corresponds to sample data 701 in FIG. 14 is calculated using equations (1) and (2), and sample data 701bthat correspond to the sample data 701aare generated for image 302. The sample data may be a pixel value at a specific point, or they may be the average of the pixel values in an arbitrary area, for which a specific point is designated as the center point, calculated while taking into account known coordinate transformation parameters errors.

[0109] FIG. 15 is a graph showing a correction coefficient. Sample data sets 701ato 704aand 701bto 704bin FIG. 14 are generated and are plotted as is shown in FIG. 15 (x points in FIG. 15). The pixel values for the sample data are generated based on the gradation correction table that is held for each image. Therefore, sample data Sa for image 301 and sample data Sb for image 302 are calculated using equations (3) and (4) in which the original pixel value is represented by i:

[0110] Sa=Ka [i]  (3)

[0111] Sb=Kb [i]  (4).

[0112] Ka and Kb denote values in the gradation correction tables for images 301 and 302.

[0113] The plotted data are employed to generate a correction coefficient f that is used to match the gradations of image 302 with those of image 301. Based on the plotted data, the correction coefficient f can be obtained using the least square method.

[0114] The gradation correction processing unit 204 employs the obtained correction coefficient f to prepare a gradation correction table. For an RGB color image, the correction coefficient f that is obtained for an individual color is fR, fG or fB. In the gradation correction table, the values are generated as data R[256], G[256] or B[256], each of which values are represented by 8-bit data.

[0115] As for an R signal, the gradation correction table is prepared in the following manner:

[0116] R[i]=fR(i) (i=0 to 255)  (5).

[0117] FIG. 16 is a diagram showing a gradation correction table. Pixel values before correction are entered in the left column, and pixel values after correction, which are generated using equation (5), are entered in the right column. The gradation correction processing unit 204 performs the above described processing to prepare the gradation correction table.

[0118] It should be noted that at this time the gradation correction table is only temporary. Only when the synthesis result has been determined to be adequate is the gradation correction table stored in correlation with the images. When the synthesis result is determined to be inadequate, the gradation correction table is abandoned and the processing is retried.

[0119] The synthesis processing unit 205 synthesizes the two images 301 and 302 using the coordinate transformation parameters and the gradation correction table. FIG. 17 is a diagram showing the synthesis processing. In accordance with the coordinates that employ as a reference the left upper corner of image 301, the pixel value at coordinates (i, j) in the synthesized image G is generated from images 301 and 302.

[0120] With a centerline 150 in the overlapping areas of images 301 and 302 being as a joint, the pixel value for image 301 is used for the area to the left of the center 150, and the pixel value for image 302 is used for the area to the right.

[0121] Since image 301 is used as a reference, an area 151 is represented by G(i, j)=Ta(i, j), in which Ta is employed to represent the pixel value for image 301. In accordance with equations (1) and (2), and by employing a pixel value that is obtained by using the gradation correction table to convert the pixel value Tb for image 302, an area 152 is represented by G(i, j)=K(Tb(i′, j′)).

[0122] It should be noted that i′=A·i+B·j+C and j′=−B·i+A·j+D, and that K( ) denotes a value in the gradation correction table.

[0123] For a color image, the generation of a pixel value for a synthesized value is performed for each of the RGB colors. The coordinates and the gradations of image 302 are changed by employing image 301 as a reference, and a synthesized image is generated. The obtained synthesized image is written in the memory unit 211 and is output to the display unit 105.

[0124] The synthesized image is displayed by the display unit 103, and at the same time a confirmation button is displayed on the display unit 105. When a user determines that the synthesized image is adequate, the user uses the confirmation button to transmit a signal to that effect to the image processing unit 104 via the operating unit 106.

[0125] When notification information to the effect that the synthesized image is inadequate is received from the operating unit 106, the two images on the display unit 105 are erased, and the image information in the memory unit 211 is erased. When the image processing unit 104 receives from the operating unit 106 notification information to the effect that the synthesized image is adequate, the coordinate transformation parameter synthesis processing unit 206 initiates the synthesis processing for the coordinate transformation parameters.

[0126] FIG. 18 is a diagram showing coordinate transformation parameter synthesis processing. In FIG. 18, eight images are to be synthesized. Assume that the upper left corner of an image 1 is the reference point O(0, 0).

[0127] When the synthesis result obtained for the two images by the image synthesis processing unit 205 is determined to be adequate, the coordinate transformation parameter synthesis processing unit 206 converts a parameter that is obtained by the coordinate transformation parameter generating unit 203 into a coordinate transformation parameter that is based on the reference point O.

[0128] The coordinate points are detected and the coordinate transformation parameters are generated for images 2 and 3. However, since the parameters for image 3 are those that are generated while image 2 is employed as a reference, the coordinate transformation parameter synthesis processing unit 206 changes these parameters to those for which point O for image 1 is employed as a reference. This process is performed once for all the images except image 1.

[0129] The gradation correction table renewal processing unit 207 updates the gradation correction table for image 3 based on the gradation correction table that is obtained by the gradation correction processing unit 204. That is, a data area is acquired for storing for image 3 the gradation correction table obtained by the gradation correction processing unit 204, and the table data are written to the data area. When information to the effect that the synthesis result for images 2 and 3 is adequate is not transmitted via the operating unit 106, the gradation correction table updating process is not performed.

[0130] When the gradation correction table has been updated, the drawing area generating unit 208 sets a drawing area for each final synthesized image.

[0131] For the drawing area for the synthesized image G for image 5, inverse conversion is performed for synthesis parameters A₅, B₅, C₅and D₅obtained by the coordinate transformation parameter synthesis processing unit 206, and the parameters A′₅, B′₅,C′₅and D′₅that are obtained are employed to calculate coordinates in the synthesized image for the original four corners of image 5 using equations (6) and (7):

[0132] x=A′₅x′+B′₅y′+C′₅  (6)

[0133] y=−B′₅x′+A′₅y′+D′₅  (7).

[0134] (x′, y′) represents the coordinates for each of the original four corners, and (x, y) represents the corresponding coordinates for the synthesized image G.

[0135] FIG. 19 is a diagram showing a drawing area for the synthesized image G. The drawing area for the synthesized image G is set as the minimum rectangular area for enclosing the four corners after conversion.

[0136] The synthesis parameters, obtained by the coordinate transformation parameter synthesis processing unit 206, and the gradation correction table and drawing area data, obtained by the gradation correction table renewal processing unit 207, are stored with the image information in the image synthesis information memory unit 209.

[0137] The information stored in the image synthesis information memory unit 209 includes (1) image identification information (e.g., a file name), (2) a coordinate transformation parameter relative to the reference point O, (3) gradation correction tables for RGB colors, (4) a drawing area for a synthesized area, and (5) a check flag.

[0138] The image processing unit 104 does not constantly store, in the temporary memory unit 103, n images or images that are being sequentially synthesized, but instead stores, therein, only the image synthesis information. When the image synthesis information has been set, the check flag is set to a value of “1” so that the information will be stored for each image.

[0139] In accordance with the paths shown in FIG. 12, the image setting unit 201 selects image 1-image 2, image 2-image 3, image 1-image 4, image 4-image 5 and image 5-image 6, and performs the above described processing series. Basically, only two images are required for this process, and when the image setting unit 201 reads from the external recording medium 101 only the two images required for the corresponding point detection, the amount of memory that is required can be reduced.

[0140] When the above processing is completed for images 5 and 6 and the image synthesis information for image 6 is stored in the image synthesis information memory unit 209, the coordinate transformation parameters for all the images are converted into those that are based on the reference point O of the image 1, and the gradation correction table is set so that the gradations are matched with those for image 1. Furthermore, a drawing area is set for each image.

[0141] The all image synthesis processing unit 210 employs the information in the image synthesis information memory unit 209 and the images stored in the temporary memory unit 103 to generate the synthesized image G for all the images. Once the drawing areas for the individual images have been acquired as is shown in FIG. 12, images 1 to 8 are sequentially read from the memory unit 211 and the pixel values in the drawing areas for the images are written. The writing of the images is performed in accordance with the synthesized coordinate transformation parameter and the gradation table.

[0142] In addition, since images are written in order beginning with image 1, when image 1 has been written in the drawing area, the information stored in the temporary memory unit 103 for image 1 is abandoned. Since the same process is performed for images 2 to 8, only sufficient memory for storing eight images need be provided for the temporary memory unit 103.

[0143] Since essentially only the original images are employed to generate information for synthesis processing, the load imposed on the processing is unchanged, regardless of how many images are employed. In addition, since the position detection is performed based on the original images, the precision is not deteriorated due to an increase in the number of images.

[0144] FIG. 20 is a block diagram illustrating the arrangement of an image processing unit for an image synthesis apparatus according to a second embodiment of the present invention. Except for the image processing unit, the individual sections are the same as those for the first embodiment.

[0145] The same references as are used in FIG. 10 are also used to denote corresponding components in an image processing unit 104A in FIG. 20, and no explanation for them will be given. The specific features of the image synthesis apparatus in the second embodiment includes an image setting unit 214, a gradation correction processing unit 215, and a gradation correction table renewal processing unit 213.

[0146] FIG. 21 is a diagram for the setting procedures employed for two images. The image setting unit 214 sets two images using procedures that differ from those in the first embodiment. Specifically, using image 1 at the upper left corner as a reference, peripheral images to the left and to the right, and above and below are searched for in order. When an image is found, the image is set as in the conventional manner, and is output to a corresponding point detecting unit 202. In FIG. 21, image 1-image 2 (arrow path (1)) is set. Then, the search is continued using image 1 as a reference, and image 4, below image 1 (arrow path (2)) is set. Processing is then initiated for images 1 and 2.

[0147] In addition, the image setting unit 214 prepares a process list in accordance with the performance order for the processes. FIG. 22 is a diagram showing the process list. Beginning with image 1, image information is added to a process list 900 in the order in which the images were used for the process.

[0148] When a search for images has been made to the right and to the left, and above and below image 1, and the processing has been completed for images that were found, images 1, 2 and 4 are entered in the process list 900. Then, using the second image 2 in the process list 900 as a reference, the search and the processing is performed in order to the left and to the right, and above and below. Although image 1 is located to the left of image 2, as a pair of images 1 and 2 has already been processed, image 1 is skipped. Whether or not the image should be skipped is determined by examining the check flag to determine whether it holds a value of 1 (already processed) or a value of 0 (unprocessed).

[0149] Since unprocessed image 3 is present to the right of image 2, image 2-image 3 is set and the succeeding process is performed. When the process for which image 2 is used as a reference has been performed, the processing is moved to the third entry in the process list 900, and the search and the process is performed while using image 4 as a reference. The above described processing is repeated to obtain the image synthesis information.

[0150] When this search method is employed, the synthesis information can be efficiently set and all images can be processed, regardless of how complicated the arrangement of the plurality of images is.

[0151] The processing performed by the gradation correction processing unit 215 will now be described. In the first embodiment, the gradation correction for matching the gradations of image 2 with those for image 1 has been explained. In the second embodiment, however, the gradations are corrected for by compounding the gradations of individual images.

[0152] Sample data are acquired in the same manner as in the first embodiment. FIG. 23 is a graph showing a correction coefficient. Based on individual sampling data, correction coefficients fa and fb are so generated by compounding the gradations of the individual images.

[0153] The gradation correction table renewal processing unit 213 generates a correction coefficient by compounding the gradations of the individual images to prepare the gradation correction table. Specifically, when the gradations between images 1 and 2 are corrected to prepare the gradation correction table, and when a gradation correction table is prepared for images 2 and 3 later, converting at the same time the gradation correction table for image 1, which contacts image 2, and the gradation correction table for image 4, which contacts image 1, is indispensable. In the first embodiment, however, this process is not required because only the gradations for an image that is newly linked are converted and corrected for while using image 1 as a reference.

[0154] To implement the above described processing, the gradation correction table renewal processing unit 213 accesses the process list 900 via a microcomputer. Since during the gradation correction for images 2 and 3, images 1, 2 and 4 are included in the process list, the correction coefficient is generated for images 2 and 3. After a notification to the effect that the synthesis result of the two images is adequate is received via the operating unit 106, the gradation correction table renewal processing unit 213 updates the gradation correction tables for images 1, 2 and 4.

[0155] In the updating process for the gradation correction table, the gradation correction coefficient fb for image 3 is employed to obtain R₃[i]=fb(i) (i=0 to 255). Similarly, for images 1, 2 and 4, R₁[i]=fa(R₁[i]), R₂[i]=fa(R₂[i]) and R₄[i]=fa(R₄[i]) are employed.

[0156] When the image synthesis information has been set through the processing series, the synthesis information is added for each image to the header portion of the process list 900 in the image synthesis information memory unit 209.

[0157] In this embodiment, the process list 900 is stored by the CPU 102 as a single file format in the recording medium. When a device (not shown) that includes a means for reproducing the format file and the all image synthesis processing unit are employed, a synthesized image can be generated from the recording medium on which information is recorded.

[0158] With this recording medium, even if a synthesized image is not recorded, image synthesis can be performed using the synthesis information as needed in order to generate a synthesized image. Furthermore, since the image synthesis device is constituted by the reproduction means and the all image synthesis processing unit, the structure can be simplified and can provide a synthesized image at a low cost.

[0159] FIG. 24 is a block diagram illustrating the arrangement of an image processing unit for an image synthesis apparatus according to a third embodiment of the present invention. The same references as are used in FIG. 10 are also used to denote corresponding components in an image processing unit 104B in FIG. 24, and no explanation for them will be given.

[0160] The image synthesis apparatus according to the third embodiment includes a feature whereby a user is permitted to select a reference image for synthesizing a plurality of images, and whereby the generating of coordinate transformation parameters and gradation corrections for other images are performed based on the user selected reference image.

[0161] FIG. 25 is a diagram showing an arrangement of a plurality of images. An image setting unit 216 reads a plurality of images from a temporary memory unit 103, and assigns numbers to the thus arranged images beginning with the image at the upper left corner.

[0162] When a shaded image 5, for example, is selected via an operating unit 106, the selected image information is set as a reference image. As well as in the first embodiment, two images are sequentially set, and the corresponding point detection process and the succeeding process are performed for these images.

[0163] Image pairs used for the corresponding point detection process are, along arrow paths in FIG. 25, (1) image 1-image 2, (2) image 2-image 3, (3) image 1-image 4, (4) image 4-image 5, (5) image 5-image 6, (6) image 4-image 7, (7) image 7-image 8 and (8) image 8-image 9.

[0164] In this embodiment, the same processing series as is performed in the first embodiment is performed until the reference image 5 is set. When the reference image 5 is set, the coordinate transformation parameters and the gradation correction tables are changed based on the reference image 5. That is, the coordinate transformation parameters and the gradation correction tables obtained while using image 1 as a reference are held until the process for the arrow path (3) is initiated, and when the process for the arrow path (4), i.e., the process for images 4 and 5, is initiated, the coordinate transformation parameters and the gradation correction tables are converted into those for which image 5 is used as a reference.

[0165] When a synthesized image for images 4 and 5 is generated, and when a notification indicating that the synthesis result is adequate is transmitted from the operating unit 106 to the microcomputer, a coordinate transformation parameter synthesis processing unit 217 converts the coordinate transformation parameter for image 4 into that obtained when image 5 is used as a reference. Then, the coordinate transformation parameter for image 1, which contacts image 4, is converted into a value while using image 5 as a reference, and the coordinate transformation parameters for images 2 and 3 are also converted into values which are based on image 5.

[0166] A gradation correction table renewal processing unit 218 changes the gradations of images 1 to 4 using a correction coefficient f′ that is employed to convert the gradations for image 4 into those for image 5. A drawing area generating unit 219 employs the coordinate transformation parameter that is obtained by a coordinate transformation parameter synthesis processing unit 217 to reset drawing areas in the synthesized image for images 1 to 5.

[0167] The results of the processing are stored in the image synthesis information memory unit 209. Information for the pair of images 5 and 6 and the succeeding pairs is converted into the information obtained when image 5 was used as a reference, and is then stored. When the image synthesis information for all the images has been set, an all image synthesis processing unit 210 synthesizes all the images in accordance with the synthesis information.

[0168] FIG. 26 is a diagram showing image synthesis processing. Since image synthesis is performed while using an image in the center as the reference, an image can be generated in which distortion is dispersed. When an image which includes the main subject is selected as a reference image, the synthesis result for all the images can be obtained while the gradations existing when the main subject was photographed are maintained.

[0169] In the third embodiment, a user selects a reference image. However, a reference image may also be automatically selected in accordance with the arrangement of the images. Since, for example, the arrangement in FIG. 26 consists of a 3×3 images matrix, an image at the vertical and horizontal center position (1, 1) may be automatically determined to be a reference image.

[0170] The functions of the image processing unit 104, 104A or 104B in the first, the second or the third embodiment can be stored as a program on a recording medium, such as a floppy disk, and the program can be loaded into a memory unit by using a system or an apparatus, such as a personal computer, and executed as application software. In this case, when the system or the apparatus reads from the recording medium the program, which is represented by software, for accomplishing the present invention, the system or the apparatus can provide the effects of the present invention.

[0171] Such a memory medium can be, for example, a ROM, a hard disk, an optical disk, a magneto optical disk, a CD-ROM, a CD-R, a DVD, a magnetic tape, or a nonvolatile memory card.

[0172] A fourth embodiment of the present invention will now be described while referring to the accompanying drawings.

[0173] FIG. 27 is a block diagram illustrating the arrangement of an image synthesis apparatus according to the fourth embodiment of the present invention. FIG. 28 is a diagram showing the external appearance of a personal computer that constitutes the image synthesis apparatus in FIG. 27.

[0174] The personal computer that serves as the image synthesis apparatus of the present invention will now be described while referring to FIG. 28. As is shown in FIG. 28, the personal computer comprises: a computer main body 1101, incorporating a CPU, a ROM, a RAM and an interface for connecting these devices; a display unit 1102, for displaying the processing results obtained by an application program; and a mouse 1103 and a keyboard 1105, with which instructions and data are entered. A mouse button 1103a is provided for the mouse 1103, which with the keyboard 1105 is connected via a cable to the computer main body 1101.

[0175] The arrangement of the image synthesis apparatus in this embodiment will now be described while referring to FIG. 27.

[0176] As is shown in FIG. 27, the image synthesis apparatus constituted by the personal computer comprises: a hardware unit 1209; an operating system (OS) 1205, to be operated by the hardware unit 1209; and application software 1204, to be operated by the operating system 1205.

[0177] The hardware unit 1209 includes a keyboard interface 1210 for connecting a keyboard 1221; a mouse interface 1212 for connecting a mouse 1216; a video interface 1213 for connecting a display device 1217; and a disk IO interface 1214 for connecting a hard disk 1215.

[0178] The operating system 1205 includes an input device management system 1206, a drawing management system 1207, and a file system 1208. The input device management system 1206, via the keyboard interface 1210, receives data input at the keyboard 1211, and, via the mouse interface 1212, receives data input with the mouse 1216, and transmits the received data to the application software 1204, as needed. The drawing management system 1207 permits the application software 1204 to execute drawings on the display device 1217 via the video interface 1213 without the application software 1204 being aware of the hardware unit 1209. The file system 1208 permits the hard disk 1215 and the application software 1204 to exchange data via the disk IO interface 1214.

[0179] The application software 1204 includes an application program for building an image synthesis system 1201. The image synthesis system 1201 includes an image processing unit 1202 for performing various processes for synthesizing a plurality of images to generate a single synthesized image, and an image display unit 1203 for displaying on the display device 1217 the processing condition and the results that are obtained during the various processes. These units are built by a program that is stored in advance. The image processing unit 1203 performs: a designation process for designating a position for each image in accordance with an instruction entered at the keyboard 1211 or with the mouse 1216; a positional relationship calculation process for extracting, for each image, two mutually adjacent images based on the designated arrangement for the images, and for calculating the positional relationship where the overlapping portions of each two extracted images match; a synthesized image generation process for synthesizing the images in accordance with the obtained positional relationship and for generating a synthesized image; and a positional relationship recalculation process for recalculating the positional relationship for two mutually adjacent images that are designated. It also controls the image display means 1203.

[0180] The positional relationship calculation process can be so set that two mutually adjacent images are designated by manipulating the keyboard 1211 or the mouse 1216; the approximate positional relationship is entered for these two images by manipulating the keyboard 1211 or the mouse 1216; and the positional relationship is calculated for the two designated adjacent images by employing the approximate positional relationship.

[0181] Furthermore, the positional relationship calculation process can be so set that the positional relationship for two mutually adjacent images is automatically calculated en bloc.

[0182] The retry processing for the synthesis of the individual images can be so set that two mutually adjacent images is designated in the image designation process; the positional relationship between the designated images is recalculated by the positional relationship recalculation process; the recalculated positional relationship for the two designated images and the previously obtained positional relationship for the other two adjacent images is employed for the image synthesis generation process to perform image synthesis processing. The positional relationship recalculation process, as well as in the positional relationship calculation process, can be so set that two mutually adjacent images can be designated by manipulating the keyboard 1211 or the mouse 1216; the approximate positional relationship is entered for the two designated images by manipulating the keyboard 1211 or the mouse 1216; and the positional relationship for the two designated adjacent images can be recalculated by employing the approximate positional relationship entered for the designated adjacent images.

[0183] The image display unit 1203 is so controlled that it displays only two mutually adjacent images on the display device 1217 when calculating the positional relationship for these images using the positional relationship calculation process. In addition, the image display unit 1203 is so controlled that it displays on the display device 1217 a synthesized image obtained in accordance with the positional relationship existing between the two adjacent images that is acquired by the positional relationship calculation process. To display a synthesized image of the two adjacent images, in the display the two images must overlap each other, and the overlapping portion must be transparent.

[0184] The image synthesis processing performed by the image synthesis apparatus will now be described while referring to FIGS. 29 through 39. FIG. 29 is a flowchart showing the image synthesis processing performed by the image synthesis apparatus in FIG. 27. FIG. 30 is a diagram showing an example image arrangement designation window at step S102 in FIG. 29. FIGS. 33A to 33D are diagrams showing an example combination of two adjacent images at step S103 in FIG. 29. FIGS. 34 and 35 are diagrams showing an example two image synthesis window at step S103 in FIG. 29. FIGS. 36 and 37 are diagrams showing an example two image synthesis result window at step S104 in FIG. 29. FIG. 38 is a diagram showing an example all images synthesized result window at step S104 in FIG. 29. And FIG. 39 is a flowchart showing the image synthesis retrying processing performed by the image synthesis apparatus in FIG. 27.

[0185] In FIG. 29, first, at step S101 a request is issued for the selection of images to be synthesized. A user employs the keyboard 1211 or the mouse 1216 to enter the file names of the images to be synthesized. The selected images are those that have been photographed by a digital camera, and that have portions that overlap at least one of the other images.

[0186] When the images to be synthesized are selected, at step S102 a request is issued to designate the arrangement for the selected images. In this embodiment, as is shown in FIG. 30, an arrangement designation window is displayed on the display device 1217, and a user is requested to select one of several predetermined arrangements for the selected images. In the arrangement designation window are displayed a message “Please designate arrangement for images”; buttons 1401, 1402 and 1403, for designating an image arrangement; a button 1405, for establishing the current process and instructing a shift to the next process; a button 1406, for instructing batch synthesis; and selected images 1404a, 1404b, 1404cand 1404d. It should be noted that the individual images 1404a, 1404b, 1404cand 1404dthat are displayed are small.

[0187] The button 1401 is used to designate the arrangement wherein the selected images are arranged horizontally in one row. This arrangement is selected for a case in which a subject was divided horizontally when it was photographed, and the obtained images are to be synthesized. When the button 1401 is selected to designate the horizontal image arrangement, as is shown in FIG. 30 the images 1404a, 1404b, 1404cand 1404dare horizontally arranged and displayed.

[0188] The button 1402 is used to designate an arrangement wherein the selected images are vertically arranged in a line. This arrangement is selected for a case in which a subject was divided vertically when it was photographed, and the obtained images are to be synthesized. When the button 1402 is selected to designate the vertical image arrangement, as is shown in FIG. 31 the images 1404a, 1404b, 1404cand 1404dare vertically arranged and displayed.

[0189] The button 1403 is used to designate an arrangement wherein the selected images are arranged two-dimensionally. This arrangement is selected for a case in which a subject was divided vertically and horizontally when it was photographed, and the obtained images are to be synthesized. When the button 1403 is selected to designate the two-dimensional image arrangement, as is shown in FIG. 32 images 1404a, 1404b, 1404cand 1404dare arranged two-dimensionally and displayed.

[0190] Since in these windows the displayed images can be rearranged as desired by drug-and-drop manipulation using the mouse 1216, a user can rearrange the images as needed. When, for example, the order of the arranged images does not match the order in which they were photographed, the displayed images can be rearranged in the order in which they were photographed.

[0191] When the user confirms that the images are appropriately arranged, he or she selects the button 1405 to designate this image arrangement. Through the selection of the button 1405, the process is shifted to the next step S103. At step S103, all the pairs of adjacent images are synthesized based on the image arrangement designated at step S102. For the four images shown in FIG. 33A that are arranged horizontally, for example, there are three the pairs of adjacent images: image 1 and image 2, image 2 and image 3, and image 3 and image 4. For the four images shown in FIG. 33B that are vertically arranged, there are also three image pairs: image 1 and image 2, image 2 and image 3, and image 3 and image 4. For the four images in FIG. 33C that are arranged two-dimensionally, there are four image pairs: image 1 and image 2, image 2 and image 4, image 1 and image 3, and image 3 and image 4. And for the nine images in FIG. 33D that are arranged two-dimensionally, as is indicated by the arrows, there are twelve image pairs .

[0192] In the synthesis processing for two adjacent images, a two image synthesis window is displayed on the display device 1217, and a user is requested to select either an auto mode or a manual mode for synthesizing the two adjacent images. As is shown in FIG. 34, in the two images synthesis window are displayed a message “Two images are synthesized”; buttons 1801 and 1802, for designating a mode for synthesizing the two adjacent images; a button 1803, for instructing a return from the current process to the previous process; a button 1804, for establishing the current process and for instructing a shift to the next process; a button 1805, for instructing batch synthesis; and two adjacent images, images 1404aand 1404b, for example. The images, 1404aand 1404b, that are displayed constitute one of the adjacent image pairs formed by the four images in FIG. 30 that are horizontally arranged.

[0193] Upon the selection of the button 1801, the auto mode is designated, and upon the selection of the button 1804, the manual mode is executed. In the auto mode, first, template matching is performed; a minute area is extracted from one of the images; and matching of this area with the other image is performed to search for a common portion in the two images. Then, the positional relationship for the two images is calculated based on the result obtained by the template matching. In accordance with the positional relationship, the two images are synthesized to generate a new image (i.e., a synthesized image is generated using the two images). The positional relationship for the two images can be obtained as a parameter that is used to synthesize one image with the other employed as a reference. The positional relationship is represented by equation (8):

[0194] u₀=Mu₁  (8),

[0195] wherein u₀=(x₀, y₀, 1)^T, u₁=(x₁, y₁, 1)^T, $M=(\begin{array}{ccc}{m}_1& m_2& m_3\\ m_4& m_5& m_6\end{array}).$

[0196] In the auto mode, the above processing is automatically performed and two adjacent images are automatically synthesized.

[0197] Upon the selection of the button 1802, the manual mode is designated, and upon the selection of the button 1804, the manual mode is executed. In the manual mode, a user designates the common portion for the two adjacent images. Based on the designated common portion, the positional relationship for the two images is calculated using, for example, equation (8), and based on the obtained positional relationship the two images are synthesized to generate a new image (i.e., a synthesized image is generated using the two images). In the manual mode, as is shown in FIG. 35, the mouse 1216 is employed to designate a common portion 901 for the image 1404aand the image 1404bon the two image synthesis window. Although the manual mode requires a user perform many manipulations, the common portion for the two images can be precisely designated, so that a more accurate image synthesis can be preformed.

[0198] When the synthesis for one image pair has been completed, as is shown in FIG. 36, a window for the synthesis result is presented on the display device 1217. Displayed in this window are a message “Please select ‘NEXT’ button to synthesize next image. For a retry, please select ‘BACK’ button”; a button 2002 for returning to the previous process; a button 2003 for establishing the current process and for instructing a shift to the next process; a button 2004 for instructing a batch synthesis; and a synthesis result 2001 obtained for two images. When a desired synthesis result is obtained, the button 2003 is selected to shift to the process for synthesizing the next two adjacent images. Upon the execution of this shift, the two image synthesis window is displayed for the next two adjacent images, and the same synthesis process is performed for them. When a desired synthesis result is not obtained, the button 2002 is selected to return to the two image synthesis window for the current images, and the image synthesis is retried.

[0199] Another example for the synthesized result display window is shown in FIG. 37. In this window are displayed the same message as in FIG. 36; a button 2102 for returning to the previous process; a button 2103 for establishing the current process and instructing a shift to the next process; a button 2104 for instructing a batch synthesis; and a synthesis result 2101 obtained for two images. This window differs from the window in FIG. 36 in that, as is indicated by the synthesis result 2101, in accordance with the positional relationship the images are displayed with an overlapping transparent portion 2105.

[0200] When all the adjacent, paired images have been synthesized, at step S104 all the obtained images are synthesized in accordance with the synthesis results obtained for all the image pairs, i.e., in accordance with the positional relationships of all the adjacent, paired images. The shift to step S104 is automatically performed when the “NEXT” button is selected in the two image synthesized result window and when all the adjacent, paired images have been synthesized.

[0201] When all the images have been synthesized, a window displaying the all images synthesized result is opened on the display device 1217. As is shown in FIG. 38, in this window are displayed a message “Please select ‘START’ button when storing this image. Please select ‘SYNTHESIS RETRY’ button when retrying synthesis.”; a button 2202 for instructing that an image be stored; a button 2203 for instructing an image synthesis retry; and an obtained synthesized result 2201. When a desired synthesized result is obtained, the user selects the button 2203 to store the obtained synthesized image 2201. The processing is thereafter terminated.

[0202] When a desired synthesized image is not obtained, the user can instruct a synthesis retry for the two adjacent images by selecting the button 2202. In this case, the window in FIG. 34 is opened on the display device 1217, and the process used in the manual mode for retrying the synthesis of two adjacent images is performed in order to acquire a desired synthesized image.

[0203] As is shown in FIG. 39, to retry the all images synthesis, first, at step S901, upon the selection of the button 2201 in the window in FIG. 38 instructing the image synthesis retry, the synthesis window (e.g., the one in FIG. 34) for the first pair of adjacent images is displayed. Then, at step S902 a check is performed to determine whether the user has instructed a synthesis retry for the two images in the two image synthesis window. When a synthesis retry instruction has been issued, at step S903, based on the user's instruction, the template matching employed in the manual mode is performed, or, based on the common portion of the two images that is redesignated by the user, template matching is performed and the positional relationship of the two images is recalculated to synthesize the images. When a synthesis retry instruction is not issued, at step S904 the current positional relationship for the two images is not calculated, and instead, the previously obtained positional relationship is employed to synthesize the two images. In this case, the parameter that was previously obtained using equation (8) and that is stored can be read in order to employ the previous positional relationship.

[0204] After the process at step S903 or S904 is completed, at step S905 a check is performed to determine whether any adjacent image pairs remain. When any adjacent image pairs remain, program control returns to step S902, and the process is repeated until all adjacent image pairs have been synthesized. When no more adjacent image pair remain, it is assumed that all the adjacent image pairs have been synthesized. At step S906 all the obtained images are synthesized, and the synthesis result is displayed in the window in FIG. 38. The processing is thereafter terminated.

[0205] The batch synthesis function that is activated upon the selection of the “BATCH SYNTHESIS” button will now be described while referring to FIGS. 40A and 40B through 43A and 43B. FIGS. 40A and 40B through 43A and 43B are diagrams showing the shifting of the state of a window employed for explaining the batch synthesis function of the image synthesis apparatus in FIG. 27.

[0206] As is described above, when the “NEXT” button or the “BACK” button in a window is selected, the currently displayed window is exchanged for the next window or for the previous window.

[0207] When the “BATCH SYNTHESIS” button is selected in an arrangement designation window 2401 in FIG. 40A, a two image synthesis window 2402 in FIG. 40B is opened, and the auto mode is automatically selected for the two images displayed in the window 2402. In the auto mode, as is described above, the template matching is performed automatically, and the positional relationship represented by equation (8) is calculated. That is, the user need not designate the auto mode or the manual mode in the two image synthesis window 2402. And without the “NEXT” button being selected, the positional relationship of the two images can be automatically obtained in the auto mode. When the positional relationship has been obtained, the two image synthesis result shown in FIG. 41A is not displayed, but instead, for the next two images the two image synthesis window 2404 shown in FIG. 41B is displayed. Then, the positional relationship for the two images displayed in the window 2404 is calculated in the auto mode. Similarly, the two image synthesis result shown in FIG. 42A is not opened, but instead, for the succeeding two images the two image synthesis window 2406 in FIG. 42B is displayed, and in the auto mode, the positional relationship for the two images in the window 2406 is calculated. The window 2407 in FIG. 43A, in which is presented the synthesis result obtained for two images, is also not displayed, and instead, the window 2408 in FIG. 43B, in which is presented the synthesis result for all the images, is displayed. When the synthesis result in the window 2408 is a desired image, this image is stored. When a desired synthesized image is not obtained, the synthesis of two adjacent images is retried.

[0208] An explanation will now be given for the processing performed when the “BATCH SYNTHESIS” BUTTON is selected in the two image synthesis window 2402 in FIG. 40B. The positional relationship for the two images displayed in the window 2402 is calculated in either the auto mode or the manual mode, which has been selected by the user, but the synthesis result window 2403 in FIG. 41A is not displayed. When the positional relationship is obtained, the two image synthesis window 2404 in FIG. 41B for the next two images is displayed. Then, the positional relationship for the two images displayed in the window 2404 is calculated in either the auto mode or the manual mode, which has been selected by the user. After the positional relationship is acquired, a synthesis result window 2405 in FIG. 42A is not displayed, but instead, the two image synthesis window 2406 in FIG. 42B for the next two images is displayed. Following this, the positional relationship for the two images displayed on the window 2406 is calculated, but thereafter, the synthesis result window 2407 in FIG. 43A is not displayed. When the positional relationship for all the pairs of images is obtained and the two image synthesis has been completed, the all images synthesis result window 2408 in FIG. 43B is displayed. When the synthesized result displayed in the window 2408 is a desired image, this result is stored. When a desired synthesized image can not be obtained, the synthesis of two images can be retried.

[0209] An explanation will now be given for the processing performed when the “BATCH SYNTHESIS” button is selected in the two image synthesis result window 2403 in FIG. 41A. When the “BATCH SYNTHESIS” button is selected in the window 2403, first, the window 2404 in FIG. 41B is displayed. The positional relationship for two images displayed in the window 2404 is calculated in the auto mode. Hereinafter, the synthesis result windows 2405 and 2407 in FIGS. 42A and 43A are not displayed, but instead, the two image synthesis window 2406 in FIG. 42B for the next two images is displayed, and the positional relationship for these images in the window 2406 is calculated in the auto mode. When the positional relationship for all the pairs of images has been obtained in the auto mode, the all images synthesis result window 2408 in FIG. 43B is opened. When the synthesis result displayed in the window 2408 is a desired image, this is stored. When a desired synthesized image can not be obtained, the synthesis of two adjacent images can be retried.

[0210] As is described above, when the “BATCH SYNTHESIS” button in the individual windows is designated, the positional relationship for two images is calculated in the auto mode, if not specifically designated, and is employed to synthesize the images. At this time, the synthesis result obtained for the two images is not displayed, and the two images for which the positional relationship is being calculated are displayed for a user.

[0211] As is described above, according to this embodiment, performed are: a designation process for designating a position for each image in accordance with an instruction entered at the keyboard 1211 or with the mouse 1216; a positional relationship calculation process for extracting, for each image, two adjacent images based on the designated arrangement for the images and for calculating the positional relationship wherein the overlapping portions of each two extracted images match; a synthesized image generation process for synthesizing the images in accordance with the obtained positional relationship and for generating a synthesized image; and a positional relationship recalculation process for recalculating the positional relationship for two adjacent images that are designated. Therefore, a large memory capacity is not required to perform the image synthesis processing.

[0212] The positional relationship calculation process is set so that two adjacent images can be designated by manipulating the keyboard 1211 or the mouse 1216; the approximate positional relationship is entered for these two images by manipulating the keyboard 1211 or the mouse 1216; and the positional relationship for the designated two adjacent images is calculated by employing the approximate positional relationship. Therefore, a more accurate positional relationship can be obtained for two images.

[0213] Furthermore, the positional relationship calculation process can be so set that the positional relationship for two adjacent images is automatically calculated en bloc. Therefore, the work required for inputting operations associated with the calculation of the positional relationship can be reduced.

[0214] The retry processing for the synthesis of the individual images can be so set that two adjacent images are designated in the image designation process; the positional relationship between the designated images is recalculated during the positional relationship recalculation process; and the recalculated positional relationship for the two designated images and the previously obtained positional relationship for the other two adjacent images is employed in the image synthesis generation process to perform image synthesis processing. Therefore, the image synthesis can be easily retried.

[0215] Since only two adjacent images are displayed on the display device 1217 for calculating the positional relationship for these images during the positional relationship calculation process, the two images that are to be synthesized can be easily confirmed. In addition, in accordance with the positional relationship existing between the two adjacent images that is acquired during the positional relationship calculation process, the two images are synthesized and the synthesized image is displayed on the display device 1217, so that the synthesized image for the two images can be easily confirmed. To display the synthesized image obtained for the two adjacent images, in the display the two images are overlapped, and the overlapping portion is transparent. As a result, at the overlapping portion the ease with which details can be discerned is superior to that when three or four images are overlapped, and the overlapped portion can be easily confirmed.