Image processing apparatus, method of processing images, and storage medium

[0002] The present invention relates to an image processing apparatus and a method for capturing wide-angle images and presenting them to a viewer, and a storage medium for such images.

[0003] A technology is known where a series of panoramic images along a route are captured and stored in a memory, and the stored panoramic images are presented to a viewer. A panoramic image is a series of pictures stitched together sequentially, the pictures being landscapes surrounding a viewer through 360 degrees span and being captured with a camera. A panoramic image offers the images of landscapes in a view direction of a viewer when the viewer stands at a position on a route and looks at the landscape from there.

[0004] However, in the conventional art, only one series of panoramic images was available for viewing. Therefore, when a viewer observes the sequence of panoramic images, the viewpoint of the viewer is confined on the route in which the sequence of images were captured. Thus, the images of landscapes available to the viewers are limited to particular directions in which the panoramic images were captured, not allowing the viewer to move his viewpoint freely.

[0005] The sequence of images may be interpolated so as to allow the viewer to move his viewpoint to some extent. However, this requires a large amount of data to be processed, imposing a heavy load on a computer and consuming a lot of time for processing the data.

[0006] Use of a three-dimensional geometric model of an environment allows the viewer to move his viewpoint freely. However, such a model results in deteriorated images as compared to the presenting of images to the viewer by switching the images of a series of panoramic images.

[0007] The present invention was made in view of the aforementioned problems. An object of the invention is to allow capturing of wide-angle images, thereby offering more freedom to the viewer in moving his viewpoint when the viewer observes the images.

[0008] In order to achieve the aforementioned objects, the image processing apparatus according to the present invention is constructed as follows.

[0009] An image processing apparatus according to the invention comprises:

[0010] storage means for capturing wide-angle images at positions on multidimensional grids by using a plurality of imaging means each of which captures a predetermined sight, the storage means storing the positions at which the images are captured and a reference direction with respect to which the images are captured, the positions and reference direction being stored corresponding to the images;

[0011] control means for specifying a viewpoint and a view direction of a viewer; and

[0012] presenting means for identifying an image that is captured at a point closest to a viewpoint of a viewer from among the images stored in the storage means, and identifying an image region according to a view direction of the viewer by using the reference direction, and presenting to the viewer an image that falls in the image region.

[0013] The image processing apparatus further comprises sync signal generating means that provides a sync signal to the plurality of imaging means, wherein the plurality of imaging means capture the images in synchronism with the sync signal.

[0014] The image processing apparatus further comprises index generating means that generates and attaches an index to images captured by each of the plurality of imaging means, the images having a common index being stitched together to form a wide-angle image.

[0015] Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0017] FIG. 1 illustrates the production of panoramic images;

[0018] FIG. 2 illustrates panoramic images and a route along which the respective panoramic images are captured;

[0019] FIG. 3 illustrates panoramic images captured along respective routes when panoramic images are captured on a plurality of routes;

[0020] FIG. 4 illustrates a basic configuration of an apparatus that produces a series of panoramic images;

[0021] FIG. 5 illustrates an example of a file that contains images captured;

[0022] FIG. 6 illustrates the movement of a viewer's viewpoint and images presented to the viewer correspondingly;

[0023] FIG. 7 is a flowchart illustrating the procedure for producing panoramic images;

[0024] FIG. 8 illustrates a panoramic image; and

[0025] FIG. 9 is a flowchart illustrating the procedure for presenting a panoramic image to the viewer.

[0026] Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings.

[0027] [First Embodiment]

[0028] This embodiment is directed to an image processing apparatus that allows the viewpoint of a viewer to move in two-dimensional directions and presents panoramic images according to the view direction of the viewer.

[0029] The production of the panoramic pictures will be described. Referring to FIG. 1, a moving base such as a car is equipped with a plurality of image-capturing devices, for example, CCD (charge coupled device) cameras. The plurality of CCD cameras are used to capture landscapes surrounding the car through 360 degrees.

[0030] The car is equipped with a GPS (global positioning system) antenna for a GPS receiver (not shown) that determines an absolute position of the camera, a three-axis angle sensor that measures an inclination of the car with respect to horizontal direction when the cameras are capturing landscapes, and corrects values measured with a later described geomagnetic angle sensor, and the geomagnetic angle sensor that detects a direction (reference direction) in which the car is oriented when the camera captures the landscape.

[0031] The car equipped with the aforementioned devices runs along, for example, a route 201 shown in FIG. 2 while the CCD cameras capture landscapes at respective positions on the route 201 (each point on the route 201 of FIG. 2). The images captured at the same time are stitched together to produce a panoramic picture, thereby yielding a series of panoramic images 200a-200has shown in FIG. 2. The route shown in FIG. 2 is only exemplary. The respective panoramic images are captured in the aforementioned manner.

[0032] Likewise, a plurality of routes as shown in FIG. 3 can be provided to produce corresponding series of panoramic images (more generally by capturing panoramic images on two-dimensional grids).

[0033] As a result, the viewer can move his viewpoint more freely to view the panoramic images of FIG. 6 than when he views the panoramic images of FIG. 2, being allowed to observe panoramic images over a wider range of sight. In FIG. 6, reference numeral 601 denotes a track along which the viewpoint of the viewer moves. As the viewer's viewpoint moves along the track, the viewer observes panoramic images captured at positions shown by larger dots than other positions. As the viewer's viewpoint moves, the panoramic images are switched such that the viewer observes a panoramic image captured at a point closest to the viewer's viewpoint. The specific manner in which the panoramic images are presented to the viewer according to the viewpoint of the viewer will be described later.

[0034] FIG. 4 illustrates a basic configuration of an apparatus that produces a series of panoramic images. The respective devices shown in FIG. 1 are carried on a moving body such as a car.

[0035] Reference numeral 401 denotes video cameras that correspond to CCD cameras (video camera) of FIG. 1. Reference numeral 403 denotes a general computer (referred to as PC hereinafter) that receives images captured by the video cameras 401 to produce the panoramic images. The production of a panoramic image will be described later in detail.

[0036] The PC 403 includes a sync signal generator 404 and a time code generator 405. The sync signal generator 404 provides a sync signal to each of the video cameras 401. The time code generator 405 produces an index that is attached to the images produced by the respective video cameras 401. The time code is used as the index in the first embodiment. The time code generator 405 times at all times so that the video images are assigned time code as the index. The images captured at the same time have a common time code and therefore referring to the time code allows identifying of images that should be connected when a panoramic image is produced.

[0037] In the first embodiment, the aforementioned sync signal generator 404 and the time code generator 405 reside in the PC 403. However, the configuration is not limited to this, and the generators 404 and 405 may be independent of the PC 403 and may be connected to the PC 403.

[0038] The PC 403 incorporates sections 403a-403h.

[0039] Reference numeral 403adenotes a CPU that reads programs and data from a ROM 403band RAM 403c,executes various procedures, and performs an overall control of the PC 403.

[0040] Reference numeral 403bdenotes a ROM that stores a boot program for the PC 403 and various settings for controlling the PC 403. The ROM 403balso stores character codes.

[0041] Reference numeral 403cis the RAM that includes an area in which, for example, programs and data are temporarily loaded from an external storage device 403d.The RAM serves as a work area when the CPU 403aperforms signal processing.

[0042] Reference numeral 403ddenotes the external storage device that holds programs and data that are installed from storage media such as a CD-ROM and a floppy disk. The external storage device 403dcan store a file that serves as an auxiliary work area when the CPU 403aneeds a work area larger than the work area of the RAM 403c.

[0043] Reference numeral 403edenotes an operating section that includes a pointing device such as a key board and a mouse and is used to input various commands into the PC 403.

[0044] Reference numeral 403fdenotes a display in the form of a CRT or an LCD, and displays messages from the PC 403 and results of various signal processes.

[0045] Reference numeral 403gdenotes an interface (I/F) that includes a serial interface and/or a network interface, so that the apparatus can be connected through a serial cable and/or a network cable to peripheral equipment and/or a network.

[0046] Reference numeral 403hdenotes a bus that connects the aforementioned sections.

[0047] Reference numeral 406 denotes a GPS. As described with reference to FIG. 1, the GPS measures an absolute position of a point at which a landscape is captured. The method of measuring the position based on the GPS is well known and the description thereof is omitted.

[0048] Reference numeral 407 denotes a three-axis angle sensor. As described with reference to FIG. 1, the three-axis angle sensor measures an inclination of the car with respect to horizontal direction when the cameras are capturing landscapes, and corrects values measured by the later described geomagnetic angle sensor. The method of correction is not in the scope of the present invention and the description thereof is omitted.

[0049] Reference numeral 408 denotes a geomagnetic angle sensor. As described with reference to FIG. 1, when the cameras capture landscapes, the geomagnetic angle sensor measures a direction in which the car is oriented.

[0050] The signal processing for producing panoramic images performed by the image processing apparatus of the aforementioned configuration will be described.

[0051] As a preparatory work of the capturing of a landscape, the car is positioned at a point on a route along which a series of panoramic images are to be captured. Then, the landscapes surrounding the point are taken by the use of the video cameras 401. The video cameras 401 capture corresponding landscapes in synchronism with a sync signal received from the sync signal generator 404 in the PC 403.

[0052] The images captured by the respective video cameras 401 are input into the PC 403, which in turn attaches a time code generated by the time code generator 405 to the images. When the images are being captured, the GPS 406, three-axis angle sensor 407, and geomagnetic angle sensor 408 measure the position and orientation of the car, which are also attached to the images.

[0053] FIG. 5 illustrates an example of a configuration of a file in which the aforementioned information is attached to the respective images. Reference numeral 500 denotes a file that contains the image 501, time code 502, direction/inclination 503, and position 504. The time code 502 is the information that is outputted from the time code generator 405 when the image 501 was captured and then input into the PC 403. Using the information, images having the same time code can be connected together to produce a panoramic image. When the images captured by the respective cameras 401 are input sequentially into the PC 403, the PC 403 is required to control the order in which the images are input into the PC 403 such that a panoramic image can be produced by connecting the input images in the order in which they are input.

[0054] The direction/inclination 503 is information on the orientation of the car measured by the geomagnetic angle sensor 408 and inclination of the car relative to the ground measured by the three-axis angle sensor 407, both being measured when the image 501 is taken.

[0055] The position 504 is a position of the car measured by the GPS 406 when the image 501 is captured, i.e., a position of a panoramic image on a route produced based on the image 501. By referring to the position information 504, a point on the route closest to the viewpoint of the viewer can be identified, and a panoramic image corresponding to the identified point can be identified.

[0056] The aforementioned operations are performed at each point (e.g., 200ato 200hof FIG. 2) to capture images surrounding the point. Likewise, the aforementioned operations may be performed at each point on a plurality of routes (e.g., respective positions on the routes 301 to 303 of FIG. 3) to capture images surrounding the point.

[0057] After the above-described capturings, panoramic images are produced from images in the files of FIG. 5 that contain the images captured by the video cameras 401. Specifically, images in the files having the same time code are stitched together in the order in which they were input into the PC 403. When the thus produced panoramic images are stored in a storage device such as the external storage device 403d,the panoramic images may be subjected to image compression such as JPEG.

[0058] FIG. 7 is a flowchart that illustrates the above-described signal processing. At step S701, images surrounding a point on a route are captured and the position and orientation of the car at that time are measured. Then, at step S702, the PC 403 receives information such as the images captured, position of the car, and orientation of the car. At step S703, the position of the car and orientation of the car are attached to images captured. The steps S701-S703 are performed for each of the positions on the route. After having captured images at all positions at step S704, the images having the same time code are stitched together to produce a panoramic image (step S705).

[0059] Using the series of panoramic images in a plurality of routes produced through the aforementioned processing, as described with reference to FIG. 6, the panoramic image captured at a point that is closest to a viewpoint of the viewer may be identified, thereby presenting panoramic images to the viewer according to the viewpoint of the viewer. The viewpoints of the viewer may be inputted to the PC 403 by using, for example, a joystick as the operating section 403e.The joystick is also used to input the view direction of the viewer into the PC 403.

[0060] Then, the method of identifying panoramic images presented to the viewer in accordance with the view direction of the viewer (i.e., the method of identifying a region of panoramic image according to the view direction of the viewer) will now be described. The image presented to the viewer according to the view direction of the viewer has h dots in a vertical dimension of the image and d dots in a horizontal dimension of the image.

[0061] FIG. 8 illustrates a panoramic image 800 having a horizontal dimension of w dots and a vertical dimension of h dots, with an origin (0, 0) of the coordinate at an upper left corner of the image. As described previously, the panoramic image 800 is an image expressed by w dots in the horizontal direction that covers a sight of 360 degrees span. Assuming that the car runs in a direction shown by θ, an image 801 as seen in the front direction can be produced by reading an image portion of a rectangular area of the panoramic image 800, the rectangular image portion having an upper left corner thereof at a coordinate ((θ/360)×w, 0) and a lower right corner thereof at a coordinate ((θ/360)×w+d, h). Thus, if the view direction of the viewer deviates by a from the front direction (i.e., θ+α), the image 802 presented to the viewer can be produced by reading another rectangular image portion of the panoramic image 800, the image portion having an upper left corner thereof at a coordinate ((θ+α/360)×w, 0) and a lower right corner thereof at a coordinate (((θ+α)/360)×w+d, h).

[0062] The direction of angle θ in which the car runs is measured by the geomagnetic angle sensor 408. Thus, an image according to the view direction of the viewer can be determined from the panoramic image having a width w. The image presented to the viewer is displayed on the display 403f.

[0063] FIG. 9 is a flowchart illustrating a procedure in which an image according to the viewpoint and view direction of the viewer is determined from a series of the produced panoramic images and presented to the viewer.

[0064] At step S901, the viewpoint and view direction of the viewer are set by an operating section, for example, a joystick. At step S902, a panoramic image is selected from among a plurality of panoramic images, the selected panoramic image being an image captured at a point that is the closest to the viewer's viewpoint. At step S903, an image according to the view direction set at step S901 is determined from the image selected at step S902. At step S904, the image according to the viewpoint and view direction of the viewer, determined at steps S902 and S903, is displayed on a display 403f.

[0065] The method of processing images and image processing apparatus therefor permit more freedom of the movement of the viewer's viewpoint in the two-dimensional directions than the method and image processing apparatus in which panoramic images are taken at a plurality of positions along a single route and presented to the viewer.

[0066] Because the images presented to the viewer are those actually captured, images presented to the viewer are better than those produced by using, for example, a three-dimensional virtual object.

[0067] [Second Embodiment]

[0068] In the first embodiment, a plurality of panoramic images are taken along a plurality of routes. In other words, the images are captured on two-dimensional grids. However, capturing images on three-dimensional grids permits more freedom to the movement of viewer's viewpoint.

[0069] [Third embodiment]

[0070] In the first and second embodiments, images that were captured at the respective positions in the past may be stored in the external storage device 403dso that the viewer obtains freedom in time domain. In other words, when the viewer wants to view an image of one year ago at a desired photographing location, the joystick, if used as the operating section 403e,maybe operated to set the viewpoint of the viewer in the time domain so that the viewer can view the panoramic image of one year ago at that point.

[0071] Note that the present invention may be applied to either a system constituted by a plurality of devices (e.g., a host computer, an interface device, a reader, a printer, and the like), or an apparatus consisting of a single equipment (e.g., a copying machine, a facsimile apparatus, or the like).

[0072] The objects of the present invention are also achieved by supplying a storage medium, which records a program code of a software program that can implement the functions of the above-mentioned embodiments to the system or apparatus, and reading out and executing the program code stored in the storage medium by a computer (or a CPU or MPU) of the system or apparatus.

[0073] In this case, the program code itself read out from the storage medium implements the functions of the above-mentioned embodiments, and the storage medium which stores the program code constitutes the present invention.

[0074] As the storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, nonvolatile memory card, ROM, and the like may be used.

[0075] The functions of the above-mentioned embodiments may be implemented not only by executing the readout program code by the computer but also by some or all of actual processing operations executed by an OS (operating system) running on the computer on the basis of an instruction of the program code.

[0076] The present invention includes a product, e.g., a printout, obtained by the image processing method of the present invention.

[0077] Furthermore, the present invention also includes a case where, after the program codes read from the storage medium are written in a function expansion card which is inserted into the computer or in a memory provided in a function expansion unit which is connected to the computer, a CPU or the like contained in the function expansion card or unit performs a part of or an entire process in accordance with designations of the program codes and realizes functions of the above embodiments.

[0078] In a case where the present invention is applied to the aforesaid storage medium, the storage medium stores program codes corresponding to the flowcharts (FIG. 7 and/or FIG. 9) described in the embodiments.

[0079] As many apparently widely different embodiments of the present invention can be made without departing from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments thereof except as defined in the appended claims.