Method and Apparatus for Merging Real and Synthetic Images

1. Field of the Invention

The present invention is directed generally to the field signal processing and, more particularly, to the field of video signal processing.

2. Description of the Background

It is known in the art to use color as a basis for merging pixel by pixel, two video signals. Chroma-keying, as the process is known, uses color information to control a switching process. A person, for example a weatherperson, stands in front of a blue screen while a first video signal is made. A second video signal is provided which is to be combined with just the weatherperson from the first video signal. The first video signal is analyzed pixel by pixel for the key, i.e., the blue of the background screen. For each blue pixel, the corresponding pixel from the second video signal is selected for the final image. Each non-blue pixel from the first video signal is selected for the final image. Through that simple process, the weatherperson is extracted from the background screen and superimposed on top of the images in the second video signal. The chroma-key process is simple enough to be performed in real time. However, the extracted image is always considered to be in front of the images from the second video signal. That may or may not result in a realistic looking final product. Furthermore, because the extracted image is always in front of the images from the second video signal, there is no physical or geometrical interaction allowed between the extracted image and the images of the second video signal.

Other methods for combining video signals are disclosed in U.S. Pat. No. 5,400,080 entitled Apparatus And Method For Combining Video Signals Representing Images Having Different Depths, U.S. Pat. No. 5,353,068 entitled Video Signal Combining Apparatus And Method, and U.S. Pat. No. 5,280,337 entitled Depth-Based Video Combining. Each of those patents discloses a method which uses depth information for combining one or more video signals. However, the patents do not disclose how to obtain the needed depth information, which is a non-trivial problem if pixel by pixel depth information is to be provided in real time for standard video signals having a frame rate of thirty frames per second. Moreover, there is no recognition in the aforementioned patents of how images from the combined video signals might interact, e.g. one image might cast a shadow on an image which is behind it in the combined video. Thus, the need exists for a method of using depth as the key for merging two video signals in real time in a way which enables the images from the combined video to interact with one another.

The present invention is directed to a method, and to an apparatus for carrying out the method, by which objects in real images and synthetic images can dynamically interact with each other in a geometrically correct manner. In this invention, a 3-D camera is used for acquiring video images and depth information for each pixel of a video image in real time. The 3-D geometry of every pixel obtained in real time is used for calculation of the geometrical relationship with objects in a synthetic image. The method is comprised of the steps of providing a first signal containing depth and image information per pixel about a real image. A second signal containing depth and image information per pixel about a synthetic image (which may include virtual images) is provided. The depth information corresponding to the real image and the depth information corresponding to the virtual or synthetic image for each pixel is compared. Based on the comparison, either the image information corresponding to the real image or the image information corresponding to the synthetic image is selected. The selected information is then combined on a per pixel basis

The method and apparatus of the present invention allow for the combination, based on depth information, of two (or more) images, one real and one synthetic, in real time. Because the depth information for each pixel is known, any interaction between pixels can be determined. For example, if two pixels have the same depth information, they occupy the same point in space, which means that the sound of a collision may be generated. By knowing the relationship of a synthetic object with respect to a light source, the synthetic object may cast a shadow on a real object, or vice versa. By having real time depth information for the images to be combined, the images can be combined in a much more realistic manner. Those advantages and benefits of the present invention, and others, will become apparent from the Description of the Preferred Embodiments hereinbelow.

For the present invention to be clearly understood and readily practiced, the present invention will be described in conjunction with the following figures wherein:

FIG. 1 is a diagram illustrating an implementation of the merging and interaction of a real image and a synthetic image according to the present invention;

FIGS. 2A and 2B are examples of an intensity map and a depth map, respectively, for a synthesized image;

FIGS. 3A and 3B are examples of an intensity map and a depth map, respectively, for a real image;

FIG. 4 is an example of an output image;

FIG. 5 is a diagram illustrating an implementation of the present invention using a pixel-by-pixel depth key switch.

FIG. 6 is a diagram illustrating an implementation of the present invention using a pixel-by-pixel depth key switch and a image mask for a real object;

FIG. 7 is a block diagram illustrating the process for calculating a shadow cast by a synthetic image onto a real image;

FIG. 8 is an example of an image with a shadow cast by a synthetic object;

FIG. 9 is a diagram illustrating the process for calculating the generation of a sound resulting from the collision of a synthetic object and a real object;

FIG. 10 is a diagram illustrating the creation of a reflection in a synthetic mirror; and

FIG. 11 is a diagram which aids in the description of the process of creating a reflection in a synthetic mirror.

As illustrated in FIG. 1, the present invention is directed to a system 10 and method of combining, in real time, objects in real images and synthetic images. The system 10 is comprised of an image rendering system 12 which may be a commercially available CAD system for producing images for which depth information is available for each pixel. The synthetic image may be comprised of a depth image or depth map 14 (an example of which is seen in FIG. 2B) and a map 15 (an example of which is seen in FIG. 2A) which contains color, intensity, or other image information.

A 3-D camera 17 is provided to produce real time images of a real object 19. The 3-D camera 17 outputs video images and 3-D geometry in real time. The real-time acquisition of images and depth information is essential for dynamic interaction between real and synthetic objects. At the output of the 3-D camera 17 the 3-D geometry of a real scene is represented by a pixel-by-pixel depth map 20 (an example of which is seen in FIG. 3B) while the color, intensity, etc. is represented by a pixel-by-pixel map 21 (an example of which is seen in FIG. 3A). A video rate stereo machine is one type of device which may be used for such a 3-D camera 17. Such machines are disclosed in, for example, T. Kanada et al., A Stereo Machine for Video Rate Dense Depth Mapping and Its New Applications, CVPR 96, 1996, which is hereby incorporated by reference.

The 3-D coordinate (x,y,z) of an object at image coordinates (i,j) can be calculated by the following equation:

where DR(i,j) is the depth value of the real object 19 at (i,j) and XR, YR, and ZR are the camera geometry functions of the 3-D camera 17. Those camera geometry functions can be determined by existing camera calibration procedures. See, for example, R. Tsai, A versatile camera calibration technique for high-accuracy 3D machine vision meteorology using off-the-shelf TV cameras and lenses, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, August 1987, and S. Kimura, et al., CMU Video-Rate Stereo Machine, Mobile Mapping Symposium, Columbus, Ohio, May 24-26, 1995, both of which are hereby incorporated by reference. If the z axis is parallel to the camera direction, then the z component can be simply written as:

As mentioned, synthetic images with their own 3-D geometry can be created by existing computer graphics technique. At the output of the image rendering system 12, the 3-D geometry of a synthetic scene is represented by pixel-by-pixel depth information DS(i,j). The viewing conditions of the synthetic images should be the same as the viewing conditions of the 3-D camera 17. In other words, the "camera geometry functions" of the synthetic image (XS, YS, ZS) have to be the same as those of the 3-D camera's (XR, YR, ZR). The synthesized images should be synthesized in real time if objects in the images move or the3-D camera geometry changes. Surface features of synthetic objects such as transparency FS1, reflectivity FS2, etc. are defined explicitly, so that kind of information can be output pixel-by-pixel if needed and stored with map 15.

If there is a specific object in a synthetic image which to be combined, an image mask MS(i,j) with the following definition can be created by the image rendering system 12: ##EQU1## If there is no specific object for processing, and the entire image is to be used, an image mask value should be set at "1" for all values of (i,j).

If there is a specific object in a real image which is to be combined, an image mask MR(i,j) with the following definition can be created by an image mask generator 22, which may be a specific software routine: ##EQU2## An image mask for an object in a real image can be created using color information or depth information from the 3-D camera 17.

A typical method of creating a mask image with color information is chroma keying which extracts the region which is not in the specific color. An image mask MR(i,j) also can be created by comparing the depth information DR(i,j) and a background depth information DRb(i,j) which is captured by a 3-D camera without the specific real object: ##EQU3## If there is no specific object for processing and the entire image is to be used, an image mask value should be set at "1" for all values of (i,j).

After the maps 14, 15, 20, 21 have been created, the geometrical relationship between objects in the real and the synthetic images is examined and interaction between objects is determined as shown by box 23 in FIG. 1. The typical output is one or more output images IO28 (an example of which is seen in FIG. 4) where real and synthetic images are merged and a real object and a synthetic object interact with each other. Other output signals 28 such as sounds, lights, etc., are also possible. Sounds and other output signals can be generated with existing signal generators 29. The images IO28, whether they are individual frames or a series of frames forming a video, can be displayed using any suitable monitor 27.

In the most general form, the output image IO(i,j) is determined with the following function:

where IR is a real image from the 3-D camera 17, DR is the pixel-by-pixel depth information corresponding to IR, IS is the synthetic image, DS is the pixel-by-pixel depth information corresponding to IS, MR is the image mask of an object in the real image, MS is the image mask of an object in the synthetic image, and FS.sub.1, FS.sub.2 . . . are additional pixel-by-pixel information attached to the synthetic image. Note that on the right hand side of equation (1), IR, IS, etc. are arrays which have dimensions of image size and not the local values at (i,j) (IR(i,j), IS(i,j), etc.). IR, IS, etc. also can be sequences arrays if needed. If IR and IS are color images, IR is the bundle of three image arrays IRr, IRg, and IRb which stand for red, green, and blue bands of the image IR, respectively, and is the bundle of three image arrays ISr, ISg, and ISb.

Examining objects in the real and the synthetic images can be accomplished because both the real and synthetic images have pixel-by-pixel depth information. Therefore, each pixel of the real and the synthetic images can have 3-D coordinates. That means that an output image can be created to reflect the 3-D relationship of all image pixels of the real and the synthetic images. Inputs are received from both the real and synthetic world in real time. Therefore, an object (typically a person) in the real world can dynamically interact with the synthetic (or virtual) world in real time.

The following illustrates some examples of how the geometrical relationship may be calculated.

Depth keying is a technique to merge real and synthetic images in a geometrically correct manner by using the pixel-by-pixel depth information (depth map) to control the image information used in the final image. As shown in FIG. 5, this technique is implemented by a depth key switch 24 which performs the function of determining the geometrical relationship of box 23. For each pixel, the depth key switch 24 compares depth information of the real and the synthetic images and routes the pixel value of the image which is nearer to the camera. Thus, the foreground image for each pixel can be determined and image created where each part of a real and a synthetic object occlude each other correctly as illustrated in the output image 28 of FIG. 4.

The function of the depth key switch 24 can be written in the following form. ##EQU4## As shown in FIG. 6, the depth key switch 24 can also be used with the image mask generator 22. Thus, when there is some specific object in the real image, equation (2) can be written as following: ##EQU5## An example of a software routine, written in C language, for implementing the functionality of equations (2) and (3) is four in Appendix A.

A shadow cast by a synthetic object onto a real object can be created by considering the geometrical relationship between a synthetic object, a real object, and a light source. FIG. 7 is a diagram of that process. In FIG. 7, the function of the block 23 is performed by a software routine 38, an example of which, written in C language, is found in Appendix B. Basically, the shadow is created by exchanging the relevant portions of the real and synthetic images.

In FIG. 7, the routine 38 begins at step 40 where the position of the shadow on the real image IR is determined. A pixel on a real object is considered as being in the shadow of a synthetic object when a pixel on a synthetic object lays in the direction to a light source. If V.sub.rs (i,j,s,t) is a 3-D vector from the position of IR(i,j) to the position of IS(s,t), and V.sub.ro (i,j) is a 3-D vector from the position of IR(i,j) to the position of the light source, then a mask Msh(i,j) can be created which has a value of "1" if IR(i,j) is in the shadow of a synthetic object: ##EQU6## where the operator (.sym.) is the inner product and .crclbar..sub.th is a threshold angle.

A new image IR' containing the shadow is created at step 42 with the following equation. ##EQU7## where Sh is a function for calculating a pixel value in a shadow. For example, Sh can be as below:

where k is a constant value between 0 and 1.

Finally, at step 44, IR' and IS are merged by, for example, the depth keying method described above. An example of such an image 46 is illustrated in FIG. 8.

In the geometrical calculation process, it is possible to compute the geometrical condition for generating signals other than images. For example, a sound can be generated when a collision between a synthetic and a real object occurs. That process is illustrated in FIG. 9 wherein the computation of the geometrical condition for generating signals other than images is performed by a software routine 48 which may be incorporated into the functionality of box 23.

The condition of the collision of a real object and a synthetic object can be denoted with a pixel count threshold N.sub.th and a collision distance threshold D.sub.th :

where ##EQU8## A sound generator 50, or other appropriate signal generator, may be provided to produce an audible output.

A virtual mirror which reflects both real and synthetic objects can be created by considering the geometrical relationship between a synthetic object, a real object, a virtual mirror, and the camera position. That process can be implemented as a software routine 52 as part of the functionality of box 23 as shown in FIG. 10. The process begins at step 54 by defining a virtual mirror (56 in FIG. 11), which contains the following information:

Mm(i,j): an image mask which is 1 if a pixel (i,j) is on the mirror, otherwise 0.

Vm(i,j): unit vectors of a mirror surface direction at a pixel (i,j).

Dm(i,j): depth information of a mirror surface at a pixel (i,j).

Each 3-D position of a mirror surface at a pixel (i,j) is calculated with the camera geometry functions and depth information. Referring to FIG. 11, Vcm(i,j) is a vector from the camera position to the mirror's 56 surface. If Mm(i,j)=1, a unit vector Vref(i,j) of the direction from the mirror's 56 surface to an object which is supposed to be reflected by the mirror 56 is: ##EQU9## Thus, we can denote the condition that an image pixel whose position from the camera (V) is on the direction Vref(i,j) from the mirror's 56 surface (Vcm(i,j)) as follows: ##EQU10## where .crclbar..sub.th is a threshold angle. That condition can be examined for each pixel of IR(i,j) and IS(i,j) using pixel-by-pixel depth information DR and DS, respectively. That examination can be eliminated for the pixels whose image mask value MR(i,j) or MS(i,j) is 1. The reflected image pixel is the pixel which has the smallest value of .vertline.V-Vcm(i,j)] among the pixels which satisfies that condition. Applying that condition for all IR and IS, we get a reflected image Im(i,j) at step 58. If no pixel of IR or IS satisfies that condition, Im(i,j) is assigned a default intensity (or color), for example, zero (or black).

With an image mask Mm and pixel-by-pixel depth information Dm, the reflected image IM can be merged with a synthetic image IS by, for example, depth keying as shown by step 60. The image information IS', DS' output by step 60 may then be merged, for example, by depth keying as shown by step 62. This time, the depth information of IS' (i.e., DS') has the following definition: ##EQU11##

The same principals may be used to enable synthetic images to be reflected in real mirrors or other similarly reflective objects.

Set forth in the Appendices hereto is specific software, i.e. specific program code segments, that are employed to configure a general purpose microprocessor to create specific logic circuits. Those circuits are intended to be "means" corresponding to any claimed means limitations. Those of ordinary skill in the art will recognize that many aspects of the present invention such as the image rendering system 12, image mask generator 22, and the functionality of box 23, as well as the generation of appropriate output signals, may be implemented in software. When implemented in software, the software may be carried by any appropriate computer media, such as disks, CD ROMs, tapes, etc.

While the present invention has been described in conjunction with preferred embodiments thereof, many modifications and variations will be apparent to those of ordinary skill in the art. The foregoing description and the following claims are intended to cover all such modifications and variations.