Display techniques for three-dimensional virtual reality

[0002] This invention relates to the integration of computer graphics and video to provide a realistic three dimensional virtual reality experience.

[0003] The display of a three dimensional world to a viewer requires considerable computation power, and it is typically costly to develop the necessary highly detailed models required for doing so. In order to simplify the problem, a portion of the world that is in the distance may be represented as a video displayed on a surface, e.g., a screen. By video it is meant the common usage of the term, such as the placing or projecting of predefined images on the surface, e.g., the electronic version of filmed moving pictures. Thus, such a world is essentially truncated in length to the screen on which the video is displayed. A great reduction in computation power and cost can be achieved by such an arrangement.

[0004] A limitation of such a world occurs when an object represented by computer graphics undergoes a trajectory that takes it to a location in the world that is not represented as computer graphics but instead is within the field represented by the video. If the object is generated at its proper location using computer graphic techniques, without further processing such an object becomes invisible to a viewer in front of the video screen, because the object is blocked from a viewer's view by the video screen. In U.S. patent application Ser. No. 08/906215 it was recognized that, when a computer graphic object undergoes a trajectory that takes it to a location in the world that is not represented as computer graphics, but instead is within the field represented by the video, that such an object should be represented as video on the screen, rather than computer graphics. Doing so prevents the object from becoming invisible as it would if it was generated at its proper location using computer graphic techniques, since it would be blocked from view by the video screen. Thus, the computer graphics object “goes into the video” as video and remains visible to a viewer in front of the video screen.

[0005] We have recognized that when a computer graphic object is located at, or undergoes a trajectory that takes it to, a location in the world that is not represented as computer graphics, but instead is within a field represented by video, one may wish to maintain the object as computer graphics, yet still have the object be visible to a viewer and appear as if the object was at its proper location within the video. This can be achieved, in accordance with the principles of the invention, by a) determining when a computer graphic object is located at, or undergoes a trajectory that takes it to, a location in the world that is not represented as computer graphics, but instead is within the field represented by the video, and b) representing such an object as a computer graphic which is in front of the screen and to which at least one transformation, e.g., a perspective transformation, is applied. The perspective transformation may be at least a function of the distance beyond the video screen at which the object would actually be located if it were to be drawn using the same representation employed prior to its crossing the video screen. In other words, instead of the computer graphics object “going into the video” as video, the object as computer graphics is made to appear as if it went into the video by using perspective techniques. Thus, the object remains visible to a viewer in front of the video screen rather than becoming invisible because it is blocked from view by the video screen as would have occurred had the object been generated at its proper location.

[0006] The file of this patent contains at least one drawing executed in color. Copies of this patent with color drawing(s) will be provided by the Patent and Trademark Office upon request and payment of the necessary fee.

[0007] FIGS. 1-4 show an example, in accordance with the principles of the invention, of a computer graphics object that is made to appear using perspective techniques as if it went into the video as computer graphics;

[0008] FIG. 5 shows an exemplary process for displaying the world to a user, in accordance with the principles of the invention;

[0009] FIG. 6 shows an exemplary process, in accordance with the principles of the invention, for determining the set of objects and/or transformed objects that are included in the set of objects displayed in by the process of FIG. 5 in the computer graphics part of the world; and

[0010] FIG. 7 shows a simplified representation of FIG. 4

[0011] FIGS. 1-4 show an example, in accordance with the principles of the invention, of a computer graphics object that is made to appear using perspective techniques as if it went into the video as computer graphics, and the computer graphics object remains visible to a viewer in front of the video screen rather than becoming invisible because it was generated at its proper location, and so was blocked from view by the video screen. More specifically, FIGS. 1-4 show what happens when a computer graphic object is at a location in the world that is not represented as computer graphics, but instead is within the field represented by the video, and such an object is represented according to the invention in front of the screen as computer graphics to which at least one perspective transformation is applied. The perspective transformation shown is at least a function of the distance between the viewer and the location at which the object would actually be located if it were to be drawn using the same representation employed prior to its crossing the video screen.

[0012] FIG. 1 shows world 101, which is a bicycle path in a park, e.g., Central Park in New York City. World 101 is divided into two portions, video screen 103, on which is shown the current frame of a video and the remainder of the world 105, which is represented using computer graphics techniques, and is thus referred to herein as computer graphics part (CG Part) 105. Within CG Part 105 there are various elements, such as bicyclist 107 representing the user, another bicyclist 109, and road 111. In FIG. 1 bicyclists 107 and 109 are shown using their untransformed, or base, appearance. For clarity purposes, the border of video screen 103 is highlighted. However, as portions of video screen 103 are occluded by other objects, such as portions of the bodies of bicyclists 107 and 109 as well as sides of road 111, the border section of such occluded portions cannot be seen, and so there is no highlighting in that area.

[0013] Note that the viewpoint of the user is actually behind the representation of both bicyclists 107 and 109. Also note that bicyclist 107 is moving slower than bicyclist 109. If this continues, eventually, bicyclist 109 will reach screen 103 while bicyclist 107 lags behind. This occurs in FIG. 2, which shows world 101 of FIG. 1, but at a later time and from a side perspective. If the relative motion difference continues between bicyclists 107 and 109, part, and then all, of bicyclist 109 will intersect with, or go past, video screen 103. If this happens, part, and then all, of bicyclist 109 will become occluded by video screen 103 from the view of a user in front of video screen 103, and so bicyclist 109 will no longer be visible to such a user. Therefore, in accordance with the principles of the invention, the appearance of bicyclist 109 is transformed into transformed bicyclist 309, e.g., using perspective techniques, so that it gives the illusion that bicyclist 309 appears to be within the video that is displayed on video screen 103 rather than being represented using the same appearance as is used for bicyclist 109 within CG Part 105, as shown in FIG. 3. Note that by perspective techniques it is meant techniques that apply at least one of scaling, translation, and orientation manipulation to the bicyclist. Also note that transformed bicyclist 309 is referred to as a so-called “trace” of bicyclist 109.

[0014] The determination to represent an object, such as bicyclist 109, using a transformed appearance may be made when the object is within a selected distance from one or more points on the video screen rather than requiring the change in representation to be at the exact point when the object intersects the video screen. This is especially useful since the video screen may be any surface and need not be simply a planar surface. Furthermore, the distance employed may be determined dynamically, e.g., as a function of the speed and trajectory of the bicyclist. The change in appearance may also take place only once it is determined that a predetermined portion of the object has actually intersected, or passed through, the video screen.

[0015] In other implementations of the invention, the computer untransformed representation of bicyclist 109 may remain visible from other viewpoints, such as a side view. Thus, in accordance with an aspect of the invention, the determination as to whether or not to transform an object that is being represented using computer graphic techniques rather than continuing to be represented untransformed may be additionally a function of the viewpoint of the user.

[0016] A corresponding side view of the world as it appears in FIG. 3 is shown in FIG. 4. Also shown in FIG. 4 is “ghost” bicyclist 409, which represents the actual location and appearance of bicyclist 109 from which transformed bicyclist 309 is generated. Although not done so for FIG. 4, which is merely for pedagogical purposes, preferably, but not necessarily, both transformed bicyclist 309 and the untransformed bicyclist 409 should not visible to a viewer at the same time.

[0017] In the event that bicyclist 107 speeds up or bicyclist 109 slows down, so that based on the relative motion between them bicyclist 109 would no longer be at or beyond video screen 103, bicyclist 109 may again be rendered using the untransformed appearance, as shown in FIG. 1.

[0018] As described in connection with FIGS. 1-4, the transformed representation, e.g., bicyclist 301, is displayed immediately adjacent to video screen 103. However, doing so is not necessary. Instead, the transformed representation may be placed anywhere between video screen 103 and the viewer's viewpoint—provided the transformation is properly conformed to the location of the transformed object. However, an advantage of placing the transformed representation adjacent to video screen 103 is that by doing so any items that are between video screen 103 and the view's viewpoint that should occlude a portion of the transformed object will do so. Otherwise, if the transformed object is placed closer to the user's viewpoint, although the object will look correct to a viewer, objects between the location of the transformed object and the screen will not occlude the transformed object as they would if the untransformed object could be rendered and remain visible at its actual expected location.

[0019] FIG. 5 shows an exemplary process for displaying the world to a user, in accordance with the principles of the invention. The process is entered in step 501 whenever the world view of the user needs to be updated. It may be necessary to update the world view of the user a) whenever a new frame of video must be displayed; b) when an object within computer graphics part of the world, e.g., CG Part 105, has moved; c) when the user's viewpoint is changed; or d) for any combination of the foregoing reasons. Thereafter, steps 503 and 505 are performed, and these steps may be performed in parallel. In step 503, each object, or a transformed version of the object if the object has gone past the video screen, that is part of the set of objects that is within the currently displayed portion of the computer graphics portion of the world is rendered within the computer graphics part of the world. In step 505 a frame of video is displayed on video screen. This may be either a new frame of video, if it is time to update the video frame being displayed, or it may be the same frame of video that was previously displayed, if the world view of the user is being updated because of a need to change the representation of the computer graphics part of the world. The process then exits in step 507.

[0020] FIG. 6 shows an exemplary process, in accordance with the principles of the invention, for determining the set of objects and/or transformed objects from which are selected the objects and/or transformed objects displayed in step 503 of FIG. 5. The process is entered in step 601 whenever set of objects and/or transformed objects for display in the computer graphics part of the world must be updated. Thus, execution of the process of FIG. 6 may be linked to execution of step 503 of FIG. 5 or it may be independent thereof. Next, in step 603, the next previously unchecked object is obtained, so that the process may check if it needs to be transformed and to do so if necessary. Thereafter, conditional branch point 605 tests to determine if the object has intersected, or gone beyond, video screen 103. This determination may be made for the object on the basis of a designated point on the object, which may be prespecified, e.g., as part of the model of the object.

[0021] If the test result in step 605 is NO, control passes to step 607 in which the object is added to a so-called “display set” for use in step 503. Control next passes to conditional branch point 609, which tests to determine if there remains more unchecked objects. If the test result in step 609 is YES, control passes back to step 603 to check the next object. If the test result in step 609 is NO, control passes to step 611 and the process is exited.

[0022] If the test result in step 605 is YES, indicating that the object has passed beyond the video screen and will otherwise become occluded from the viewer's view, a process is undertaken in steps 613 through 621 to develop the transformed representation of the object, in accordance with the principles of the invention. More specifically, in step 613, a duplicate of the object is developed, and in step 615 a reference point for the object is selected. The point chosen is at the discretion of the implementer. It may correspond to the predesignated point of object that was employed in step 605. Typically, the point chosen will—although it is not required to—be within the bounding box of the object. The remaining steps are more easily understood in connection with the example shown in FIG. 7.

[0023] FIG. 7 shows a simplified representation of FIG. 4, including a) video screen 103, b) bicyclist 107, c) transformed bicyclist 309, and d) original bicyclist 409 for which transformed bicyclist 309 is developed. Also shown is a line extending from point VP, which is the location of the viewpoint of FIGS. 1 and 3, to point B, which is the reference point chosen in step 615 for bicyclist 409. Along the VB-B line is point S, which is the point at which the VB-B line intersects video screen 103. Point T, also along the VB-B line, corresponds to the point at which the duplicate is translated in step 617, in accordance with the principles of the invention. Specifically, point T corresponds to the reference point of the duplicate but it is located at a point, selected by the implementer, that lies between S and VP, preferably, but not necessarily, as close to S as possible, while insuring that that the duplicate will never intersect the video screen when it is rotated about any axis passing through point T, based on the size of the transformed object when it is located at point T. A conservative way to select point T is to use the full sized object to be transformed, e.g., bicyclist 409.

[0024] In step 619, the duplicate is scaled to match its translated location, so that it appears to a viewer at VP to be in proper perspective, in accordance with the principles of the invention. In an exemplary embodiment of the invention, the scaled size of the duplicate object is determined by using a scale factor which is the ratio of the distance from S to VP with respect to the distance from B to VP. The scale factor is then multiplied by the original size of the object, to determine the size of the transformed object.

[0025] Note that in this embodiment of the invention, there is no need to change the orientation of the transformed object from that of the original object, although both its location and it size are transformed.

[0026] The duplicate object as scaled and translated is added to the display set. The process then continues at step 609 as described above.

[0027] Using a duplicate of the original object gives a seamless appearance to the object as its actual position crosses between the computer graphics area and the video screen. However, a duplicate of the original object need not be employed. Instead, an alternate object may be used to represent the original object as, or as part, of the transformation.

[0028] The processes of the invention may be implemented by a system which includes a processor running software to execute the functionality of the processes described herein, and the resulting images displayed on a display, such as a computer monitor.

[0029] The foregoing merely illustrates the principles of the inventions. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.