System and method for enhancing the realism of a displayed image

[0002] This invention relates to image displays and more particularly to enhancing the realism of images by creating an illusion of environmental lighting effects from and on the images.

[0003] The present invention relates to the field of optics, and this application makes use of several terms of art used throughout the field. The inventors have adopted and use the following definitions, and they are therefore included in this patent. A ‘real image’ is an optical image formed of real foci (Merriam-Webster's Online Collegiate Dictionary). In other words, real images are made by light rays that are brought to a focus at the image position, and therefore can be made visible on a surface, such as a screen or smoke. See Warren J. Smith, Modem Optical Engineering, 8 (McGraw-Hill 1966). A ‘virtual image,’ on the other hand, is an image formed of points from which divergent light rays seem to emanate without actually doing so (Merriam-Webster's Online Collegiate Dictionary) . For example, the virtual image in a plane mirror is at some distance behind the mirror. Virtual images cannot be made visible on a screen because they are made by light rays that do not actually come from the location at which the image appears. See Warren J. Smith, Modem Optical Engineering, 9 (McGraw-Hill 1966). As used throughout the specification, a ‘negative image’ is a negative of an image, or an image having the light and dark parts in approximately inverse order to those of the original subject (Merriam-Webster's Online Collegiate Dictionary). A negative image may be created by reversing the polarity of an image, digital signal processing, or forming a two-dimensional ‘gobo,’ which is an opaque cut-out resembling the original subject to be placed near a light source. A ‘focal plane’ is a plane that is perpendicular to the axis of a lens or mirror and passes through the focus of the lens or mirror (Merriam-Webster's Online Collegiate Dictionary).

[0004] Movies, slides, video, and other displayed images have long been a major source of audience entertainment. Much effort has been expended to make images appear more like objects in three-dimensional space in order to make them more compelling to the viewer, and thereby increase the entertainment value of such images. A variety of techniques have been employed to cause displayed images to appear more like objects in three-dimensional space. One technique is to create “aerial” images, which are virtual images that are projected into free space, producing the illusion of a three-dimensional hologram. Such virtual images are typically created using concave mirrors and positive lenses that form images of, for example, physical objects, film projections or video screens. Examples of such a technique can be found in Krause et al., U.S. Pat. No. 5,944,403 and Orlandi, European Patent No. EP-B1-0 410 968.

[0005] Another technique to enhance the illusion of realism of displayed images is to clearly establish the distance between the viewer and the image to create an illusion of image depth. One method for creating such an illusion is to place physical objects near the plane of focus of the image in order to establish the relative depth from the image and the object to the viewer. Another method is to place a semi-transparent scrim between the optical objective and the image, and to project an image on the scrim that emphasizes the position of the image relative to the scrim.

[0006] Although these techniques succeed to one degree or another in making images more compelling, the images still appear unrealistic because the lighting in and around the images does not match the surrounding environment. In particular, images do not cast a shadow on their surroundings, nor do they contain shadows or reflect lighting effects from light sources in their environments.

[0007] To overcome the limitations in the prior art described above, the present invention discloses a system and method for enhancing the realism of displayed images by reflecting lighting effects from the displayed image's environment in and around the image.

[0008] Specifically, in one embodiment of the invention, the system comprises an image display subsystem and a shadow projector. The image display subsystem may be, for example, a virtual or real image projector or video display, and it may display an aerial image or a real image. The shadow projector may be, for example, a slide, film, or video projector or a theatrical lighting fixture with gobo. The shadow projector projects a second image, which may be from a photographic negative or an inversion of the signal of the displayed image, near the displayed image such that the second image appears to be a shadow of the displayed image. Since real shadows are created only by physical objects, the displayed image appears more like an object in three-dimensional space.

[0009] The “shadow” projected by the shadow projector can be made to correspond with the shape, alignment, movement, and perspective of an object or image relative to a real or virtual light source. By a “virtual light source,” the inventors mean a light source within an image. For example, a virtual light source may be a sun projected on a movie screen. With a second image source comprising fixed media, for example, a gobo or slide, the source can be changed in discrete steps with the movement of a light source or the image. When the second image source comprises continuous media, for example, video or film, the image correspondence may take place continuously. By corresponding the shadow image to the displayed image, the image appears to cast a realistic shadow, further enhancing the illusion of realism.

[0010] In another embodiment of the invention, the displayed image contains a virtual light source. The shadow projector projects a shadow near either an object image within the displayed image or a real object near the displayed image, such that the “shadow” appears to be cast from the virtual light source. This creates an illusion that the virtual light source is a real light source, and further enhances the realism of the image.

[0011] An alternative embodiment comprises an image display subsystem and a first image source. The lighting and shadows from light sources and physical objects in the viewer's environment are integrated into the image. When displayed, the image contains realistic lighting and shadows from the viewer's environment. This enhances the realism of the display by creating an illusion that the image is lighted by the environmental light sources or shadowed by real objects in its environment.

[0012] Other features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the invention.

[0013] FIG. 1 illustrates one embodiment of the invention, wherein a virtual image projector projects an image of an object and a shadow projector casts the negative of the image in the direction of the image such that it appears to be a shadow of the image;

[0014] FIG. 2 illustrates an alternative embodiment of the system shown in FIG. 1, wherein the apparent shadow moves as the shadow projector changes its projection angle;

[0015] FIG. 3 illustrates another embodiment of the invention, wherein both the image display projected by the virtual image projector and an inverter are driven by the same video storage device;

[0016] FIG. 4 illustrates another embodiment, wherein a real object is placed near an image containing a virtual light source. A shadow projector projects a shadow of the object that appears to be cast from the virtual light source;

[0017] FIG. 5 illustrates another embodiment of the invention, wherein the projected image appears to contain a shadow cast by the shadow-casting object from an environmental light source;

[0018] FIG. 6 illustrates an alternative embodiment of the system in FIG. 5, wherein the object, shadow, and virtual light source are rendered on an electronic display device by a video source before being projected; and

[0019] FIG. 7 illustrates an alternative embodiment of the system of FIG. 6, wherein the virtual light source is rendered to correspond with a moving environmental light source.

[0020] The invention summarized above and defined by the enumerated claims may be better understood by referring to the following detailed description, which should be read in conjunction with the accompanying drawings. This detailed description of particular preferred embodiments, set out below to enable one to practice the invention, is not intended to limit the enumerated claims, but to serve as particular examples thereof. It is to be understood that other embodiments may be utilized as changes may be made without departing from the scope of the present invention.

[0021] FIG. 1 illustrates one embodiment of the present invention. An aerial image 10 of a real object 11 is displayed by an image display subsystem 12. In this embodiment, the image display subsystem 12 is a virtual image projector, using a concave mirror. A shadow projector 14 has a focal plane containing a negative image 16 of the aerial image 10. The shadow projector 14 is positioned in the proximity of the aerial image 10 such that projection of the negative image 16 appears to create a shadow 18 of the image 10. In this embodiment, the shadow projector 14 is a film slide projector. The shape and size of the negative image 16 and resulting projected image or shadow 18 are configured to be appropriate for the geometrical relationship of the shadow projector 14 and the image 10. A baffle or scrim 20 is added as shown to shield the object 11 from view. The object 11 may be replaced by a real image displayed or projected on a screen. The image 10 may alternatively be a real image. The image display subsystem 12 may alternatively be a virtual image projector using a positive lens, or a real image display, for example, a film slide projector, motion picture film projector, video projector, or a display from an electronic display device. The shadow projector 14 can also be, for example, a theatrical lighting fixture with a gobo, motion picture film projector, or video projector displaying an electronically stored negative image.

[0022] FIG. 2 illustrates the movement of the shadow projector 14 shown in FIG. 1 to a new position and alignment. The corresponding change in shape and orientation of the projected shadow 19 is also shown. As the shadow projector 14 moves, the negative image 15 changes as needed to maintain proper perspective. If the shadow projector 14 uses fixed media, such as a transparency slide or gobo, then the transformation of the negative image 15 can be accomplished in discrete steps by substituting different fixed negative images (15, 17) as necessary at each new position of the shadow projector 14. If the negative image 15 is continuously variable, as with motion picture film or video, then the motion of the shadow projector 14 and the shadow 19 can be accomplished continuously. By changing the shape of the negative image 15 as the shadow projector 14 is moved, such that the contour of the shadow 19 accurately represents the perspective of the geometry of the shadow projector and image, the realism of the aerial image can be further enhanced.

[0023] FIG. 3 illustrates another embodiment of the invention. The image display subsystem 12 displays an aerial image 9 of a real image 13, for example, a moving picture displayed on a screen. A video source 22 drives both the real image 13 display and an inverter 24. The inverter 24 reverses the polarity of the video source 22 using digital signal processing, and outputs the signal to the shadow projector 21. The shadow projector 21 projects the inverted signal of the real image 13 as a shadow 23 in the proximity of the displayed image 9, such that the shadow 23 appears to be cast by the displayed image 9. By using the video source 22 as a source for both the displayed image 9 and the shadow 23, moving pictures and their shadows can be easily synchronized. The simplicity of the system is also increased, avoiding the need for a separate image source 25 for the shadow projector 21.

[0024] Synchronization may also be accomplished without using an inverter 24. When both the image display subsystem 12 and the shadow projector 21 are film or slide projectors, the negative of the film used for the image display can be created and used as the source 25 for the shadow projector 21. The image 9 may alternatively be a real image. The image display subsystem 12 may alternatively be a virtual image projector using a positive lens, or a real image display, for example, a film slide projector, motion picture film projector, video projector, or a display from an electronic display device.

[0025] FIG. 4 illustrates another embodiment of the invention. An image display subsystem 26 displays a real image 30 containing a virtual light source 32. An object 34 is placed near the real image 30. The shadow projector 14 projects a shadow 27 near the object 34, such that the shadow appears to be cast by the object 34 from the virtual light source 32. A shadow 27 may also be projected near an image of an object that is also contained in the real image 30. This system and method allow both real objects and images to appear to cast shadows from a virtual light source. As discussed above, the shadow projector 14 may also be a motion picture or video projector displaying a negative 16 of the film or inverted digital signal used to display the first image 28.

[0026] FIG. 5 illustrates an embodiment of the invention comprising an image display subsystem 12, an image source 37, and an aerial image 41. A real shadow 36 is cast onto the image source 37 by a real object 38 from a real light source 40. The shadow 36 is aligned on the image source 37 such that, when displayed, the image 41 appears to be shadowed by a second real object 42 that is near the displayed image 41. The light source 40 is positioned so as to illuminate the image source 37 such that, when displayed, the image 41 appears to be lighted by a second light source 44 that is in the environment of the displayed image 41. The light source 40 also corresponds with the movement, brightness and color of the environmental light source 44. By reflecting environmental light sources and shadows from real objects near the displayed image in the image itself, the illusion of realism is further enhanced. The displayed image 41 may appear to be a physical object, capable of reflecting the lighting and shading of its environment.

[0027] FIG. 6 illustrates that the image source 37 of FIG. 5 may be the output 39 of a video source 22. A virtual shadow 36 is rendered on a real image display according to the perspective of a virtual light source 40 and an object image 38. The virtual light source 40 and the object image 38 are rendered on the display so as to correspond with the real light source 44 and real object 42 when the image 41 is displayed. The image source 39 may be film, slides, or an electrical signal. Rendering may be accomplished by recording images of real or computer generated objects with similar lighting on film, slides, or video storage media such as a video disc or tape, so that when the image is played back through the image display subsystem 12, the environmental lighting and shading are reflected in the image 41. Rendering may also be accomplished by generating and displaying the images in real-time. Rendering the shadows and lighting effects within an image, rather than using real objects, allows moving and more complicated images to be easily displayed.

[0028] In FIG. 7, a light source position sensor 46 senses the movement of the environmental light source 44, and outputs the position data to a light source position processor 48. The processor 48 then outputs the data to the video source 22, which renders a corresponding change in position for the virtual light source 40. Where a digital storage medium, such as a disc or solid state memory is employed, the position of the light source 44 can be used as an index to retrieve and display a prerecorded image 39 that is appropriate for the lighting geometry in the environment. Sensors may also be used to sense and output color and brightness data to the processor 48. By dynamically changing the illumination of the displayed image 41 to correspond to the environmental lighting and shading, the realism of the image is further enhanced.

[0029] In embodiments where a real-time three-dimensional computer image generator is employed as the video source 22, the position data from the light source 44 can control the position of the virtual light source 40 used to render the three-dimensional object 36, which is displayed on the display 39 and imaged by the image display subsystem 12 as an aerial image 41. The image display subsystem 12 may alternatively be a real image display, such as a display of an electronic device, slides, film or video projected on a screen, or a television screen. Such real-time image rendering enables the image 41 to reflect lighting changes as they occur, without the need of prerecording those changes.

[0030] In any of the above embodiments, the image display subsystem 12 may be a film or slide projector and screen, a television, video, or liquid crystal display screen, or a virtual image projector, projecting aerial images of the real image displays described above or of real objects.