Autostereo projection system

[0002] This application is a Continuation of U.S. patent application Ser. No. 09/103,955 filed Jun. 24, 1998, now Pat. No. 6,224,214, which is a Continuation-in-Part of International Application No. PCT/US98/06304 filed Mar. 27, 1998, which is a continuation and claims priority to U.S. Ser. No. 09/003,614 filed Jan. 7, 1998, which claims the benefit of U.S. Provisional Application Nos. 60/042,424 filed Mar. 27, 1997, and 60/050,603 filed Jun. 24, 1997, the entire teachings of which are all incorporated herein by reference in their entirety.

[0003] The world is a three-dimensional (3-D) place and the human brain is very effective at interpreting data when it is presented in a 3-D format. The human brain interprets a scene as being three-dimensional by a variety of visual clues from which it can extract information for depth perception. There have been many attempts to find an effective way of creating displays that take advantage of visual clues to provide a true 3-D window on the world with practical real-time motion.

[0004] The visual clues can be broken down to four main categories which may be replicated by a 3-D display technology. Although an individual eye only sees a two dimensional (2-D) image created on the retina, the observer's brain works with the slightly different views from the two eyes to construct the third dimension of depth from which the brain can create a 3-D image. This process is called stereopsis. To achieve this result, a display must somehow channel a different view into each eye.

[0005] Another category is motion parallax, which relies on the relative apparent motion of objects in the field of view as the observer's head moves vertically or laterally. To achieve this result, a display typically gauges and responds to the observer's head motion.

[0006] The two other categories provide weaker visual clues. Using eye focusing and convergence feedback, the brain can obtain some information about the depth position of an object by sensing how strongly the eye lens is squeezed to bring the object into focus and the relative angles of the two eyes needed to make the object views converge within the brain. Because, eye focusing and convergence are weak visual clues, most 3-D display systems can operate with the observer focusing at a fixed distance, such as infinity or on a screen close to the observer.

[0007] In addition, software-based visual illusions can create weak visual clues within an image generator for display on a 2-D surface. These illusions include object perspective and shading, distance haze and movement parallax from camera motion. Software-based visual illusions are the easiest category of visual clues to produce. Indeed, most simulator and computer games produce these visual illusions by using 3-D worlds within the computer memory and processor before rendering on a 2-D surface.

[0008] One example of a three-dimensional display taking advantage of stereopsis includes a movable vertical slit in combination with an image source. The slit is moved between a plurality of positions in sequence with images displayed by the image source. The images can be viewed by a viewer with each image displaced in space based on the position of the slit.

[0009] An example of a 3-D display device which can provide both stereopsis and motion parallax is a head-mounted display (HMD) device. Miniature display screens within the HMD device are registered to and provide a respective image to each eye. A head tracking device can be combined with the HMD device to provide lookaround capability with observer head motion. Such systems are, however, limited to use by a single observer at any one time and transport delays can result in nausea and loss of equilibrium.

[0010] In most 3-D viewing situations, it is only necessary to provide parallax for motion in a plane occupied by both of the observer's eyes (e.g. a horizontal plane). Although prior art systems can provide a stereoscopic image to an observer, they have a number of significant limitations. Such systems suffer from low image brightness, a narrow field of view, a small number of pupil slices, and a small image size. Although any one of these deficiencies could be improved using prior art techniques, such improvement would be at the expense of severe degradation of the other considerations.

[0011] In accordance with preferred embodiments of the invention, a three-dimensional display device includes a multiple pupil, multiple projection system which has its pupils abutted as close together as possible. Each projection system simultaneously images a respective image from an image source onto a common viewing optic. The viewing optic can include a Fresnel lens or a concave mirror in a folded system. The viewing optic images the exit pupil of the projection lenses onto a continuous array of viewing ports at a viewing space. Such a system provides a brighter, multiple view display with a wide viewing space projection.

[0012] In accordance with a first aspect of the invention, a display device includes a plurality of image sources and an image space having a plurality of discrete views. Each image source displays a plurality of sequential images of an object. Each view is optically coupled to a respective sequential image of the object.

[0013] A multi-projection optical system is preferably used to couple the views to the sequential images. The multi-projector optical system includes a plurality of projector lens assemblies, each registered to a respective image source. A plurality of addressable shutters are registered to each projector lens assembly. A common viewing assembly is registered to the projector lens assemblies to provide the multiple views. The common viewing assembly preferably includes a tunable optical diffuser to smear seams between the projector systems. The optical diffuser preferably includes an index matching medium.

[0014] Preferably, a controller is coupled to the image sources and the shutter elements. Each image source provides a respective view of a scene at a specific time. The shutters are arranged such that there are a plurality of slits optically aligned with each image source. The controller operates the image sources and the shutters to form a plurality of simultaneous images. An observer views two of these images at a time, one with each eye.

[0015] The above and other features of the invention, including various novel details of construction and combination of parts, will be more particularly described with reference to the accompanying drawings and pointed out in the claims. It will be understood that the particular embodiments of the invention are shown by illustration only and not as a limitation of the invention. The principal and features of this invention may be embodied in varied and numerous embodiments without departing from the scope of the invention.

[0016] FIG. 1 is a schematic diagram of a simulating 3-D view using discrete parallax changes.

[0017] FIG. 2 is a schematic diagram of a preferred autostereoscopic projection system in accordance with a preferred embodiment of the invention.

[0018] FIG. 3 is a schematic diagram of a preferred lenticular sandwich diffuser in accordance with the invention.

[0019] FIG. 4 is a schematic diagram of a transparent image generating device employable in accordance with preferred embodiments of the invention.

[0020] FIG. 1 is a schematic diagram of a simulating 3-D view using discrete parallax changes. In the real world 1, as an observer 7 moves horizontally the viewing angle image of an object 5 changes smoothly and continuously.

[0021] In practice, however, the human brain is comfortable with stepped changes as long as the step increments are suitably small. In a practical simulation 3, discrete view slices V₁, . . . , V_Nof the object 5 are provided to the observer 7.

[0022] The simplest form of this “quantized” approach is a lenticular raster. Static images from different camera angles are divided into narrow strips and interlaced with one another in an ordered manner. When a lenticular lens of the correct number of lines per unit length, according to the strip width, is laid over the mingled images, observers 7 can see a different complete image, depending on their angle of view.

[0023] Spatially multiplexed systems based on a lenticular raster have two main drawbacks. First, because all the images are displayed simultaneously on the image generating device, the area available for each is limited and the image resolution is restricted. Second, each view requires its own optical projection system, making the scheme complex and difficult to scale for more views.

[0024] In accordance with preferred embodiments of the invention, time multiplexing is used to avoid the deficiencies found in spatial multiplexing systems. Briefly, the images corresponding to the various viewing angles are flashed on the image generating device sequentially in time, each occupying the full screen.

[0025] FIG. 2 is a schematic diagram of a preferred autostereoscopic projection system in accordance with a preferred embodiment of the invention. In one embodiment, views of an object 5 are captured by an array of still or video cameras 12₁, 12₂, . . . , 12_N. An imaging system 10 prepares the captured images for display. The images and video synchronization signals are provided to a control system 20 over a video bus VID and synchronization connections HSync, VSync, ZSync. The control system 20 can control operation of the camera 12₁, . . . , 12_Nvia a control bus CTL to the imaging system 10. In another embodiment, the views are computer generated and stored by the imaging system 10.

[0026] The control system 20 provides video signals to an imaging system 30. Preferably, the imaging system 30 comprises an array of image sources 32₁, 32₂, 32₃which can be cathode ray tube (CRT) or liquid crystal display devices. Although only three CRTs are illustrated, it will be understood that any number of CRTs can be used, with certain optical constraints, depending on the total number of discrete views desired. The image sources 32 also receive control signals from the control system 20.

[0027] A projection lens system 40 is optically coupled to the imaging system 30. In particular, the projection system 40 includes a plurality of projection subsystems 40₁, 40₂, 40₃which are each coupled to a respective image source 32₁, 32₂, 32₃. Each projection lens subsystem 40 includes a plurality of projection lenses 41, 43, 45, 47, 49. The exit pupils 52₁, 52₂, 52₃of the image sources 32₁, 32₂, 32₃are defined by the front projection lenses 49₁, 49₂, 49₃. As illustrated, the front projection lenses 49₂directly abut adjacent front projection lenses 49₁, 49₃. In accordance with a preferred embodiment of the invention the front projection lenses 49₁, 49₂, 49₃are fabricated by molding the lenses from a sheet of plastic.

[0028] Light from the exit pupils 52₁, 52₂, 52₃is processed by respective shutter elements 50₁, 50₂, 50₃. The shutter elements 50 are spatial light modulators which include a moveable slit controlled by the control system 20. Preferably, the shutter elements 50 are liquid crystal devices and the slit is a vertical light modulating cell which can be selected from about 5-8 vertically arranged positions in each shutter element 50. Alternatively, the light modulating cell can be selected from a plurality of two-dimensionally arranged windows in each shutter element.

[0029] Although the shutter elements 50₁, 50₂, 50₃are shown to be forward of the exit pupils 52₁, 52₂, 52₃, that arrangement is not required. Indeed, the shutter elements 50₁, 50₂, 50₃can be positioned behind the front projection lenses 49₁, 49₂, 49₃. Preferably, the shutter elements 50₁, 50₂, 50₃are positioned as close as possible to the exit pupils 52₁, 52₂, 52₃. The further the shutter elements are positioned forward of the exit pupils, more optical efficiency is lost. In a particular embodiment of the invention, the shutter elements are placed less than one inch from the front projection lenses with about one-half inch being preferred.

[0030] In operation, the control system 20 controls the image sources 32 and the respective shutter element 50 such that a different video frame is provided on the displays for each slit position in the shutter element. In particular, the video frames are time-multiplexed in step with the shutters. In that way, there are plurality of pupils created for each image source 32.

[0031] As shown, a plurality of video signals VID₁, VID₂, VID₃are provided by a display driver 22 of the control system 20 to respective image sources 32₁, 32₂, 32₃. In addition, horizontal synchronization (HSync), vertical synchronization (VSync), and video field synchronization (ZSync) signals are processed by the control system 20. In particular, a display control module 24 receives the HSync and VSync signals to drive the image sources 32. A shutter control module 26 receives the VSync and ZSync signals and drives the shutter elements 50. The VSync signal is used to synchronize the video frames of the image sources 32 with the slits in the shutter elements 50.

[0032] Although not shown, a separate red, green, blue (RGB) filter element can be placed over each image source 32. These color filter elements are preferably used to sequentially create a color video frame from a broad spectrum image generator. The display control module 24 would then operate the color filter elements.

[0033] For a particular frame of an image source 32, the projection system 40 projects an image projection P₁, . . . ,P₉, . . . , P₁₇, . . . , P₂₄to a common viewing optic 55. In accordance with one aspect of the invention, the viewing optic 55 is a Fresnel lens. The viewing optic 55 focuses each exit pupil onto a virtual exit pupil or viewing port V₁, . . . , V₂₄on an imaging plane 60. These viewing ports provide a view of the frame image on the image source.

[0034] At any one time, the viewing optic 55 provides the views from the selected pupil of each image source 32 to a viewing space 65 for viewing by an observer 7 or a plurality of observers. Each observer 7 typically maintains the left eye 70L and the right eye 70R in the viewing space. The eyes are spaced apart by an intra-pupillary distance (IPD). As shown, there an a plurality of viewing ports V₁, . . . , V₂₄at the imaging plane 60, each providing a view of a respective video frame of a respective image source. The optics are optimized so the views of the slits at the viewing space 65 abut each adjacent view and are spaced apart by a distance D on center. To promote seamless viewing, where the images appear continuous as the viewer's head moves, the distance D is preferably less than or equal to one-half the IPD. Typically, the slits are between 22-25 mm wide at the observer's eye. As such, blackened areas of the image at the observer are substantially eliminated.

[0035] Each eye 70L, 70R thus sees a different image source 32 to create an illusion of three-dimensional objects. In addition to essentially seamless viewing, the optical system 3 provides extended viewing through many viewing ports.

[0036] To maximize the field of view, the lens array is as large as practical. For example, in accordance with a preferred embodiment of the invention, the optical system has a focal ratio of f/1 and is dimensioned to be equivalent to a 25-inch diagonal monitor.

[0037] To minimize the dimensions of the device, a folded optical system having a concave mirror is preferably utilized for the viewing optics 55. The concave mirror operates as a viewing screen, focusing the light onto the respective viewing ports. By employing mirrors, the optical path can be folded to increase the focal length of the system while maintaining a relatively compact size.

[0038] A preferred system includes a ferroelectric directing strip shutter having an electronically-controlled slit that lets light pass through only one segment at a time. The images displayed on the CRT and the switching of the directing shutter segments are synchronized so the appropriate image is channeled to each viewing slice. The projection lens system creates an essentially distortion-free real image at the position of the field lens. The size of this image determines the effective screen size seen by the observer.

[0039] The field lens 55, which is generally a Fresnel lens equal in size to the effective screen dimensions, has no effect on the real image of the CRT. Its function is to create a real image of the directing shutter at the optimum viewing plane for the observer. Thus the Fresnel lens determines the size of the region where an observer can see a full-screen 3-D image. For practical systems, the depth of the viewing region may extend several feet beyond the optimum viewing plane.

[0040] In a prototype system, a refresh rate of 30 frames per second has been obtained on a 25 inch diagonal screen at a 512×384 pixel resolution in an interlaced mode. The preferred viewing distance is about five feet, so the screen subtends an angle of about 20 degrees. The viewing eyebox for a full-screen 3-D image is about 2 feet wide, several feet deep and can provide 28 different camera angles, which are smoothed together by a weak diffuser.

[0041] One characteristic of a preferred display is a multiple-view mode, where different sections of the viewing region can image completely different full-screen animation. This is possible because the view slices need not be limited to displaying a series of slightly different camera angles. For instance the leftmost slices could provide a full-screen 3D view of the overall battle scene in a simulator, while the rightmost slices could be used to zero in on a specific area of the battlefield. This multiple view feature is of particular interest to the entertainment industry since, using one machine, two game players sitting side-by-side are provided with their own 3D full-screen independent perspective of the game. A 50 inch screen version for the arcade game market with two side-by-side viewing boxes, each about a foot wide, can be produced.

[0042] There are, however, practical constraints in making the exit pupils of the projection lens systems abut without any gap. A lenticular lens can be used to lengthen the exit pupil unidirectionally and horizontally. A lenticular lens is a flat sheet, usually acrylic plastic, having a series of lines, each being in effect a very narrow cylindrical lens with semicircular profile. A typical lenticular lens may have 50 to 100 lines per inch and functions as a unidirectional diffuser. Such lenses are commonly used as the front screen on a large-screen projection television where the lines are mounted vertically so as to widen laterally the angle of view for the screen. As long as the image from the projection lens system 40 creates a real image on the lenticular lens, the image itself is not diffused by the lens, only the exit pupil. A similar effect is seen when viewing objects through a regular diffuser, such as a piece of frosted glass. Objects far from the glass are completely fogged when viewed through it, but a hand pushed up against the glass is clearly visible on the other side.

[0043] As with a projection television system, a lenticular lens can be positioned to produce horizontal stretching and so merge the eyeboxes for the multiple projector system. However, this would destroy the 3D effect by merging together the different camera views. A goal is to provide a very small amount of horizontal smearing; enough to remove the seams and soften the edges of the various views, but not enough to destroy the 3D effect. If the seams are initially minimized by carefully abutting the projector lens systems as close as possible, then typically a unidirectional diffuser with about one tenth of a degree of angular spread is needed. Practical lenticular lenses are made out of acrylic plastic with a refractive index of 1.491 and produce between 20 and 40 degrees of angular diffusion. It may be possible to produce an acrylic lenticular lens with considerably less angular diffusion but currently the tooling needed to produce a lens of that particular design is very expensive and, in a preferred embodiment of the invention, needs the capability to tune the amount of diffusion the lens produces to produce the optimum balance between seam removal and clear 3D imagery.

[0044] The difficulty in producing an acrylic lenticular lens with very small angular diffusion arises from the large refractive index difference between the acrylic at 1.491 and the air surrounding the lens at 1.000. The focusing power of a lens depends of the refractive index differential between the lens material and the surrounding medium, usually air.

[0045] FIG. 3 is a schematic diagram of a preferred lenticular sandwich diffuser in accordance with the invention. The lenticular sandwich 100 includes an acrylic lenticular lens 102, a sheet of glass 106 and a layer of trapped index matching fluid or other medium 104 pressed up against the lenticular surface of the lens. Sealing walls 108 provide an outer boundary for the fluid 104.

[0046] The fluid 104 has an index close to that of the acrylic lens 102 and can be easily changed so as to tune the overall lenticular system to produce the desired optical effect within the 3D display. The amount of diffusion is controlled by the refractive index of the matching fluid 104, available commercially in sets with increments down to 0.004. If the index matching fluid 104 has the same index as the acrylic lenticular lens 102 material then the whole device acts as a transparent plate with no diffusion.

[0047] The refractive index of the liquid, nl, can be calculated by the equation: $\mathrm{nl}=-2·\left[\tan \left(\mathrm{da}\right)-\frac{1}{\left[2\left[\mathrm{fl}\left[\left(n-1\right)\left(\mathrm{lpi}\right)\right]\right]\right]}·n\right]·\left(\mathrm{fl}\right)\left(n-1\right)\left(\mathrm{lpi}\right)$

[0048] where

[0049] lpi is the number of lines per inch for the lenticular lens;

[0050] fl is the focal lengths of the lenticular lens segments;

[0051] n is the refractive index of the lenticular lens material (e.g., acrylic=1.491)

[0052] da is the desired half angular spread for the lenticular lens

[0053] This lenticular sandwich has been used in a preferred 3D display system and tuned to reduce the seams to an acceptable level while maintaining good image quality and 3D realism. An additional benefit is that the transition between the seen views V₁, . . . , V₂₄(FIG. 2) is smoothed to produce a less jerky and much more visually pleasing effects as the observer's head 7 moves from side to side.

[0054] Although preferred embodiments of the invention employ CRT image sources, a light valve display device can also be used to practice the invention. An example of such display devices is a liquid crystal display device.

[0055] FIG. 4 is a schematic diagram of a transparent image generating device employable in accordance with preferred embodiments of the invention. As illustrated, a strip backlight 112 is separated from a transmissive display imaging surface 110. The images corresponding to the various viewing angles are flashed on the image generating device 110 sequentially in time, each occupying the full screen. A common backlight imaging lens 115 creates an image of the display strip backlight in the observer viewing plane. As illustrated, an image I₁, for view 1 (V₁) on the image plane 60 is displayed on the transmissive display device 110. As illustrated, the first backlight 112₁of the display strip backlight 112 is flashed for projecting the view 1 image (P₁). Operation of the backlight 112 and transmissive display 110 are controlled by suitable synchronizing electronics 120.

[0056] The transmission display 110 is placed in the plane of the common lens 115 and is unaffected by it. The various views are channeled into the appropriate viewing slice in space by a display backlight arrangement or the use of an electronic shutter scheme as described above. In this configuration, the views can also share a common optical train, reducing complexity and making scaling of the system for more views easier.

[0057] Liquid crystal displays are not presently capable of producing the frame rates needed for a time-multiplexed system. Current CRT technology, however, can operate at the faster frame rates preferred for time multiplexing. It will be understood that time-multiplexed systems in accordance with preferred embodiments of the invention can include multiple projections of images generated on transmissive display devices.

[0058] Those skilled in the art will know, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments of the invention described herein. These and all other equivalents are intended to be encompassed by the following claims.