Parallax panoramagram having improved depth and sharpness

[00002] This application is a 371 of PCT/US99/26676 filed Nov. 12, 1999 which claims benefit of 60/108,268 filed Nov. 12, 1998.

[00003] This invention relates to parallax panoramagrams, also known as lenticular stereograms, and more particularly, to a method for increasing the depth and sharpness of images in lenticular stereograms.

[00004] For many decades, inventors have studied and attempted to improve the lenticular stereogram, also known as the parallax panoramagram. The art has been very well traveled and is very well understood. By incorporating a cylindrical lenticular screen, corduroy-like, over the surface of a properly encoded print, a stereoscopic depth effect may be achieved. As shown in FIG. 1, the lenticules 102 have semi-cylindrical surfaces aligned so that their length is in the vertical direction. The lenticules are in intimate juxtaposition with a print surface 105, which contains columns of encoded visual information. Each column is associated with a particular lenticule, and within each column there will be found a series of views ranging from a leftmost to a rightmost perspective. Thus, the observer of a panoramagram print will see two perspectives rather than one due to the refractive surface of the panoramagram.

[00005] There are many prior art suggestions which are similar to the lenticular stereogram in their effect. These autostereoscopic displays have several features in common. For one thing, the selection device is in intimate juxtaposition with the display or print surface. For another, a large number of views is required, rather than just two views as is the case for a plano-stereoscopic display like the polarized light method for motion picture projection, or in the occlusion principle which is incorporated in the CrystalEyes® line of products manufactured by StereoGraphics Corporation.

[00006] It has been noted that the panoramagram depth content must be restricted because of the lack of sharpness as depth increases. Inventors have addressed this problem with various suggestions for making improvements. Indeed, in the case of a series of patents assigned to the Nimslo Corporation in the early 1980's, the essence of their art for a mass-consumer lenticular and camera print system is to provide a means for the operator to make an adjustment to place the most important part of the picture at the print surface, with zero parallax, so it would be sharp. It was understood at that time and even to this day, that only the subject which is at the plane of the display will be sharply in focus. Prior patents which discuss this problem and/or solutions include U.S. Pat. Nos. 3,960,563; 4,037,950; 4,063,265; 4,086,585; and 4,124,291.

[00007] Stereographers have accepted this limitation. Those working in the field have been able to mask the defect, because it can be made to appear to be identical to the loss of sharpness which occurs in normal photographic reproduction due to depth of field effects. Indeed, in a typical panoramagram having a distant background and powerful stereoscopic effects which reach above the plane of the display, there will be an attempt to mask the lack of sharpness by suggesting that loss of focus is attributable to depth of field. This is a limitation that those skilled in the art of creating such displays have gotten around because of their artistic skill. Yet it is clear that the major benefit of the parallax panoramagram is to offer depth content. As long as the depth content is compromised, the display cannot be considered to be a fully perfected visual medium.

[00008] The present invention sets out to vastly increase the sharpness of the parallax panoramagram, and do so with means that are inexpensive and which produce an unequivocal improvement in sharpness for objects both in the distance and apparently close to the observer.

[00013] FIG. 1A is a perspective view of a parallax panoramagram, also known as a lenticular stereogram, having a lenticular screen 101 and its associated print image 105. Screen 101 is comprised of individual cylindrical lenticules 102. The front surface of the print 105 is in intimate juxtaposition with the back of the lenticular screen 101. The optics and organization of lenticular displays are generally well known to those in the art and are therefore not described in any detail. Our discussion is confined to descriptions that are relevant to the improvement of depth and sharpness disclosed herein.

[00014] FIG. 1B is a top view of the lenticular screen 101 and associated print surface 105 of FIG. 1A. The dotted lines 104a and 104b are for reference only and are perpendicular to the print surface 105. These lines 104a and 104b are meant to indicate the area under a particular lenticule which is devoted to the image that is specifically associated with that lenticule. Brackets 103a, 103b and 103c are drawn at the surface of the print surface 105 to indicate the extent and width of a column.

[00015] In FIG. 1C columns 103a, 103b and 103c each have five individual image stripes, each stripe representing a particular section of a perspective view. It is well understood in the art that the leftmost and rightmost views are interchanged because of the refraction of the lenticular screen, which by its nature takes place only in the horizontal direction. For example, stripe 1 a is the rightmost view and stripe 5c is the leftmost view. For didactic purposes, we are restricting the number of image stripes to five. It should also be pointed out that if a panoramagram camera is employed which might produce, in effect, an optical continuum across the width of a column, the discussion below would still apply because the effect of such a continuum being wiped across a screen by means of a slit shutter, or by optical means, can always be resolved in terms of discrete pixels and hence in terms of discrete stripes.

[00016] The term “pixel” is generally understood to mean a discrete picture, element of a single color value which, in large quantities, make up a raster display, or a raster bitmap, or a raster printout. We are using the term “pixel cluster” to mean an image element derived from a component bitmap, several of which make up a row of a lenticule column. Our pixel clusters are square with their height and width the same dimension as the width of a stripe.

[00017] The pitch of the lenticular screen is defined as the number of lenticules in the horizontal direction. For example, a screen with 58 lenticules to the inch has a pitch of 58, and the width of the lenticule is the reciprocal of the pitch.

[00018] As has been noted, parallax panoramagrams may be produced in a number of ways. In the past, the means available for producing a panoramagram have been restricted to cumbersome cameras, which might require motion of the camera across several feet over a period of many seconds. This effectively restricted the subject matter to still lifes or to people who were required to hold still for a long time.

[00019] The later variations of the art, notably work done by Dudley, introduced the concept of interdigitating images for multiple camera views, thus replicating the effect of the panoramagram and which, in essence, has look-around capability (which is another way of saying that the position of the observer's eyes is not tightly restricted).

[00020] To make panoramagram viewing enjoyable, there must be a reasonable angle of view over which the image may be viewed. As is well known in the art, there is a central or primary zone which is followed by a pseudo zone in which, because of the refractive property of the optics combined with the location of the observer, over a certain angle of view in space there will be a mixed combination of stripes from adjacent columns. Then follows a secondary zone which is stereoscopic and a pseudo zone and then a stereoscopic tertiary zone and these zones are symmetrically placed on either side of the primary zone.

[00021] It has been observed that the panoramagram suffers a significant loss of sharpness which is more or less proportional to parallax, and yet parallax is the principal information which determines the stereoscopic depth effect. Therefore, parallax panoramagrams in the prior art have been limited in terms of their depth effect.

[00022] In order to better understand this problem, we refer to FIG. 2, which shows the optical path of wave train 204 (on-axis) and wave train 205 (off-axis) intersecting a single lenticule 201 of a lenticular sheet or screen. These wave trains represent the image forming rays that will intersect with the eyes of an observer looking at the lenticular stereogram. Surface 202 has a cylindrical curvature and in this horizontal section is a sector of a circle. The focal length of the lenticule is approximately the length of the line 208. Wave trains 204 and 205 come to focus at points 206 and 207 respectively. Bracket 203 denotes the width of lenticule 201, which corresponds to any one of the columns 103a through 103c shown in FIGS. 1B and 1C. Focal point 206 corresponds to a central stripe, for example stripe 3, and focal point 207 corresponds to another stripe, for example stripe 2.

[00023] Each eye of the observer sees its own perspective view when looking at a lenticular stereogram. For example, the observer may see image stripes 2a, 2b, 2c and so on for the right perspective view and image stripes 3a, 3b, 3c and so on for the left perspective view. With only five stripes per column (in this example), one might conclude that the observer only sees twenty percent of the image with each eye and thus sees a series of stripes rather than an entire image, but that is not the case. Further, if each eye is only seeing twenty percent of the image, one might conclude that the image would be substantially reduced in brightness, but that is not the case.

[00024] What we have not considered is the focal spot size formed by the optics. In this case, the spot size is an ellipse which may have a width greater than the width of a single stripe because of optical limitations. It is well known that even the best corrected optics cannot focus to a “perfect” point because of aberrations. The size of the spot is influenced by how well the optics are focused and how well they are corrected, since some aberrations may cause a spread of the spot size. Assuming that the focal length of the lens matches the distance from its optical center to the surface 203, such a simple lens cannot be well corrected because it cannot bring all wavelengths to focus at the same point. This lack of color correction is only one issue.

[00025] The lenticules in a typical lenticular sheet are poorly corrected compared with, for example, a box camera lens. In addition, off-axial focal spots may be even more spread out than axial spots for both optical and geometric reasons. Nevertheless, the optics do a decent job as will be observed when viewing the secondary zone image, which usually looks about as sharp as the primary zone. If the diminution in performance of off-axial rays are the key to the loss of sharpness in a lenticular stereogram, then this loss of sharpness would be exacerbated when viewing secondary zone images, and this is simply not the case.

[00026] However, the questions about missing information and loss of brightness because of stripe width and focal spot size get right to the heart of the matter. The observer does not see a series of stripes because of the imperfections of the optical system. This is to a great extent helpful because it would be unpleasant to have the experience of looking through a vertical grating. Also, there is little difference between prints with and without the lenticular screen having been applied. This tells us that the image is formed from a number of adjacent stripes having been combined rather than from a particular stripe being available for each eye as would be implied by the simple model. Although the device produces the necessary stereoscopic effect, because of the unwanted combining of adjacent stripes we have created a significant optical problem which manifests itself as image blurring for corresponding points with large parallax values.

[00027] To help understand why this is so, consider the case of an image residing entirely at the plane of a print with all of the stripes having zero parallax. The image stripes would all be identical and it does not matter if adjacent stripes are combined by the imperfect optical system. In such a case, the image will be sharp because like stripes are added together. But with large values of parallax, stripes adjacent to each other are pictorially out of context with each other. They represent disparate segments of the puzzle that make up a panoramagram image. In such a case, the effect of the spot size overlapping adjacent stripes will result in image blur. Although the corresponding point information may be emphasized from the stripe sitting at the center of the focal spot, images contributed by the stripes on either side of the desired or center stripe (relatively speaking) contribute to a degradation of sharpness. When unwanted image information is added, the result is a lack of sharpness, and the effect is proportional to parallax since the greater the parallax the lesser the correlation between adjacent stripes within a column.

[00028] Presently, panoramagrams are produced by digital techniques using a computer. Graphics workstations are frequently used although a modern personal computer has adequate resources to perform the required techniques by using off-the-shelf software such as Adobe Photoshop. Typically, a digital image is produced as a Windows bitmap (BMP) having a 24 bit RGB format.

[00029] The images may be taken with standard cameras using silver-based materials, or with digital cameras. If taken with standard silver-based cameras, the images must first be digitized. Each image is sliced up into stripes, and these stripes are then ordered into columns. The columns are then ordered into groups of stripes, as shown in FIG. 1C, which shows portions of three adjacent columns made up of stripes numbered 1 through 5.

[00030] The five stripes represent five image elements, or pixel clusters, derived from five original image components. Because the lenticule pitch tends to not be a precise multiple of the pitch of the printing device, the image elements typically do not correspond to actual pixels. Thus, the original image components need to be interdigitated using one of several possible techniques. One technique, based on standard anti-aliasing, is to compute how each image element should map to the actual pixel grid, and have that image element contribute to the pixel(s) that it maps to based on weighted averaging with other image elements. Another technique is to find the single image element that maps to a particular pixel better than any other image element, and assign the best image element's color value to that pixel.

[00031] Within each column, a stereo pair is represented by the combination of any two stripes. For example, stripe 1 and stripe 2 is a stereo pair, and stripe 2 and stripe 5 is a stereo pair. The subscripts appended to the stripe number in the drawing denotes that the stripe belongs to a particular column 103a, 103b or 103c.

[00032] The technique for enhancing the sharpness of the panoramagram, or enhancing its depth (the two go together, for improving the sharpness actually enhances the ability to create deeper images), consists of applying a sharpening filter to the interdigitated image, such as the filters found in Adobe Photoshop. That is, after the individual perspective views have been sliced up and ordered into stripes and then ordered into columns, it is at this point that a sharpening filter is used. In the course of creating this invention, all of the sharpening filters in Photoshop were used in experiments, including the Unsharp Mask, which in particular allows one to vary the amount of sharpness applied on an incremental basis. In brief, a sharpening filter increases the contrast between the adjacent pixels. Using this technique produces an astonishing improvement in the lenticular stereogram, to the extent that images with very large values of parallax and depth effect are now gratifyingly sharp for all parts of the image. This is a completely unexpected result and one that produces a tremendous improvement in the sharpness of images with large parallax values.

[00033] Although the image sharpness and depth of the panoramagram is vastly enhanced using the technique described herein, we found that a disturbing artifact was produced by the process. As described above, lenticular stereogram images have a good primary zone, followed by a pseudo zone, then a secondary zone, and then a tertiary zone. We found that in the transitions between zones after the sharpening filter had been applied, there appeared to be harsh vertical lines which made the transitions extremely obtrusive. This is an undesirable artifact of the beneficial improvement. Happily, we have found a way to eliminate the visibility of the coarse vertical structure, but before describing it, we shall describe why sharpening filtration applied to the interdigitated image produces the beneficial result.

[00034] We described above why sharpness is lost as a result of the unwanted blending together of information from adjacent stripes. While such blending has the salutary effect of producing an integral image free from annoying vertical stripes, this blending of image information from adjacent stripes containing image points with high parallax values is the root cause of the blurring of the lenticular stereogram. Our process, which involves sharpening of the interdigitated image, produces greater contrast or differentiation between adjacent stripes. In this way, the primary stripe, i.e. the stripe which is at the center of the focal spot, is emphasized with respect to the adjacent stripes. The observation that increasing the sharpening filtration for large values of parallax is required tends to support this hypothesis. In addition, the fact that treating the images that make up the source of the lenticular stereogram prior to interdigitation has no beneficial effect also tends to support this hypothesis because the beneficial sharpening must take place not within the pixel clusters but rather between the stripes.

[00035] We also get a feeling for why the harsh vertical stripe artifact occurs between columns. The sharpening filtration when applied treats all adjacent stripes, even those at column boundaries and these may have very low correlation. This treatment produces an exacerbation of the boundary between columns. Thus, this harsh artifact is caused by the apparent edge that the sharpness filter's edge detection algorithm frequently finds at a lenticule column boundary. Since such apparent edges are caused by parallax discontinuity, and not by actual edge features in the scene, it is not desirable for the sharpness filter to enhance their sharpness. The cure, as described below, is to soften the boundary.

[00036] Boundary stripes, for example, stripes 5a and 1b or stripes 5b and 1c, are relatively unrelated in terms of image context, and thus the sharpening filter produces the harsh vertical artifact described. The cure is to use an averaging technique to soften the boundary stripes. This can be accomplished in a number of different ways.

[00037] One way to accomplish this is to shrink the image elements in each lenticular column such that each lenticular column now fits two additional image element fields thus, in our ongoing example, we would now have seven image elements per lenticular column instead of the original five. (In experiments, we have actually used more like twenty original image elements per lenticular column). The two new image elements are added along each edge of each lenticular column. Thus, the original five image elements per column, originally numbered 1 through five, became image element numbers 2 through 6 out of 7 total, and the newly added image elements become image element number 1 and 7. With any given pair of columns, the last image element (#7 in this example) of the first column and the first image element (#1) of the next column are each assigned a color value which equals the average of the second to last image element (#6 in this example) and the second image element (#2) of the next column. Color value averaging is accomplished by separately averaging red, green and blue color component values.

[00038] Overall, the important technique is to essentially soften the discontinuity between adjacent lenticules. Once this has been accomplished, the artifact is greatly reduced or eliminated. To reiterate, the important steps are: interdigitate image components while applying an averaging technique, or a softening technique, to the lenticule column boundaries, and then apply a sharpening filter after the image has been interdigitated. It should be pointed out that this technique does not add any additional artifacts and does not reduce either the depth content of the stereogram or the extent of the viewing zone.

[00039] A further refinement would be to add sharpening which is proportional to the parallax of the image points. This could be done by creating a table of parallax values using the extreme left and right perspective views for the creation of such a table. The table could then be used to add an increasing amount of sharpness as a function, most beneficially as a linear function in many instances, to the image points. In point of fact it is rarely required since we have observed that a single value for the sharpening filtration usually produces a decent effect.

[00040] The results of the techniques described herein are entirely unexpected, are not predicted by the prior art, and have the great advantage of allowing the use of standard lenticular screens without any additional optical correction. Such correction would involve additional expense in terms of designing and calculating such a screen, if indeed it is possible to put such a design into practice, and then a great deal of money would have to be spent in actually fabricating such a lenticular screen.

[00041] The present invention, by using computer techniques as described herein, avoids a costly and uncertain development process which would have undoubtedly resulted in lenticular screens that would be even more costly than the present product.

[00042] The present invention produces an entirely unexpected and startling improvement in the panoramagram, and does so for a very low cost. The result is beneficial, and any observer would agree that such panoramagrams have a significantly extended depth content as a result of the application of this technique. In addition to the technique described here, it is also possible to apply sharpening in the vertical direction only. When this approach is used, there is a noticeable improvement in sharpening, and little or none of the artifact described above.

[00043] Yet another approach or variation on the technique described above is to reverse the order of the stripes left to right within a column. As has been described above, right now each column has stripes which are ordered in a mirror image because of the optical inversion of the lenticular screen. As an intermediate step in using this technique, if the stripes are left uninverted with respect to left and right orientation, and the sharpness filter is applied, because the material between adjacent columns is related in terms of context, then the artifact described here will not be produced. After the sharpening filter has been applied, the columns must now have their stripes inverted left to right to produce a panoramagram with the normal viewing characteristics.

[00044] We believe the increase in sharpening comes about because the aberration defects as described in FIG. 2 have been overcome. Principally, the increased aberration in marginal or non-axial stripes, that is, stripes which are away from the center of the column, leads to a reduction in contrast, or a reduction in modulation transfer function. The reduced efficiency of the optic with regard to the ability to reproduce spatial frequencies with contrast is most marked for the marginal stripes. We believe that the higher spatial frequencies may well be lost, but the lower frequencies have boosted contrast as a result of the sharpness correction.