Methods, systems and devices for displaying live 3-D, parallax and panoramic images

[0002] Video cameras are used to record and transmit images, such as live sporting events. Live images are transmitted from traditional video cameras which are fixed, relative to the scene they are recording and transmitting, at certain camera angles. Distinct from an image captured from a single camera angle are panoramic images. Panoramic images are typically generated using a number of camera angles. To date, there has been no known live transmission or reception of panoramic images. The display of panoramic images has been limited to non-live video films, such as those seen at various science and historical museums throughout the United States. These non-live, panoramic films have been created using a number of existing techniques. The first technique involves recording images on a film moving in one direction while a camera moves in an opposite direction. The second technique involves rotating a camera through the angles 0 to 360 degrees in a single plane, all the while taking “still” images at each angle. These are just two examples. Other techniques exist, but it would be impractical to discuss each and every one of them in a few short paragraphs. In every case, however, existing techniques either require the movement of a camera or the movement of the film/storage mechanism in order to create and store panoramic images. None of these existing techniques involve the display of live, 3-D panoramic images.

[0003] In addition, none of the existing techniques involve the display of live, 3-D “parallax” images. Parallax images are created by the human visual system. To accurately reproduce images this parallax effect must be taken into account.

[0004] Accordingly, it is an object of the present invention to provide for devices and methods for displaying live, 3-D parallax and panoramic images over a substantially 360°×360° range of angles.

[0005] It is another object of the present invention to provide devices and methods for displaying a plurality of live parallax and panoramic images from more than one viewpoint,

[0006] It is yet a further object of the present invention to generate novel video signals from live 3-D panoramic images.

[0007] Other objectives, features and advantages of the present invention will become apparent to those skilled in the art from the following description taken in conjunction with the accompanying drawings.

[0008] In accordance with the present invention there are provided methods, systems and devices for generating video signals which can be used to display live, 3-D parallax and panoramic images.

[0009] One such device comprises a novel spherical lens and spherical charge-coupled device for detecting live, 3-D images over a 360°×360° range of angles. The detected images are then broken up into positioning and image data. This data is swept by novel positioning and image sweep generators. The output of the sweep generators is an unformatted video signal.

[0010] The unformatted video signal is then formatted to create a novel, composite video signal. The composite video signal is then used to display live, 3-D parallax and panoramic images.

[0011] The present invention and its advantages can be best understood with reference to the drawings, detailed description of the preferred embodiments and claims that follow.

[0012] FIG. 1 depicts a detection unit for detecting live, 3-D panoramic images over a 360°×360° range of angles according to one embodiment of the present invention.

[0013] FIG. 2 illustrates how points located along the circumference of a detection unit, such as the unit shown in FIG. 1, can be determined by positioning angles “φ” and “θ” according to one embodiment of the present invention.

[0014] FIG. 3 depicts an example of how multiple detection units located at different viewpoints may be utilized to detect a plurality of live, 3-D panoramic images according to one embodiment of the present invention.

[0015] FIG. 4 depicts a block diagram of detection and processing units adapted to detect and generate live, 3-D panoramic images according to one embodiment of the present invention.

[0016] FIG. 5 depicts an example of a portion of a novel video signal according to one embodiment of the invention.

[0017] Referring to FIG. 1, there is shown a detection unit or means 1 adapted to detect live, 3-D panoramic images over a substantially 360°×360° range of angles. The detection unit 1 is adapted to detect light images originating from objects which surround the unit 1 and impinge upon the surface of the detection unit 1, for example, at points “D” and “E”. The detection unit 1 is centered at a position “A”. In addition to detecting images originating from surrounding objects, the detection unit 1 can be further adapted to generate image and positioning data indicating the image content and the image's position along the circumference of the detection unit 1. In an illustrative embodiment of the invention the detection unit 1 comprises a spherical detection unit, such as a spherical CCD or the like and a spherical lens or the like. In one embodiment of the invention, the spherical lens is adapted to focus images from surrounding objects onto the spherical CCD. Thereafter, the spherical CCD is adapted to convert the images into video and positioning data. The present invention envisions the use of a novel spherical lens and CCD. The spherical lens and CCD combination are adapted to detect the widest possible viewing angle. The spherical lens may be constructed using holographic technology or other technology adapted to produce left eye (abbreviated “LE”) and right eye (abbreviated “RE”) images. More will be said about LE and RE images below. Though it is assumed for the present discussion that the images detected are from a part of the visible spectrum, the present invention is not so limited. Images based on any frequency or wavelength of light along the electromagnetic spectrum, such as infrared and ultra-violet light, may be detected by a detection unit adapted to detect such light.

[0018] In an illustrative embodiment of the invention, the range of angles detected by the detection unit 1 is 360°×360°. It should be realized, however, that the detection unit can detect any range of angles less than 360°×360° as well.

[0019] Continuing, the detection unit 1 is further adapted to transmit the positioning and image data to processing unit or means 2 via pathway 3. Once this data is received by the processing unit 2, the processing unit 2 is adapted to generate live, 3-D parallax and panoramic images based on images from position “A”. At position “A”, a plurality of images are generated covering a 360°×360° range of angles. It should be understood that lines 4 and 5 shown in FIG. 1 are for illustrative purposes only. These lines illustrate the fact that the detection unit 1 is capable of detecting live, 3-D panoramic images from all angles surrounding the unit 1. Though shown as hard-wired to processing unit 2, it should be understood that the detection unit 1 may be adapted to transmit data via wireless transmissions or the like. In still another embodiment of the invention, the detection unit 1 is adapted to optically transmit data to processing unit 2 with little or no delay.

[0020] Detection unit 1 must operate using some known means of power. If power is supplied via a hard-wired electrical cord then the position where the power cord connects to the detection unit 1 creates a “blind spot”. That is, the surface of the detection unit which is adapted to receive the power cord cannot be used as a lens or CCD. Therefore, images originating from objects which would normally be detected at this position will escape detection causing a blind spot or “gap” in the range of angles detectable by the detection unit 1. In illustrative embodiments of the invention, the processing unit 2 or detection unit 1 may be adapted to compensate for the existence of this blind spot.

[0021] The present invention makes it possible to display live, 3-D parallax and panoramic images using the positioning and image data. For example, suppose an object located at position “G” in FIG. 1 is fixed and the detection unit 1 is also fixed and centered around position “A”. Though the detection unit 1 is centered around position “A” it is adapted to collect images of the object located at position “G” from all angles. Any image which impinges on the circumference of the spherical lens and CCD will be captured by the detection unit 1. Thus, the detection unit 1 is capable of detecting multiple images of the same object located at “G”. For example, different images of the same object located at “G”, will be detected by the detection unit at points “D” and “E”. The images reaching points “D” and “E” follow paths D₁and E₁, respectively. After these images are detected they are then processed by the processing unit 2 to generate live, 3-D parallax and panoramic images.

[0022] FIG.2 shows a different view of the object located at “G”. Angles “φ” and “θ” help determine the location of an image of the object located at “G” which impinges on a point located along the circumference of spherical detection unit 1. For this reason these angles will be referred to as positioning angles. The positioning data generated by detection unit 1 is derived from positioning angles “φ” and “θ”.

[0023] A few more comments need to be introduced before a more detailed discussion of the detection unit 1 and processing unit 2 can begin. One comment concerns an issue related to mankind, namely, the fact that people who have two working eyes actually see two images; one image with their right eye and another with their left eye. These two images are processed by the brain to form one “parallax” image. To demonstrate this phenomena, close your left eye. Next, extend your right arm out so that your arm is perpendicular to your body. Then extend your right fore-finger so that it is pointing upwards. Your extended fore-finger should be visible to your right eye only at this point. Now close your right eye and open your left. You should see that the image of your fore-finger appears to move. In fact, the closer you move your finger to your nose the bigger the movement or shift appears to be. The present invention takes this phenomena into account.

[0024] Referring again to FIG. 2 there is shown a point “H” located along a spherical detection unit. At this point, and at each point along the circumference of the spherical detection unit having a clear line of sight, there exists two images of the object located at “G”; one RE image as seen by a right eye and one LE image as seen by a left eye. The RE and LE images are contained within the data which is transmitted to processing unit 2.

[0025] The present invention also envisions the generation of live, 3-D parallax and panoramic images from multiple detection units located at different reference viewpoints. For example, referring again to FIG. 1, suppose one detection unit is centered at point “A” and a second detection unit is centered at point “B”. Both detection units are adapted to detect live, 3-D panoramic images of an object located at “G”. Because they are located at two different reference viewpoints, however, the images detected will be different. The advantages of using multiple detection units can be further illustrated with reference to FIG. 3.

[0026] FIG. 3 depicts an object located at position “I” which is hidden from the detection unit located at position “B”. However, if images from detection units located at positions “A”, and “B” are combined, the object located at “I” will become visible. Said another way, the two images are combined to form an image of the object located at “I”. Thus, the use of multiple detection units located at different viewpoints makes it possible to detect a plurality of live, 3-D panoramic images which may not be detectable by a single unit.

[0027] FIG. 4 depicts a block diagram of units used in the detection, processing and generation of live, 3-D images according to one embodiment of the present invention. As shown, the processing unit 2 comprises image processor or image processing means 6 and positioning processor or positioning processor means 7. The image processor 6 is adapted to receive data via pathway 3 from detection unit 1 and to generate 3-D image data 12 representative of RE and LE images or image data. In one embodiment of the invention, the image processor 6 comprises a 3-D parallax interpreter adapted to generate RE and LE image data. The same data input into image processor 6 is also input into positioning processor 7. The positioning processor 7 is adapted to receive the data and to generate positioning data 13 which correspond to the points along the circumference of detection unit 1 where live 3-D images are detected. These points correspond to the positions of objects which originate the images. In an illustrative embodiment of the invention the positioning processor 7 comprises a viewpoint, bias controller adapted to generate positioning data representative of points derived from positioning angles “φ” and “θ”. Said another way, the image processor 6 generates the RE and LE image data 12 while the positioning processor 7 generates the “φ” and “θ” positioning data 13. The “φ” and “θ” data are used to insure that the image or images corresponding to a given point is inserted into its proper position in a composite video signal 19aoutput by the processing unit.

[0028] In an illustrative embodiment of the invention, the positioning processor 7 comprises a filter or filters, such as a window aperture, adapted to filter out either “φ” or “θ” positioning data, respectively. The data input into positioning processor 7 may comprise both horizontal (“φ”) and vertical (“θ”) positioning data for both a RE and LE image. To correctly display such signals, the positioning processor 7 must be adapted to filter out the horizontal data from the vertical and vice-versa. Window apertures, filters or the like can be used to achieve this function. The exact design or method of filtering may vary. For instance, a single, switchable filter may be used to filter out horizontal data followed by vertical data, or vice-versa. Or, a first filter may filter out horizontal data while a second filter filters out vertical data.

[0029] In an additional embodiment of the present invention, the positioning processor 7 further comprises an integrator. The integrator is adapted to integrate or otherwise link the positioning data (i.e, φ, θ) with the correct RE and LE image data.

[0030] In the beginning of this discussion we noted the existence of a “blind spot”. To compensate for this blind spot the detection unit 1 or processing unit 2 (as indicated by dotted lines) comprises a blind spot compensation section or means 11 adapted to compensate for such a blind spot. The blind spot compensation section 11 is adapted to estimate positioning and image data corresponding to the position of the blind spot. When two detection units are used, positioning and image data corresponding to the blind spot can be estimated by using data from other units. In an illustrative embodiment of the invention, each detection unit or camera has a different blind spot. Suppose we are using three detection units. To estimate the positioning and image data which corresponds to the blind spot of the first unit we can use data taken from the second and third. Similarly, because the blind spot of the second unit is different than that of the first and third, we can estimate the positioning and image data corresponding to the second unit's blind spot using data from the first and third units, and so on. Though the compensation section 11 will typically be made a part of the detection unit 1 or processing unit 2, it should be understood that it may also be separated out as a separate unit.

[0031] The RE and LE 3-D image data 12 is sent to image sweep generator or means 8 which is adapted to generate a first, unformatted live 3-D video signal 12afrom the RE and LE 3-D data 12 which falls within the adjustable “sweep” range of the sweep generator 8. The sweep generator 8 is then adapted to output the first 3-D video signal 12a. Similarly, the positioning data, i.e., “φ” and “θ”, 13 is sent to positioning sweep generator or means 9 which is adapted to sweep or scan certain positions from among all of the positions lying on the circumference of detection unit 1. So it is clear, the “φ” and “θ” data 13 output from positioning processor 7 represents data from all of the points along the circumference of detection unit 1. At each point there is an image made up of RE and LE data. Only some of these points (i.e., a range of points) will be used to create a final live, 3-D image, however. In an illustrative embodiment of the invention the processing unit 2 further comprises resolution control section or means 14 adapted to control the “resolution” of any final live, 3-D parallax and panoramic image or images by controlling the sweep generator 9. The resolution of any final, live, 3-D parallax and panoramic image depends upon the range or number of points swept by the positioning sweep generator 9. The greater the number of points, the greater the resolution and vice-versa. In an alternative embodiment of the invention, the resolution control section 14 is adapted to control the positioning sweep generator 9 such that the sweep generator 9 only sweeps a certain range or number of points. In yet a further embodiment of the invention, the resolution control section 14 is adapted to control the sweep generator 9 according to stored reference point data. The reference point data may be stored in the resolution control section 14, in a memory device or in a combination of the two. The reference point data represents the resolution range of the processing unit 2. In sum, the resolution control section 14 determines which points will be swept by the sweep generator 9. In reality, the resolution control section 14 also determines which portion of the first, unformatted video signal 12awill be used to create a second, unformatted swept 3-D video signal 15. Only that portion of video signal 12awhich corresponds to points which have been selected to be swept by sweep generators 8 and 9 will be used to create the second, unformatted video signal 15.

[0032] Generally, in a multiple detection unit/processing unit system, each detection unit/processing unit combination will use the same resolution but this need not be the case.

[0033] In an illustrative embodiment of the invention, the second 3-D video signal 15 generated by sweep generators 8,9 is input into a video formatting section or means 10. The formatting section is adapted to format the second, 3-D video signal so that it can be used to display a live, 3-D parallax and panoramic image or images 15. In an illustrative embodiment of the invention, a display unit or means 20 is adapted to display the formatted, live 3-D parallax and panoramic images. This display 20 may comprise a computer, television or other display, such as one found in a theater, adapted to receive the live, 3-D parallax and panoramic video signal.

[0034] FIG. 5 depicts an example of a portion 16 of a novel composite video signal which is output at point “J” as a result of such formatting. The present invention envisions the generation of the novel, formatted composite video signal which can be used by a display 20 to receive and display a live, 3-D parallax and panoramic image or images. Signal portion 16 comprises a single scan line of the display 20. It should be understood that the processing unit 2 may be adapted to output any number of scan lines, however. As shown, the signal portion 16 comprises start and stop pulses 17a,17b. In between these two pulses is one video scan line 18. This single video scan line 18 may comprise a plurality of formatted video signals 19a-19c. For example, formatted composite video signal 19amay comprise a video signal from viewpoint “A”, signal 19bfrom viewpoint “B” and signal 19cfrom still a third viewpoint.

[0035] In another embodiment of the invention, signal portion 16 comprises formatted composite video signals 21aand 21bfrom separate detection units 100,1000. In this manner, images from more than one detection unit may be displayed simultaneously. Each of the signals 19a-19cand 21a,21bcomprises RE and LE data of an image relative to its position. Thus, live 3-D parallax panoramic images from more than one viewpoint and from more than one detection unit may be displayed. The processing unit 2 is adapted to produce a live, parallax and panoramic image or images from live, 3-D parallax and panoramic composite video signals 21a, 21b.

[0036] As shown in FIG. 4, the detection units 1, 100, 1000 may be “cascaded” together. In an illustrative embodiment of the invention, each detection unit 1, 100, 1000 is adapted to combine the signals from every other detection unit in order to generate a final image without any overlapping. In a sense, the final image is “stitched” together from each unit.

[0037] Though the terms detection unit, processing unit, spherical lens and spherical CCD have been used it should be understood that these components may be called by a more general term, such as a 3-D camera. It should also be understood that the sections making up the detection unit 1 and processing unit 2 may be combined or further broken down into additional sections. Further, parts or all of the detection unit 1 may be made a part of the processing unit 2, or vice-versa.

[0038] Though the discussion above involved the use of a single processing unit 2, image processor 6, positioning processor 7, sweep generators 8, 9 resolution control section 14, video formatting section 10 and display 20 it should be understood that the invention is not so limited. Though at least one of each of the above units/sections is needed to carry out its respective function, more than one of each unit/section may be used, such as when multiple detection units are cascaded. Similarly, one or more unformatted or formatted video signals may be generated by a system.

[0039] It is to be understood that changes and variations may be made without departing from the spirit and scope of this invention as defined by the claims that follow.