Apparatus and method for non-sequential image data transmission and display

[0002] This application claims benefit of Provisional application Ser. No. 60/065,433, filed Oct. 30, 1997.

[0003] The present invention relates to an apparatus and method for image data transmision and display, including image display and transmission of image data over the Internet.

[0004] In many applications currently utilized within the art, it is common for graphical or visual data to be transmitted to a user over telecommunications lines such as phone lines. Such image transmissions include transmissions of video images, images of a scene, of a three dimensional object, and so forth. Particularly in low bandwidth environments, image transmission and display can be extremely slow. Accordingly, it is a goal of the present invention to address certain of the drawbacks of the transmission systems used in the present state of the art, while particularly taking into account current bandwidth limitations.

[0005] In accordance with the present invention, an apparatus and method is provided for image display and transmission, including display and transmission of image data over the Internet. Image transmission is effected by nonsequential streaming of sequential images, or or of multiple views of a scene or object. In accordance therewith, rotation of objects or scenes and viewing of multiple perspectives can be accomplished more rapidly, since views can be preliminarily displayed before completion of data streaming (which itself is limited in speed due to the low bandwidth environment). Accordingly, the present invention is particularly useful for transmission of image data over the Internet.

[0006] FIG. 1 is a first perspective view of an image for transmission in accordance with the present invention.

[0007] FIG. 2 is a second perspective view of the image of FIG. 1, from a different angle of view than that shown in FIG. 1.

[0008] FIG. 3 is a third perspective view of the image of FIG. 1, from a different angle of view than those shown in FIGS. 1 and 2.

[0009] FIG. 4 is a table illustrating a sample embodiment of the nonsequential streaming of multiple frames, in accordance with the present invention.

[0010] FIG. 5 is a second embodiment of the nonsequential streaming disclosed herein.

[0011] FIG. 6 is a first perspective view of an image transmitted and displayed to the viewer in accordance with the present invention.

[0012] FIG. 7 is a second perspective view of the image of FIG. 1, displayed to the viewer when the cursor is located in the shown position.

[0013] FIG. 8 is a third perspective view of the image of FIG. 1, displayed to the viewer when the cursor is located in the shown position.

[0014] FIG. 9 is a third perspective view of the image of FIG. 1, displayed to the viewer when the cursor is located in the shown position.

[0015] In accordance with the present invention, a streaming and image display technology is provided for display of images transmitted over telecommunications lines, including phone lines or even satellite transmissions. The invention is particularly suited for low-bandwidth environments such as the Internet. The present invention (also called TrueMotion VR™ or TMVR™ by the present inventors), allows a person viewing a web page (“the viewer”) to select and move among the images in a streamed video sequence before all the frames of that video sequence have been transmitted to the viewer. Software based on this technology therefore accepts a stream of video images, and begins displaying those images before receiving all of them. The viewer can then, for example, rotate an object early on during transmission without waiting for the full transmission to be completed. In this way, the viewer can quickly move a cursor over the TrueMotion VR™ image to quickly view an object and gain a better understanding of its shape, color and form in time and space.

[0016] In the preferred embodiment, the invention is implemented in software as a Netscape™ client plugin or ActiveX control program; although, alternatively, the technology described herein can be applied in many different environments. In the preferred environment, the program resides within an Internet/Intranet HTML page and displays image frames depending on where the viewer of these images has placed a cursor inside the image on his or her computer screen. This navigation technique allows the content developer to produce images of an object that can be displayed based on the location of a cursor within the image.

[0017] The simplest content architecture consists of a series of frames shot from different angles of an object, as shown for example in FIGS. 1-3. The object can be made to appear to rotate from side to side if the viewer moves the cursor right and left. As the cursor moves to the right, the plugin displays images from camera angles progressively to the left so the object appears to rotate to the right. As the cursor moves to the left, the plugin displays images from camera angles progressively to the right so that the object appears to rotate to the left. The images shown in FIGS. 1 through 3, when viewed sequentially, show the cursor moving to the right and the object appearing to rotate to the right.

[0018] A more complex content architecture consists of several series of frames called “rows” or “rings”. As the cursor moves up and down over the face of the plugin, it displays images from different rings. Depending on the content of the different rings, the image could “zoom” to become larger and smaller as the cursor moves up and down. Alternatively, the camera could appear to move up over the object. Simply changing the way the image is built can allow the content developer to make the image appear to move closer and farther, or to rotate up and down.

[0019] In accordance with the present invention, a method for data transmission to the user is provided, which is based on a unique approach to streaming video. A video clip is normally comprised of a linear collection of images (or frames) arranged in time order (i.e. a sequential stream of image frames). Video streaming software transmits the clip from one computer to another. In the method of the prior art, the software transmits the frames of the clip in the same order in which the clip was created. A video player, as it plays these frames in order, gives the impression of coherent motion because frames that are earlier in time are displayed before those that are later in time.

[0020] In what the present inventors call ordered streaming, frames are sent from the server computer to the client computer (which is running a browser) in the order that they will be displayed. Frame 1 is followed by Frame 2, Frame 2 is followed by Frame 3, and so forth. For example, with reference to the figures, FIG. 1 would be transmitted first, followed by FIG. 2, then by FIG. 3.

[0021] In low bandwidth situations, when it takes a significant length of time to transmit each frame, ordered streaming is not considered by the present inventors to be a sufficiently effective strategy. The transmission medium, typically a fairly slow, analog telephone line, gets clogged up with frames that are similar to one another. Yet the viewer ultimately wants the ability to see all sides of the object as quickly as possible. In accordance with the present invention, it is believed that the viewer would be willing to sacrifice some angular resolution and thus smoothness of motion as the clip of images begins to stream into the browsers.

[0022] Accordingly, in the method and software of the present invention, TrueMotion VR™ sends the frames from server to browser in an out of order sequence (i.e. as a non-sequential stream of image frames). Any number of frames can be so transmitted, whether 5, 10, 20, 50, 100, 1000, 10,000, or more frames. Wherever the cursor is positioned, the browser displays the nearest frame that it has received from the server (i.e. the proximal image). When all the frames are received, the browser can display the correct frame; when frames are still arriving, the browser displays the proximal image, i.e. the closest frame that it can. The software modifications required for modification of the known software for conducting transmission and display of images on screen in accordance with the methods of the present invention are well within the skill of one of ordinary skill in the art.

[0023] One example of such an out-of-order strategy for 24 frames in the AVI file and one row, reorders the frames from line to scattered. This strategy is shown in FIG. 4. Frames 1 through 24 (as shown on the upper line of the figure) represents the in sequence, or linear order of the complete series of images, while Frames 1, 13, 7, 20, 4, etc. represents the out of sequence order that those images are transmitted to the user. If the Frames 1 through 24, for example, represent views of an object taken at successive angles, the result for the viewer is that all sides of the object can be seen quickly and, as the viewer waits, the angular resolution between “steps” improves.

[0024] As an example, we can examine the behavior of the plugin program after four frames have been received by the browser. With an ordered stream, Frames 1, 2, 3 and 4 would be transmitted to the viewer in sequence and the plugin would display:

[0025] frame 1 if the cursor was in the left hand 24th of the window (position 1),

[0026] frame 2 if the cursor was in the second 24th (position 2),

[0027] frame 3 if the cursor was in the third 24th,

[0028] frame 4 if the cursor was in the next ⅞ of the screen.

[0029] Most of the time, frame four would be displayed. Eventually all of the frames would be transmitted to the user. In the meantime, however, during the transmission period, the overall viewable image would be limited, as shown in the above example.

[0030] With a scattered stream, on the other hand, four out of sequence frames would be transmitted to the user, e.g. Frames 1, 13, 7 and 20, in that order, as shown in FIG. 4. After transmission of the four out of sequence frames, a broad range of angles of the object would be viewable. Specifically, the plugin would display:

[0031] frame 1 if the cursor was in positions 1-4, as shown in FIG. 6;

[0032] frame 7 if the cursor was in positions 5-10, as shown in FIG. 8;

[0033] frame 13 if the cursor was in positions 11-16, as shown in FIG. 7;

[0034] frame 20 if the cursor was in positions 17-24, as shown in FIG. 9.

[0035] This scattered stream display methodology, therefore, allows the viewer to see more of the object sooner. Although the full series of frames has not yet been transmitted, the viewer nonetheless has a broader spectrum of views available early on than was provided with the ordered stream transmission.

[0036] Many other re-ordering strategies are possible. The scattered stream provided above is just one example of a particularly useful scattering of the frames. Another very useful scheme arranges the sequential stream of image frames into subsets of image frames or “slices”. In this example, illustrated in FIG. 5, four slices are sent—the first frame of each slice, followed by the second frame of each slice, and so forth. For Frames 1-24, the first slice includes Frames 1-6, the second slice includes Frames 7-12, the third slice includes Frames 13-18, and the fourth slice includes Frames 19-24. Thus, the first four frames transmitted to the user correspond to the first frames of each of the slices, or Frames 1, 7, 13, and 19. Likewise, the next four frames transmitted to the user correspond to the second frames of each of the slices, or Frames 2, 8, 14, and 20. The remaining frames are transmitted in similar fashion. In other words, the linear sequence of images is subdivided into some number of adjacent subsequences or “slices”. A first frame is sent from each subsequence or slice, then a second frame from each slice, and so on.

[0037] Finally, custom orderings by the content developer can allow particularly interesting viewing angles to be sent early in the stream.

[0038] Having described this invention with regard to specific embodiments, it is to be understood that the description is not meant as a limitation since further modifications and variations may be apparent or may suggest themselves to those skilled in the art. It is intended that the present application cover all such modifications and variations.