Decompression System for Compressed Video Data for Providing Uninterrupted Decompressed Video Data Output

This invention is in the field of video data and the provision of video data to users. In particular, it relates to methods and apparatus for decompresing compressed video data.

For purposes of this description, video data typically comprises the video and audio data contained in a stored video program. However, other data including but not limited to text and graphics may be included in the video data without in any way affecting the operation of the present invention or the substance of this description. All references herein to video data should therefore be considered in the broadest sense.

Video servers for providing video data to users are known. Although uncompressed video data can be stored in such servers and sent to users, the sheer amount of data in even a short video program usually requires that the video data be stored and manipulated in a compressed form. Methods and apparatus for accomplishing the compression and decompression of video data are known.

One known compression format is sponsored by the Motion Picture Expert Group and is known as MPEG. Although both an MPEG-1 and an MPEG-2 compression format are known, their differences are not relevant to the present invention.

MPEG compression is based on the fact that from one frame of video data to the next, there are comparatively few changes, even when objects or persons are in motion. It is therefore not necessary to store all of the video data contained in each frame. Rather, after a base frame has been stored, each successive frame can be recreated by storing only the video data that describes objects or persons that have either changed or moved. Periodically, a complete frame of video data must be stored to re-initialize the process. This type of data compression is called motion compensation.

MPEG compressed video data consists of three types of frames. The first, an intra coded frame (hence, an I-frame) provides all the video data needed to fully describe that particular frame. A predicted frame (hence, a P-frame) provides only information about how the P-frame differs from the last I- or P-frame. Finally, a bi-directional frame (hence, a B-frame) provides information about how the B-frame differs from both the preceding I- or P-frame and the next I- or P-frame. The decompression of the video data contained in a B-frame requires the decompression of two frames, either both I-frames, an I-frame and P-frame, or both P-frames. Decompressing a P-frame requires the video data contained in the preceding P- or I-frame. Typically, storing an I-frame requires three times as many bits as a P-frame, and storing a P-frame requires roughly three times as many bits as a B-frame. These relative storage requirements of the I-, B-, and P-frames are provided only for comparison purposes and no limitation of the present invention to the stated relative storage requirements should be implied.

FIG. 1 shows the typical transmission and display order of a series of MPEG compressed video data frames. As P-frames need the video data contained in a decoded I-frame to be decoded, and as B-frames need the video data contained in either or both a decoded I-frame and a P-frame, the transmission order of compressed MPEG frames differs from the display order of decompressed MPEG frames. Both the transmitted and displayed frames begin with an I-frame, and another I-frame occurs roughly every fifteen frames thereafter. During transmission, two B-frames are preceded by a P-frame. When displayed, after the first I-frame, two B-frames follow, and then a P-frame. As neither a B-frame or a P-frame can be decompressed without reference to an I-frame, all compressed video data streams must begin with an I-frame.

A known architecture for decoding MPEG video data streams is shown in FIG. 2. Decompression system 10 consists of compressed video data buffer 11, decoder 13, and decompressed video data buffer 17. An input bus 15 provides a stream of compressed MPEG video data to buffer 11. As bus 15 provides video data at a fixed rate, some time elapses before enough video data is stored in buffer 11 for decoder 13 to begin decompressing the video data.

A latency time exists before enough video data enters buffer 11 for decoder 13 to begin decompression. This latency time is herein called buffer filling latency time. An even longer latency time occurs due to the nature of the MPEG video data. Either an I-frame and a P-frame, two P-frames, or two I-frames must be decompressed and available before a B-frame can be decompressed. Typically two B-frames are transmitted after a P-frame. The system must receive and decompress these I- and/or P-frames before the B-frames can be decompressed. The time required for this decoding and reordering is herein called a reordering latency time.

The effect of the reordering latency time is noticeable every time a new video program begins. The reordering latency time and the buffer filling latency time together result in the system generating several blank frames between the old and new video programs while the new video program is decompressed sufficiently for display. In some known systems, as many as eight such blank frames are generated between two consecutive video programs. These blank frames are highly undesirable.

At present, no known system corrects this deficiency at acceptable cost.

A first preferred embodiment of the present invention comprises a video decompression system that can accept multiple compressed video data streams as input. For purposes of this description a compressed video data stream can be comprised of a single video program or multiple video programs. Different video data streams will therefore comprises different individual video programs. The video data streams may be available simultaneously at the input of the present invention or they can be received at different times. The system will decompress and display a first video data stream. Prior to the end of the first video data stream, the system will begin accepting as input and decompressing another video data stream. At least several frames of the MPEG compressed video data comprising the second video data stream will be decompressed and available for display as soon as the first video data stream ends. The buffer filling and reordering latency times which occur when the present invention begins to decompress its first video data stream occur off-line, so the user has no direct experience of it. As successive video data streams begin, their buffer filling and reordering latency time occur while the previous video data stream is being decompressed and displayed. The user does not experience these latencies either. In known systems, the user experiences both the buffer filling and the reordering latency times as blank frames between successive video data streams every time a new video data stream begins.

In the preferred embodiment, the input switch controls the flow rate of two video data streams on two video data input lines which comprise the inputs of the preferred embodiment. Initially, the first video data stream flows into the present invention at a first rate. When the second video data stream is later allowed to flow, it flows into the system at a second rate on the second line. In the preferred embodiment, the first rate is higher than the second rate. In other embodiments, this need not be the case. The input switch can halt the flow of either stream.

The input switch is in turn coupled to two decompression circuits, each circuit comprising a compressed video data buffer, a decoder, and a decompressed video data buffer. Incoming compressed video data is stored in the compressed video data buffer, decompressed in the decoder, and stored temporarily in the decompressed video data buffer. The decompressed video data buffer from both circuits is in turn coupled to an output switch.

Both decompression circuits and the input and output switch are coupled to a microcontroller. Under instructions from a stream scheduler, the microcontroller determines which decompression circuit will accept video data at the first rate and which will accept it the second rate. The first decompression circuit receives the first video data stream at the first rate, decompress it, and provides it to the output bus.

Prior to the end of the first video data stream, the system will instruct the input switch to begin flowing the second video data stream into the second decompression circuit at the second rate. Several frames of the second video data stream will be decompressed and stored in the decompressed video data buffer before the first video data stream ends. When the first video data streams ends, the system immediately begins to display the decompressed second video data stream and simultaneously increases the rate of video data flow into the second decompression circuit to the first rate. In this manner, successive video data streams are displayed without the user seeing any blank frames between the video data streams or experiencing the buffer filling and reordering latency times directly.

Processing at first and second rates by alternate decompression circuits continues indefinitely until all video data streams have been displayed.

The preferred embodiment will now be described in detail with reference to the figures listed and described below.

FIG. 1 shows typical MPEG compressed frame transmission and decompressed frame display sequences (Prior Art);

FIG. 2 shows a known system for decompressing video data (Prior Art);

FIG. 3 is a block diagram of the preferred embodiment of the present invention; and

FIG. 4 is an example showing the relative rates of video data transmission through the first and second decompression circuits of the present invention during typical use.

A first preferred embodiment of the present invention is illustrated in FIG. 3. Decompression system 100 comprises input switch 105, first and second decompression circuits 120 and 130, which in turn are further comprised respectively of compressed video data buffers 121 and 131, first and second decoders 123 and 133, decompressed video data buffers 125 and 135, output switch 115 and microcontroller 110. Stream scheduler 150 is coupled to microcontroller 110. Microcontroller 110 is a commercially available Motorola 68331 microcontroller and requires no further description herein.

In each decompression circuit, the compressed video data buffer is coupled to the decoder, which is in turn coupled to the decompressed video data buffer. The decoders in the preferred embodiment are commercially available STI 3500 MPEG decoders from SGS Thompson and require no further description herein.

Input switch 105 is coupled to two compressed video data input lines, both compressed video data buffers 121 and 131, and to microcontroller 110. Output switch 115 is similarly coupled to decompressed video data buffers 125 and 135 and to microcontroller 110. Microcontroller 110 is further coupled to both decoders 123 and 133, the two compressed video data buffers 121 and 131, as well as to steam scheduler 150. Although microcontroller 110 receives instructions from stream scheduler 150, stream scheduler 150 does not form part of the present invention.

Although shown as separate buffers in FIG. 3, the compressed and decompressed video data buffers can be realized as a single buffer, which would be accessed in a known manner. exact configuration of the buffers can therefore be varied considerably from the illustrated embodiment without changing the present invention in a material way. In the preferred embodiment, the buffers comprise eight 256K words by 16 bits dynamic random access memories ("DRAMs") coupled to the decoder. The DRAMs are commercially available from many vendors, including Hitachi, and require no further description herein.

In operation, two separate compressed video data streams enter input switch 105. Under command of microcontroller 110, video data flows into one of the decompression circuits at a first rate and flows into the other at a second rate. For purposes of this description only, and without implying any limitation, video data will be assumed to flow initially into decompression circuit 120 at a high rate and into decompression circuit 130 at a lower rate. It should be understood that nothing herein constrains the second rate to be less than the first rate. Although the preferred first embodiment uses a first rate of 15 megabits per second and a second rate of 7.5 megabits per second, these rates were chosen to reduce overall bandwidth demands. The second rate in other embodiments could be the same or indeed higher than the first rate.

As compressed video data buffer 121 begins to fill with the video data it is receiving at a high rate, decoder 123 begins decompressing that video data. Decompressed frames of video data are stored in decompressed video data buffer 125. After the first I- and P-frames have been decompressed and stored in decompressed video data buffer 125, decompression of the B-frames that separated the I- and P-frames begins. Decompressed video data frames are removed from decompressed video data buffer 125 in the proper order under the direction of microcontroller 110 and sent through output switch 115 to an output bus.

When the first compressed video data stream is nearly finished, the process of decompression begins in decompression circuit 120. Microcontroller 110 instructs input switch 105 to begin flowing the second compressed video data stream into decompression circuit 130, albeit at the second, lower rate. If the video stream being decompressed in decompression circuit 120 is very long, it is possible that decompression circuit 130 will fill its decompressed video data buffer to capacity prior to the completion of the first video data stream, despite the microcontroller only ordering the second decompression circuit to begin operation shortly before the first video data stream is predicted to end. Microcontroller 110 will in that case instruct input switch 105 to stop the flow of video data into decompression circuit 130.

Once the video program flowing into decompression circuit 120 ends, decompression circuit 130 has several frames of video data decompressed and ready for immediate display or output. Output switch 115 would be instructed by microcontroller 110 to switch the output to decompression circuit 130 as the output from decompression circuit 120 ends, insuring a continuous generation of decompressed video data without any blank frames between video streams. Input switch 105 also begins to flow the second video data stream into decompression circuit 130 at the higher, first rate of video data flow. Input switch 105 may provide decompression circuit 120 with a low rate of video data immediately, but it is more likely that no video data will be flowed into decompression circuit 120 for at least some interval of time. This process would continue alternatively, with each decompression circuit alternatively providing the output from the system. In this manner, the buffer filling and reordering latency times are not experienced by system users and no blank frames occur between video programs.

An example of the process of alternatively providing video data to decompression circuits 120 and 130 is illustrated graphically in FIG. 4. At time T.sub.1, decompression circuit 120 is receiving and decompressing a first video data stream at a high rate. Decompression circuit 130 is not receiving video data. At time T.sub.2, microcontroller 110 has instructed input switch 105 to begin flowing the second video data stream into decompression circuit 130 at the second, lower rate. At time T.sub.3, the first video data stream being processed by decompression circuit 120 ends. Immediately, decompression circuit 130 begins receiving video data at a high rate, while simultaneously sending its stored decompressed video data frames to the output bus. At time T.sub.4, microcontroller 110 instructs input switch 105 to begin flowing the next video data stream into first decompression circuit 120 at the second, lower rate. At time T.sub.5, the buffers in decompression circuit 120 are full and video data flow to that circuit ceases. At time T.sub.6, the video data stream being decompressed by decompression circuit 130 ends. Decompression circuit 120 now receives its next video data stream at the high, first rate, while providing its stored decompressed video data frames to the output bus. Video data stream input to decompression circuit 130 ends until time T.sub.7, when microcontroller 110 again instructs input switch 105 to begin flowing the next video data stream into decompression circuit 130 at the second, lower rate. At time T.sub.8, the video data stream being decompressed by decompression circuit 120 ends and data stream input to that circuit is also ended, while simultaneously video data stream input to decompression circuit 130 is increased to the first, high rate and the stored frames of decompressed video data in decompression circuit 130's buffer are provided as output. This sequential process continues until all video data streams provided as input have been decompressed and provided as output in a continuous output stream.

Although the present invention has been described in detail with reference to only two decompression circuits, nothing herein should be taken to limit the present invention to only two such circuits. The expansion of the system to more decompression circuits would be a straightforward process and would provide even greater system flexibility and usefulness. In a system with more than two decompression circuits, the switching process would obviously not have to occur alternatively or sequentially.

The present invention can be used in a real time display system, where different video programs must be displayed consecutively without pause. Video "clips" can be played from random start points to random end points followed by another video "clip" with random start and end points, with no delay or blank frames between the "clips". It should be noted that such random start points will require additional flexibility from the system, as decompression will have to begin early enough to decompress the video data stream up to the "random" start point, as all preceding decompressed video data streams will have to be used to decompress the video data stream up to the chosen start point and then discarded. The present invention is also useful in video data editing systems, where video data programs or portions of programs must be spliced together. Video program insertion, advertisement insertion and video editing are all environments within which the present invention would be useful.