US 6,937,770 B1Grant
Adaptive bit rate control for rate reduction of MPEG coded video
Issue Date:2005-08-30
•19 Claims
•34 Drawing Sheets
Abstract
Original-quality MPEG coded video is processed to produce reduced-quality MPEG coded video at a reduced bit rate. The processing is based on a scale factor between average frame size of the original-quality MPEG coded video and a desired average frame size of the reduced-quality MPEG coded video. For each Discrete Cosine Transform (DCT) block of each frame, the processing calculates a size of the block of the reduced frame by scaling the original block size by the scale factor, and removes a sufficient number of bits from the original block to obtain substantially the calculated size. In addition, the processing accumulates excess bits when the block size reduction eliminates more bits from a block than are necessary for the desired reduction of the size of the block, and any excess bits are used for processing a number of following blocks.
Metadata
Assignee
- EMC Corporation
Inventors
- Seyfullah H. Oguz
- Sorin Faibish
- Wayne W. Duso
Application Information
Application Number:US 09/750,565
Filing Date:2000-12-28
Priority Date:2000-12-28
Art Unit:2623
Patent Drawings (34 sheets)
Description
Background of the Invention
[0001] 1. Field of the Invention
[0002] The present invention relates to processing and storage of compressed visual data, and in particular the processing and storage of compressed visual data for bit rate reduction.
[0003] 2. Background Art
[0004] It has become common practice to compress audio/visual data in order to reduce the capacity and bandwidth requirements for storage and transmission. One of the most popular audio/video compression techniques is MPEG. MPEG is an acronym for the Moving Picture Experts Group, which was set up by the International Standards Organization (ISO) to work on compression. MPEG provides a number of different variations (MPEG-1, MPEG-2, etc.) to suit different bandwidth and quality constraints. MPEG-2, for example, is especially suited to the storage and transmission of broadcast quality television programs.
[0005] For the video data, MPEG provides a high degree of compression (up to 200:1) by encoding 8×8 blocks of pixels into a set of discrete cosine transform (DCT) coefficients, quantizing and encoding the coefficients, and using motion compensation techniques to encode most video frames as predictions from or between other frames. In particular, the encoded MPEG video stream is comprised of a series of groups of pictures (GOPs), and each GOP begins with an independently encoded (intra) I frame and may include one or more following P frames and B frames. Each I frame can be decoded without information from any preceding and/or following frame. Decoding of a P frame requires information from a preceding frame in the GOP. Decoding of a B frame requires information from both a preceding and a following frame in the GOP. To minimize decoder buffer requirements, transmission orders differ from presentation orders for some frames, so that all the information of the other frames required for decoding a B frame will arrive at the decoder before the B frame.
[0006] In addition to the motion compensation techniques for video compression, the MPEG standard provides a generic framework for combining one or more elementary streams of digital video and audio, as well as system data, into single or multiple program transport streams (TS) which are suitable for storage or transmission. The system data includes information about synchronization, random access, management of buffers to prevent overflow and underflow, and time stamps for video frames and audio packetized elementary stream packets embedded in video and audio elementary streams as well as program description, conditional access and network related information carried in other independent elementary streams. The standard specifies the organization of the elementary streams and the transport streams, and imposes constraints to enable synchronized decoding from the audio and video decoding buffers under various conditions.
[0007] The MPEG-2 standard is documented in ISO/IEC International Standard (IS) 13818-1, “Information Technology-Generic Coding of Moving Pictures and Associated Audio Information: Systems,” ISO/IEC IS 13818-2, “Information Technology-Generic Coding of Moving Pictures and Associated Audio Information: Video,” and ISO/IEC IS 13818-3, “Information Technology-Generic Coding of Moving Pictures and Associated Audio Information: Audio,” which are incorporated herein by reference. A concise introduction to MPEG is given in “A Guide to MPEG Fundamentals and Protocol Analysis (Including DVB and ATSC),” Tektronix Inc., 1997, incorporated herein by reference.
[0008] MPEG-2 provides several optional techniques that allow video coding to be performed in such a way that the coded MPEG-2 stream can be decoded at more than one quality simultaneously. In this context, the word “quality” refers collectively to features of a video signal such as spatial resolution, frame rate, and signal-to-noise ratio (SNR) with respect to the original uncompressed video signal. These optional techniques are known as MPEG-2 scalability techniques. In the absence of the optional coding for such a scalability technique, the coded MPEG-2 stream is said to be nonscalable. The MPEG-2 scalability techniques are varieties of layered or hierarchical coding techniques, because the scalable coded MPEG-2 stream includes a base layer that can be decoded to provide low quality video, and one or more enhancement layers that can be decoded to provide additional information that can be used to enhance the quality of the video information decoded from the base layer. Such a layered coding approach is an improvement over a simulcast approach in which a coded bit stream for a low quality video is transmitted simultaneously with an independently coded bit stream for high quality video. The use of video information decoded from the base layer for reconstructing the high quality video permits the scalable coded MPEG-2 stream to have a reduced bit rate and data storage requirement than a comparable simulcast data stream.
[0009] The MPEG-2 scalability techniques are useful for addressing a variety of applications, some of which do not need the high quality video that can be decoded from a nonscalable coded MPEG stream. For example, applications such as video conferencing, video database browsing, and windowed video on computer workstations do not need the high quality provided by a nonscalable coded MPEG-2 stream. For applications where the high quality video is not needed, the ability to receive, store, and decode an MPEG-2 base-layer stream having a reduced bit rate or data storage capacity may provide a more efficient bandwidth versus quality tradeoff, and a more efficient complexity versus quality tradeoff. A scalable coded MPEG-2 stream provides compatibility for a variety of decoders and services. For example, a reduced complexity decoder for standard television could decode a scalable coded MPEG-2 stream produced for high definition television. Moreover, the base layer can be coded for enhanced error resilience and can provide video at reduced-quality when the error rate is high enough to preclude decoding at high quality.
[0010] The MPEG scaling techniques are set out in sections 7.7 to 7.11 of the MPEG-2 standard video encoding chapter 13818-2. They are further explained in Barry G. Haskell et al., Digital Video: An Introduction to MPEG-2, Chapter 9, entitled “MPEG -2 Scalability Techniques,” pp. 183-229, Chapman & Hall, International Thomson Publishing, New York, N.Y., 1997, incorporated herein by reference. The MPEG scalability techniques include four basic techniques, and a hybrid technique that combines at least two of the four basic techniques. The four basic techniques are called data partitioning, signal-to-noise ratio (SNR) scalability, spatial scalability, and temporal scalability.
[0011] The conventional MPEG scalability techniques permit transmission of the coded video to be switched from a high quality, high bit rate stream to a low quality, low bit rate stream when transmission of the high quality, high bit rate stream is either precluded by network congestion or is not needed at the destination of the stream. However, the conventional MPEG scalability techniques do not permit the bit rate reduction to be freely selected. For many applications, the required bit rate is intermediate that of the high quality, high bit rate stream, and the low quality, low bit rate stream provided by the conventional MPEG scalability techniques. In some applications, the required bit rate will fluctuate between a high rate and a low rate. In any case, it is desired to create a valid MPEG data stream with the best video quality given the required bit rate.
[0012] Much research has been done addressing the problem of encoding a valid MPEG data stream given a required constant or variable bit rate. In general, to satisfy the MPEG rate control requirements, all of the data for each picture must be within the video buffer at the instant it is needed by the decoder. This requirement usually translates to upper and lower bounds on the number of bits allowed in each picture. For example, a number of bits are allocated to each picture based on the picture type, and the bits for each picture are allocated to 8×8 blocks in each picture based on a measure of local coding complexity in each picture. A quantization scale is selected for each 8×8 block to encode the 8×8 block with the number of bits allocated to the block. See Gonzales et al., U.S. Pat. No. 5,231,484 issued Jul. 27, 1993, on Motion Video Compression System with Adaptive Bit Allocation and Quantization, and Ramamurthy et al., U.S. Pat. No. 5,675,384, issued Oct. 7, 1997.
[0013] Unfortunately, when the bit rate was reduced to a small fraction of the standard bit rate, the bit rate control provided by allocating bits based on picture type, and selecting a quantization scale for each block, became rather imprecise. Sometimes the poor bit rate control was compensated for by liberal use of stuffing to make up for a difference between the allocated number of bits and the actual number of bits for each 8×8 block. For low bit rates, however, the stuffing represents a waste of information capacity resulting in a significant loss of picture quality. Efforts toward improving the bit rate control has focused on more sophisticated methods of estimating the complexity of image segments in comparison to average complexity in order to more precisely allocate bits to the image segments to obtain a rather constant visual picture quality and thus minimal degradation in picture quality. One solution was to perform two-pass encoding, which cannot be performed in real time. In a first pass, the video sequence is encoded with constant-bit-rate (CBR) encoding, while statistics concerning coding complexity are gathered. Next, the first pass data is processed to prepare control parameters for the second pass, which performs the actual VBR compression. See Westerink et al., “Two pass MPEG-2 variable-bit-rate encoding,” IBM J. Res. Develop., Vol. 43, No. 4, July 1999, pp. 471-488. Real-time, single-pass encoding has also been proposed which is said to adapt to the complexity of the image segments. However, substantial processing resources are required for good performance from a single-pass encoder. See Mohsenian et al., “Single-pass constant- and variable-bit-rate MPEG-2 video compression,” IBM J. Res. Develop., Vol. 43, No. 4, July 1999, pp. 489-509.
Summary of the Invention
[0014] The basic objective of the present invention is to provide an improved bit rate control for producing reduced-quality MPEG coded video from original quality MPEG coded video. It is desired to produce the reduced-quality MPEG coded video at a given constant or variable bit rate with a minimum of computing resources, yet ensure that the reduced-quality MPEG coded video will have a relatively constant visual picture quality over the picture segments and thus minimal degradation in picture quality.
[0015] In accordance with one aspect, the invention provides a method of producing reduced-quality MPEG coded video from original-quality MPEG coded video. The original-quality MPEG coded video includes a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG coded video. The reduced-quality MPEG coded video also has frames of 8×8 blocks. Each frame in the reduced-quality MPEG coded video has a corresponding frame in the original-quality MPEG coded video, and each 8×8 block in each frame of the reduced-quality MPEG coded video has a corresponding block in a corresponding frame in the original-quality MPEG coded video. The method includes the steps of selecting non-zero AC DCT coefficients from corresponding blocks in the original-quality MPEG coded video to be included in the blocks of the reduced-quality MPEG coded video so that each 8×8 block in each frame of the reduced-quality MPEG coded video has a number of bits encoding non-zero AC DCT coefficients that is generally proportional to the number of bits encoding non-zero AC DCT coefficients for the corresponding 8×8 block in the corresponding frame of original-quality MPEG coded video.
[0016] In accordance with another aspect, the invention provides a method of producing reduced-quality MPEG coded video from original-quality MPEG coded video. The original-quality MPEG coded video includes a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG coded video. The reduced-quality MPEG coded video also has frames of 8×8 blocks. Each frame in the reduced-quality MPEG coded video has a corresponding frame in the original-quality MPEG coded video, and each 8×8 block in each frame of the reduced-quality MPEG coded video has a corresponding block in a corresponding frame in the original-quality MPEG coded video. The method includes a number of steps for each block in the reduced-quality MPEG coded video. These steps include (a) determining the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG coded video; (b) computing a number of bits available for encoding AC DCT coefficients in the original-quality MPEG coded video by scaling the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG coded video with a scale factor; and (c) selecting non-zero AC DCT coefficients in a certain order from the corresponding block in the original-quality MPEG coded video to be included in said each block of the reduced-quality MPEG coded video until the number of bits available for encoding the AC DCT coefficients in the block in the reduced-quality encoded video is not sufficient for encoding, in the block of the reduced-quality MPEG coded video, any more of the AC DCT coefficients in the corresponding block of original-quality MPEG coded video.
[0017] In accordance with yet another aspect, the invention provides a method of producing in real-time a stream of reduced-quality MPEG-2 coded video from a source of original-quality MPEG-2 coded video. The original-quality MPEG-2 coded video includes a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG-2 coded video. The reduced-quality MPEG-2 coded video also has frames of 8×8 blocks. Each frame in the reduced-quality MPEG-2 coded video has a corresponding frame in the original-quality MPEG-2 coded video, and each 8×8 block in each frame of the reduced-quality MPEG-2 coded video having a corresponding block in a corresponding frame in the original-quality MPEG-2 coded video. The method includes a number of steps for each frame in the reduced-quality MPEG-2 coded video. The steps for each frame include (a) computing a moving average of the size of the corresponding frame in the original-quality MPEG-2 coded video; (b) computing a scale factor from the moving average of the size of the corresponding frame in the original-quality MPEG-2 coded video and a desired size of said each frame of the reduced-quality MPEG-2 coded video; and (c) for each 8×8 block in said each frame: (i) determining the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG-2 coded video; (ii) computing a number of bits available for encoding AC DCT coefficients in the original-quality MPEG-2 coded video by scaling the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG-2 coded video with a scale factor, and (iii) selecting non-zero AC DCT coefficients in a parsing order from the corresponding block in the original-quality MPEG-2 coded video to be included in said each block of the reduced-quality MPEG-2 coded video until the number of bits available for encoding the AC DCT coefficients in the block in the reduced-quality encoded video is not sufficient for encoding, in the block of the reduced-quality MPEG-2 coded video, any more of the AC DCT coefficients in the corresponding block of original-quality MPEG-2 coded video.
Brief Description of the Drawings
[0018] Other objects and advantages of the invention will become apparent upon reading the following detailed description with reference to the accompanying drawings, in which:
[0019] FIG. 1 is a block diagram of a data network including a video file server implementing various aspects of the present invention;
[0020] FIG. 2 is a flowchart of a procedure executed by a stream server computer in the video file server of FIG. 1 to service client requests;
[0021] FIG. 3 is a flowchart of a procedure for splicing MPEG clips;
[0022] FIG. 4 is a flowchart of a procedure for seamless video splicing of MPEG clips;
[0023] FIG. 5 is a more detailed flowchart of the procedure for seamless video splicing of MPEG clips;
[0024] FIG. 6 is a continuation of the flowchart begun in FIG. 5 ;
[0025] FIG. 7 is a timing diagram showing a timing relationship between video presentation units (VPUs) and associated audio presentation units (APUs) in an original MPEG-2 coded data stream;
[0026] FIG. 8 is a timing diagram showing a timing relationship between video presentation units (VPUs) and associated audio presentation units (APUs) for a fast-forward trick-mode stream;
[0027] FIG. 9 is a flowchart of a procedure for selection and alignment of audio presentation units (APUs) in the fast-forward trick-mode stream;
[0028] FIG. 10 is a flowchart of a procedure for producing a trick-mode MPEG-2 transport stream from a regular MPEG-2 transport stream (TS);
[0029] FIG. 11 is a diagram illustrating relationships between the MPEG discrete cosine transform (DCT) coefficients, spatial frequency, and the typical zig-zag scan order;
[0030] FIG. 12 is a diagram illustrating a relationship between an MPEG-2 coded bit stream and a reduced-quality MPEG-2 coded bit stream resulting from truncation of high-order DCT coefficients;
[0031] FIG. 13 is a flowchart of a procedure for scaling MPEG-2 coded video using a variety of techniques;
[0032] FIG. 14 is a flowchart of a procedure for signal-to-noise ratio scaling MPEG-2 coded video using a frequency-domain low-pass truncation (FDSNR_LP) technique;
[0033] FIG. 15 is a flowchart of a procedure for signal-to-noise ratio scaling MPEG-2 coded video using a frequency-domain largest-magnitude coefficient selection (FDSNR_LM) technique;
[0034] FIG. 16 is a flowchart of a procedure that selects one of a number of techniques for finding a certain number “k” of largest values out of a set of “n” values;
[0035] FIG. 17 is a flowchart of a procedure for finding a certain number “k” of largest values from a set of “n” values, which is used in the procedure of FIG. 16 for the case of k<<½ n;
[0036] FIG. 18 is a diagram of a hash table and associated hash lists;
[0037] FIG. 19 is a flowchart of a procedure for finding a certain number “k” of values that are not less than the smallest of the “k” largest values in a set of “n” values beyond a certain amount.
[0038] FIG. 20 is a flowchart of modification of the procedure of FIG. 15 in order to possibly eliminate escape sequences in the (run, level) coding of the largest magnitude coefficients;
[0039] FIG. 21 is a flowchart of a subroutine called in the flowchart of FIG. 20 in order to possibly eliminate an escape sequence;
[0040] FIG. 22 is a first portion of a flowchart of a procedure for scaling an MPEG-2 coded video data stream using the modified procedure of FIG. 20 while adjusting the parameter “k” to achieve a desired bit rate, and adjusting a quantization scaling factor (QSF) to achieve a desired frequency of occurrence of escape sequences;
[0041] FIG. 23 is a second portion of the flowchart begun in FIG. 22 ;
[0042] FIG. 24 is a simplified block diagram of a volume containing a main file, a corresponding fast forward file for trick mode operation, and a corresponding fast reverse file for trick mode operation;
[0043] FIG. 25 is a more detailed block diagram of the volume introduced in FIG. 24 ;
[0044] FIG. 26A is a diagram showing video file access during a sequence of video operations including transitions between the main file, the related fast forward file, and the related fast reverse file;
[0045] FIG. 26B shows a script of a video command sequence producing the sequence of video play shown in FIG. 26A ;
[0046] FIG. 27 is a table of read and write access operations upon the volume of FIG. 24 and access modes that are used for the read and write access operations;
[0047] FIG. 28 is a hierarchy of video service classes associated with the fast forward file and the fast reverse file in the volume of FIG. 25 ;
[0048] FIG. 29 shows a system for modifying and combining an MPEG-2 audio-visual transport stream with an MPEG-2 closed-captioning transport stream to produce a multiplexed MPEG-2 transport stream having the same bit rate as the original MPEG-2 audio-visual transport stream;
[0049] FIG. 30 is a diagram showing blocks in a frame of original-quality MPEG coded video being proportionally reduced to obtain a frame of reduced-quality MPEG coded video in an adaptive bit rate reduction method;
[0050] FIG. 31 shows a graph of frame size in bits per frame as a function of frame number for original-quality MPEG coded video and corresponding reduced-quality MPEG coded video obtained using the adaptive bit rate reduction method introduced in FIG. 30 ;
[0051] FIG. 32 shows a graph of frame size in bits per frame as a function of frame number for original-sized MPEG I frames and corresponding reduced-size I frames for trick file generation using the adaptive bit rate reduction method introduced in FIG. 30 ;
[0052] FIG. 33 is a first sheet of a flowchart for programming a digital computer to implement the adaptive bit rate reduction method introduced in FIG. 30 ;
[0053] FIG. 34 is a second sheet of the flowchart begun in FIG. 33 ;
[0054] FIG. 35 is a third sheet of the flowchart begun in FIG. 33 ;
[0055] FIG. 36 is a subroutine for programming a digital computer to determine a coefficient bit rate reduction factor for a reduction from an MPEG source having a known constant bit rate;
[0056] FIG. 37 is a subroutine for programming a digital computer to determine a coefficient bit rate reduction factor for a reduction from an MPEG source having an unknown or variable bit rate; and
[0057] FIG. 38 is a block diagram illustrating the use of the adaptive bit rate reduction method introduced in FIG. 30 for the real-time production of a reduced-quality MPEG-2 stream from an original-quality MPEG-2 source.
[0058] While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described in detail. It should be understood, however, that it is not intended to limit the form of the invention to the particular forms shown, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the invention as defined by the appended claims.
Description of Illustrative Embodiments
[0059] With reference to FIG. 1 , there is shown a block diagram of a data network 20 linking a number of clients 21, 22, 23 to a video file server 24 implementing various aspects of the present invention. The video file server 24 includes at least one stream server computer 25 and a data storage system 26. The stream server computer 25 has a processor 27 and a network link adapter 28 interfacing the processor to the data network 20. The processor 27 executes a data streaming program 29 in memory 30 in order to stream MPEG coded video in real-time to the clients.
[0060] Client requests for real-time video are placed in client play lists 31 in order to schedule in advance video file server resources for the real-time streaming of the MPEG coded video. The play lists 31 specify a sequence of video clips, which are segments of MPEG-2 files 32, 33 in data storage 34 of the data storage system 26. The stream server processor 27 accesses a client play list in advance of the time to begin streaming MPEG coded video from a clip, and sends a video prefetch command to a storage controller 35 in the data storage system 26. The storage controller responds to the video prefetch command by accessing the clip in the data storage 34 to transfer a segment of the clip to cache memory 36. When the video data of the segment needs to be sent to the client, the stream server processor 27 requests the data from the storage controller 35, and the storage controller immediately provides the video data from the cache memory 36. Further details regarding a preferred construction and programming of the video file server 24 are disclosed in Duso et al., U.S. Pat. No. 5,892,915 issued Apr. 6, 1999, entitled “System Having Client Sending Edit Commands to Server During Transmission Of Continuous Media From One Clip in Play List for Editing the Play List,” incorporated herein by reference.
[0061] In accordance with an aspect of the invention, the stream server computer 25 executes an MPEG scaling program 38 to produce reduced-quality MPEG coded video from nonscalable MPEG-2 coded video by truncating discrete cosine transform (DCT) AC coefficients from the coded blocks in the MPEG-2 coded video data. The reduced-quality MPEG coded video can be produced during ingestion of an MPEG-2 file 32 from the network 20, and stored in one or more associated files 37. Alternatively, the reduced-quality MPEG coded video in the files 37 could be produced as a background task from the MPEG-2 file 32. Reduced quality MPEG coded video could also be produced in real-time from an MPEG-2 file 33 during streaming of the reduced-quality MPEG coded video from the stream server computer 25 to the network 20. The reduced-quality MPEG coded video is useful for a variety of applications, such as browsing and review of stored MPEG-2 assets for search and play-list generation, bit stream scaling for splicing, and bit-rate adjustment via video quality alteration for services with limited resources.
[0062] A typical example of browsing for play-list generation involves searching stored assets in a multi-media data base for segments of a desired content to be included in the play list, and in particular selecting the beginning frame and ending frame of each segment to be included. Such editing occurs often in the broadcast environment for inserting commercials and news clips into pre-recorded television programming, and for editing movies for content and time compression. The decoding technique of the present invention permits a PC workstation 23 to perform the decoding and display in real-time by execution of a software program. An operator can view the video content in a display window 39 in a fast-forward or fast-reverse mode, stop at and resume from freeze frames that are valid “in points” and “out points” for seamless splicing, and select an in-point and out-point for a next segment to be included in the play list. The stream server computer 25 could also include a seamless splicing program 40 providing seamless transitions between video segments that are contiguous in a play list and are from different video clips.
[0063] For seamless splicing, it is often necessary to reduce the bitrate for one or more frames at the end of a first segment prior to splicing to a second segment. In this case the bitrate must be reduced to avoid buffer overflow as a result of displaying the original frames at the end of the first segment. One method of reducing the bitrate is to insert a freeze frame at the end of the first segment, but this has the disadvantage of introducing distortion in the temporal presentation of the frames and precluding frame accuracy. A less disruptive method is to use the present invention for reducing the bitrate for a lower-quality presentation of one or more frames at the end of the first segment.
[0064] The present invention can also reduce the bit transmission rate and storage requirements for MPEG-2 applications by altering the video quality. For example, different clients may present different bandwidth access requests for video from nonscalable MPEG-2 files 32, 33 in the video file server. Also, temporary network congestion may limit the bandwidth available to satisfy a request for real-time streaming of video data. In each case, the present invention can alter the video quality to meet the desired or available bandwidth to satisfy the request.
[0065] With reference to FIG. 2 , there is shown a flowchart of a procedure executed by a stream server computer in the video file server of FIG. 1 to service client requests. In a first step 50, execution branches to step 51 when a client request is not a request for real-time streaming. If the request is a request to input a new MPEG-2 file, then execution branches to step 52 to input the new MPEG-2 file and to create a reduced-quality version of the MPEG-2 file as available resources permit. If the request is not a request to input a new MPEG-2 file, then execution continues from step 51 to step 53. In step 53, execution branches to step 54 if the request is for play list editing. In step 54, the client may browse through the reduced-quality MPEG file to select in-points and out-points of clips to be spliced.
[0066] In step 50, when the request is for real-time streaming, then execution branches to step 55. In step 55, if there is network congestion so that there is insufficient bandwidth to transmit a stream of original-quality MPEG-2 coded video, then execution branches to step 56 to stream compressed video from the reduced-quality MPEG file. If no reduced-quality MPEG file is available for the desired clip, then the reduced-quality MPEG coded video to be streamed is produced in real-time from the original-quality MPEG-2 coded video. There are also applications, such as the display of spatially down-sampled video in a small display window (39 in FIG. 1), for which the client may request reduced-quality MPEG coded video. In this case, in the absence of network congestion, execution will continue from step 55 to step 57, and branch from step 57 to step 56 for streaming of reduced-quality MPEG coded video to the client.
[0067] Reduced-quality MPEG coded video is also useful for “trick-mode” operation. Trick-mode refers to fast forward or fast reverse display of video, in a fashion analogous to the fast forward and fast reverse playback functions of a video cassette recorder (VCR). The problem with trick-mode operation is that the speed of the MPEG stream cannot simply be speeded up because the transmission bandwidth would be excessive and a conventional MPEG-2 decoder will not be able to handle the increased data rate or even if the decoder would have been able to support the increased data rate, such a change in the original operating conditions is not allowable. For this reason, in trick-mode, neither the original display rate of 29.97 frames per second (for NTSC or 25 frames per second for PAL) nor the original transport stream (TS) multiplex rate should change. Nor is it possible to simply decimate frames since only the I frames are independently coded, and the P frames and B frames need the content of certain other frames for proper decoding. The I frames typically occur once for every 15 frames. Assuming that this convention is followed in the encoding process, it would be possible to preserve and play each I frame from each and every group of pictures (GOP), resulting in a 15 times slower temporal sampling rate, or a 1 to 15 speeding up of motion if the I frames only are played back at the nominal NTSC rate of approximately 30 frames per second. Consequently, the content of a 60 minutes duration clip will be covered in 4 minutes. Unfortunately the average information content per frame for the I frames is more than four times the average information content of the P and B frames. Therefore, the trick-mode cannot be implemented simply by transmitting only the I frames for a speed-up by a factor of 15, because this would need an increase in the TS multiplex rate over the nominal rate.
[0068] In particular, the average information content of an I frame has been measured to be about 56,374.6 bytes. If the I frames only are transmitted at the standard NTSC rate, then the bit transmission rate would be: 8(bits per byte)*56,374.6(bytes per frame)*29.97(frames per sec.) or about 13,516,374.1 bits per second only for the video stream, which is significantly above—almost 3.38 times—the original rate of 4 megabits per second used in this test. This calculation, being based on an average quantity, is ignoring the indispensable need for an actually higher transport rate to provide some safety margin to handle short-term-sustained large size I frame chains (bursts) which practically always happen. Clearly, some form of modification in the trick-mode operation definition is required to handle this problem and pull the bit-rate requirement down to the nominal 4 megabits per second.
[0069] Two degrees of freedom are available to achieve such a reduction in the required bit-rate for trick-mode operation. The first is I frame compression quality and the second is a motion speed-up ratio. With respect to compression quality, it is well known that human observers' perception of image detail degrades with increasing motion speed of objects in the scene. Based on this fact, the type of D pictures were introduced in MPEG-1 video syntax for fast visible (forward or reverse) search purposes. (See ISO/IEC 11172-2: 1993 Information Technology—Coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mbits/s—Part 2: Video, Annex D.6.6. Coding D-Pictures, p. 102). D pictures make use of only the DC coefficients in intra coding to produce very low quality (in terms of SNR) reproductions of desired frames which were judged to be of adequate quality in fast search mode.
[0070] In order to provide support for enhanced quality trick-mode operation, the quality of the original I frames can be reduced by the preservation of just a sufficient number of AC DCT coefficients to meet the bit-rate limitation. Based on experiments with two standard video test sequences (one encoded at 15 Mbits/sec. and the other at 24 Mbits/sec. and both with I frames only), it is observed that the bandwidth for I frames can be scaled to one half by keeping about 9 lowest order AC coefficients and eliminating the rest. This scheme provides good quality even at the full spatial and temporal resolution, much better than D pictures.
[0071] The inherent speed-up ratio lower bound imposed by the GOP structure can be relaxed and further lowered by freeze (P) frame substitution in between genuine (SNR scaled or non-scaled) I frames. The maximum number of freeze frames that can be inserted before visually disturbing motion jerkiness occurs, is very likely to depend heavily on the original GOP structure (equivalently the separation between I frames of the original sequence) and the original amount of motion in the clip. However, 1, 2 or 3 freeze frame substitutions in between genuine I frames present reasonable choices which will yield speed-up ratios of 1 to 7.5, 1 to 5 and 1 to 3.75 respectively instead of the 1 to 15 speed-up ratio provided by the genuine I frames only implementation. (These ratios are computed by a first-order approximation that neglects a slight increase in bandwidth required by the consecutive freeze frames, which are inserted in between genuine I frames and can typically be made very small in size in comparison to the average size of a genuine I frame. Therefore, the insertion of 1, 2, 3 freeze frames will result in bandwidth reductions of 2 to 1, 3 to 1 and 4 to I respectively. The accuracy of this approximation degrades as more consecutive freeze frames and/or SNR scaling is employed.) An easy way to see the validity of these approximate figures is to note for example that in the case of I freeze frame insertion, the total presentation time of the trick-mode clip for an originally 60 minutes duration asset will increase from 4 minutes to 8 minutes. Since due to the underlying assumption of the first-order approximation stated above, the same amount of data (I frames only) will be transmitted in this doubled time interval, the bandwidth requirement will be halved. The final choice for trick-mode implementation should reflect a balanced trade-off along these two degrees of freedom. For example, SNR scaling of I frames down to 9 AC coefficients can be used along with single freeze frame insertion between I frames. These two choices, both of which are individually capable of providing a 2 to 1 bandwidth reduction as discussed before, will yield a combined 4 to 1 bandwidth reduction which will comfortably bring the non-scaled I frame-only bit-rate of 13,516,374.1 bits/sec. down to below the 4 Mbits/sec. quota. If the visual quality provided by 9 AC coefficients is not considered adequate, then SNR scaling could be tuned to keep more AC coefficients at the expense of a smaller bandwidth reduction. This, however, could be compensated consequently by increasing the number of freeze frames to be used in between I frames. Coarser quantization (and therefore poorer visual quality) can be tolerated at high trick-mode speeds and better visual quality should be retained at lower trick-mode speeds.
[0072] With reference to FIG. 2 , if the client has requested trick-mode operation, execution branches from step 58 to step 59. In step 59, execution branches to step 60 for a low value of speed-up. In step 60, the trick-mode stream is produced by streaming original-quality I frames and inserting three freeze frames per I frame, to yield a speed-up factor of 15/4=3.75 based on an original MPEG-2 coded stream having one I frame for every 15 frames. For a higher speed-up factor, execution branches from step 59 to step 61. In step 61, either one or two freeze frames are selected per I frame to provide a speed-up factor of 15/2=7.5, or 15/3=5 respectively. Then in step 62 the trick-mode stream is produced by streaming reduced-quality I frames and inserting the selected number of freeze frames between the reduced-quality I frames. If a trick-mode operation is not requested in step 58, then execution continues from step 58 to step 63. In step 63, the stream server computer streams original-quality MPEG-2 coded data to the client. Further details regarding trick-mode operation are described below with reference to FIGS. 7 to 10.
[0073] FIGS. 3 to 6 show further details regarding use of the present invention for MPEG splicing. In particular, reduced-quality frames are substituted for the freeze frames used in the seamless splicing procedure found in the common disclosure of Peter Bixby et al., U.S. application Ser. No. 09/539,747 filed Mar. 31, 2000; Daniel Gardere et al., U.S. application Ser. 09/540,347 filed Mar. 31, 2000; and John Forecast et al., U.S. application Ser. 09/540,306 filed Mar. 31, 2000; which are all incorporated by reference herein. The common disclosure in these U.S. applications considered pertinent to the present invention is included in the written description below with reference to FIGS. 3 to 6 in the present application (which correspond to FIGS. 19 , 22, 23, and 24 in each of the cited U.S. applications).
[0074] FIG. 3 shows a basic procedure for MPEG splicing. In the first step 121, the splicing procedure receives an indication of a desired end frame of the first clip and a desired start frame of the second clip. Next, in step 122, the splicing procedure finds the closest I frame preceding the desired start frame to be the In Point for splicing. In step 123, the splicing procedure adjusts content of the first clip near the end frame of the first clip and adjusts content of the second clip near the In Point in order to reduce presentation discontinuity (due to decoder buffer underflow) and also to prevent decoder buffer overflow when decoding the spliced MPEG stream. Finally, in step 124, the concatenation of the first clip up to about the Out Point and the second clip subsequent to about the In Point is re-formatted, including re-stamping of the presentation time stamps (PTS), decoding time stamps (DTS), and program clock reference (PCR) values for the audio and video streams in the second clip.
[0075] Considering now video splicing, the splicing procedure should ensure the absence of objectionable video artifacts, preserve the duration of the spliced stream, and if possible, keep all of the desired frames in the spliced stream. The duration of the spliced stream should be preserved in order to prevent any time drift in the scheduled play-list. In some cases, it is not possible to keep all of the original video frames due to buffer problems.
[0076] Management of the video buffer is an important consideration in ensuring the absence of objectionable video artifacts. In a constant bit rate (CBR) and uniform picture quality sequence, subsequent pictures typically have coded representations of drastically different sizes. The encoder must manage the decoder's buffer within several constraints. The buffer should be assumed to have a certain size defined in the MPEG-2 standard. The decoder buffer should neither overflow nor underflow. Furthermore, the decoder cannot decode a picture before it receives it in full (i.e. completely). Moreover, the decoder should not be made to “wait” for the next picture to decode; this means that every 40 ms in PAL and 1/29.97 second in NTSC, the decoder must have access to a full picture ready to be decoded.
[0077] The MPEG encoder manages the video decoder buffer through decode time stamps (DTS), presentation time stamps (PTS), and program clock reference (PCR) values. When splicing the end of a first clip to the beginning of a second clip, there will be a problem of video buffer management if a duration of time DTSL1−Te is different from a duration of time DTSF2−PCRe2 minus one video frame (presentation) interval, where DTSL1 is the DTS at the end of the first clip and indicates the time at which the video decoder buffer is emptied of video data from the first clip, Te is the time at which the last video frame's data is finished being loaded into the video decoder buffer, DTSF2 is the DTS of the first frame of the second clip, and PCRe2 is the PCR of the second clip extrapolated from the value of the most recent received genuine PCR record, to the first byte of the picture header sync word of the first video frame in the clip to start. The extrapolation adjusts this most recently received genuine PCR record value by the quotient of the displacement in data bits of the clip from the position where it appears in the second clip to the position at which video data of the first frame of the second clip begins, divided by the data transmission bit rate for transmission of the clip to the decoder. Because the time PCRe2 must immediately follow Te, there will be a gap in the decoding and presentation of video frames if DTSF2−PCRe2 is substantially greater than DTSL1−Te plus one video frame interval. In this case, the buffer will not be properly full to begin decoding of the second clip one video frame interval after the last frame of the first clip has been decoded. Consequently, either the second clip will be prematurely started to be decoded or the decoder will be forced to repeat a frame one or more times after the end of the display of the last frame from the first clip to provide the required delay for the second clip's buffer build-up. In the case of a premature start for decoding the second clip, a video buffer underflow risk is generated. On the other hand, in case of repeated frames, the desired frame accuracy for scheduled play-lists is lost besides the fact that neither a precise timing adjustment can be achieved through this procedure. If DTSF2−PCRe2 is substantially less than DTSL1-Te plus one video frame interval, then the decoder will not be able to decode the first frame of the second clip at the specified time DTSF2 because the last frame of the first clip will not yet have been removed from the video buffer. In this case a video buffer overflow risk is generated. Video buffer overflow may present a problem not only at the beginning of the second clip, but also at a subsequent location of the second clip. If the second clip is encoded by an MPEG-2 compliant encoder, then video buffer underflow or buffer overflow will not occur at any time during the decoding of the clip. However, this guarantee is no longer valid if the DTSF2−PCRe2 relationship at the beginning of the second clip is altered. Consequently, to avoid buffer problems, the buffer occupancy at the end of the first clip must be modified in some fashion. This problem is inevitable when splicing between clips having significantly different ending and starting buffer levels. This is why the Society of Motion Picture and Television Engineers (SMPTE) has defined some splice types corresponding to well-defined buffer levels. (See SMPTE Standard 312M, entitled “Splice Points for MPEG-2 Transport Streams,” SMPTE Journal, November 1998.) In order to seamlessly splice the first clip to the second clip, the content of the first clip (towards its end) is modified so that PCRe2 can immediately follow Te (by one byte transmission time) and DTSF2 can just follow DTSL1 (by one video frame presentation interval).
[0078] FIG. 4 shows a flow chart of a seamless video splicing procedure that attains the desired condition just described above. In a first step 141, the first DTS of the second clip is anchored at one frame interval later than the last DTS of the first clip in order to prevent a video decoding discontinuity. Then, in step 142, the procedure branches depending on whether the PCR extrapolated to the beginning frame of the second clip falls just after the ending time of the first clip. If so, then the splice will be seamless with respect to the original video content. Otherwise, the procedure branches to step 143. In step 143, the content of the first clip is adjusted so that the PCR extrapolated to the beginning frame of the second clip falls just after the ending time of the first clip. Therefore the desired conditions for seamless video splicing are achieved.
[0079] With reference to FIG. 5 , there is shown a more detailed flow chart of a seamless video splicing procedure. In a first step 151, the procedure inspects the content of the first clip to determine the last DTS/PTS of the first clip. This last DTS/PTS of the first clip is designated DTSL1. Next, in step 152, the procedure inspects the content of the first clip to determine the time of arrival (Te) of the last byte of the first clip. In step 153, the procedure adds one frame interval to DTSL1 to find the desired first DTS location for the second clip. The sum, designated DTSF1, is equal to DTSL1+1/FR, where FR is the video frame rate. In step 154, while keeping the DTS−PCRe relationship unaltered for the second clip, the procedure finds the time instant, designated TS, at which the first byte of the second clip should arrive at the decoder buffer. This is done by calculating TSTART=DTSF2−PCRe2, and TS=DTSF1−TSTART.
[0080] Continuing in FIG. 6 , in step 155, execution branches depending on whether TS is equal to Te plus 8 divided by the bit rate. If not, then the clips to be spliced need modification before concatenation, and execution branches to step 156. In step 156, execution branches depending on whether TS is less than Te plus 8 divided by the bit rate. If not, then there is an undesired gap in between the clips to be spliced, and execution branches to step 157. In step 157, null packets are inserted into the clips to be spliced to compensate for the gap. The gap to be compensated has a number of bytes, designated Gr, equal to (TS−Te)(BIT RATE)/8 minus one. If in step 156, TS is less than Te plus 8 divided by the bit rate, then execution continues from step 156 to step 158 to open up a certain amount of space in the first clip to achieve TS=Te+8/(BIT RATE). The number of bytes to drop is one plus (Te-TS)(BIT RATE)/8. If possible, the bytes are dropped by removing null packets. Otherwise, one or more frames at the end of the first clip are replaced with corresponding reduced-quality frames, which have fewer bytes than the original-quality frames at the end of the first clip.
[0081] If in step 155 TS is found to be equal to Te plus 8 divided by the bit rate, then execution continues to step 159. Execution also continues to step 159 from steps 157 and 158. In step 159, the transport streams from the two clips are concatenated. Finally, in step 160, a subroutine is called to compute a video time stamp offset, designated as VOFFSET. This subroutine finds the DTS of the last video frame (in decode order) of the first clip. This DTS of the last video frame of the first clip is denoted DTSVL1. Then the subroutine finds the original DTS of the first frame to be decoded in the second clip. This DTS of the first frame to be decoded in the second clip is denoted DTSVF2. Finally, the subroutine computes the video time stamp offset VOFFSET as DTSVL1−DTSVF2 plus one video frame duration.
[0082] FIGS. 7 to 10 show further details regarding trick-mode operation. FIG. 7 shows a timing relationship between video presentation units (VPUs) and associated audio presentation units (APUs) in an original MPEG-2 coded data stream, and FIG. 8 shows similar timing for the fast-forward trick-mode stream produced from the original data stream of FIG. 7. (The fast-forward trick-mode stream is an example of a trick-mode stream that could be produced in step 60 of FIG. 2. ) The original data stream has successive video presentation units for video frames of type I, B, B, P, B respectively. The trick-mode stream has successive video presentation units for video frames of types I, F, F, I, F where “F” denotes a freeze P (or possibly B) frame. Each I frame and immediately following F frames produce the same video presentation units as a respective I frame in the original data stream of FIG. 7 , and in this example, one in every 15 frames in the original data stream is an I frame. Each freeze frame is coded, for example, as a P frame repeating the previous I frame or the previous P-type freeze-frame (in display order). In each freeze frame, the frame is coded as a series of maximum-size slices of macroblocks, with an initial command in each slice indicating that the first macroblock is an exact copy of the corresponding macroblock in the previous frame (achieved by predictive encoding with a zero valued forward motion compensation vector and no encoded prediction error), and two consequent commands indicating that the following macroblocks in the slice until and including the last macroblock of the slice are all coded in the same way as the first macroblock.
[0083] For trick-mode operation, there is also a problem of how to select audio presentation units (APU) to accompany the video presentation units that are preserved in the trick-mode stream. Because the video presentation units (VPU) have a duration of ( 1/29.97) sec. or about 33.37 msec. and the audio presentation units (APU) have a duration of 24 msec., there is neither a one-to-one correspondence nor alignment between VPUs and APUs. In a preferred implementation, the audio content of a trick-mode clip is constructed as follows. Given the total presentation duration ( 1/29.97) sec. or about 33.37 msec. for a single video frame, it is clear that always at least one and at most two 24 msec. long audio presentation units (APU) will start being presented during the end-to-end presentation interval of each video frame. This statement refers to the original clip and does not consider any audio presentation unit whose presentation is possibly continuing as the video frame under consideration is just put on the display. The first of the above mentioned possibly two audio presentation units will be referred to as the aligned audio presentation unit with respect to the video frame under consideration. For example, in FIG. 8 , the APUj is the aligned audio presentation unit with respect to the VPUi. Now, when the I frames are extracted and possibly SNR scaled and possibly further interleaved with a number of freeze P frames in between them to produce the trick-mode video packetized elementary stream (PES), the associated trick-mode audio stream is constructed as follows. For each I type video frame presentation interval (and for that matter also for freeze P type video frames) in this trick-mode clip, the above stated fact of at least one (and at most two) audio presentation unit being started, holds. Then for each I frame presentation interval in the trick-mode clip, once any possibly previously started and continuing audio presentation unit ends, insert its aligned audio presentation unit (from the original clip) and continue inserting APUs from the original clip subsequent to the aligned one until covering the rest of the I frame presentation interval and also any possibly following freeze P frame presentation intervals until crossing into and overlapping (or less likely aligning) with the next I frame's presentation interval. In FIG. 8 , for example, the audio presentation units APUj, APUj+1, APUj+2, and APUj+3 are inserted, until crossing into and overlapping with the next I frame VPUi+15. Following APUj+3 is inserted APUk, which designates the APU aligned with VPUi+15 in the original stream. Clearly, the final alignment of (the aligned and consequent) audio presentation units with respect to their associated I frames will be slightly different in the trick-mode clip as compared to the original clip. However, considering how the trick-mode audio component will sound like, this poses no problem at all.
[0084] FIG. 9 is a flowchart of a procedure for producing the desired sequencing of audio presentation units (APUs) in the fast-forward trick-mode stream. This procedure scans the audio elementary stream in the original MPEG-2 stream to determine the sequence of APUs in the original stream and their presentation-time alignment with the I frames in the video elementary stream of the original MPEG-2 transport stream, while selecting APUs to include in the trick-mode stream. In a first step 171, execution proceeds once the end of the current APU is reached. If the end of the current APU has not entered a new VPU (i.e., the beginning of the current APU is within the presentation time of one VPU and the end of the current APU is within the presentation time of the same VPU), or if it has entered a new VPU (i.e., the beginning of the current APU is within the presentation time of one VPU and the end of the current APU is within the presentation time of a new (next) VPU) but the new VPU is not an I frame, then execution branches to step 174. In step 174, an APU pointer is incremented, and in step 175 execution proceeds into this next APU. If in step 173 the end of the current APU extends into an I frame, then in step 176 the APU pointer is advanced to point to the first APU beginning within the duration of the VPU of the I frame in the original MPEG-2 stream.
[0085] FIG. 10 is a flowchart of a procedure for producing a trick-mode stream from an MPEG-2 transport stream (TS). In a first step 181, the MPEG-2 TS is inputted. In step 182, the video elementary stream (VES) is extracted from the TS. In step 183, a concurrent task extracts the audio elementary stream (AES) from the TS. In step 184, I frames are extracted from the VES and valid packetized elementary stream (PES) packets are formed encapsulating the I frames. In step 185, the I frames are SNR scaled, for the high speed cases of the trick-mode. In step 186, P-type freeze frames are inserted into the stream of SNR scaled I frames (in between the scaled I frames), and valid PES packets are formed for the trick-mode VES encapsulating the P-type freeze frames and the SNR scaled I frames. Concurrently, in step 187, appropriate audio access units (from the originally input MPEG-2 TS asset) are selected and concatenated based on the structure of the VES being formed for the trick-mode clip, as described above with reference to FIG. 9 , and valid PES packet encapsulation is formed around these audio access units. Finally, in step 188, the trick-mode TS stream is generated by multiplexing the trick-mode VES from step 186 into a system information (SI) and audio PES carrying TS skeleton including the audio PES packets from step 187.
[0086] FIGS. 11 to 19 include details of the preferred techniques for truncating AC DCT coefficients for producing low-quality MPEG coded video from original-quality MPEG-2 coded video. Most of these techniques exploit the fact that in the typical (default) zig-zag scan order, the basis functions for the high-order AC DCT coefficients have an increasing frequency content. FIG. 11 , for example, shows a matrix of the DCT coefficients Cij. The row index (i) increases with increasing vertical spatial frequency in a corresponding 8×8 coefficient block, and the column index (1) increases with increasing horizontal spatial frequency in the corresponding 8×8 coefficient block. The coefficient C11 has zero frequency associated with it in both vertical and horizontal directions, and therefore it is referred to as the DC coefficient of the block. The other coefficients have non-zero spatial frequencies associated with their respective basis functions, and therefore they are referred to as AC coefficients. Each coefficient has an associated basis function fij(x,y) that is separable into x and y components such that fij(x,y)=fi(y)fj(x). The x and y component functions fi(y) and fj(x) are shown graphically in FIG. 11 as cosine functions in order to illustrate their associated spatial frequencies. In practice, the component functions are evaluated at discrete points for the 64 pixel positions in the 8×8 blocks, so that each of the DCT basis functions is an 8×8 array of real numbers. In particular, the component functions are:
fi(y)=SQRT((2−δi-1)/8)(cos((π/8)(y−½)(i−1))) for y=1, 2, 3, . . . , 8
fj(x)=SQRT((2−δj-1)/8)(cos((π/8)(x−½)(j−1))) for x=1, 2, 3, . . . , 8
The heavy black line through the matrix of coefficients inFIG. 11 denotes the default zig-zag scan order typically used for MPEG-2 encoding. Listed in this order, the coefficients are C11, C12, C21, C31, C22, C13, C14, C23, C32, C41, . . . , C86, C77, C68, C78, C87, C88. The first coefficient in this zig-zag scan order is the DC coefficient C11 providing the lowest spatial frequency content in the 8×8 block of pixels, and the last coefficient in this zig-zag scan order is the coefficient C88 providing the highest spatial frequency content in the 8×8 block of pixels.
fi(y)=SQRT((2−δi-1)/8)(cos((π/8)(y−½)(i−1))) for y=1, 2, 3, . . . , 8
fj(x)=SQRT((2−δj-1)/8)(cos((π/8)(x−½)(j−1))) for x=1, 2, 3, . . . , 8
The heavy black line through the matrix of coefficients in
[0087] FIG. 12 is a diagram illustrating a relationship between an original MPEG-2 coded bit stream 200 and a reduced-quality MPEG-2 coded bit stream 210 resulting from truncation of high-order DCT coefficients from the original MPEG-2 coded bit stream. Shown in the original MPEG-2 coded bit stream 200 is a portion of a video PES packet including DCT coefficients for an 8×8 pixel block. The DCT coefficients include a differentially coded DC coefficient 201, and three (run, level) events 202, 203, 204 encoding three respective nonzero AC coefficients possibly along with some zero valued AC coefficients preceding the three nonzero valued ones. The DCT coefficients are ordered according to the zig-zag scan order shown in FIG. 11 (or possibly according to an alternate zig-zag scan pattern also supported by the MPEG-2 standard), and AC coefficients having zero magnitude are described in terms of total counts of consecutive zero valued coefficients lying in between two nonzero valued coefficients, in the MPEG-2 coded bit stream. An end-of-block (EOB) code 205 signals the end of the encoded DCT coefficients for the current block. The reduced-quality MPEG-2 coded bit stream 210 includes a DC coefficient 201′ identical to the DC coefficient 201 in the original MPEG-2 coded bit stream 200, and a (run, level) event 202′ identical to the (run, level) event 202 in the original MPEG-2 coded bit stream 200. Second and third (run, level) events, however, have been omitted from the reduced-quality MPEG-2 bit stream 210, because an EOB code 205′ immediately follows the (run, level) event 202′. Therefore, the two nonzero high-order AC DCT coefficients encoded by the second and third (run, level) events 203, 204 have been omitted from the reduced-quality MPEG-2 bit stream 210.
[0088] FIG. 13 is a flowchart of a procedure for scaling MPEG-2 coded video using a variety of techniques including the omission of AC DCT coefficients. The procedure operates upon an original-quality MPEG-2 coded video stream by removing AC DCT coefficients in this stream to produce a lower quality MPEG coded video stream. In a first step 221, execution branches to step 222 if the scaled MPEG coded video is to be spatially subsampled. In step 222, the procedure removes any and all DCT coefficients for spatial frequencies in excess of the Nyquist frequency for the downsampled video. For example, if the low-quality video stream will be downsampled by a factor of two in both the vertical and the horizontal directions, then the procedure removes any and all DCT coefficients having a row index (i) greater than four and any and all DCT coefficients having a column index (j) greater than four. This requires the decoding of the (run, level) coded coefficients to the extent necessary to obtain an indication of the coefficient indices. If a sufficient number of the original AC DCT coefficients are removed for a desired bandwidth reduction, then the scaling procedure is finished. Otherwise, execution branches from step 223 to step 224. Execution also continues from step 221 to step 224 if spatial downsampling is not intended.
[0089] In step 224, execution branches to step 225 if low-pass scaling is desired. Low-pass scaling requires the least computational resources and may produce the best results if the scaled, low-quality MPEG coded video is spatially downsampled. In step 225, the procedure retains up to a certain number of lowest-order AC DCT coefficients for each block and removes any additional DCT coefficients for each block. This is a kind of frequency domain signal-to-noise ratio scaling (FDSNR) that will be designated FDSNR_LP. A specific example of the procedure for step 225 will be described below with reference to FIG. 14.
[0090] Execution continues from step 224 to step 226 if low-pass scaling is not desired. In step 226, execution branches to step 227 if largest magnitude based scaling is desired. Largest magnitude based scaling produces the least squared error or difference between the original-quality MPEG-2 coded video and the reduced-quality MPEG coded video for a given number of nonzero AC coefficients to preserve, but it requires more computational resources than the low-pass scaling of step 225. More computational resources are needed because if there are more nonzero AC coefficients than the desired number of AC coefficients for a block, then the (run, level) events must be decoded fully to obtain the coefficient magnitudes, and additional resources are required to find the largest magnitude coefficients. In step 227, the procedure retains up to a certain number of largest magnitude AC DCT coefficients for each block, and removes any and all additional AC DCT coefficients for each block. This is a kind of frequency domain signal-to-noise ratio scaling (FDSNR) that will be designated FDSNR_LM. A specific example of the procedure for step 227 will be described below with reference to FIG. 15.
[0091] If in step 226 largest magnitude based scaling is not desired, then execution continues to step 228. In step 228, execution branches to step 229 to retain up to a certain number of AC DCT coefficients that differ in magnitude from up to that number of largest magnitude AC DCT coefficients by no more than a certain limit. This permits a kind of approximation to FDSNR_LM in which an approximate search is undertaken for the largest magnitude AC DCT coefficients if there are more nonzero AC DCT coefficients than the desired number of AC DCT coefficients in a block. The approximate search can be undertaken using a coefficient magnitude classification technique such as a hashing technique, and the low-pass scaling technique can be applied to the classification level that is incapable of discriminating between the desired number of largest magnitude AC DCT coefficients. A specific example is described below with reference to FIG. 19.
[0092] With reference to FIG. 14 , there is shown a flowchart of a procedure for scaling MPEG-2 coded video using the low-pass frequency-domain signal-to-noise (FDSNR_LP) scaling technique. This procedure scans and selectively copies components of an input stream of original-quality MPEG-2 coded data to produce an output stream of reduced-quality MPEG-2 coded video. The procedure is successively called, and each call processes coefficient data in the input stream for one 8×8 block of pixels. No more than a selected number “k” of coded lowest order (nonzero or zero valued) AC coefficients are copied for the block where the parameter “k” can be specified for each block.
[0093] In a first step 241 of FIG. 14 , the procedure parses and copies the stream of original-quality MPEG-2 coded data up to and including the differential DC coefficient variable-length code (VLC). Next, in step 242, a counter variable “l” is set to zero. In step 243, the procedure parses the next (run, level) event VLC in the stream of original-quality MPEG-2 coded data. In step 244, if the VLC just parsed is an end-of-block (EOB) marker, execution branches to step 245 to copy the VLC to the stream of reduced-quality MPEG-2 coded video, and the procedure is finished for the current block.
[0094] In step 244, if the VLC just parsed is not an EOB marker, then execution continues to step 246. In step 246, a variable “r” is set equal to the run length of zeroes for the current (run, level) event, in order to compute a new counter value l+r+1. In step 247, if the new counter value l+r+1 is greater than the parameter “k”, then the procedure branches to step 248 to copy an EOB marker to the stream of reduced-quality MPEG coded data. After step 248, execution continues to step 249, where the procedure parses the input stream of original-quality MPEG-2 coded data until the end of the next EOB marker, and the procedure is finished for the current block.
[0095] In step 247, if the new counter value l+r+1 is not greater than the parameter “k”, then execution continues to step 250. In step 250, execution branches to step 251 if the new counter value l+r+1 is not equal to “k” (which would be the case if the new counter value is less than “k”). In step 251, the counter state l is set equal to the new counter value l+r+1. Then, in step 252, the VLC just parsed (which will be a VLC encoding a (run, level) event) is copied from the stream of original-quality MPEG-2 coded data to the stream of reduced-quality MPEG-2 coded data. After step 252, execution loops back to step 243 to continue the scanning of the stream of original-quality MPEG-2 coded data.
[0096] In step 250, if the new counter value i+r+1 is equal to “k”, then execution branches from step 250 to step 253, to copy the VLC just parsed (which will be a VLC encoding a (run, level) event) from the stream of original-quality MPEG-2 coded data to the stream of reduced-quality MPEG-2 coded data. Next, in step 254, the procedure copies an EOB marker to the stream of reduced-quality MPEG-2 coded data. After step 254, execution continues to step 249, where the procedure parses the input stream of original-quality MPEG-2 coded data until the end of the next EOB marker, and the procedure is finished for the current block.
[0097] FIG. 15 is a flowchart of a procedure for scaling MPEG-2 coded video using the largest magnitude based frequency-domain signal-to-noise ratio (FDSNR LM) scaling technique. This routine is successively called, and each call processes coefficient data in the input stream for one 8×8 block of pixels. No more than a specified number “k” of largest magnitude AC DCT coefficients are copied for the block, and a different number “k” can be specified for each block.
[0098] In a first step 261 in FIG. 15 , the procedure parses and copies the input stream of original-quality MPEG-2 coded data to the output stream of lower-quality MPEG-2 data up to and including the differential DC coefficient variable-length code (VLC). Then in step 262 all (run, level) event VLCs are parsed and decoded until and including the EOB marker of the current block. The decoding produces coefficient identifiers and corresponding quantization indices representing the quantized coefficient values. In step 263, the quantization indices are transformed to quantized coefficient values. In step 264, the (quantized) coefficients are sorted in descending order of their magnitudes. In step 265, the first “k” coefficients of the sorted list are preserved and the last 63-k AC DCT coefficients in the sorted list are set to zero. In step 266, (run, level) event formation and entropy coding (VLC encoding) are applied to the new set of coefficient values. Finally, in step 267, the VLCs resulting from step 266 are copied to the output stream until and including the EOB marker.
[0099] II The sorting step 264 of the FDSNR_LM procedure can consume considerable computational resources. It is important to notice that not a full sorting of the quantized AC coefficients with respect to their magnitudes but rather a search for a specified number “k” of largest magnitude AC coefficients is all that is required. This task can be performed exactly or approximately in different ways so as to avoid the complexity associated with a conventional sorting procedure. In general, a relatively large number of the 63 AC DCT coefficients will have a quantized value of zero. Only the non-zero coefficients need be included in the sorting process. Moreover, if there are “n” non-zero coefficients and only “k” of them having the largest magnitudes are to be preserved in the output stream, then the sorting process may be terminated immediately after only the largest magnitude “k” coefficients have been found, or equivalently immediately after only the smallest magnitude “n−k” coefficients have been found. Moreover, the sorting procedure itself can be different depending on a comparison of “k” to “n” in order to minimize computations.
[0100] With reference to FIG. 16 , there is shown a flowchart of a procedure that selects one of a number of techniques for finding a certain number “k” of largest values out of a set of “n” values. In a first step 271, execution branches to step 272 if “k” is less than ½ “n.” In step 272, execution branches to step 273 if “k” is much less than ½ “n.” In step 273, the first “k” values are sorted to produce a list of “k” sorted values, and then the last “n−k” values are scanned for any value greater than the minimum of the sorted “k” values. If a value greater than the minimum of the sorted “k” values is found, then that minimum value is removed and the value greater than the minimum value is inserted into the list of “k” sorted values. At the end of this procedure, the list of sorted “k” values will contain the maximum “k” values out of the original “n” values. A specific example of this procedure is described below with reference to FIG. 17.
[0101] In step 272, if “k” is not much less than ½ “n”, then execution branches to step 274. In step 274, a bubble-sort procedure is used, including “k” bottom-up bubble-sort passes over the “n” values to put “k” maximum values on top of a sorting table. An example of such a bubble-sort procedure is listed below:
[0102]
| /* TABLE(0) to TABLE(n−1) INCLUDES n VALUES */ | |
| /* MOVE THE k LARGEST OF THE n VALUES IN TABLE TO THE | |
| RANGE | |
| TABLE(0) TO TABLE(k−1) IN THE TABLE */ | |
| /* k <= ½ n */ | |
| FOR i=1 to k | |
| FOR j=1 to n−i | |
| IF (TABLE(n−j) > TABLE(n−j−1)) THEN( | |
| /* SWAP TABLE(n−j) WITH TABLE(n−j−1) */ | |
| TEMP ← TABLE(n−j) | |
| TABLE(n−j) ← TABLE(n−j−1) | |
| TABLE(n−j−1) ← TEMP | |
| NEXT j | |
| NEXT I | |
[0103] In step 271, if “k” is not less than ½“n”, then execution branches to step 275. In step 275, if “k” is much greater than ½ “n”, then execution branches to step 276. In step 276, a procedure similar to step 273 is used, except the “n−k” minimum values are maintained in a sorted list, instead of the “k” maximum values. In step 276, the last “n-k” values are placed in the sort list and sorted, and then the first “k” values are scanned for any value less than the maximum value in the sorted list. If a value less than the maximum value in the sorted list is found, then the maximum value in the sorted list is removed, and the value less than this maximum value is inserted into the sorted list. At the end of this procedure, the values in the sorted list are the “n−k” smallest values, and the “k” values excluded from the sorted list are the “k” largest values.
[0104] In step 275, if “k” is not much greater than ½“n”, then execution branches to step 277. In step 277, a bubble-sort procedure is used, including “n−k” top-down bubble-sort passes over the “n” values to put “n-k” minimum values at the bottom of a sorting table. Consequently, the k maximum values will appear in the top “k” entries of the table. An example of such a bubble-sort procedure is listed below:
[0105]
| /* TABLE(0) to TABLE(n−1) INCLUDES n VALUES */ | |
| /* MOVE THE n-k SMALLEST OF THE n VALUES IN THE TABLE */ | |
| /* TO THE RANGE TABLE(k) TO TABLE(n−1) IN THE TABLE */ | |
| /* n > k >= ½ n */ | |
| FOR i=1 to n-k | |
| FOR j=0 to n−i−1 | |
| IF (TABLE(j) < TABLE(j+1)) THEN( | |
| /* SWAP TABLE(j) WITH TABLE(j+1)*/ | |
| TEMP ← TABLE(j) | |
| TABLE(j) ← TABLE(j+1) | |
| TABLE(j+1) ← TEMP | |
| NEXT j | |
| NEXT I | |
[0106] Turning now to FIG. 17 , there is shown a flowchart of a procedure for finding up to a specified number “k” of largest magnitude AC DCT coefficients from a set of “n” coefficients, corresponding to the procedure of FIG. 16 for the case of k<<½n. In a first step 281, a counter “i” is set to zero. In step 282, the next AC DCT coefficient is obtained from the input stream of original-quality MPEG-2 coded data. If an EOB marker is reached, as tested in step 283, then execution returns. In step 284, the counter “i” is compared to the specified number “k”, and if “i” is less than “k”, execution continues to step 285. In step 285, a coefficient index and magnitude for the AC DCI coefficient is placed on a sort list. In step 286, the counter “i” is incremented, and execution loops back to step 282.
[0107] Once the sort list has been loaded with indices and magnitudes for “k” AC DCT coefficients and one additional coefficient has been obtained from the input stream, execution branches from step 284 to step 287. In step 287 the list is sorted by magnitude, so that the minimum magnitude appears at the end of the list. Then in step 288 the coefficient magnitude of the current coefficient last obtained from the input stream is compared to the magnitude at the end of the list. If the coefficient magnitude of the current coefficient is not greater than the magnitude appearing at the end of the list, then execution continues to step 289 to get the next AC DCT coefficient from the input stream. If an EOB marker is reached, as tested in step 290, then execution returns. Otherwise, execution loops back to step 288.
[0108] In step 288, if the magnitude of the current coefficient is greater than the magnitude at the end of the list, then execution branches to step 291. In step 291, the entry at the end of the list is removed. In step 292, a binary search is performed to determine the rank position of the magnitude of the current coefficient, and in step 293, the current coefficient index and magnitude are inserted into the list at the rank position. The list, for example, is a linked list in the conventional fashion to facilitate the insertion of an entry for the current coefficient at any position in the list. After step 293, execution loops back to step 288.
[0109] An approximation technique of coefficient magnitude classification can be used to reduce the computational burden of sorting by coefficient magnitude. A specific example is the use of hashing of the coefficient magnitude and maintaining lists of the indices of coefficients having the same magnitude classifications. As shown in FIG. 18 , a hash table 300 is linked to hash lists 301 storing the indices of classified coefficients. As shown, the hash table 300 is a list of 2M entries, where “M” is three, and an entry has a value of zero if its associated list is empty, and otherwise the entry has a pointer to the end of the coefficients in its associated list. The lists shown in FIG. 18 have fixed memory allocations in which the pointers in the hash table also indicate the number of coefficient indices in the respective hash lists. Alternatively, the hash lists could be dynamically allocated and linked in the conventional fashion.
[0110] FIG. 19 shows a flowchart of a procedure for using the hash table 300 and hash lists 301 of FIG. 18 to perform a sort of “k” coefficients having approximately the largest magnitudes from a set of “n” coefficients. This approximation technique ensures that none of the “k” coefficients selected will have a magnitude that differs by more than a certain error limit from the smallest magnitude value of “k” coefficients having the largest magnitude. The error limit is established by the number of hash table entries, and it is the range of the magnitudes that can be hashed to the same hash table entry.
[0111] In a first step 311 in FIG. 19 , the hash table is cleared. Then in step 312, the next AC DCT coefficient is obtained from the input stream. If an EOB marker is not reached, as tested in step 313, then execution continues to step 314. In step 314, a hash table index is stripped from the most significant bits (MSBs) of the coefficient magnitude. For the hash table in FIG. 18 having eight entries, the three most significant bits of the coefficient magnitude are stripped from the coefficient magnitude. This is done by a bit masking operation together with a logical arithmetic shift operation. Then in step 315, the coefficient index is inserted on the hash list of the indexed hash table entry. For example, the hash table entry is indexed to find the pointer to where the coefficient index should be inserted, and then the pointer in the hash table entry is incremented. After step 315, execution loops back to step 312. Once all of the AC coefficients for the block have been classified by inserting them in the appropriate hash lists, an EOB marker will be reached, and execution will branch from step 313 to step 316.
[0112] Beginning in step 316, the hash table and hash lists are scanned to find approximately the “k” largest magnitude coefficients. The hash lists linked to the bottom entries of the hash table will have the indices for the largest magnitude coefficients. Each hash list is scanned from its first entry to its last entry, so that each hash list is accessed as a first-in-first-out queue. Therefore, in each magnitude classification, the coefficient ordering in the output stream will be the same as the coefficient ordering in the input stream, and the approximation will have a “low pass” effect in which possibly some lower-frequency coefficients having slightly smaller magnitudes will be retained at the expense of discarding some higher-frequency coefficients having slightly larger magnitudes. (The approximation results from the fact that the last hash list to be scanned is not itself sorted, and to eliminate the error of the approximation, the last hash list to be scanned could be sorted.)
[0113] In step 316, a scan index “i” is set to 2M−1 in order to index the hash table beginning at the bottom of the table, and a counter “j” is set equal to “k” in order to stop the scanning process after finding “k” coefficients. Next, in step 317, the hash table is indexed with “i”. In step 318, if the indexed entry of the hash table is zero, then execution branches to step 319. In step 319, the procedure is finished if “i” is equal to zero; otherwise, execution continues to step 320. In step 320, the index “i” is decremented, and execution loops back to step 317.
[0114] If in step 318 the indexed hash table entry is not zero, then execution continues to step 321. In step 321, the next entry is obtained from the indexed hash list, and the coefficient index in the entry is used to put the indexed coefficient in the output stream. Then in step 322 execution branches to step 319 if the end of the indexed hash list is reached in the previous step 321. If the end of the list was not reached in step 321, then execution continues from step 322 to step 323. In step 323 the counter “j” is decremented, and in step 324 the counter “j” is compared to zero. In step 324, if the counter “j” is less than or equal to zero, then the procedure is finished. Otherwise, if the counter “j” is not less than or equal to zero in step 324, execution loops back to step 321.
[0115] The FDSNR_LM procedure, as described above, in general provides a significant improvement in peak signal-to-noise ratio (PSNR) over the FDSNR_LP procedure when each procedure retains the same number of non-zero AC DCT coefficients. It has been found, however, that substantially more bits are required for the (run, level) coding of the non-zero AC DCT coefficients resulting from the FDSNR_LM procedure than those resulting from the FDSNR_LP procedure, provided that the same coefficient quantization and scanning method is used. Therefore, the FDSNR_LM procedure provides at best a marginal improvement in rate-distortion (PSNR as a function of bit rate) over the FDSNR_LP procedure unless the non-zero AC DCT coefficients for the FDSNR_LM procedure are quantized, scanned, and/or (run, level) coded in a fashion different from the quantization, scanning, and/or (run, level) coding of the coefficients in the original MPEG-2 clip. A study of this problem resulted in a discovery that it is sometimes possible to reduce the number of bits for (run, level) coding of coefficients for an 8×8 block including a given number of the non-zero largest magnitude AC DCT coefficients if additional coefficients are also (run, level) coded for the block.
[0116] The (run, level) coding of the non-zero AC DCT coefficients from the FDSNR_LM procedure has been found to require more bits than from the FDSNR_LP procedure due to an increased occurrence frequency of escape sequences for the (run, level) coding. The increased frequency of escape sequences is an indication that the statistical likelihood of possible (run, level) combinations for the non-zero AC DCT coefficients selected by the FDSNR_LM procedure is different from the statistical likelihood of possible (run, level) combinations for the non-zero AC DCT coefficients produced by the standard MPEG-2 coding process and in particular those selected by the FDSNR_LP procedure.
[0117] The MPEG-2 coding scheme assigns special symbols to the (run, level) combinations that occur very frequently in ordinary MPEG-2 coded video. The most frequent (run, level) combinations occur for short run lengths (within the range of about 0 to 5, where the run length can range from 0 to 63) and relatively low levels (about 1 to 10, where the level can range from 1 to 2048). The most frequent of these special symbols are assigned the shortest variable-length code words (VLCs). If a (run, level) combination does not have such a special symbol, then it is coded as an escape sequence including a 6-bit escape sequence header code word followed by a 6-bit run length followed by a 12 bit signed level. An escape sequence requires a much greater number of bits than the special symbols, which have varying lengths depending on their relative frequency. In particular, the escape sequences each have 24 bits, and the special symbols have variable-length code words having a maximum of 17 bits.
[0118] There are two (run, level) VLC tables. The first coding table is designated TABLE 0, and the second is designated TABLE 1. These tables specify the (run, level) combinations having special symbols, and the special symbol for each such combination. For each table, the (run, level) combinations having special symbols, and the range of the VLC bit lengths of the special symbols, are summarized below:
[0119]
| SUMMARY OF PROPERTIES OF DCT COEFFICIENT TABLE ZERO | ||
| (Table Zero is Table B.14, p. 135 of ISO/IEC 13818-2 1996E) | ||
| Run | Range of Levels | Range of Code Lengths |
| 0 | 1 to 40 | 2 to 16 |
| 1 | 1 to 18 | 4 to 17 |
| 2 | 1 to 5 | 5 to 14 |
| 3 | 1 to 4 | 6 to 14 |
| 4 | 1 to 3 | 6 to 13 |
| 5 | 1 to 3 | 7 to 14 |
| 6 | 1 to 3 | 7 to 17 |
| 7 | 1 to 2 | 7 to 13 |
| 8 | 1 to 2 | 8 to 13 |
| 9 | 1 to 2 | 8 to 14 |
| 10 | 1 to 2 | 9 to 14 |
| 11 | 1 to 2 | 9 to 17 |
| 12 | 1 to 2 | 9 to 17 |
| 13 | 1 to 2 | 9 to 17 |
| 14 | 1 to 2 | 11 to 17 |
| 15 | 1 to 2 | 11 to 17 |
| 16 | 1 to 2 | 11 to 17 |
| 17 | 1 | 13 |
| 18 | 1 | 13 |
| 19 | 1 | 13 |
| 20 | 1 | 13 |
| 21 | 1 | 13 |
| 22 | 1 | 14 |
| 23 | 1 | 14 |
| 24 | 1 | 14 |
| 25 | 1 | 14 |
| 26 | 1 | 14 |
| 27 | 1 | 17 |
| 28 | 1 | 17 |
| 29 | 1 | 17 |
| 30 | 1 | 17 |
| 31 | 1 | 17 |
[0120]
| SUMMARY OF PROPERTIES OF DCT COEFFICIENT TABLE ONE | ||
| (Table One is Table B.15, p. 139 of ISO/IEC 13818-2 1996E) | ||
| Run | Range of Levels | Range of Code Lengths |
| 0 | 1 to 40 | 3 to 16 |
| 1 | 1 to 18 | 4 to 17 |
| 2 | 1 to 5 | 6 to 14 |
| 3 | 1 to 4 | 6 to 14 |
| 4 | 1 to 3 | 7 to 13 |
| 5 | 1 to 3 | 7 to 14 |
| 6 | 1 to 3 | 8 to 17 |
| 7 | 1 to 2 | 8 to 13 |
| 8 | 1 to 2 | 8 to 13 |
| 9 | 1 to 2 | 8 to 14 |
| 10 | 1 to 2 | 8 to 14 |
| 11 | 1 to 2 | 9 to 17 |
| 12 | 1 to 2 | 9 to 17 |
| 13 | 1 to 2 | 9 to 17 |
| 14 | 1 to 2 | 10 to 17 |
| 15 | 1 to 2 | 10 to 17 |
| 16 | 1 to 2 | 11 to 17 |
| 17 | 1 | 13 |
| 18 | 1 | 13 |
| 19 | 1 | 13 |
| 20 | 1 | 13 |
| 21 | 1 | 13 |
| 22 | 1 | 14 |
| 23 | 1 | 14 |
| 24 | 1 | 14 |
| 25 | 1 | 14 |
| 26 | 1 | 14 |
| 27 | 1 | 17 |
| 28 | 1 | 17 |
| 29 | 1 | 17 |
| 30 | 1 | 17 |
| 31 | 1 | 17 |
[0121] The FDSNR_LP procedure selected AC DCT coefficients have (run, level) symbol statistics that are similar to the statistics of ordinary MPEG-2 coded video, and therefore the FDSNR_LP AC DCT coefficients have a similar frequency of occurrence for escape sequences in comparison to the ordinary MPEG-2 coded video. In contrast, the FDSNR_LM procedure selects AC DCT coefficients resulting in (run, level) combinations that are less likely than the combinations for ordinary MPEG-2 coded video. This is due to two reasons. First, the FDSNR-LM procedure selects AC DCT coefficients having the highest levels. Second, the FDSNR_LM procedure introduces higher run lengths due to the elimination of coefficients over the entire range of coefficient indices. The result is a significantly increased rate of occurrence for escape sequences. Escape sequences form the most inefficient mode of coefficient information encoding in MPEG-2 incorporated into the standard so as to cover important but very rarely occurring coefficient information.
[0122] In order to improve the rate-distortion performance of the scaled-quality MPEG-2 coded video from the FDSNR_LM procedure, the non-zero AC DCT coefficients selected by the FDSNR_LM procedure should be quantized, scanned, and/or (run, level) coded in such a way that tends to reduce the frequency of the escape sequences. For example, if the original-quality MPEG-2 coded video was (run, level) coded using TABLE 0, then the largest magnitude coefficients should be re-coded using TABLE 1 because TABLE 1 provides shorter length VLCs for some (run, level) combinations having higher run lengths and higher levels. It is also possible that re-coding using the alternate scan method instead of the zig-zag scan method may result in a lower frequency of occurrence for escape sequences. For example, each picture could be (run, level) coded for both zig-zag scanning and alternate scanning, and the scanning method providing the fewest escape sequences, or the least number of bits total, could be selected for the coding of the reduced-quality coded MPEG video.
[0123] There are two methods having general applicability for reducing the frequency of escape sequences resulting from the FDSNR_LM procedure. The first method is to introduce a non-zero, “non-qualifying” AC DCT coefficient of the 8×8 block into the list of non-zero qualifying AC DCT coefficients to be coded for the block. In this context, a “qualifying” coefficient is one of the k largest magnitude coefficients selected by the FDSNR_LM procedure. The non-qualifying coefficient referred to above, must be lying in between two qualifying AC DCT coefficients (in the coefficient scanning order) that generate the (run, level) combination causing the escape sequence. Moreover, this non-qualifying coefficient must cause the escape sequence to be replaced with two shorter length VLCs when the AC DCT coefficients are (run, level) coded. This first method has the effect of not only decreasing the number of bits in the coded reduced-quality MPEG video in most cases, but also increasing the PSNR.
[0124] The qualifying AC DCT coefficient causing the escape sequence that is first in the coefficient scanning order will be simply referred to as the first qualifying coefficient. The qualifying AC DCT coefficient causing the escape sequence that is second in the coefficient scanning order will be simply referred to as the second qualifying coefficient. For example, suppose the qualifying coefficients in zig-zag scan order for an 8×8 block include C51 followed by C15 having a level of 40. If only the qualifying coefficients were (run, level) coded for the microblock, C15 would result in a run length of 3, because there are a total of three non-qualifying coefficients (C42, C33, and C24) between C51 and C15 in the scan order. Therefore, C15 would have to be coded as an escape sequence, because a run of 3 and level of 40 does not have a special symbol. In this example, the escape sequence is in effect caused by a first qualifying coefficient, which is C51, and a second qualifying coefficient, which is C15. This escape sequence can possibly be eliminated say, if C24 is a non-zero, non-qualifying coefficient of the block, C24 has a level of 5 or less, and C24 is (run, level) coded together with the qualifying coefficients. For example, assuming that C24 has a level of 5, and using the MPEG-2 (run, level) coding TABLE 1, then C24 has a run length of two and is coded as the special symbol 0000 0000 1010 0s, where “s” is a sign bit, and C15 now has a run length of 0 and is coded as the special symbol 0000 0000 0010 00s. Such a consideration clearly applies to the rest of the non-zero non-qualifying coefficients lying in between the two qualifying coefficients producing the escape sequence. In the above example, these non-qualifying coefficients are C42 and C33.
[0125] Whether or not an escape sequence can be eliminated from the (run, level) coding of the qualifying coefficients can be determined by testing a sequence of conditions. The first condition is that the second qualifying coefficient must have a level that is not greater than the maximum level of 40 for the special (run, level) symbols. If this condition is satisfied, then there must be a non-zero non-qualifying AC DCT coefficient that is between the first and second qualifying coefficients in the coefficient scanning order. If there is such a non-qualifying coefficient, then the combination of its level and the run length between the first qualifying coefficient and the non-qualifying coefficient in the coefficient scanning order must be one of the special (run, level) symbols. If so, then the combination of the level of the second qualifying coefficient and the run length between the non-qualifying coefficient and the second qualifying coefficient must also be a special (run, level) symbol, and if so, all required conditions have been satisfied. If not, then the conditions with respect to the non-qualifying coefficient are successively applied to any other non-zero non-qualifying AC DCT coefficient of the block lying in between the two qualifying coefficients, until either all conditions are found to be satisfied or all such non-qualifying coefficients are tested and failed. If there are sufficient computational resources, this search procedure should be continued to find all such non-qualifying coefficients that would eliminate the escape sequence, and to select the non-qualifying coefficient that converts the escape sequence to the pair of special symbols having respective code words that in combination have the shortest length.
[0126] A flow chart for a modified FDSNR_LM procedure using the first method is shown in FIGS. 20 and 21 . In a first step 331 of FIG. 20 , the procedure finds up to “k” largest magnitude non-zero AC DCT coefficients (i.e., the “qualifying coefficients”) for the block. (This first step 331 is similar to steps 261 to 265 of FIG. 15 , as described above.) In step 332, (run, level) coding of the qualifying coefficients is begun in the scan order using the second coding table (Table 1). This (run, level) coding continues until an escape sequence is reached in step 333, or the end of the block is reached in step 336. If an escape sequence is reached, execution branches from step 333 to step 334. If the level of the second qualifying coefficient causing the escape sequence is greater than 40, execution continues from step 334 to step 336. Otherwise, execution branches from step 334 to step 335 to invoke a subroutine (as further described below with reference to FIG. 21 ) to possibly include a non-zero non-qualifying AC DCT coefficient in the (run, level) coding to eliminate the escape sequence. The subroutine either returns without success, or returns such a non-qualifying coefficient so that the escape sequence is replaced with the two new (run, level) codings of the first qualifying coefficient and the non-qualifying coefficient and then the non-qualifying coefficient and the second qualifying coefficient. From step 335, execution continues to step 336. Execution returns from step 336 if the end of the block is reached. Otherwise, execution continues from step 336 to step 337, to continue (run, level) coding of the qualifying coefficients in the scan order using the second coding table (TABLE 1). This (run, level) coding continues until an escape sequence results, as tested in step 333, or until the end of the block is reached, as tested in step 336.
[0127] With reference to FIG. 21 , there is shown a flow chart of the subroutine (that was called in step 335 of FIG. 20 ) for attempting to find a non-zero, non-qualifying AC DCT coefficient that can be (run, level) coded to eliminate an escape sequence for a qualifying coefficient. In a first step 341, the procedure identifies the first qualifying coefficient and the second qualifying coefficient causing the escape sequence. For example, the subroutine of FIG. 21 can be programmed as a function having, as parameters, a pointer to a list of the non-zero AC DCT coefficients in the scan order, an index to the first qualifying coefficient in the list, and an index to the second qualifying coefficient in the list. In step 342, the subroutine looks for a non-zero non-qualifying AC DCT coefficient between the first and the second qualifying coefficients in the scan order. For example, the value of the index to the first qualifying coefficient is incremented and compared to the value of the index for the second qualifying coefficient, and if they are the same, there is no such non-qualifying coefficient. Otherwise, if the new coefficient pointed to (by incrementing the index of the first qualifying coefficient) is a non-zero coefficient then it becomes a candidate non-qualifying coefficient deserving further testing. If however the new coefficient pointed to (by incrementing the index of the first qualifying coefficient) has a value zero then it is not a candidate non-qualifying coefficient. If no such (candidate) non-qualifying coefficients are found, as tested in step 343, then execution returns from the subroutine with a return code indicating that the search has been unsuccessful. Otherwise, execution continues to step 344.
[0128] In step 344, the non-qualifying coefficient is (run, level) coded, to determine in step 345 whether it codes to an escape sequence. If it codes to an escape sequence, then execution loops back from step 345 to step 342 to look for another non-zero non-qualifying AC DCT coefficient in the scan order between the first and second qualifying coefficients. If it does not code to an escape sequence, then execution continues from step 345 to step 346. In step 346, the second qualifying coefficient is (run, level) coded, using the new run length, which is the number of coefficients in the scan order between the non-qualifying coefficient and the second qualifying coefficient. If it codes to an escape sequence, as tested in step 347, then execution loops back from step 347 to step 342 to look for another non-zero non-qualifying AC DCT coefficient in the scan order between the first and second qualifying coefficients. If it does not code to an escape sequence, then execution continues from step 347 to step 348.
[0129] In step 348, execution returns with a successful search result unless a continue search option has been selected. If the continue search option has been selected, then execution branches from step 348 to step 349 to search for additional non-zero non-qualifying AC DCT coefficients that would eliminate the escape sequence. In other words, steps 342 to 347 are repeated in an attempt to find additional non-zero non-qualifying AC DCT coefficients that would eliminate the escape sequence. If no more such non-qualifying coefficients are found, as tested in step 350, execution returns with a successful search result. Otherwise, execution branches from step 350 to step 351 to select the non-qualifying coefficient giving the shortest overall code word length and/or the largest magnitude for the best PSNR, and execution returns with a successful search result. For example, for each non-qualifying coefficient that would eliminate the escape sequence, the total bit length is computed for the (run, level) coding of the non-qualifying coefficient and the second qualifying coefficient. Then a search is made for the non-qualifying coefficient producing the shortest total bit length, and if two non-qualifying coefficients which produce the same total bit length are found, then the one having the largest level is selected for the elimination of the escape sequence.
[0130] A second method of reducing the frequency of occurrence of the escape sequences in the (run, level) coding of largest magnitude AC DCT coefficients for an 8×8 block is to change the mapping of coefficient magnitudes to the levels so as to reduce the levels. Reduction of the levels increases the likelihood that the (run, level) combinations will have special symbols and therefore will not generate escape sequences. This second method has the potential of achieving a greater reduction in bit rate than the first method, because each escape sequence can now be replaced by the codeword for one special symbol, rather than by the two codewords as is the case for the first method. The second method, however, may reduce the PSNR due to increased quantization noise resulting from the process producing the lower levels. Therefore, if a desired reduction of escape sequences can be achieved using the first method, then there is no need to perform the second method, which is likely to reduce the PSNR. If the first method is used but not all of the escape sequences have been eliminated, then the second method could be used to possibly eliminate the remaining escape sequences.
[0131] The mapping of coefficient magnitudes to the levels can be changed by decoding the levels to coefficient magnitudes, changing the quantization scale factor (qsi), and then re-coding the levels in accordance with the new quantization scale factor (qsi). The quantization scale factor is initialized in each slice header and can also be updated in the macroblock header on a macroblock basis. Therefore it is a constant for all blocks in the same macroblock. In particular, the quantization scale factor is a function of a q—scale_type parameter and a quantizer_scale_code parameter. If q_scale_type=0, then the quantizer scale factor (qsi) is twice the value of q_scale_code. If q_scale_type=1, then the quantizer scale factor (qsi) is given by the following table, which is the right half of Table 7-6 on page 70 of ISO/IEC 13838-2:1996(E):
[0132]
| quantizer scale code | quantization scale factor (qsi) | |
|---|---|---|
| 1 | 1 | |
| 2 | 2 | |
| 3 | 3 | |
| 4 | 4 | |
| 5 | 5 | |
| 6 | 6 | |
| 7 | 7 | |
| 8 | 8 | |
| 9 | 10 | |
| 10 | 12 | |
| 11 | 14 | |
| 12 | 16 | |
| 13 | 18 | |
| 14 | 20 | |
| 15 | 22 | |
| 16 | 24 | |
| 17 | 28 | |
| 18 | 32 | |
| 19 | 36 | |
| 20 | 40 | |
| 21 | 44 | |
| 22 | 48 | |
| 23 | 52 | |
| 24 | 56 | |
| 25 | 64 | |
| 26 | 72 | |
| 27 | 80 | |
| 28 | 88 | |
| 29 | 96 | |
| 30 | 104 | |
| 31 | 112 | |
[0133] In a preferred implementation, to reduce the coefficient levels, the quantization scale factor is increased by a factor of two, and the levels of the non-zero AC DCT coefficients are reduced by a factor of two, so long as the original value of the quantization scale factor is less than or equal to one-half of the maximum possible quantization scale factor. For q_scale_type=1, a factor of two increase in the quantization scale factor (qsi) is most easily performed by a table lookup of a new quantization_scale_code using the following conversion table:
[0134]
| Original quantization scale code | New quaitization scale code | |
|---|---|---|
| 1 | 2 | |
| 2 | 4 | |
| 3 | 6 | |
| 4 | 8 | |
| 5 | 9 | |
| 6 | 10 | |
| 7 | 11 | |
| 8 | 12 | |
| 9 | 14 | |
| 10 | 16 | |
| 11 | 17 | |
| 12 | 18 | |
| 13 | 19 | |
| 14 | 20 | |
| 15 | 21 | |
| 16 | 22 | |
| 17 | 24 | |
| 18 | 25 | |
| 19 | 26 | |
| 20 | 27 | |
| 21 | 28 | |
| 22 | 29 | |
| 23 | 30 | |
| 24 | 31 | |
[0135] In a preferred method for generation of trick mode files, the quantization scale factor is adjusted in order to achieve a desired reduction in the escape sequence occurrence frequency resulting from the modified FDSNR_LM procedure, and the number (k) of largest magnitude coefficients is adjusted in order to achieve a desired reduction in bit rate. A specific implementation is shown in the flow chart of FIGS. 22-23 . In a first step 361, the number (k) of largest magnitude AC coefficients per 8×8 block is initially set to a value of 9, and the quantization scaling factor (QSF) is initially set to a value of 2. Then conversion of the I frames of an original-quality MPEG-2 coded video clip to a lower quality level begins. When a picture header is encountered in step 362, indicating the beginning of a new I frame, execution continues to step 363. In step 363, execution branches depending on the value of the intra_vlc_format parameter in the picture header of the original-quality MPEG-2 coded video clip. This value is either 0, indicating that the first (run, level) coding table (TABLE 0) was used for coding the picture, or 1, indicating that the second (run, level) coding table (TABLE 1) was used for coding the picture. In either case, the down scaled quality picture will be coded with the second (run, level) coding table. If the intra_vlc_format parameter is equal to 0 execution continues from step 363 to step 364 where TABLE 0 is read in for (run, level) symbol decoding in the original-quality MPEG-2 coded clip. Otherwise, if the intra_vlc_format parameter is equal to 1, then execution continues from step 363 to step 365 where TABLE 1 is read in for (run, level) symbol decoding in the original-quality MPEG-2 coded clip.
[0136] After steps 364 and 365, execution continues to step 366. In step 366, the modified FDSNRS_LM procedure is applied to the 8×8 blocks of the current slice, using the adjusted quantization scale index, if the adjusted quantization scale index is less than the maximum possible quantization scale index. In step 367, execution loops back to step 362 to continue 8×8 block conversion until a new slice header is encountered, indicating the beginning of a new slice. Once a new slice is encountered, execution continues from step 367 to step 368. In step 368, the average escape sequence occurrence frequency per block for the last slice is compared to a threshold TH1. If the escape sequence occurrence frequency is greater than the threshold, then execution branches to step 369. In step 369, if the quantization scaling factor (QSF) is less than or equal to a limit value such as 2, then execution branches to step 370 to increase the quantization scaling factor (QSF) by a factor of two.
[0137] In step 368, if the escape sequence occurrence frequency is not greater than the threshold TH1, then execution continues to step 371 of FIG. 23. In step 371, the average escape sequence occurrence frequency per 8×8 block for the last slice is compared to a threshold TH2. If the escape sequence occurrence frequency is less than the threshold TH2, then execution branches to step 372. In step 372, if the quantization scaling factor (QSF) is greater than or equal to a limit value such as 2, then execution branches to step 373 to decrease the quantization scaling factor (QSF) by a factor of two. After step 373, also after step 370 of FIG. 22 , execution continues to step 374 of FIG. 23. In step 374, execution continues to step 375 if a backtrack option has been selected. In step 375, re-coding for the last slice is attempted using the adjusted quantization scale factor. The new coding, or the coding that gives the best results in terms of the desired reduction of escape sequence occurrence frequency, is selected for use in the scaled quality picture. After step 375, execution continues to step 376. Execution also continues to step 376 from: step 369 in FIG. 22 if the quantization scaling factor (QSF) is not less than or equal to 2; step 371 in FIG. 23 if the escape sequence occurrence frequency is not less than the threshold TH2; step 372 in FIG. 23 if the quantization scaling factor (QSF) is not greater than or equal to 2; and from step 374 in FIG. 23 if the backtrack option has not been selected.
[0138] In step 376, the average bit rate of the (run, level) coding per 8×8 block for at least the last slice is compared to a high threshold TH3. Preferably this average bit rate is a running average over the already processed segment of the current scaled quality I-frame, and the high threshold TH3 is selected to prevent video buffer overflow in accordance with the MPEG-2 Video Buffer Verifier restrictions. If the average bit rate exceeds the high threshold TH3, then execution continues to step 377, where the number (k) of non-zero largest magnitude AC coefficients per 8×8 block is compared to a lower limit value such as 6. If the number (k) is greater than or equal to 6, then execution continues to step 378 to decrement the number (k).
[0139] In step 376, if the average bit rate is not greater than the threshold TH3, then execution continues to step 379. In step 379, the average bit rate is compared to a lower threshold TH4. If the average bit rate is less than the threshold TH4, then execution branches from step 379 to step 380, where the number (k) of non-zero largest magnitude AC DCT coefficients per 8×8 block is compared to a limit value of 13. If the number (k) is less than or equal to 13, then execution continues to step 381 to increment the number (k). After step 378 or 381, execution continues to step 382. In step 382, execution continues to step 383 if a backtrack option is selected. In step 383, an attempt is made to re-code the last slice for the scaled quality picture using the adjusted value of the number (k) of non-zero largest magnitude AC DCT coefficients per block. After step 383, execution loops back to step 362 of FIG. 22 to continue generation of the scaled quality clip. Execution also loops back to step 362 of FIG. 22 after: step 377 if the value of (k) is not greater than or equal to 6; step 379 if the average bit rate is not less than the threshold TH4; step 380 if the value of (k) is not less than or equal to 13; and step 382 if the backtrack option has not been selected. Coding of the scaled quality clip continues until the end of the original quality clip is reached in step 364 of FIG. 22 , in which case execution returns.
[0140] In a preferred implementation, a fast forward trick mode file and a fast reverse trick mode file are produced from an original-quality MPEG-2 coded video main file when the main file is ingested into the video file server. As shown in FIG. 24 , a volume generally designated 390 is allocated to store the main file 391. The volume 390 includes an allocated amount of storage that exceeds the real file size of the main file 391 in order to provide additional storage for meta-data 392, the fast forward trick file 393, and the fast reverse trick file 394. The trick files are not directly accessible to clients as files; instead, the clients may access them thorough trick-mode video service functions. With this strategy, the impact on the asset management is a minimum. No modification is needed for delete or rename functions.
[0141] Because the volume allocation is done once for the main file and its fast forward and fast reverse trick mode files, there is no risk of lack of disk space for production of the trick files. The amount of disk blocks to allocate for these files is computed by the video service using a volume parameter (vsparams) specifying the percentage of size to allocate for trick files. A new encoding type is created in addition to types RAW for direct access and MPEG2 for access to the main file. The new encoding type is called EMPEG2, for extended MPEG2, for reference to the main file plus the trick files. The video service allocates the extra file size only for these files.
[0142] For the transfer of these files to archive or to another video file server, it would be useful to transfer all the data even if it is a non-standard format. For the FTP copy-in, a new option is added to specify if the source is in the EMPEG2 format or if it is a standard MPEG2 file. In the first case, the copy-in should provide the complete file 390. In the second case, the video service allocates the extra size and the processing is the same as for a record. For the copy-out, the same option can be used to export the complete file 390 or only the main part 391. The archiving is always done on the complete file 390.
[0143] The trick mode file production is done by a new video service procedure. This procedure takes as input the speed-up factor (or the target trick mode file size) along with the number of freeze (P or B) frames to insert in between the scaled I frames and then generates both the fast forward file 393 and the fast reverse file 394 for this speed-up factor (or target trick mode file size) and with the specified number of interleaving freeze frames. Since the bandwidth of the original clip (in the main file) and the bandwidths of the two trick mode clips (in the fast forward and fast reverse files) are the same, the speed-up factor and the target trick mode file size are equivalent pieces of information. A default speed-up factor (system parameter) can be used. The main file is read and the trick mode files are produced. If a trick mode file already exists with the same speed-up factor, it is rewritten or nothing is done depending on an option. Multiple trick mode files could be created with different speed-up factors. But it is preferred to permit only one set of fast forward and fast reverse trick mode files to be produced at a time (i.e., no parallel generation with different speed-up factors). The current speed-up factor is a parameter of the volume parameters (vsparams).
[0144] As stated above another parameter to be provided to the video service procedure in charge of trick mode file generation is the number of freeze frames to be inserted in between consequent scaled I frames. The preferred values for this parameter are 0 and 1, although other positive integer values greater than 1 are also possible. The inclusion of freeze frames due to their very small sizes spare some bandwidth which can then be used to improve the quality of scaled I frames. Hence, the freeze frames in this context provide a mechanism to achieve a trade-off between the scaled I frame quality and the temporal (motion) sampling. Depending on the speed-up factor (or the target trick mode file size) and also the number of interleaving freeze frames to be inserted, the video service procedure in charge of trick mode file generation determines a sub-sampling pattern (closest to uniform) to choose the original I frames which will be scaled and included in the trick mode files. For example, the case of an original clip with 10 frames per GOP, a trick mode file size which is 10% of the main file together with 0 freeze frames, implies the use of all original I frames for being scaled and included in the trick mode file. This will typically result in a low quality scaling. As another example, the case of an original clip with 10 frames per GOP, a trick mode file size which is 10% of the main file together with 1 freeze frame, implies the use of a 2 to 1 (2:1) sub-sampling on the original I frames which will choose every other original I frame for being scaled and included in the trick mode file.
[0145] FIG. 25 is a more detailed diagram of the volume 390, showing additional meta-data and related data structures. The Inode 401 includes 4 disk blocks containing a file-system oriented description of the file. The Meta-data (MD) directory 402 includes 4 disk blocks describing each entry of the meta-data area 392. The entries of the meta-data area 392 include a description of the MPEG-2 meta-data 403, a description of the trick files header meta-data 404, and a description of the GOP index meta-data 405. The MPEG-2 meta-data 403 includes 15 disk blocks maximum.
[0146] The trick files header 404 includes I disk block, which specifies the beginning of free area (end of last trick file) in blocks, the number of trick files couple (FF FR), and for each trick file, a speed-up factor, a block address of the GOP index, a block address of the trick file forward, a byte length of the trick file forward, a block address of the trick file reverse, a byte length of the trick file reverse, a frames number of the trick file, and a number of GOP of each trick files.
[0147] The GOP index includes 2024 disk blocks. The GOP index specifies, for each GOP, a frame number, a pointer to the MPEG-2 data for the GOP in the main file, and various flags and other attributes of the GOP. The flags indicate whether the GOP entry is valid and whether the GOP is open or closed. The other attributes of the GOP include the maximum bit rate, the average bit rate, the AAU size in bytes, the APU duration in seconds, the audio PES packet starting locations, the AAU starting locations, the AAU PTS values, and the decode time stamp (DTS) and the value of the program clock reference (PCR) extrapolated to the first frame of the GOP. The size of all the data preceding the main file is, for example, 1 megabyte.
[0148] There is one GOP index 406 for both the fast forward file 393 and the fast reverse file 394. The GOP index 406 of the trick files is different than the GOP index 405 of the main file. The GOP index 406 of the trick files contains, for each GOP, the byte offset in the trick file forward of the TS packet containing the first byte of the SEQ header, the frame number in the fast forward file of the GOP (the same value for the fast reverse file can be computed from this value for the fast forward file), the frame number in the original file of the first frame of the GOP, and the byte offset in the original file of the same frame (to resume after fast forward or reverse without reading the main GOP index).
[0149] The GOP index 405 for the main file and the GOP index 406 for the fast forward and fast reverse trick files provides a means for rapidly switching between the normal video-on-demand play operation during the reading of the main file, and the fast-forward play during the reading of the fast-forward file, and the fast-reverse play during the reading of the fast reverse file. For example, FIG. 26A illustrates the read access to various GOPs in the main file, fast forward file, and fast reverse file, during a play sequence listed in FIG. 26B. Due to the presence of down-scaled I frames and possibly consequent freeze frames in the trick mode files, the video buffer verifier (VBV) model for a trick mode file is different than the VBV model of the main file. Consequently, the mean video decoder main buffer fullness levels can be significantly different for these files. For example, a transition from the main file to one of the trick files will usually involve a discontinuity in the mean video decoder main buffer fullness level, because only the I frames of the main file correspond to frames in the trick files, and the corresponding I frames have different bit rates when the trick mode I frames are scaled down for a reduced bit rate. An instantaneous transition from a trick file back to the main file may also involve a discontinuity especially when freeze frames are inserted between the I frames for trick mode operation. To avoid these discontinuities, the seamless splicing procedure of FIGS. 3 to 6 as described above is used during the transitions from regular play mode into trick mode and similarly from trick mode back into the regular play mode. Through the use of the seamless splicing procedure to modify the video stream content, for example for the “Seamless Splice” locations identified in FIG. 26A , the video decoder main buffer level will be managed so as to avoid both overflows and underflows leading to visual artifacts.
[0150] It is desired to copy in and out of the volume 390 with or without the meta-data 392 and the trick files 393, 394. This is useful to export and/or import complete files without regenerating the trick files. The file encoding type is now recognized as a part of the volume name. Therefore there can be multiple kinds of access to these files. The read and write operations are done by derivations of the class file system input/output (FSIO) which takes into account the proper block offset of the data to read or write. There is one derivation of FSIO per encoding type, providing three different access modes. EMGP3, MPEG2, and RAW. EMPEG2 accesses the whole volume from the beginning of the meta-data array, and in fact provides access to the entire volume except the inode 401, but no processing is done. MPEG2 access only the main part of the asset with MPEG processing, including file analyze and meta-data generation in a write access. RAW access only the main part of the asset without processing. These access modes are operative for read and write operations for various access functions as further shown in FIG. 27.
[0151] During a record operation, the video service allocates a volume and computes the number of block to allocate using the volume parameter giving the percentage to add for the trick files. Then, the size in blocks given to the stream server is the main part size only without the extension for the trick files. This avoids using the reserved part of the volume when the effective bit rate is higher than the requested bit rate. At the end of a record operation or a FTP copyin operation, the video service calls a procedure CMSPROC_GETATTR, and the stream server returns the actual number of bytes received and the actual number of blocks used by the main file plus the meta-data. The same values are returned for both MPEG2 and EMPEG2 files. The video service computes again the file extension to manage the trick files and adjust the number of allocated blocks.
[0152] Both trick files forward and reverse are generated by the same command. First, the trick file forward is generated by reading the main file. The trick file GOP index is built and kept in memory. During this generation, only the video packets are kept. PCR, PAT and PMT will be regenerated by the MUX in play as for any other streams. The audio packets are discarded. This ensures that there is enough stuffing packets for the PCR reinsertion. For this, a stuffing packet is inserted every 30 milliseconds.
[0153] Then using the GOP index, the trick file forward is read GOP by GOP in reverse order to generate the trick file reverse. The same GOPs are present in both files. The only modification done is an update of the video PTS, which must be continuous. Then, the GOP index is written on disk. This avoids reading again the file while generating the second trick file. The GOP index size is: 24 times the GOP number. In the worst case (the file is assumed not to be I frame only), there are 2 frames per GOP and 30 frames per second. So for 1 hour in fast forward, the GOP index size is: (24×3600×30) 2=1296000 bytes. This will be the case for a 4 hour film played at 4 times the normal speed. Therefore, this GOP index can be kept in memory during the trick file generations without risk of memory overflow.
[0154] The read and write rate are controlled to conserve bandwidth on the cached disk array. The bandwidth reserved for these generations is a parameter given by the video service. It is a global bandwidth for both read and writes. The number of disk 1/0 per seconds is counted so as not to exceed this bandwidth.
[0155] The trick files header update is done once when both the fast forward and fast reverse trick files and the GOP index have been successfully written.
[0156] Playing a file is done with the CM_MpegPlayStream class. Fast forward (reverse) can only be requested when we are in the paused state. The current frame on which we are paused is known from the MpegPause class. This frame is located in the GOP index of the trick file. Then the clip start point and length are modified in the Clip instance with the trick file position computed from the beginning of the clip. So, the Clip class handle these trick files in a manner similar to the main file. The current logical block number is updated with the block address in the trick file recomputed from the beginning of the main clip. In fact, a seek is performed in the trick file as it was part of the main file, which is totally transparent for the ClipList and Clip classes. The transition from fast forward to pause is handled in a similar fashion. The clip start and length and the logical block number are again updated. The smooth transitions from pause to fast forward and from fast forward to pause are done in the same way as for regular play. There is a splicing from the pause stream to the play stream.
[0157] The class hierarchy for trick file handling is shown in FIG. 28. The MpegFast, MpegFastForward and MpegFastReverse class handles the GOP generation from the initial file. This is the common procedure for building the GOP whatever the source and the destination. RealTimcFastFwd and RealTimeFastRev are the class instantiated when a real time fast forward (reverse) has to be done. They manage the real-time buffer flow to the player. There is a derivation of the methods takeBuffer and returnBuffer which uses the base class to build the GOP in the buffer to be played. The main file access is done using a buffer pool.
[0158] TrickFilesGenerate is the class instantiated to generate trick files forward and reverse. It inherits from TrickFileAccess the methods for reading the original file some buffers and for writing the trick file and its meta-data. It inherits from MpegFastForward the methods for building the GOP and for managing the advance in the file.
[0159] The computation of the next 1 frame to play is done by MpegFast, MpegFastForward and RealTimeFastFwd. When a trick file generation command is invoked, a thread is created and started and the generation itself is done off-line. A call-back is sent to the video service when the generation is completed. The class TrickFilesGenerate generates the trick file forward, and then, using the GOP index built in memory, the class TrickFiles Generate generates the trick file reverse.
[0160] When there is a transition from play to pause, the only latency issue is related to the buffer queue handled by the player and to the GOP size. The stream can build immediately the active pause GOP, and then this GOP will be sent at the end of the current GOP with a splicing between these two streams.
[0161] When there are transitions from pause to regular play or fast forward and fast reverse, a seek in the file is done. This means that the current buffer pool content is invalidated and the buffer pool is filled again. Play can start again while the buffer pool is not completely full, as soon as the first buffer is read. The buffer pool prefilling can continue as a background process. The issue here is that there is a risk to generate an extra load on the cached disk array as well as on the stream server side when the buffer pool is being prefilled.
[0162] To avoid too frequent transitions from play to fast forward and fast reverse, there is a limitation of the number of requests per second for each stream. This limitation is part of the management of the video access commands. A minimum delay between two commands is defined as a parameter. If the delay between a request and the previous one is too small, the request is delayed. If a new request is received during this delay, the new request replaces the waiting one. So the last received request is always executed. The volume parameter (vsparams) file contains these new parameters for the trick mode files:
[0163] TrickFileExtensionSize:<percent>:
[0164] DefaultFastAcceleration:<acceleration>:
[0165] DMtrickFileGen:<mask of reserved DM> (This parameter is a mask of the stream servers that can be chosen to perform the trick file generation. The default value is 0xfffc: all of the stream servers.)
[0166] DMtrickFileGenBW:<bandwidth used for trick file generation> (This parameter is the value of the bandwidth effectively used by the stream server for the trick files generation.)
[0167] The video service routines are modified to operate upon the EMPG2 files, and in particular to compute the size of the EMPG2 files, to allocate the volume for the main file and the trick files, and to generate the trick files. The volume creation functions (VAPP) and volume access functions (RRP) use the EMPEG2 files in the same way as MPEG2 files. This means that a MPEG2 volume is created on the stream server. Both MPEG2 and EMPEG2 files can be used in the same session or play-list. The session encoding type is MPEG2. In record (or copy-in), the number of blocks allocated for an EMPEG2 file is computed using the percentage of size to add. At the end of record (or copy-in), the number of blocks is adjusted using the number of blocks returned by the stream server (by CMSPROC_GETATTR) and adding the percentage for trick files. The trick files validity and generation date are stored by the video service in the asset structure. The bandwidth allocated to the TrickFilesGenerate command is defined in the volume parameters (vsparams or vssiteparams). The selection of a stream server to generate the trick files takes into account this bandwidth only. If preferred stream servers are specified in vsparams (or vssiteparams), then the selected stream server will be one of these specified stream servers.
[0168] In a preferred implementation of the video service software, a new encoding type is created. The encoding type enum becomes:
[0169]
| enum encoding-t{ | |||
| ENC_UNKNOWN | = 0, | /* unknown format */ | |
| ENC_RAW | = 1, | /* uninterpreted data */ | |
| ENC_MPEG1 | = 2, | /* constrained MPEG1 */ | |
| EMC_MPEG | = 3, | /* generic MPEG */ | |
| ENC_EMPEG2 | = 4, | /* MPBG2 with trick files exten- | |
| sion */ | |||
| }; | |||
[0170] The encoding information accessible by VCMP_EXTENDEDINFO includes information about trick files:
[0171]
The video service software includes a new procedure (VCMP_TRICKFILESGEN) for trick file generation, which uses the following structures:
| struct trickFilesInfo_t{ | |||
| ulong_t | generationDate; | /* date/time of the generation of | |
| the trick files */ | |||
| rate_factor_t | acceleration; | /* acceleration factor */ | |
| ulong_t | framesNumber; | /* frames number in each trick | |
| file (FWD and REV) */ | |||
| ulong_t | gopNumber; | /* GOP number of each file */ | |
| }; | |||
| struct EMPEG2info_t{ | |||
| MPEG2info_t | MPEG2info; | ||
| trickFilesInfo_t | trickFiles<>; | ||
| }; | |||
| union encodingInfo_t switch (encoding-t enc) { | |||
| caseENC_MPEG: | |||
| MPEG2info_t | MPEG2info; | ||
| case ENC_EMPEG2: | |||
| EMPEG2info_t | EMPEG2info; | ||
| default: | |||
| void; | |||
| }; | |||
The video service software includes a new procedure (VCMP_TRICKFILESGEN) for trick file generation, which uses the following structures:
[0172]
| struct VCMPtrickgenres_t{ | ||
| VCMPstatus_t | status; | |
| tHandle_t | handle; | |
| }; | ||
| struct VCMPtrickfilesargs_t{ | ||
| name_t | clipname; | |
| bool_t | overwriteIfExists; | |
| rate_factor_t | acceleration; | |
| }; | ||
| VCMPtrickgenres_t | VCMP_TRICKFILESGEN | |
| (VCMPtrickfilesargs_t) = 36, | ||
[0173] If the trick files already exist and if the boolean overwriteIfExists is true, then the trick files are generated again, in the other case nothing is done. Acceleration is the acceleration as defined and used for the controlled speed play function. It is a percentage of the normal speed, it must be greater than 200 and smaller than 2000. The special value of 0 can be used to generate files with the default acceleration defined in vssiteparams. The procedure starts the generation process. The completion is notified by a callback.
[0174] The video service includes a new option to copyin and copyout. The option is added to allow a user to copy all the file or the main asset only. For compatibility with old client programs, the following new procedures are added:
[0175]
These new procedures take the same interface as the existing one, but are used to copy-in the complete file: meta−data+Asset+trick files.
| VCMPcopyres_t | VCMP_FULL_COPYIN (copyinargs2_t) = 37, |
| VCMPcopyres_t | VCMP_FULL_COPYOUT (copyoutargs2_t) = 38, |
These new procedures take the same interface as the existing one, but are used to copy-in the complete file: meta−data+Asset+trick files.
[0176] The video service includes a new procedure VCMP_TRICKFILESGENCOMPLETED, which uses the following structures:
[0177]
| struct VCMPtrickfilescomplete_t | ||
| tHandle_t | handle; | |
| VCMPstatus_t | status; | |
| }; | ||
| VCMPstatus_t TRICKFILESGENCOMPLETED | ||
| (VCMPtrickfilescomplete_t) = 10, | ||
[0178] The video service includes new procedures are added for handling trick mode generation arguments, which uses the following structures:
[0179]
| struct cms_trick_gen_args { | ||
| Handle_t | Vshandle; | |
| name_t | name; | |
| bool_t | overwriteIfExists; | |
| rate_factor_t | acceleration; | |
| bandwidth_t | reservedBw; | |
| }; | ||
| cms_status | CMSPROC_GEN_TRICK_FILES | |
| (cms_trick_gen_args) = 34, | ||
| struct trick_gen_completed_args { | ||
| Handle_t | Vshandle; | |
| cms_status | status; | |
| }; | ||
| void CTLPROC_TRICKGENCOMPLETED | ||
| (trick_gen_completed_args) = 8, | ||
[0180] The video service includes the following option to force the regeneration of trick files even if they exist:
-
- [0181] nms_content−gentrick<name> [<−f>] [acceleration]
Without this option, an error code is returned if the trick files exist. “Acceleration” is an acceleration factor. If it is not present, the default value is taken in vsparams.
- [0181] nms_content−gentrick<name> [<−f>] [acceleration]
[0182] The video services include a encoding information access function (nms_content−m). This function produces a displayed output containing, for each trick file generated, the acceleration, the generation date and time, the frames number, and the GOP number.
[0183] For the use of an FTP copy function with the trick files, the following new commands are added:
- [0184] nms_content−copyinfull<same arguments as−copyin>
- [0185] nms_content−copyoutfull<same arguments as−copyout>
[0186] Another application of the SNR scaling of the invention is to reduce the bit rate of an MPEG-2 transport stream in order to allow combining multiple MPEG-2 transport streams to match a target bit rate for a multiple program transport stream. For example, FIG. 29 shows a system for combining an MPEG-2 audio-visual transport stream 411 with an MPEG-2 closed-captioning transport stream 412 to produce a multiplexed MPEG-2 transport stream 413. In this case, the closed captioning transport stream 412, containing alphanumeric characters and some control data instead of audio-visual information, has a very low bit rate compared to the audio-visual transport stream 411. Assuming that the target bit rate for the multiplexed transport stream 413 is the same as the bit rate of the audio-visual transport stream 411, there need be only a slight decrease in the bit rate of the audio-visual transport stream, and this slight decrease can be obtained by occasionally removing one non-zero AC DCT coefficient per 8×8 block. Therefore, in the system of FIG. 29 , the audio-visual transport stream 411 is processed by a program module 414 for selective elimination of non-zero AC DCT coefficients to slightly reduce the average bit rate of this transport stream. A transport stream multiplexer 415 then combines the modified audio-visual transport stream with the closed captioning transport stream 412 to produce the multiplexed MPEG-2 transport stream 413.
[0187] In order to determine whether or not any non-zero AC DCT coefficient should be eliminated from a next 8×8 block in the audio-visual transport stream 411, a module 421 is executed periodically to compute a desired bit rate change in the audio-visual transport stream 411. For example, respective bit rate monitors 416, 417 may measure the actual bit rate of the audio-visual transport stream 411 and the closed captioning transport stream 412. Alternatively, if it is known precisely how these transport streams are generated, presumed values for the bit rates of these transport streams may be used in lieu of measured bit rates. The computation of the desired bit rate change also includes the desired bit rate 418 for the multiplexed MPEG-2 transport stream, and a bit rate 419 of multiplexer overhead, representing any net increase in bit rate related to the multiplexing of the audio-visual transport stream 411 with the closed captioning transport stream 412. An adder/subtractor 420 combines the various bit rate values from the inputs 416, 417, 418, and 419 to compute the desired bit rate change in the audio-visual transport stream 411. From the adder/subtractor 420, the module 421 converts the desired change in bit rate to a desired number of bits to be removed per computational cycle (e.g., per millisecond). This number of bits to be removed per computational cycle is received in an adder/subtractor 422, and the output of the adder/subtractor is received in an integrator 423. A limiter 424 takes the sign (positive or negative) of the integrated value to produce a flag indicating whether or not one non-zero AC DCT coefficient should be removed from the coefficients for the next 8×8 block, assuming that the next block has at least one non-zero AC DCT coefficient. (Alternatively, a non-zero AC DCT coefficient could be removed only if the 8×8 block has more than a predetermined fraction of the average number of AC DCT coefficients per 8×8 block.) The particular non-zero AC DCT coefficient to remove in each case can be selected using any of the methods discussed above with reference to FIGS. 14 , 15, or FIG. 20. For example, the coefficient to remove could be the last non-zero AC DCT coefficient in the scan order. Alternatively, the non-zero AC DCT coefficient having the smallest magnitude could be removed so long as its removal does not cause an escape sequence.
[0188] When the module 414 removes a non-zero AC DCT coefficient from a 8×8 block, it sends the number of bits removed to the adder/subtractor 422. In a preferred implementation, the operations of the adder/subtractor 422, integrator 423, and limiter 424 are performed by a subroutine having a variable representing the integrated value. During each computational cycle, the variable is incremented by the number of bits to be removed per computational interval, and whenever the module 414 removes a non-zero AC DCT coefficient from a 8×8 block of the audio-visual transport stream, the variable is decremented by the number of bits removed.
[0189] Although the system in FIG. 29 has been described for achieving a slight reduction in bit rate of the MPEG-2 audio-visual transport stream 411 for combining multiple transport stream to produce a multiplexed MPEG-2 transport stream, it should be apparent that it could be used for obtaining relatively large reductions in bit rate. In this case, the module 414 would use the procedure of FIGS. 14 , 15 or preferably FIG. 20 , and a multi-level comparator 424 would be used instead of a single-level comparator 424. The multi-level comparator would determine a desired number of non-zero coefficients to discard per 8×8 block based on the value of the output of the integrator 423. The maximum number of non-zero AC coefficients to keep for each 8×8 block (i.e., the value of the parameter “k”), for example, would be determined by subtracting the number of non-zero AC DCT coefficients in the 8×8 block from the desired number to discard, and limiting this difference to no less than a predetermined fraction of the average number of non-zero AC coefficients per 8×8 block.
[0190] In order to produce reduced-quality MPEG-2 data having a bit rate that is a small fraction of the bit rate of the original-quality MPEG-2 data, the bits for each frame should be allocated to each DCT coded block (i.e., an 8×8 pixel block) in the frame in such a way that more complex blocks are allocated more bits than simple blocks. With a relatively large reduction in bit rate, there will be degradation in picture quality, and the degradation will be less noticeable if each 8×8 block in the frame suffers approximately the same degree of degradation. If the same number of bits is allocated to each 8×8 block regardless of complexity, then the complex blocks will be allocated fewer bits than needed for a uniform degradation, and the simple blocks will be allocated more bits than needed for a uniform degradation. Consequently, the picture degradation will become more noticeable in the more complex blocks.
[0191] In the original-quality MPEG-2 data, the number of bits allocated to the AC DCT coefficients of each 8×8 block is a good estimate of the complexity of the block, and therefore one can allocate coefficient bits to each 8×8 block in the reduced quality MPEG-2 data in proportion to block complexity by proportionally scaling the number of bits allocated to coefficient bits in each 8×8 block of the original quality MPEG-2 block. For example, a coefficient bit rate reduction factor of RF is computed, where RF is the ratio of the desired AC DCT coefficient bit rate of the reduced-quality MPEG-2 data divided by the AC DCT coefficient bit rate of the original quality MPEG-2 data. The number of coefficient bits available for encoding the non-zero AC DCT coefficients in each 8×8 block in the reduced-quality MPEG-2 data is then calculated by multiplying the rate reduction factor RF by the number of coefficient bits in the corresponding 8×8 block in the original-quality MPEG-2 data.
[0192] The final reduced video bit rate is smaller than the total bit rate of the original-quality MPEG-2 data, divided by the reduction factor, by an offset S representing the rate of bits in the MPEG-2 data that are not video bits or padding bits. The padding bits may be removed in the reduced rate file. The preferred rate reduction technique of dropping DCT coefficients has an advantage that the reduced-quality MPEG-2 data will have the same offset S of bits in the reduced rate MPEG-2 data that are not coefficient bits. Therefore, if the original-quality MPEG-2 data has a total bit rate of BO, a padding PD and offset S, and it is desired for the reduced-quality MPEG-2 data to have a total bit rate of BR, then the coefficient bit reduction factor RF is RF=(BR−S)/(BO−PD−S). BR−S represents the video bit rate of the reduced bit rate file and BO−PD−S represents the video bit rate of the original quality MPEG-2 data. In many cases the offset S will be relatively small compared to BR and BO so that any change or uncertainty in the value of the offset has a small effect on the value of the reduction factor. Therefore, the offset S could be estimated based on frame size and frame type (e.g., I, P or B) depending on the desired uncertainty in the value of the reduction factor RF. S is always greater than zero.
[0193] In general, the number of bits allocated to the non-zero AC DCT coefficients of an 8×8 block in a frame in the reduced-quality MPEG-2 data will be less than the number of bits allocated to the non-zero AC DCT coefficients of the corresponding 8×8 block of the same frame in the original-quality MPEG-2 data. This is due to the fact that the preferred process of bit rate reduction is eliminating non-zero AC DCT coefficients. Therefore, the number of bits allocated for the 8×8 block is reduced by a variable number each time a coefficient is eliminated. It is possible although not likely that the dropping of one or more coefficients will cause the number of bits allocated to the 8×8 block to become equal to the number of bits that are available for encoding the non-zero AC DCT coefficients for the reduced 8×8 block. It is most likely that some of the bits available for encoding the non-zero AC DCT coefficients for the reduced 8×8 block will not be used for encoding the non-zero AC DCT coefficients for the reduced 8×8 block. So that the actual bit rate for the reduced-quality MPEG-2 data will not become significantly less than the desired rate due to available bits that would not be used for encoding the non-zero AC DCT coefficients, the available bits that are not used for encoding non-zero AC DCT coefficients for a reduced 8×8 block are made available for encoding AC DCT coefficients in following 8×8 blocks. In particular, a count of the bits that were available but not yet used for encoding AC DCT coefficients for the reduced blocks is maintained in a memory location or program variable that will be referred to as a “bucket.” Initially, this bucket is empty. At the start of selecting a reduced number of coefficients for the next reduced-quality 8×8 block, the coefficient bit rate reduction factor RF is multiplied by the number of coefficient bits in the original-quality MPEG-2 block. This product represents a minimum number of bits that are available for encoding non-zero AC DCT coefficients selected for the reduced-quality block. In addition, a certain fraction of any bits in the bucket are also available for encoding non-zero AC DCT coefficients selected for the reduce-quality block. The fraction represents the extent to which the bits in the bucket can be shared among following blocks. For example, if the bucket is not cleared and the fraction is the reciprocal of the number of blocks in a frame, then bits in the bucket will be shared over at least a frame of following coefficients. Moreover, at the end of a frame, the bucket need not be cleared, and excess bits from coefficients in one frame will be available for encoding AC DCT coefficients in a following frame.
[0194] FIG. 30 illustrates graphically how the adaptive bit rate method achieves a generally proportional reduction in the number of non-zero AC coefficient bits in each 8×8 block of a frame, and a smoothing of deviations from a proportional reduction among the 8×8 blocks. A frame 501 in the original-quality MPEG coded video is represented as a matrix of 8×8 blocks. In this example, for the sake of illustration, the frame has 8 rows and 8 columns of 8×8 blocks, although in practice, the frame would usually have a larger number of rows and columns of 8×8 blocks. Inside the cells of the matrix of the frame 501 are compressed 8×8 blocks, such as the compressed block 502. The compressed blocks are shown having a varying size. The size of each compressed block represents the total number of bits used in the variable-length encoding of the non-zero AC DCT coefficients in the block.
[0195] Shown below the frame 501 is a corresponding reduced size frame 503 representing a frame in the reduced-quality MPEG coded video produced from the original-quality MPEG coded video. In particular, the reduced size frame 503 is a matrix having the same number of rows and columns of 8×8 blocks. Each 8×8 block in the reduced frame 503 is produced from a corresponding 8×8 block in the original frame 501 by a process of selective removal of non-zero AC DCT coefficients so that the 8×8 block in the reduced frame 504 has a proportionately reduced number of bits encoding non-zero AC DCT coefficients, subject to smoothing of deviations among the reduced 8×8 blocks. For example, the reduced 8×8 block 504 is produced by selective removal of non-zero AC DCT coefficients from the compressed block 502.
[0196] It is desired to keep the ratio between each original block size and the reduced rate block size constant and equal to the ratio of the original frame size and the reduced bit rate frame size. This is not always possible because the bit allocation per block differ from frame to frame due to local simple blocks and a variable proportion of complex sub frames in a frame. For example, there are blocks for which the maximum number if bits is smaller than the average block size computed as:
Breduced(j)=Boriginal(j)/MAoriginal(n)*BRreduced
Where (j) is the index of the block in the frame, (n) is the present frame number, Breduced(j) is the number of bits allocated for the reduced bit rate block (j) and it is proportional to the original block size Boriginal(j), MAreduced(n) is the moving average of original frame size after (n) frames and BRreduced is the target reduced video bit rate. The average block size of the reduced bit rate frame is calculated as:
AVBreduced=Freduced/NB
Where Freduced is the frame size of the reduced-quality file, and NB is the number of blocks per frame and it is calculated to conform to the MPEG-2 standard from the frame resolution as follows:
NB=ceil(FR(horizontal)/8)*ceil(FR(vertical)/8)
where “ceil” is the conversion to a rounded-up integer, FR(horizontal) is the horizontal frame resolution in pixels (720 for NTSC and 528 for PAL), and FR(vertical) is the vertical frame resolution in pixels (480 for NTSC and PAL).
Breduced(j)=Boriginal(j)/MAoriginal(n)*BRreduced
Where (j) is the index of the block in the frame, (n) is the present frame number, Breduced(j) is the number of bits allocated for the reduced bit rate block (j) and it is proportional to the original block size Boriginal(j), MAreduced(n) is the moving average of original frame size after (n) frames and BRreduced is the target reduced video bit rate. The average block size of the reduced bit rate frame is calculated as:
AVBreduced=Freduced/NB
Where Freduced is the frame size of the reduced-quality file, and NB is the number of blocks per frame and it is calculated to conform to the MPEG-2 standard from the frame resolution as follows:
NB=ceil(FR(horizontal)/8)*ceil(FR(vertical)/8)
where “ceil” is the conversion to a rounded-up integer, FR(horizontal) is the horizontal frame resolution in pixels (720 for NTSC and 528 for PAL), and FR(vertical) is the vertical frame resolution in pixels (480 for NTSC and PAL).
[0197] In other cases the number of bits is greater than the average size allocated as shown in the reduced frame 503. It should also be apparent from the above discussion that sometimes all of the bits allocated to the reduced frame cannot be used. Bits that are allocated but not used in the reduced frame should be distributed by a smoothing mechanism to another reduced frame that may be more complex.
[0198] FIG. 30 shows how some reduced blocks are smaller than a proportionally-reduced size and the bucket is filled, and some reduced blocks are larger than the proportionally-reduced size and the bucket is emptied. This methodology is especially efficient in cases when there are very large variations between adjoining blocks based on the details of the block, for example hair strings near a talking head. For example, the reduced frame 503 includes a “bucket filling” block 505 and a “bucket emptying” block 507. A rectangle 506 around the bucket filling block 505 indicates a proportional reduction in the number of bits of the non-zero AC DCT coefficients in the corresponding compressed block 509 in the original frame 501. The difference in area between the outer rectangle 506 and the bucket filling block 505 indicate bits that are available for encoding non-zero AC DCT coefficients of the bucket filling block 505 but were not used and instead were placed in the bucket. A rectangle 508 inside the bucket emptying block 507 indicates a proportional reduction in the number of bits of the non-zero AC DCT coefficients in the corresponding compressed block 510 in the original frame 501. The difference in area between the inner rectangle 508 and the bucket emptying block 507 represents bits that were taken from the bucket and allocated to the bucket emptying block 507.
[0199] When a block of a frame is simple, the “bucket” will fill, and when a block of the frame is very complex the “bucket” will empty. All this is done by including as many coefficients for a block as possible given a target size of bits in proportion to the non-zero AC DCT coefficient bits in the corresponding block of the original encoded frame 501, plus a share of any bits in the bucket. In general, the number (m) of non-zero AC DCT coefficients to keep for the block is found such that:
ABS(Sum(Bits(DCT(1))+ . . . +Bits(DCT(m))−Breduced(j))<Bucket/NBR(j)
Where Bucket is the number of bits available in the “bucket” (size of the “bucket”), and NBR(j) is the remaining number of blocks in the frame including block (O), calculated as NB−j, so that the share of any bits in the bucket is Bucket/NBR(j). The DCT coefficients are obtained by parsing the original encoded block. The non-zero AC DCT coefficients that are retained in the reduced frame could be selected in the order of parsing, or in the order of the largest magnitude coefficients. In any case, the “bucket” remains as empty as possible considering that excess bits from one block will be shared with at least a certain number (NB) of following blocks.
ABS(Sum(Bits(DCT(1))+ . . . +Bits(DCT(m))−Breduced(j))<Bucket/NBR(j)
Where Bucket is the number of bits available in the “bucket” (size of the “bucket”), and NBR(j) is the remaining number of blocks in the frame including block (O), calculated as NB−j, so that the share of any bits in the bucket is Bucket/NBR(j). The DCT coefficients are obtained by parsing the original encoded block. The non-zero AC DCT coefficients that are retained in the reduced frame could be selected in the order of parsing, or in the order of the largest magnitude coefficients. In any case, the “bucket” remains as empty as possible considering that excess bits from one block will be shared with at least a certain number (NB) of following blocks.
[0200] The use of the bucket as a smoothing mechanism with a frame and between frames provides a significant improvement in picture quality because there is a large variability between the number of bits per DCT coefficient due to the MPEG-2 encoding code books and the number of escape sequences. For example, suppose Bucket=36000 bits, NB=48*30=1440, SUM(DCT(1)+ . . . +DCT(11))=200, SUM(DCT(1)+ . . . +DCT(12))=300, and Breduced(j)=280. The formula above will result in:
ABS(200−280)=80>36000/1440=25
ABS(300−280)=20<36000/1440=25
So we keep 12 DCT coefficients. In some cases there are only 11 DCT coefficients in the example and there is no choice but to fill the “bucket” with 80 more bits.
ABS(200−280)=80>36000/1440=25
ABS(300−280)=20<36000/1440=25
So we keep 12 DCT coefficients. In some cases there are only 11 DCT coefficients in the example and there is no choice but to fill the “bucket” with 80 more bits.
[0201] FIG. 31 shows a graph of frame size (i.e., number of bits per frame) as a function of frame number for various traces. A first trace 521 is for the original-quality MPEG coded video, and a second trace 522 is for the reduced-quality MPEG coded video produced from the original-quality MPEG coded video using the adaptive bit rate reduction method as introduced above and as further described below. FIG. 31 also shows a moving average 523 of the first trace 521, the average 524 of the first trace, and the target frame size 524 which is the average of the second trace 522. It can be noticed from FIG. 31 that the size of the reduced frames are proportional to the size in bits of the original encoded file. In the case presented in FIG. 31 the rate reduction is done on all the encoded frames including I, B and P type frames. It should be seen that the moving average frame size over about 15 frames for the original-quality MPEG coded video is very smooth and almost constant in time. This property is not required or used in any way in the adaptive bit rate reduction method. For other applications, this property could be exploited for example to ease calculations because the size of the GOP is kept constant by the encoder and combines with the smoothness of the moving average to ensure that the transport stream (TS) bit rate is constant in time (i.e., CBR). Because this property is not required for the adaptive rate reduction method, the method may be performed with the same accuracy for frame sequences that are not encoded for CBR. For example, the method may be used upon I frames removed from original-quality MPEG coded video to generate trick files for which there is no correlation between I frames separated by large time intervals and making prediction of frame size impossible.
[0202] FIG. 32 illustrates such a case of independent I frames. FIG. 32 includes a first trace 531 for the I frames from original-quality MPEG coded video, a second trace 532 which is a moving average of the first trace 531, a third trace 533 for the reduced-quality I frames, and a fourth trace 534 which is the target frame size for the reduced-quality I frames. It should be apparent that the moving average frame size in this case is far from being constant and still the reduced rate frame sizes are proportional to the sizes of the original encoded I frames.
[0203] FIGS. 33 to 35 show a flowchart of programming of a digital computer for adaptive bit rate reduction of MPEG coded video as introduced above. For example, the programming could be used in the MPEG scaling program (38 in FIG. 1 ) in the stream server computer (25 in FIG. 1 ) in the video file server 24 in FIG. 1. In a first step 541 of FIG. 33 , the bucket is cleared. In step 542, the first video frame in the original-quality MPEG clip is parsed down to the level of 8×8 DCT blocks. Then in step 543, the DCT coefficient bit rate reduction factor is determined, as will be further described below with reference to FIGS. 36 and 37 . In step 544, the first 8×8 block is identified, and a block index (J) is set to zero. In step 545, the first 8×8 block is parsed. If the block has no non-zero AC DCT coefficients, as tested in step 546, then the block cannot be reduced, and execution branches to step 547. If the end of the frame has not been reached, then execution branches from step 547 to step 548, to get the next block and increment the block counter (J), and to parse the next block in step 545. If the end of the frame is reached, then execution continues to step 549. In step 549, if the end of the clip is reached, then the adaptive bit rate reduction for the clip is finished. Otherwise, execution continues from step 549 to step 550, to get the next frame, and parse the next frame in step 542.
[0204] In step 546, if there are non-zero AC DCT coefficients in the current block, then execution continues to step 561 in FIG. 34. In step 561, the original block size (BS) is scaled by the reduction factor (RF) to compute a desired number of bits (DNB) for the reduced block size. In other words, the DNB is computed as the product of RF and BS. In step 562, the number of bits (NBB) in the bucket that are available for encoding non-zero AC DCT coefficients for the reduced block is calculated by dividing the number of bits in the bucket (BUK) by the number of remaining blocks in the frame, including the current block, calculated as (NB−J), and they are “borrowed” from the bucket by decrementing BUK by NBB. NB is the number of blocks in the frame, and J is the index of the current block. (Alternatively, NBB could be calculated as BUK/NBK, which would more freely spread the bits in the bucket from one frame to the next, rather than tending to deplete the bucket by the end of the frame.) Then in step 563, the number of bits available (NBA) for encoding non-zero AC DCT coefficients for the reduced block are calculated by adding the desired number of bits (DN) to the number of bits (NBB) borrowed from the bucket.
[0205] In step 564, the first non-zero AC DCT coefficient is obtained in the parsing order. This is for an implementation that selectively retains the low-order coefficients in the parsing order, such as the low-pass scaling method in step 225 of FIG. 13 and in FIG. 14 as described above. Other scaling methods could be used instead. For example, the preferred methods selectively retain or eliminate the non-zero coefficients. If there are sufficient computational resources, for example, then largest magnitude scaling as in step 227 of FIG. 13 and in FIG. 15 could be used, or the approximate largest magnitude scaling as in step 229 of FIG. 13 and in FIGS. 18-19 could be used. If largest magnitude scaling were used, then the first coefficient obtained in step 564 would be the AC coefficient having the largest magnitude. If approximate largest magnitude scaling were used, then the first coefficient obtained in step 564 would be the AC coefficient having the approximate largest magnitude; in other words, it would be the first AC coefficient produced by the hash table sorting procedure of FIG. 19.
[0206] In step 565, the procedure determines the number of bits (NBC) for run-length coding for the first non-zero coefficient assuming that the first coefficient will be included in the reduced block. This is a very easy task for the low-pass scaling method in step 225 of FIG. 13 , because the number of bits for the run-length coding of an AC coefficient in the reduced block will be the same as the run-length coding of the AC coefficient in the original block. For other scaling methods, the AC coefficient may have to be decoded and re-encoded to determine how many additional bits need to be used for run-length coding to include the AC coefficient in the reduced block. Moreover, for other scaling methods, for a second or subsequent AC coefficient in the block, the addition of a subsequent AC coefficient to the reduced block may change the run-length coding of prior AC coefficients as well as run-length coding for the additional coefficient. After step 565, execution continues to step 571 of FIG. 35.
[0207] In step 571 of FIG. 35 , the number of bits (NBC) required for encoding and including the non-zero AC DCT coefficient in the reduced block is compared to the number of bit available (NBA) for encoding non-zero AC DCT coefficients in the block. If NBC is not less than or equal to NBA, then execution branches to step 572 to put the remaining unallocated bits into the bucket. Execution loops back from step 572 to step 547 of FIG. 33.
[0208] In step 571 of FIG. 35 , if NBC is less than or equal to NBA, then execution continues to step 573. In step 573 the coefficient is transferred to the reduced-rate MPEG coded video. Then in step 574, the number of bits available (NBA) to encode the non-zero AC DCT coefficients of the block is decremented by the number of bits used (NBC) in encoding the non-zero AC DCT coefficients of the block. In step 575, execution branches to step 572 if there are no more non-zero AC DCT coefficients in the block. Otherwise, execution continues from step 575 to step 576 in FIG. 34. In step 576, the next non-zero AC DCT coefficient is obtained in the parsing order, for the implementation of the low-pass scaling method. For other scaling methods, the next largest-magnitude or approximate next largest magnitude non-zero AC DCT coefficient would be obtained in step 576 instead of the next non-zero AC DCT coefficient in the parsing order.
[0209] FIG. 36 shows a flowchart of a subroutine for determining the bit rate reduction factor (RF) for a reduction from an MPEG source having a known constant bit rate. In a first step 581, the offset rate (S), of bits in the original-quality MPEG source that are not bits of the AC DCT coefficients, is determined. Then in step 582, the coefficient bit rate reduction factor (RF) is computed from the known constant bit rate (BO) and padding (PD) of the original-quality MPEG source, the offset rate (S), and the desired reduced bit rate (BR) of the reduced-quality MPEG coded video. In particular, the coefficient bit rate reduction factor is computed as RF=(BR−S)/(BO−PD−S).
[0210] FIG. 37 shows a flowchart of a subroutine for determining the coefficient bit rate reduction factor (RF) for a reduction from an MPEG source having an unknown or variable bit rate. This subroutine, for example, is called in step 543 of FIG. 33. In a first step 591 of FIG. 37 , the video frame size in bits (Vs) is determined, for example from the parsing of the video frame in the previous step 542 in FIG. 33. Then in step 592 of FIG. 37 , a moving average (VAVS) of the video frame size is determined over the last N frames. For example, VAVS=(VS(I−N−1)+VS(I−N−2)+VS(I−N−3)+ . . . +VS(I))/N, where VS(I) denotes the video frame size of Ith video frame in the original-quality MPEG source, and N is the number of frames over which the moving average is computed. N defines a “frame window,” which is, for example, the number of video frames over 0.5 seconds worth of video data. In general, the moving average is the result of a low-pass digital filtering operation upon the video frame size. The low-pass digital filter is implemented by a subroutine, such as the following:
[0211]
| /* Compute Moving Average of up to N samples of VS */ | |
| /* J is initially set to N−1 */ | |
| /* K is initially set to N−1 */ | |
| /* SUM is initially set to zero */ | |
| IF (K<0) THEN CO TO CASE2 | |
| /* Handle the case at the beginning of the clip where */ | |
| /* N samples of VS are not yet available */ | |
| SUM ← SUM + VS | |
| VAVS ← SUM/(N−J) | |
| VSBUF(J) ← VS | |
| J←J−1 | |
| RETURN | |
| /* Handle the case where N samples of VS have been loaded */ | |
| /* into the buffer VSBUF */ | |
| CASE2: IF (K<0) THEN K←N−1 | |
| SUM←SUM+VS − VSBUF(K) | |
| VAVS ← SUM/N | |
| VSBUF(K) ← VS | |
| K←K−1 | |
| RETURN | |
[0212] In step 593 of FIG. 37 , a target average video frame size (VRAVS) is calculated from an accuracy rate control factor (AR), the desired reduced rate (BR) of the reduced-quality MPEG coded video, and the video frame rate (FR) according to VRAVS=AR*BR/FR. The accuracy rate control factor (AR) is a factor substantially equal to 1 that can be adjusted on a frame-by-frame basis to control the bit rate of the reduced-quality MPEG coded video. The accuracy rate control factor (AR) could also be a constant set sufficiently less than 1 to account for variability in the moving average of the frame rate so that the bit rate of the reduced-quality MPEG coded video will not exceed the desired bit rate BR, and stuffing could later be inserted in the reduced-quality MPEG coded video to obtain precisely the desired bit rate BR.
[0213] In step 594, the number of bits (BS) in the frame that are not bits of the AC DCT coefficients is determined, for example, from the parsing of the video frame in the prior step 542 of FIG. 33. In the last step 595 of FIG. 37 , the coefficient bit rate reduction factor (RF) is computed as VRAVS/VAVS.
[0214] FIG. 38 shows that a video buffer model could be used for adjusting the bit level in the bucket when the adaptive bit rate reduction procedure 603 (as in FIGS. 33-37 ) produces a reduced quality MPEG-2 stream 604 in real time from a source 605 of original-quality MPEG-2 coded video. The video buffer model 601 accumulates the video buffer level (VBV) in accordance with the time stamps (DTS, PTS) and program clock reference (PCR) values in the reduced-quality MPEG-2 stream 604 indicating when bits from the stream 604 are added to the video buffer level (VBV) and when bit from the steam 604 are removed from the video buffer level (VBV). In particular, the number of bits in the bucket 602 is incremented when the video buffer level becomes less than a low threshold TH1, and the number of bits in the bucket 602 is decremented when the video buffer level becomes greater than a high threshold TH2. The thresholds TH1 and TH2 are selected to prevent underflow of the video buffer level (i.e., VBV becoming less than zero) and to prevent overflow of the video buffer level (i.e., VBV exceeding a maximum size that is greater than TH2). The video buffer model could be implemented as a subroutine that would be called when a frame is encoded and also at the time when the frame would be decoded, relative to the time that the frame is encoded. In this case, for example, the maximum size of the buffer includes a multiplicity of the reduced size frames, the low threshold is larger than the largest reduced frame, and the difference between the size of the buffer and the high threshold is larger than the size of the largest reduced frame. The subroutine could be coded as follows:
-
- [0215] /* Video buffer model */
- [0216] /* This routine is called when a reduced frame has been encoded, */
- [0217] /* Before increasing the video buffer level for the frame just encoded, */
- [0218] /* a video buffer queue (VBUFQUEUE) is inspected. The video buffer */
- [0219] /* queue has respective entries for frames that have been encoded */
- [0220] /* but not yet decoded. Each entry includes a decode time stamp */
- [0221] /* indicating when the frame is to be removed from the video buffer */
- [0222] /* relative to the loading of the current frame into the video buffer. */
- [0223] START: IF (VBUFQUEUE IS EMPTY) THEN GO TO STEP2
- [0224] STEP1: GET DECODE TIME STAMP OF ENTRY AT HEAD OF VBUFQUEUE
- [0225] IF(DECODE TIME STAMP IS AFTER LOADING TIME OF CURRENT FRAME) THEN GO TO STEP 2
- [0226] GET FRAME SIZE (FSHE) OF FRAME OF ENTRY AT HEAD OF VBUFQUEUE
- [0227] VBUF←FSHE
- [0228] REMOVE ENTRY FROM HEAD OF VBUFQUEUE
- [0229] GO TO START
- [0230] STEP2: IF (VBUF<TH1) THEN BUK←BUK−(VBUF−TH1)
- [0231] IF (VBUF<0) THEN (VBUF←0; REPORT UNDERFLOW ERROR)
- [0232] VBV←VBV+FS
- [0233] IF (VBV)>TH2 THEN BUK←BUK−(VBV−TH2)
- [0234] IF (VBV>MAX) THEN (VBV←MAX; REPORT OVERFLOW ERROR)
- [0235] PUT FS and relative time stamp of frame on buffer queue RETURN
[0236] Although the adaptive bit rate reduction method has been shown in FIG. 38 being used for the real-time production of a reduced-quality MPEG-2 stream 604 from a source 605 of original-quality MPEG-2 coded video, the same method can be used in order to reduce the size of an MPEG-2 transport stream file to a given file size either for storage on CD ROMs or for trick file generation.
[0237] For trick file generation, sometimes there is need to insert freeze frames to compensate for large variations in the frame size. For generating trick files for video-on-demand (VOD) applications, in particular, the file size is generally limited to 10% of the original file size while the target reduced bit rate must be smaller or equal than the bit rate of the original file. When only I frames are used the equivalent bit rate of the I frames of an IBBP encoded clip is 3 to 4 times higher than the bit rate of the encoded file, for example for clips with lots of action encoded at 4 Mbps by a high quality encoder, the equivalent bit rate of the I frames of the clip is 14.5 Mbps. Therefore, the bit rate reduction factor is 3.6, which at such low original bit rate will generate bad quality video frames. To compensate for this large reduction, some of the I frames in the original file are discarded and replaced by P freeze frame that are very small in size. The selection of I frames to discard is done based on the number of bits that will be allocated per block at such a low bit rate as 4 Mbps.
[0238] In view of the above, there has been described a method of producing reduced bit rate MPEG coded video while achieving uniform proportional video quality reduction proportional to the ratio of the original frame size and the scaled down frame size by proportional reduction of block size and smoothing between frames using a “bucket” of bits. Bits are placed in the bucket when the bits available to be allocated to a block cannot be allocated to the block because there is a rather coarse granularity in the number of bits that can be allocated to a block. A certain fraction of the bits in the bucket are made available for encoding AC DCT coefficients in the block, so that the bits in the bucket after allocation to a block are available for encoding AC DCT coefficients in at least a certain number of following blocks. The method can be applied to a source of original-quality MPEG coded video having an unknown or variable bit rate by computing a moving average of the number of bits per frame over a certain number of consecutive frames. The method improves the quality of the reduced-quality MPEG coded video for a given bit rate by increasing the bit allocation to the video by reducing to a minimum the amount of padding used to precisely achieve the desired bit rate, and by obtaining a substantially uniform degradation in quality over the frames and blocks in the frames. The method can be done in the compressed domain without decoding of the run-length encoded AC DCT coefficients, so that it can be done in real time while streaming MPEG-2 data from a file server at a variable reduced rate to accommodate network congestion. The method can also be used to produce a trick file in parallel with the encoding of an MPEG-2 stream for storage of compressed video in a video file server.
Claims
1. A method of producing reduced-quality MPEG coded video from original-quality MPEG coded video, the original-quality MPEG coded video including a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG coded video, the reduced-quality MPEG coded video also having frames of 8×8 blocks, each frame in the reduced-quality MPEG coded video having a corresponding frame in the original-quality MPEG coded video, and each 8×8 block in each frame of the reduced-quality MPEG coded video having a corresponding block in a corresponding frame in the original-quality MPEG coded video, said method comprising the steps of:
selecting non-zero AC DCT coefficients from corresponding blocks in the original-quality MPEG coded video to be included in the blocks of the reduced-quality MPEG coded video so that each 8×8 block in each frame of the reduced-quality MPEG coded video has a number of bits encoding non-zero AC DCT coefficients that is generally proportional to the number of bits encoding non-zero AC DCT coefficients for the corresponding 8×8 block in the corresponding frame of original-quality MPEG coded video;
which includes computing a number of bits available for encoding the non-zero AC DCT coefficients in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video, the number of bits available for encoding being computed by scaling the number of bits encoding non-zero AC DCT coefficients in the corresponding block of the corresponding frame of the original-quality MPEG coded video by a scale factor for said each frame, the scale factor being selected to obtain a desired reduced bit rate in the MPEG coded video for said each frame; and
which includes computing a moving average of frame size of the frames in the original-quality MPEG coded video, and computing the scale factor for said each frame from the moving average of frame size and a desired frame size for the reduced-quality MPEG coded video.
2. A method of producing reduced-quality MPEG coded video from original-quality MPEG coded video, the original-quality MPEG coded video including a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG coded video, the reduced-quality MPEG coded video also having frames of 8×8 blocks, each frame in the reduced-quality MPEG coded video having a corresponding frame in the original-quality MPEG coded video, and each 8×8 block in each frame of the reduced-Quality MPEG coded video having a corresponding block in a corresponding frame in the original-quality MPEG coded video, said method comprising the steps of:
selecting non-zero AC DCT coefficients from corresponding blocks in the original-quality MPEG coded video to be included in the blocks of the reduced-quality MPEG coded video so that each 8×8 block in each frame of the reduced-quality MPEG coded video has a number of bits encoding non-zero AC DCT coefficients that is generally proportional to the number of bits encoding non-zero AC DCT coefficients for the corresponding 8×8 block in the corresponding frame of original-quality MPEG coded video;
which includes computing a number of bits available for encoding the non-zero AC DCT coefficients in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video, the number of bits available for encoding being computed by scaling the number of bits encoding non-zero AC DCT coefficients in the corresponding block of the corresponding frame of the original-quality MPEG coded video by a scale factor for said each frame, the scale factor being selected to obtain a desired reduced bit rate in the MPEG coded video for said each frame; and
which includes computing a difference between the number of bits available for encoding the non-zero AC DCT coefficients in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video and the number of bits used for encoding the non-zero AC DCT coefficients retained in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video, and making available for encoding non-zero AC DCT coefficients of following blocks said difference between the number of bits.
3. The method as claimed in claim 2, which includes accumulating said difference to produce an accumulated number of bits that were available for encoding non-zero AC DCT coefficients in prior 8×8 blocks of the reduced-quality MPEG coded video but were not used for encoding non-zero AC DCT coefficients in the prior 8×8 blocks of the reduced-quality MPEG coded video, and making said accumulated number of bits available for encoding non-zero AC DCT coefficients in a certain number of following blocks in the reduced-quality MPEG coded video by dividing said accumulated number of bits by said certain number of following blocks to compute a fraction of the accumulated number of bits that is available for encoding said each of the 8×8 blocks of said each frame of the reduced-quality MPEG coded video in addition to the number of bits computed by scaling the number of bits encoding non-zero AC DCT coefficients in the corresponding block of the corresponding frame of the original-quality MPEG coded video by the scale factor for said each frame.
4. The method as claimed in claim 3, wherein said certain number of blocks is substantially equal to the number of blocks in said each frame, so that bits that are available but not used for encoding the AC DCT coefficients for the blocks in said each frame are made available for encoding the AC DCT coefficients for the blocks in a following frame.
5. A method of producing reduced-quality MPEG coded video from original-quality MPEG coded video, the original-quality MPEG coded video including a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG coded video, the reduced-quality MPEG coded video also having frames of 8×8 blocks, each frame in the reduced-quality MPEG coded video having a corresponding frame in the original-quality MPEG coded video, and each 8×8 block in each frame of the reduced-quality MPEG coded video having a corresponding block in a corresponding frame in the original-quality MPEG coded video, said method comprising the steps of:
selecting non-zero AC DCT coefficients from corresponding blocks in the original-quality MPEG coded video to be included in the blocks of the reduced-quality MPEG coded video so that each 8×8 block in each frame of the reduced-quality MPEG coded video has a number of bits encoding non-zero AC DCT coefficients that is generally proportional to the number of bits encoding non-zero AC DCT coefficients for the corresponding 8×8 block in the corresponding frame of original-quality MPEG coded video;
which includes computing a difference between a number of bits available for encoding the non-zero AC DCT coefficients in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video and the number of bits used for encoding the non-zero AC DCT coefficients retained in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video, and making said difference available for encoding non-zero AC DCT coefficients of following blocks.
6. The method as claimed in claim 5, which includes accumulating said difference to produce an accumulated number of bits that were available for encoding non-zero AC DCT coefficients in prior 8×8 blocks of the reduced-quality MPEG coded video but were not used for encoding non-zero AC DCT coefficients in the prior 8×8 blocks of the reduced-quality MPEG coded video, and making said accumulated number of bits available for encoding non-zero AC DCT coefficients in a certain number of following blocks.
7. The method as claimed in claim 6, herein said accumulated number of bits is made available for encoding non-zero AC DCT coefficients in a certain number of following blocks of the reduced-quality MPEG coded video by dividing said accumulated number of bits by said certain number of following blocks of the reduced-quality MPEG coded video to compute a fraction of the accumulated number of bits that is available for encoding non-zero AC DCT coefficients in said each of the 8×8 blocks of the reduced-quality MPEG coded video in addition to the number of bits computed by scaling the number of bits encoding non-zero AC DCT coefficients in the corresponding block of the corresponding frame of the original-quality MPEG coded video by the scale factor for said each frame.
8. The method as claimed in claim 6, wherein said certain number of following blocks is substantially equal to the number of blocks in a frame, so that bits that are available but not used for encoding the AC DCT coefficients for the blocks in said each frame are made available for encoding the AC DCT coefficients for the blocks in a following frame.
9. A method of producing reduced-quality MPEG coded video from original-quality MPEG coded video, the original-quality MPEG coded video including a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG coded video, the reduced-quality MPEG coded video also having frames of 8×8 blocks, each frame in the reduced-quality MPEG coded video having a corresponding frame in the original-quality MPEG coded video, and each 8×8 block in each frame of the reduced-quality MPEG coded video having a corresponding block in a corresponding frame in the original-quality MPEG coded video, said method comprising, for each block in the reduced-quality MPEG coded video, the steps of:
(a) determining the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG coded video;
(b) computing a number of bits available for encoding AC DCT coefficients in the original-quality MPEG coded video by scaling the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG coded video with a scale factor; and
(c) selecting non-zero AC DCT coefficients in a certain order from the corresponding block in the original-quality MPEG coded video to be included in said each block of the reduced-quality MPEG coded video until the number of bits available for encoding the AC DCT coefficients in the block in the reduced-quality encoded video is not sufficient for encoding, in the block of the reduced-quality MPEG coded video, any more of the AC DCT coefficients in the corresponding block of original-quality MPEG coded video.
10. The method as claimed in claim 9, wherein said order is a parsing order of the non-zero AC DCT coefficients in the corresponding block in the original-quality MPEG coded video.
11. The method as claimed in claim 9, which includes computing the scale factor from a bit rate of the original-quality MPEG coded video and a desired bit rate for the reduced-quality MPEG coded video.
12. The method as claimed in claim 9, which includes computing the scale factor for each frame of the reduced-quality MPEG coded video from a moving average of the size of the corresponding frames in the original-quality MPEG coded video and a desired frame size for the reduced-quality MPEG coded video.
13. The method as claimed in claim 9, which includes computing a difference between the number of bits available for encoding the non-zero AC DCT coefficients in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video and the number of bits used for encoding the non-zero AC DCT coefficients retained in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video, and making available for encoding non-zero AC DCT coefficients of following blocks said difference between the number of bits.
14. The method as claimed in claim 13, which includes accumulating said difference to produce an accumulated number of bits that were available for encoding non-zero AC DCT coefficients in prior 8×8 blocks of the reduced-quality MPEG coded video but were not used for encoding non-zero AC DCT coefficients in the prior 8×8 blocks of the reduced-quality MPEG coded video, and making said accumulated number of bits available for encoding non-zero AC DCT coefficients in a certain number of following blocks in the reduced-quality MPEG coded video by dividing said accumulated number of bits by said certain number of following blocks to compute a fraction of the accumulated number of bits that is available for encoding said each of the 8×8 blocks of said each frame of the reduced-quality MPEG coded video in addition to the number of bits computed by scaling the number of bits encoding non-zero AC DCT coefficients in the corresponding block of the corresponding frame of the original-quality MPEG coded video by the scale factor for said each frame.
15. The method as claimed in claim 14 wherein said certain number of following blocks is substantially equal to the number of blocks in said each frame, so that bits that are available but not used for encoding the AC DCT coefficients for the blocks in said each frame are made available for encoding the AC DCT coefficients for the blocks in a following frame.
16. A method of producing in real-time a stream of reduced-quality MPEG-2 coded video from a source of original-quality MPEG-2 coded video, the original-quality MPEG-2 coded video including a set of non-zero AC discrete cosine transform (DCT) coefficients for 8×8 blocks in frames of the original-quality MPEG-2 coded video, the reduced-quality MPEG-2 coded video also having frames of 8×8 blocks, each frame in the reduced-quality MPEG-2 coded video having a corresponding frame in the original-quality MPEG-2 coded video, and each 8×8 block in each frame of the reduced-quality MPEG-2 coded video having a corresponding block in a corresponding frame in the original-quality MPEG-2 coded video, said method comprising, for each frame in the reduced-quality MPEG-2 coded video, the steps of:
(a) computing a moving average of the size of the corresponding frame in the original-quality MPEG-2 coded video;
(b) computing a scale factor from the moving average of the size of the corresponding frame in the original-quality MPEG-2 coded video and a desired size of said each frame of the reduced-quality MPEG-2 coded video; and
(c) for each 8×8 block in said each frame:
(i) determining the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG-2 coded video;
(ii) computing a number of bits available for encoding AC DCT coefficients in the original-quality MPEG-2 coded video by scaling the number of bits used in encoding non-zero AC DCT coefficients in the corresponding block of original-quality MPEG-2 coded video with a scale factor, and
(iii) selecting non-zero AC DCT coefficients in a parsing order from the corresponding block in the original-quality MPEG-2 coded video to be included in said each block of the reduced-quality MPEG-2 coded video until the number of bits available for encoding the AC DCT coefficients in the block in the reduced-quality encoded video is not sufficient for encoding, in the block of the reduced-quality MPEG-2 coded video, any more of the AC DCT coefficients in the corresponding block of original-quality MPEG-2 coded video.
17. The method as claimed in claim 16, which includes computing a difference between the number of bits available for encoding the non-zero AC DCT coefficients in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video and the number of bits used for encoding the non-zero AC DCT coefficients retained in said each of the 8×8 blocks in each frame of the reduced-quality MPEG coded video, and making available for encoding non-zero AC DCT coefficients of following blocks said difference between the number of bits.
18. The method as claimed in claim 17, which includes accumulating said difference to produce an accumulated number of bits that were available for encoding non-zero AC DCT coefficients in prior 8×8 blocks of the reduced-quality MPEG coded video but were not used for encoding non-zero AC DCT coefficients in the prior 8×8 blocks of the reduced-quality MPEG coded video, and making said accumulated number of bits available for encoding non-zero AC DCT coefficients in a certain number of following blocks in the reduced-quality MPEG coded video by dividing said accumulated number of bits by said certain number of following blocks to compute a fraction of the accumulated number of bits that is available for encoding said each of the 8×8 blocks of said each frame of the reduced-quality MPEG coded video in addition to the number of bits computed by scaling the number of bits encoding non-zero AC DCT coefficients in the corresponding block of the corresponding frame of the original-quality MPEG coded video by the scale factor for said each frame.
19. The method as claimed in claim 18, wherein said certain number of blocks is substantially equal to the number of blocks in said each frame, so that bits that are available but not used for encoding the AC DCT coefficients for the blocks in said each frame are made available for encoding the AC DCT coefficients for the blocks in a following frame.
Patent Citations (37)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US5231484(A) | 1993-07-01 | Gonzales et al. | Applicant |
| US5231487(A) | 1993-07-01 | Hurley et al. | Applicant |
| US5381144(A) | 1995-01-01 | Wilson et al. | Applicant |
| US5534944(A) | 1996-07-01 | Egawa et al. | Applicant |
| US5565998(A) | 1996-10-01 | Coombs et al. | Applicant |
| US5675384(A) | 1997-10-01 | Ramamurthy et al. | Applicant |
| US5691770(A) | 1997-11-01 | Keesman et al. | Applicant |
| US5694170(A) | 1997-12-01 | Tiwari et al. | Applicant |
| US5793897(A) | 1998-08-01 | Jo et al. | Applicant |
| US5812788(A) | 1998-09-01 | Agarwal | Applicant |
| US5838678(A) | 1998-11-01 | Davis et al. | Applicant |
| US5847761(A) | 1998-12-01 | Uz et al. | Examiner |
| US5859660(A) | 1999-01-01 | Perkins et al. | Applicant |
| US5889561(A) | 1999-03-01 | Kwok et al. | Examiner |
| US5892548(A) | 1999-04-01 | Kim | Applicant |
| US5892915(A) | 1999-04-01 | Duso et al. | Applicant |
| US5917830(A) | 1999-06-01 | Chen et al. | Applicant |
| US5929916(A) | 1999-07-01 | Legall et al. | Examiner |
| US5959690(A) | 1999-09-01 | Toebes, VIII et al. | Applicant |
| US5969650(A) | 1999-10-01 | Wilson | Applicant |
| US6038000(A) | 2000-03-01 | Hurst, Jr. | Applicant |
| US6052384(A) | 2000-04-01 | Huang et al. | Applicant |
| US6061399(A) | 2000-05-01 | Lyons et al. | Applicant |
| US6101195(A) | 2000-08-01 | Lyons et al. | Applicant |
| US6141358(A) | 2000-10-01 | Hurst et al. | Applicant |
| US6154496(A) | 2000-11-01 | Radha | Applicant |
| US6222881(B1) | 2001-04-01 | Walker | Examiner |
| US6263021(B1) | 2001-07-01 | Sethuraman et al. | Applicant |
| US6330286(B1) | 2001-12-01 | Lyons et al. | Applicant |
| US6343098(B1) | 2002-01-01 | Boyce | Examiner |
| US6414998(B1) | 2002-07-01 | Yoshinari et al. | Applicant |
| US6480547(B1) | 2002-11-01 | Chen et al. | Applicant |
| US6678332(B1) | 2004-01-01 | Gardere et al. | Applicant |
| US6771703(B1) | 2004-08-01 | Oguz et al. | Examiner |
| US2001/0036355(A1) | 2001-11-01 | Kelly et al. | Applicant |
| US2002/0087976(A1) | 2002-07-01 | Kaplan et al. | Applicant |
| US2002/0154694(A1) | 2002-10-01 | Birch | Applicant |
Non-Patent Literature (41)
- Y. Nakajima, H. Hori, and T. Kanoh, “Rate Conversion of MPEG Coded Video by Re-quantization Process,” IEEE Proc. of ICIP-95, vol. III, Sep. 1995, pp. 408-411.Applicant
- A.T. Erdem and M.I. Sezan, “Multi-generation Characteristics of the MPEG Video Compression Standards,” IEEE Proc. of ICIP-94, vol. II, 1994, pp. 933-937.Applicant
- M. Perreira, and A. Lippman, “Re-codable video,” IEEE Proc. of ICIP-94, vol. II, 1994, pp. 952-956.Applicant
- M. Mohsenian, R. Rajagopalan, and C.A. Gonzales, “Single-pass constant- and variable-bit-rate MPEG-2 video compression,” IBM J. Res. Develop., vol. 43, No. 4, Jul. 1999, pp. 489-509.Applicant
- P.H. Westerink, R. Rajagopalan, and C.A. Gonzales, “Two-pass MPEG-2 variable-bit-rate encoding,” IBM J. Res. Develop., vol. 43, No. 4, Jul. 1999, pp. 471-488.Applicant
- Jill Boyce, John Henderson, and Larry Pearlstein, “An SDTV Decoder with HDTV Capability: An All-Format ATV Decoder,” Hitachi American Ltd., file://C:Fatima\67.gif, pp. 67-75, published at least as early as Jan. 12, 2000.Applicant
- Boon-Lock Yeo, “On fast microscopic browsing of MPEG-compressed video,” IBM T.J. Watson Research Center, Jan. 1998, Multimedia Systems 7, 1999, pp. 269-281.Applicant
- Robert Mokry and Dimitris Anastassiou, “Minimal Error Drift in Frequency Scalability for Motion-Compensated DCT Coding,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 4, No. 4, Aug. 1994, pp. 392-406.Applicant
- C. Ward, C. Pecota, X. Lee, and G. Hughes, “Seamless Splicing for MPEG-2 Transport Stream Video Servers,” SMPTE Journal, Dec. 1999, pp. 873-879.Applicant
- Norm Hurst and Katie Cornog, “MPEG Splicing: A New Standard for Television—SMPTE 312M,” SMPTE Journal, Nov. 1998, pp. 978-988.Applicant
- Norm Hurst and Katie Cornog, “Splicing FAQ,” http://www.mpeg.org/MPEG/splicing-FAW.html, 8 pages, published at least as early as Oct. 13, 1999.Applicant
- SMPTE 312M-1999, SMPTE Standard for Television, “Splice Points for MPEG-2 Transport Streams,” Approved Apr. 8, 1999, The Society of Motion Picture and Television Engineers, White Plains, NY, 20 pages.Applicant
- “The Shape of MPEG,” DV Magazine, vol. 4, No. 12, Dec. 1996, http://livedv.com/Mag/Dec96/Contents/mpeg/mpeg.html, 5 pages, published at least as early as Oct. 13, 1999.Applicant
- “A Guide to MPEG Fundamentals and Protocol Analysis (Including DVB and ATSC),” Tektronix, Inc., Beaverton, Oregon, 1997, pp. 48 pages.Applicant
- Leonardo Chiariglione, “MPEG and multimedia communications,” CSELT, Torino Italy, http://www.cselt.it/ufv/leonardo/paper/isce96.htm, 50 pages, published at least as early as Oct. 13, 1999.Applicant
- Barry G. Haskell, Atul Puri, and Arun N. Netravali, MPEG-2 Video Coding and Compression, Chp. 8, pp. 156-182, and “Interactive Television,” Chp. 13, pp. 292-306, Digital Video: An Introduction to MPEG-2, Chapman & Hall, New York, NY, 1997.Applicant
- “MPEG-2: The basics of how it works,” Hewlett Packard, published at least as early as Oct. 31, 1999, 17 pages.Applicant
- Anil K. Jain, Fundamentals of Digital Image Processing, Prentice Hall, Inc., Englewood Cliffs, New Jersey, 1989, Chapter 4: Image Sampling and Quantization and Chapter 5: Image Transforms, pp. 80-188.Applicant
- “Information technology—Generic coding of moving pictures and associated audio information: Systems,” International Standard, ISO/IEC 13818-1:1996(E), 136 pages.Applicant
- “Information technology—Generic coding of moving pictures and associated audio information: Video,” International Standard, ISO/IEC 13818-2:1996(E), 211 pages.Applicant
- “Information technology—Generic coding of moving pictures and associated audio information—Part 3: Audio,” International Standard, ISO/IEC 13818-3:1995(E), 118 pages.Applicant
- Jerry D. Gibson, Toby Berger, Tom Lookabaugh, Dave Lindbergh, and Richard L. Baker, Digital Compression for Multimedia: Principles and Standards, Morgan Kaufmann Publishers, Inc., San Francisco, CA, 1998, Chapter 9: JPEG Still-Image Compression Standard, pp. 291-308, and Chapter 11: MPEG Compression, pp. 363-417.Applicant
- Barry G. Haskell, Atul Puri, and Arun N. Netravali, Digital Video: An Introduction to MPEG-2, Chapman & Hall, New York, NY, 1997, pp. 1-279, 292-306, 369-421.Applicant
- Nilesh V. Patel and Ishwar K. Sethi, Compressed Video Processing For Cut Detection, Vision and Neural Networks Laboratory, Dept. of Computer Science, Wayne State University, Detroit, MI, Oct. 1997, 26 pages.Applicant
- Nilesh V. Patel and Ishwar K. Sethi, Video Shot Detection and Characterization for Video Databases, Vision and Neural Networks Laboratory, Dept. of Computer Science, Wayne State University, Detroit, MI, Oct. 1997, 22 pages.Applicant
- Bo Shen, Ishwar K. Sethi and Vasudev Bhaskaran, DCT Convolution and Its Application In Compressed Video Editing, Dept. of Computer Science, Wayne State University, Detroit, MI and Visual Computing Dept., Hewlett-Packard Laboratories, Palo Alto, CA, To appear in SPIE VCDIP '97, also submitted to IEEE Trans. Cir. And Sys. For Video Tech., 11 pages.Applicant
- B. Shen and I.K. Sethi, Convolution-Based Edge Detection for Image/Video in Block DCT Domain, Vision & Neural Networks Laboratory, Dept. of Computer Science, Wayne State University, Detroit, MI, To appear in Journal of Visual Communications and Image Representation, 19 pages.Applicant
- Bo Shen and Ishwar K. Sethi, Direct feature extraction from compressed images, Vision and Neural Networks Laboratory, Dept. of Computer Science, Wayne State University, Detroit, MI, SPIE vol. 2670, Storage & Retrieval for Image and Video Databases IV, 1996, 12 pages.Applicant
- Bo Shen and Ishwar K. Sethi, Block-Based Manipulations On Transform-Compressed Images and Videos, Vision and Neural Networks Laboratory, Dept. of Computer Science, Wayne State University, Detroit, MI, To appear in Multimedia Systems, 26 pages.Applicant
- Bo Shen and Ishwar K. Sethi, Inner-Block Operations On Compressed Images, Vision and Neural Networks Laboratory, Dept. of Computer Science, Wayne State University, Detroit, MI, ACM Multimedia '95, San Francisco, CA, Nov. 5-9, 1995, 10 pages.Applicant
- Alexandros Eleftheriadis and Dimitris Anastassiou, Constrained and General Dyanamic Rate Shaping of Compressed Digital Video, Dept. of Electrical Engineering and Center for Telecommunications Research, Columbia University, New York, NY, Proceedings, 2nd IEEE International Conference on Image Processing (ICIP-95), Arlington, VA, Oct. 1995, 4 pages.Applicant
- Alexandros Eleftheriadis and Dimitris Anastassiou, Optimal Data Partioning of MPEG-2 Coded Video, Dept. of Electrical Engineering and Center for Telecommunications Research, Columbia University, New York, NY Processings, 1st International Conference on Image Processing (ICIP-94), Austin, Texas, Nov. 1994, 5 pages.Applicant
- Andrew B. Watson, Joshua A. Solomon, Albert Ahumada, and Alan Gale, DCT Basis Function Visibility: Effects of Viewing Distance and Contrast Masking, (1994), 11 pages, in B.E. Rogowitz (Ed.), Human Vision, Visual Processing and Digital Display IV (pp. 99-108), Billington, WA, SPIE.Applicant
- O'Reilly Network Broadcast 2000 Brings DV Editing to Linus (Aug. 11, 2000), http://www.oreillynet.com/pub/a/network/2000/08/11/magazine/broadcase2000.html, published at least as early as Mar. 27, 2001, 3 pages; Broadcast 2000, http://heroinewarrior.com/bcast2000.php3, published at least as early as Mar. 27, 2001, 4 pages.Applicant
- MPEG Wizard: MPEG Real-Time External Encoder, http://www.duplexx.com/mpgwiz.html, MPEG Wizard: MPEG Real-Time Encoder—Features and Software, http://www.duplexx.com/mpgwiz_f.html, MPEG Wizard: MPEG Real-Time Encoder—Specs & Requirements, http://www.duplexx.com/mpgwiz_r.html, published at least as early as Mar. 19, 2001, 4 pages.Applicant
- Adrienne Electronics Corporation—Home Page, http://www.adrielec.com/, 1 page; Functional Grouping of LTC/VITC, VTR Interface, and Video Products, http://www.adrielec.com/listing.htm, 2 pages; Adrienne Electronics Products and Price Listings Catalog, http://www.adrielec.com/shortfor.htm, 8 pages; AEC-BOX-8/18/28 Standalone VITC and/or LTC Time Code Generator, http://www.adrielec.com/box28lit.htm, 4 pages; AEC-BOX-8/18/28 Standalone LTC/VITC Time Code Reader, http://www.adrielec.com/box20lit.htm, 5 pages, published at least as early as Mar. 15, 2001.Applicant
- National P/N CLC020—SMPTE 259M Digital Video Serializer with Integrated Cable Driver, http://www.national.com/pf/CL/CLC020.html, published at least as early as Mar. 14, 2001, 3 pages.Applicant
- TE600 MPEG-2 DSNG Encoder, satellite uplink equipment, downlink, teleports, earth stations, amplifiers, antennas, http://www.usacanada.net/satellite/te600.htm, published at least as early as Mar. 14, 2001, 3 pages.Applicant
- TDR600/RA, satellite uplink equipment, downlink, teleports, earth stations, amplifiers, antennas, http://www.usacanada.net/satellite/tdr600-ra.htm, published at least as early as Mar. 14, 2001, 2 pages.Applicant
- TE300A MPEG-2 Encoder, Satellite uplink equipment, downlink, teleports, earth stations, amplifiers antennas, http://www.usacanada.net/satellite/te300a.htm, published at least as early as Mar. 14, 2001, 3 pages.Applicant
- TE-30, satellite uplink equipment, downlink, teleports, earth stations, amplifiers, antennas, http://www.usa.canada.net/satellite/te30.htm, published at least as early as Mar. 14, 2001, 3 pages.Applicant