US 6,298,089 B1Grant
Method for seamless and near seamless audio and non-video splicing of a digital transport stream
Issue Date:2001-10-02
•11 Claims
•3 Drawing Sheets
Abstract
In a system of seamless splicing of multimedia data streams, the method includes the steps of scanning forward in time until a first audio Packetized Elementary Stream (PES) header with an Audio Presentation Time Stamp (APTS) greater than the Video Presentation Time Stamp (VPTS) is detected. Then, continuing to scan audio packets until an audio frame header is detected, and replacing each audio packet encountered that did not contain this specific audio frame header (or Audio PES header) with a null packet. By these steps, an audio splice-in point that does, in fact, support a near-seamless audio splice is determined. The method of accomplishing a near-seamless audio splice includes the steps of scanning forward in time until the first audio PES header with APTS greater than VPTS is detected, and replacing each audio frame header with a null packet.
Metadata
Assignee
- Viewgraphics, Inc.
Inventor
- Hillel Gazit
Application Information
Application Number:US 09/357,722
Filing Date:1999-07-21
Priority Date:1998-12-10
Art Unit:7
Classifications
IPC:
H04B 166
Field of Search:
375240375240263752402837524001375240023752401234842334870534870634884534884523488453382232382236386 51386 52
Patent Drawings (3 sheets)
Description
REFERENCE
[0002] The information, data and all benefits of provisional application No. 60/111,666 filed Dec. 10, 1998 are incorporated by reference into this description.
[0003] The writing, the subject matter, and the description presented herein is protected under the provisions of United States Copyright laws by the Assignee, Viewgraphics, Inc., except only to any extent required by law in order to obtain and to continue all patent protection that is available during the term of any patent that issues hereon.
BACKGROUND OF THE INVENTION
Field of the Invention
[0004] The present invention relates generally to multimedia data stream processing and, more particularly, to apparatus and method for processing digitally encoded multimedia data streams.
[0005] A multimedia data stream refers to a binary stream that represents multimedia data, by which is meant video, opional audio streams, and optional other streams (such as subtitles, private data, etc.).
[0006] A digitally encoded multimedia data stream refers to a compressed version of a multimedia data stream. MPEG-2 is an example of a standard that defines compressed multimedia data formats. A multimedia data stream that has been formatted according to the MPEG-2 standard is referred to as a “MPEG-2 encoded data stream”.
[0007] The term “transport stream” is defined in ISO/IEC 13818-1:1996(e) page ix. The term “single program transport stream” (SPTS) is defined in ISO/IEC 13818-1:1996(E) page xi. A single program transport stream can also be abbreviated as SPTS; which is an example of a “MPEG-2 encoded data stream”.
[0008] MPEG-2 is one particular system of digitally encoding multimedia data streams, and since this system is regarded as an industry standard by many, it will be used in the following detailed description of the present invention. MPEG-2 prescribes specific protocols and capabilities, while allowing essentially unrestricted ways of implementing the prescribed protocols.
[0009] MPEG-2 encoded data streams are constructed so as not to exceed any of the limitations of the decoder buffer. Splicing involves appending a new data stream to a selected point in an existing data stream, thereby replacing the existing data stream at the splice point.
[0010] The MPEG-2 system of digitally encoding data streams usually involves a specification that has been regarded as an industry standard and, among other features, its encoding and decoding specifications prescribe an array of data stream format, timing, synchronization, and device capability parameters essentially independent of hardware and software system implementations.
[0011] However, the MPEG-2 specification, as is the specification of other systems, is deficient with regard to applications and contingencies that were unforeseen at the time the specification was first proposed. For example, it failed to anticipate the growing requirements for digitally encoded data stream splicing support with standard encoders.
[0012] The term “existing data stream” refers to an old data stream, which is a present data stream. A stream replacing the existing data stream is referred to as a “new data stream”, which is the stream that plays after a splice in.
[0013] A digitally encoded data stream is encoded such that, during decoding, a decoder buffer will continuously contain a sufficient number of frames for continuous decoding but not so many frames as to overflow the decoder buffer. Such a data stream contains a variable amount of reference data for recreating a video image.
[0014] Over time, the buffer levels in a decoder will fluctuate depending on which frames and frame types have been transmitted to the buffer and which frames have been processed for display by the decoder.
[0015] Typically, video frames of a digitally encoded data stream are received by a decoder in a different order from the order they are displayed. Decoders are required to buffer data streams so that video frames can be reconstructed and re-ordered readily.
[0016] Non-video data typically consists of audio data and other data private to the stream. Since audio is decoded separately from video, the time base of an audio packet can vary widely with respect to the time base in nearby video packets multiplexed on the same transport stream. Also, an audio buffer in a decoder is small relative to the video buffer.
[0017] Consequently, if the video and audio frames have similar time stamps, video is sent much earlier (e.g., about 100 milliseconds) before the audio. For this reason, the ideal splice-in and the ideal splice-out points with respect to video do not necessarily address the splice-in and splice-out points relative to non-video data. If audio is not spliced homogeneously with video, audio packets from the old data stream may be played with video from the new stream, or audio from the new stream may be played with video from the old stream.
[0018] Standardization efforts proposed by the Society of Motion Pictures and television suggest abandonment of conventional splicing methodologies in favor of a new extension to the MPEG-2 specification. The proposed standard contains a new integrated protocol directed specifically at performing seamless splicing of a data stream pair.
[0019] The proposed standard provides for incorporation of splice points and related splicing support information directly into the data stream at the time of encoding. Unfortunately, the proposed standard will require the expense of at least replacing a very large number of existing encoders in production today. Moreover, the proposed standard does not address splicing at arbitrary frames within an MPEG-2 encoded data stream.
[0020] Methods have been suggested recently for seamlessly splicing MPEG-2 digitally encoded data streams using the large install base of encoders without requiring new encoding protocols. However, these methods do not specifically address how to preserve the seam between two data streams for the audio and non-video components of the data stream.
[0021] Therefore, the problem persists for splicing two data streams while preserving the seam between them. Clearly, what is needed is a new and completely different approach to solving the problem, such a solution is provided by the present invention, which is described in detail hereinafter.
OBJECTS AND SUMMARY OF THE INVENTION
[0022] It is a principal object of the present invention to provide a new and improved method of processing digitally encoded-data streams.
[0023] It is a further object of the present invention to provide a method for determining splice-in and splice-out points for audio and non-video data.
[0024] Another object of the present invention is to provide a new and improved method for seamlessly splicing MPEG-2 digitally encoded data streams while preserving the seam between them.
[0025] Briefly a method in accordance with the present invention involves splicing audio and non-video data with a digitally encoded data stream. When a splicing point is identified in a program transport stream, the present invention will permit a near seamless splice-in and splice-out of data streams at these points for audio and other non-video data.
[0026] Other features, objects and advantaged of the present invention will become more readily apparent from the following detailed description of the presently preferred embodiment.
BRIEF DESCRIPTION OF THE DRAWINGS
[0027] FIG. 1 illustrates a sequence of video and non-video MPEG-2 transport packets representing a single program transport stream (SPTS) and a possible video splice-in point.
[0028] FIG. 2 illustrates a sequence of video and non-video MPEG-2 transport packets representing a SPTS and a possible video splice-out point.
[0029] FIG. 3 illustrates how audio packets from an old data stream can be injected into a new data stream for a non-video seamless splice.
DETAILED DESCRIPTION OF THE INVENTION
[0030] According to MPEG-2 Systems Specification, the presentation time stamp (PTS) and decoding time stamp (DTS) are in units of 1/90,000 seconds Therefore, the program clock reference (PCR) base is viewed as having 90,000 ticks per second.
[0031] A seamless splice is one in which no anomalies exist near the splice point, Such anomalies might include, but are not limited to, black or blank frames or non-synchronous switching of audio and video data.
[0032] There are three related methods for splicing audio and non-video data with a digitally encoded data stream, according to the present invention. These methods vary in terms of implementation, complexity and smoothness at a splice seam Each method may make use of the procedure to calculate the video splice points, and each method pertains only to audio and other non-video data. Each method described applies to SPTS's that have at least one audio or non-video data stream.
[0033] A first of these method determines the splice-in point by frame position. Referring to video splice-in point 170, FIG. 1, a video packet 125 is the first packet of a new single program transport stream (SPTS) to be spliced-in. (The new SPTS refers to the stream that plays after splice in, and the old SPTS refers to the stream that plays before splice out.) The audio splice-in point is determined by scanning forward in time,
[0034] The payload of each audio packet is scanned for the beginning of the first audio frame after video splice-in point 170. Each audio packet after the video splice-in point 170 that does not contain the next audio frame header is replaced with a null packet. A null packet is established by setting the packet identifier (PID) field in the packet header to 1FFF (hexadecimal) with the following byte set to 10 (hexadecimal).
[0035] Assume audio packet 150 in FIG. 1 contains the first audio frame header following the video splice-in point 170. Then, audio packet 135 is replaced with a null packet.
[0036] This method is straight forward to implement. However, by a modification of this method requiring more processing cycles to scan additional audio packet payloads, it is assured that audio of the new SPTS does not overlap video of the old SPTS. This modification determines the splice-in point by Presentation Time Stamp (PTS).
[0037] In other words, the audio splice-in point is determined by scanning forward in time until the first audio packetized elementary stream (PES) header with an audio PTS (APTS) greater than the video PTS (VPTS) is detected: APTS>VPTS. (The audio PTS is contained in the audio PES header.)
[0038] This modification only continues the scanning audio packets step, described above, until an audio frame header is detected. Then, each audio packet scanned that did not contain this specific audio frame header (or audio PES header) is replaced with a null packet. The audio PTS is preserved for correct decoding.
[0039] A second method determines the splice-out point by position in the old SPTS. Referring to FIG. 2, video packet 245 is the last video packet of the old SPTS before the video splice-out point 270. The audio splice-in point is determined by scanning backwardly in time. The payload of each audio packet payload is scanned for the beginning of the first audio frame before video splice-out point 270.
[0040] Each audio packet before the video splice-out point 270 that does not contain the next audio frame header is replaced with a null packet. If audio packet 215 contained such an audio frame header, then the audio packet 235 is replaced with a null packet.
[0041] A third method determines the splice-out point by PTS. Referring to the video splice-out point 270 in FIG. 2, video packet 245 is the last packet of the old SPTS. Then, the audio splice-in point is determined by scanning forward in time until the last audio PES header with APTS<VPTS is detected.
[0042] Assume this is packet 265 in FIG. 2. Each video packet between the video splice-out point 270 and packet 265 is replaced with a null packet. Video packets 250, 255, and 265 are replaced with null packets. This method supports a more accurate audio splice-out point relative to the video splice-out point.
[0043] A fourth method supports a near-seamless audio splice, while keeping the video buffer levels in the decoder sufficiently high. By combining the modification of the first method for splice-in with the third method for splice-out, any resulting gap of non-video data in the combined SPTS is minimized. The non-video gap is relative to the combined stream resulting by splicing out of the old SPTS and splicing in the new SPTS.
[0044] Those audio packets that are replaced with nulls in the old SPTS are injected into the new SPTS for a near seamless splice. (The resulting splice produces an audio gap of less than one audio frame.)
[0045] These audio packets from the old SPTS first are copied into an intermediate splice buffer prior to being transmitted to the decoder. The PTSs contained in these packets ore adjusted by adding a constant. This constant, C, can be defined, according to the present invention, by the following relationship:
[0046] C=VPTS_NEW(FIRST)−[VPTS_OLD(LAST)+PICTURE_GAP]
[0047] The constant C is used to adjust the PTS as follows:
[0048] APTS_NEW(j)=APTS_OLD(j)+C
[0049] where:
[0050] APTS_NEW(j) is the audio presentation time of the j-th audio frame stored in the intermediate non-video splice buffer;
[0051] APTS_OLD(j) is the audio presentation time of the j-th audio frame of the old SPTS;
[0052] VPTS_NEW(FIRST) is the presentation time of the first video frame spliced in from the new SPTS;
[0053] VPTS_OLD(LAST) is the presentation time of the last video frame of the old SPTS; and
[0054] PICTURE_GAP is calculated as the Ticks per Second divided by the number of pictures per second, erg., for NTSC, this is 90,000/(30,000/1001).
[0055] These audio packets, then, are injected into the new stream by replacing a combination of null packets and leading audio packets of the new SPTS. This must be done in such a way so as not to overflow the audio buffer of the decoder.
[0056] Null packets are common within SPTSs of constant bit rate (CBR). As trailing audio packets from the old packets are copied over the leading audio packets from the new SPTS, the leading audio packets are rippled through the new SPTS by replacing null packets occurring later in the new SPTS.
[0057] Referring to FIG. 3, noting video splice-out point 470 and video splice-in point 370, suppose that video packet 420 is the last video packet displayed from the old SPTS and that video packet 325 is the first video packet displayed from the new SPTS. Suppose that audio packets 435 and 450 have PTSs such that they are to be transitioned to the new SPTS.
[0058] Audio packets 335 and 355 of the new SPTS are copied into null packets farther down in the new SPTS. Audio packets 435 and 3450 of the old SPTS are copied into the original position of audio packets 335 and 355 respectively.
[0059] A fifth method is a further refinement of the fourth method by supporting a seamless non-video splice. This is achieved by calculating the APTS adjustment. Following the MPEG2 Audio Specification, the number of samples per frame is defined as 384 samples for layer 1 and 1152 for layers 2 and 3. The time of the audio frame is defined as:
[0060] time_of_audio_frame=number_of_audio_samples/sample_frequency
[0061] The time_of_audio_frame is calculated by using only the information in one audio header. Each audio PTS in the new SPTS is modified by adding a delta derived as follows:
[0062] delta=APTS_FIRST(by_new_PCR)−APTS_FIRST(actual).
[0063] Using the relationship described above:
[0064] delta=APTS_FIRST(by_old_PCR)+C−APTS_FIRST(actual).
[0065] This is equivalent to:
[0066] delta=APTS_OLD(LAST)+time_of_audio_frame+C−APTS_FIRST(actual)
[0067] Where:
[0068] by_old_PCR=the PCR of the old SPTS;
[0069] by_new_PCR=the PCR of the new SPTS;
[0070] APTS_FIRST(by_old_PCR)=the presentation time of the first audio packet of the new stream according to the PCR of the old stream;
[0071] APTS_FIRST(by_new_PCR)=the presentation time of the first audio packet of the new stream according to the PCR of the new stream;
[0072] APTS_FIRST(actual)=the original presentation time of the first audio header of the new streams; and
[0073] APTS_OLD(LAST)=the presentation time stamp of the last audio frame of the old SPTS.
[0074] The term“delta” can be positive or negative. The closer delta is to zero, the better the quality. For each subsequent splice point, a new incremental delta must be computed and added to the previous delta.
[0075] Methods 1, 2 and 3 can be implemented in a commercial splicing product with seamless audio splice results that, while not entirely seamless, are acceptable. Method 1 and 3 above support a near-seamless audio splice. The difference between VPTS and the APTS is bounded by 1152 samples divided by the sample frequency.
[0076] Method 5 preserves all non-video data associated with the video splice-out point.
[0077] The methods of the invention are best implemented in a general purpose computer. For application to stored transport streams, virtually any computer having adequate storage for the video can be used. For real time application to transport streams, the processor must be sufficiently fast to keep up with the stream rates.
[0078] While the present invention has beep described in substantial detail, other and further forms, features and modifications will occur to those that are skilled in this art. Therefore, it is understood that the present invention is not limited by the description hereinabove, but rather, it is defined by the claims appended hereto.
Claims
What is claimed is:
1. In a system of implementing splicing of at least one audio and non-video data stream for predetermined video splice-in and splice-out points, comprising the steps of:
determining an audio splice-in point that supports a near-seamless audio splice, including the steps of:
scanning a Single Program transport Stream (SPTS) forward in time from the video splice-in point;
examining each audio packet encountered to determine whether it is the first audio frame after the video splice-in point; and
replacing each audio packet encountered that is not a first audio frame with a null packet;
implementing splicing audio and non-video data streams at said audio splice-in point, including the steps of;
scanning forward in time until the last audio PES header with APTS<VPTS is detected;
replacing each video packet between the video splice-out point and this point with a null packet;
copying the audio packets for the old SPTS to an Intermediate splice buffer; and
adding a predetermined constant to the PTS of the audio packets in the intermediate splice buffer;
whereby a near-seamless audio splice is accomplished.
2. In a system of implementing splicing of at least one audio and non-video data stream for predetermined video splice-in and splice-out points, a method for determining the audio splice-in point including the steps of:
scanning a Single Program Transport Stream (SPTS) forward in time from the Video splice-in point;
examining each audio packet encountered to determine whether it is the first audio frame after the video splice-in point; and
replacing each audio packet encountered that is not a first audio frame with a null packet;
whereby an audio splice-in point that supports a near-seamless audio splice is determined.
3. The method of claim 2 where a null packet is established by setting the packet identifier field in a packet header to 1FFF (hexadecimal) with the following byte set to 10 (hexadecimal).
4. In an MPEG-2 system of splicing audio and non-video data streams, a method for determining an audio splice-in point comprising the steps of:
scanning forward in time Until the first audio Packetized Elementary Stream (PES) header with an Audio PTS (Audio Presentation Time Stamp) (APTS) greater than the Video PTS (VPTS) is detected;
continuing to scan audio packets until an audio frame header is detected; and
replacing each audio packet encountered that does not contain this specific audio frame header (or Audio Packetized Elementary Stream header) with a null packet;
whereby an audio splice-in point that supports a near-seamless audio splice is determined.
5. In a MEG-2 system of splicing at least one audio and non-video data stream, a method for determining the audio splice-out point comprising the steps of:
scanning the SPTS backward in time from the video splice-out point;
examining each audio packet thus encountered for the beginning of the first audio frame before a video splice-out point; and
replacing each audio packet encountered that does not contain an audio frame header with a null packet;
whereby an audio splice-out point that supports a near-seamless audio splice is determined.
6. In a system for splicing of at least one audio and non-video data stream, a method for determining the audio splice-out point comprising the steps of:
scanning forward in time until the last audio PES header with APTS<VPTS it detected;
replacing each video packet between the video splice-out point and this point with a null packet;
whereby an audio splice-out point that supports a near-seamless audio splice is determined.
7. A method for splicing of at least one audio and non-video data stream/comprising the steps of:
C=VPTS_NEW(FIRST)−[VPTS_OLD(LAST)+PICTURE_GAP]
where:
scanning forward in time until the first audio packetized elementary stream (PBS) header with an audio PTS (APTS) greater than the video PTS (VPTS) is detected;
continuing to scan audio packets until an audio frame header is detected;
replacing each audio packet encountered that does not contain this specific audio frame header (or audio PES header) with a null packet;
scanning forward in time until the last audio PES header with APTS<VPTS is detected;
replacing each video packet between the video splice-out point and this point with a null packet;
copying the audio packets for the old SPTS to an intermediate splice buffer;
adding a constant to the PTS of the audio packets in the intermediate splice buffer where the constant is defined by:
VPTS_NEW(FIRST) is the presentation time of the first video frame spliced in from the new SPTS;
VPTS_OLD(LAST) is the presentation time of the last video frame of the old SPTS;
PICTURE_GAP is calculated as the Ticks per Second divided by the number of pictures per second;
replacing a combination of null packets and leading audio packets of the new SPTS with audio packets from the intermediate splice buffer;
whereby a near-seamless audio splice is accomplished.
8. The method at defined by claim 7, including the step of replacing a combination of said null packets and leading audio packets of said new SPTS with audio packets from said intermediate splice buffer.
9. A method for seamless at least one splicing audio and non-video data stream, comprising the steps of;
C=VPTS_NEW(FIRST)−[VPTS_OLD(LAST)+PICTURE_GAP]
where:
scanning forward in time until a first audio packetized elementary stream (PES) header with an audio PTS (APTS) greater than a video PTS (VPTS) is detected;
continuing to scan audio packets until on audio frame header is detected;
replacing each audio packet encountered that does not contain a specific audio frame header (or audio PES header) with a null packet;
scanning forward in time until the last audio PES header with APTS<VPTS is detected;
replacing each video packet between the video splice-out point and this point with one of said null packets;
copying the audio packets for the old SPTS to an intermediate splice buffer;
adding a constant to the PTS of the audio packets in said intermediate splice buffer, where said constant is determined by the relationship:
VPTS_NEW(FIRST) is the presentation time of the first video frame spliced in from the new SPTS;
VPTS_OLD(LAST) is the presentation time of the last video frame of the old SPTS;
PICTURE_GAP is calculated as the Ticks per Second divided by the number of pictures per second;
whereby an audio splice-in point to support a near-seamless audio splice is determined.
10. The method as defined by claim 8, including the step of defining the time of said audio frame header by the relationship:
time_of_audio_frame=number_of_audio_samples/sample_frequency
whereby a seamless audio splice is accomplished.
11. The method as defined by claim 8, including the step of modifying each audio PTS in the new SPTS by adding a delta derived as follows:
delta=APTS_OLD(LAST)+time_of_audio_frame+C−APTS_FIRST(actual)
where:
whereby a seamless audio splice is accomplished.
APTS_FIRST(actual) refers to the original presentation time of the first audio header of the new stream;
APTS_OLD(LAST) is the presentation time stamp of the last audio frame of the old SPTS for subsequent splice points, computing a new incremental delta adding the computed delta to the previous delta;
Patent Citations (9)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US5859660 | 1999-01-01 | Perkins et al. | Examiner |
| US5917830 | 1999-06-01 | Chen et al. | Examiner |
| US6038000 | 2000-03-01 | Hurst, Jr. | Examiner |
| US6049569 | 2000-04-01 | Radha et al. | Examiner |
| US6137834 | 2000-10-01 | Wine et al. | Examiner |
| US6141358 | 2000-10-01 | Hurst, Jr. et al. | Examiner |
| US6154496 | 2000-11-01 | Radha | Examiner |
| US6195368 | 2001-02-01 | Gratacap | Examiner |
| US6208691 | 2001-03-01 | Balakrishnan et al. | Examiner |
Non-Patent Literature (6)
- Donovan et al., Pharse splincing and variable substitution using the IBM trainable speech synthesis system, Acoustics, Speech, and Signal Processing, 1999. Proceedings., 1999 IEEE International Conference on, vol.: 1 1999, pp. 373-376.Applicant
- Radha et al., Compressed video seamless switching using variable prarameters, Image Processing, 1998. CIP 98. Proceedings. 1998 Iternational Conference vol.: 1 1998, pp. 862-865.Applicant
- Balakrishnan et al., Rate and buffer regulation in a video encoder for seamless splicing, Image Processing, 1998.ICIP 98. Proceedings, 1998 International Conference vol.: 1 1998, pp. 880-883.Applicant
- Wee et al., Splicing MPEG video streams in the compressed domain, Multimedia Signal Processing, 1997., IEEE First Workshop on, 1997, pp. 225-230.Applicant
- Brighwell et al, Flexible switching and editing of MPEG-2 video bitstreams, Broadcasting Convention, 1997. International, 1997, pp. 547-552.Applicant
- Birch, C.H., MPEG splicing and bandwidth management, Broadcasting Convneiton 1997, International, 1997, pp. 541-546.Examiner