Non real-time delivery of MPEG-2 programs via an MPEG-2 transport stream

[0002] This application claims the benefit of U.S. Provisional Application No. 60/169,872, filed Dec. 9, 1999.

[0003] The present invention relates to digital television, and more particularly to a method and apparatus by which MPEG-2 programs or the like may be extracted from a transport stream and delivered via the same or another transport stream at a fraction of the program's original transport rate, or stored using less capacity than required to store the original transport stream. In this manner, the invention enables a receiving device to receive, store, and decode the program at some future, user-initiated time.

[0004] In bandwidth-limited systems, a tradeoff exists between the number of digital TV programs that may be delivered concurrently and their subjective quality. A need may also exist to allow the delivery of programs in non-real-time as enhanced services which do not impact the quality of programs delivered in real-time. Such enhanced services could include commercial advertisements that are applicable to the receiving device and “played out” by remote control (e.g., during a commercial break). Moreover, it would be advantageous to provide a capability to deliver programs requested by the user of the receiving device on a non-real-time basis, i.e., after the original real-time transmission of a program.

[0005] In the past, the issue of picture quality versus the number of programs to be carried over a given bandwidth has been addressed by employing statistical multiplexing (see, e.g., U.S. Pat. No. 5,216,503 to Paik et al.). Services such as video-on-demand have also been provided where the bandwidth necessary to convey such information is accommodated by additional physical channels.

[0006] Various schemes for selecting between different real-time television signals are also known in the prior art. For example, U.S. Pat. Nos. 5,155,591 and 5,231,494 describe targeted advertising that is delivered in real-time. Such advertising may be selected by a decoder employing customer “profiles”, as disclosed in U.S. Pat. No. 5,758,257. However, such schemes are predicated on the transmission of concurrent real-time (analog and digital) TV programs.

[0007] U.S. Pat. No. 5,619,337 describes a system for recording MPEG-2 transport streams. The goal is to address the recording of a single program stream extracted from a multi-program transport stream onto a device such as a digital VCR for subsequent playback.

[0008] It would be advantageous to provide a bandwidth-efficient system for conveying digital television programs and/or other content (e.g., multimedia) in their original format, which may comprise, for example, MPEG-2 transport packets. The system should allow the programs and other content to be provided on a non-real-time basis for later use by a recipient. Such content should be able to be easily received, recorded on a digital VCR or otherwise stored, and played back at the user's convenience. It would be still further advantageous to permit the communication of multiple digital TV programs and/or other content with little or no impact on the quality of real-time programs delivered via the same transport stream.

[0009] The system should enable the delivery of pre-encrypted content to one or more decoders in non-real-time, wherein authorization data is provided to the decoders at a common time to enable decryption and playback.

[0010] The present invention provides a method and apparatus enjoying the aforementioned and other advantages.

[0011] The present invention makes use of allocated bandwidth and/or bandwidth made available by variable bit rate (VBR) program streams to provide a bandwidth-efficient scheme for conveying content such as digital television programs and/or multimedia information in their original form (e.g., MPEG-2 transport packets). A large portion of currently available MPEG-2 decoder hardware and/or software may be used to decode content delivered per the invention. The invention also, in effect, allows the delivery of multiple non-real-time digital TV programs and other content with little to no impact to the quality of content delivered in real-time via the same transport stream.

[0012] A particular method in accordance with the invention provides data of at least a first program of a first data stream in an input transport stream, where the transport stream includes data of at least a second data stream. The method includes the steps of: extracting null packets and/or nonessential packets of the first data stream, providing an annotation packet that provides information regarding the extracted packets, and inserting the annotation packet and program packets of the first program into the input transport stream in place of null packets of the second data stream to form an output transport stream.

[0013] A decoding method is provided for processing an output transport stream that includes data of at least a first program and a second data stream. The method includes the steps of: recovering program packets of the first program, storing the recovered packets at a storage device, recovering an annotation packet from the output transport stream, and regenerating a number of null packets of the first program based on the recovered annotation packet to provide regenerated data that is suitable for decoding. The output transport stream is formed at an encoder by extracting null packets and/or nonessential packets of a first data stream that carries the first program, and inserting the annotation packet and the program packets of the first program into an input transport stream in place of null packets of the second data stream. The annotation packet provides information regarding the extracted packets.

[0014] Corresponding apparatuses are also presented.

[0015] FIG. 1 is a block diagram of a Data Stream Generator in accordance with the invention.

[0016] FIG. 2 is a block diagram of a Data Stream Player in accordance with the invention.

[0017] FIG. 3 depicts an example of an MPEG-2 transport stream prior to and after, processing by the Data Stream Generator in accordance with the invention.

[0018] FIG. 4 illustrates the transport streams and data provided at various points in FIG. 1, including the use of annotation packets, in accordance with the invention.

[0019] The present invention may be implemented using two key elements, namely a Data Stream Generator/encoder and a Data Stream Player/decoder. The Data Stream Generator, which is responsible for extracting real-time programs, storing them and re-inserting them as non-real-time data, can exist as a standalone product for use, e.g., with any MPEG-2 encoder or multiplexer, including the DigiCipher® II encoder products available from a division of Motorola Corporation that was formerly known as General Instrument Corporation, Horsham, Pa., U.S.A. MPEG-2 is described in ISO/IEC IS 13818-1, International Standard, MPEG-2 Systems. The invention can also be used with data that is provided according to, e.g., Draft ATSC Data Broadcast Specification, ATSC T3/S13 Doc. 010, Rev. 0.20, Feb. 9, 1999, DVB Specification for Data Broadcasting, ETSI EN 3-1 192 V1.1.1 (1997-12), or other digital packetized data formats.

[0020] Furthermore, the invention can be implemented in hardware, firmware, software, or any combination thereof. Additional processing such as transcoding (to reduce the bit rate of the original programs) can also be applied to reduce the delivery time of program data to the receiving device, reduce the required storage capacity, and/or to allow for multiple programs to be delivered concurrently.

[0021] The Data Stream Player may comprise, e.g., an MPEG-2 decoder or decoder enhancement which receives the transport stream, extracts and stores the data stream (based on criteria established by its user or the transport stream provider), and decodes the data stream when initiated by the user or transport stream provider.

[0022] Moreover, the Data Stream Generator and Player can employ access control whereby the non-real-time data may only be decoded when the proper authorization is provided to the Player, even if the non-real-time stream has already been delivered and stored. Content can then be securely provided (e.g., by using entitlement control messages—ECMs) in advance of its “official” release, allowing the content to be immediately played in its entirety upon official release. In this manner, for example, new releases of digital movies and other premium entertainment can be delivered in advance for later viewing. The present invention can also deliver content such as targeted advertisements, news and the like in non-real-time to a decoder for subsequent playback based, for example, on a customer profile.

[0023] The present invention, and in particular the use of “annotation” packets as described hereinafter, can also be used to facilitate the recording of digital content on a digital VCR or the like, as it reduces content storage requirements and allows additional information to be conveyed about the content. Such information can be used, for example, to facilitate playback control (e.g., targeted ad profile data, location of I-frames for fast forward/rewind functions, etc.), and for transmission error handling.

[0024] FIG. 1 illustrates a Data Stream Generator (DSG) 100 in accordance with the invention. A main transport stream (TS) input, provided from audio/video data at a “live” encoder/multiplexer 110, and, optionally, at least one secondary TS input, are provided to the DSG 100. A switch 115 provides either the main or secondary TS input to an optional program decryption function 120 and an optional program transcode function 125. The transcode function 125 is used when it is desired to reduce the bit rate of the input program. Decryption is used when an encrypted input is received and it is desired to transcode it.

[0025] A nonessential and null packet extraction and annotation packet generator 130 removes nonessential and/or null packets from the input program and forms corresponding annotation packets in accordance with the invention. Null packets are inserted at an encoder to maintain a desired bit rate, whereas the nonessential packets may include other program packets that are not used. For example, a TS may be distributed nationally by satellite to a number of different headends of a cable television network. Each network may have its own program line up that does not use all of the received programs. The unused program are therefore nonessential to that associated network.

[0026] The annotation packets provide information regarding the null and/or nonessential packets by providing information that describes them, such as the number of null and nonessential packets. Preferably, one annotation packet provides information for several null nonessential packets. Successive annotation packets are used for successive respective groups of null and/or nonessential packets. An associated storage device 132 may store the processed stream with the annotation packets.

[0027] The program packets (e.g., comprising audio and video data) and annotation packets are output from the function 130 to an optional program encrypt function 135. These packets together form a processed or condensed data stream, which is a condensed version of the packets that were output from the switch 115. At a data stream insertion function 140, the packets of the processed stream are inserted into the main TS, e.g., using multiplexing. Specifically, the processed stream packets replace null packets of the main TS input. Thus, there is no loss of important data in the main TS.

[0028] Note that the processed stream may be inserted back into the TS from which it was derived. For example, when the switch passes the input from path “all”, the processed stream is derived from the main TS input, and is inserted back into the main TS input. Or, the processed stream may be derived from the secondary TS input (path “c”) and inserted into the main TS input.

[0029] The components of the DSG 100 are under the control of a central controller 150, e.g., via a bus 155. The controller 150, in turn, may be responsive to a system controller 160, e.g., that receives an operator input. For example, an operator may set parameters for transcoding, decryption and encryption, or to specify nonessential programs, e.g., according to a PID.

[0030] The DSG 100 extracts content (e.g., MPEG-2 video) from any of its MPEG-2 transport stream inputs, which can be “live” (e.g., received from a broadcast) or “off-line” (e.g., retreived from storage), processes the data (as described below), and replaces null packets present in the “live” transport stream with the processed, non-real-time-data. This processed data may be delivered over an indefinite period, e.g., using carouseling, which involves periodic, repeated transmission of data.

[0031] FIG. 2 illustrates a Data Stream Player (DSP) 200, also according to the invention. The DSP 200 may be embodied as a set-top box decoder or the like at a user's location. The DSP 200 receives a transport stream (TS) input, such as the TS at path “b” in FIG. 1, at a data stream retrieval function 210 and at a switch/splicer 250. The TS includes real-time program packets of one or more programs (e.g., channels or services), along with the non-real-time insertion stream packets, including the annotation packets thereof. The data stream retrieval function 210 extracts the non-real-time program data and associated annotation packets for storage.

[0032] The program player 230 performs the function of regenerating null packets to limit the amount of storage 220 required. In particular, the player 230 analyzes the annotation packets to regenerate a number of null packets based on the number of null and/or nonessential packets that are described by the associated annotation packets. Error handling procedures may also be applied, as discussed elsewhere. The regenerated stream is forwarded to a storage device 220 for subsequent retrieval. Successive portions of the stream may be stored while additional portions continue to be received and processed at the function 210. A program player 230 initiates playback of the stored data at the storage device 220, e.g., in response to a user request, by forwarding the data to the switch/splicer 250, with optional decryption, if required, at a program decryption function 240.

[0033] In a switch mode, the switch/splicer 250 provides either the original TS input or the retrieved program data to a demux 260 for subsequent processing at a video decoder 270 and an audio decoder 280. The demux routes the video and audio packets to the appropriate decoder and, may also demux specific program packets of the TS, e.g., according to a PID and a desired channel that is tuned by the user.

[0034] In a splice mode, the switch/splicer 250 splices packets of the retrieved insertion stream with the TS input. Splicing may be performed by using splice messages (“cues”) present in the real-time or non-real-time program streams. Switching implies a “hard-switch” that may result in a brief, noticeable disruption (e.g., picture roll, audio pop, etc.).

[0035] A user interface 290 is provided for allowing the user to tune or playback desired programs.

[0036] Thus, in addition to supporting the decode and display of “live” (e.g., real-time) programs, the Data Stream Player 200 retrieves selected non-real-time programs and stores them for future playback. The storage and playback may be initiated locally or remotely, depending on the application. For example, in the case of movies, the user of the Data Stream Player 200 may specifically select which programs to store and view, e.g., via the interface 290. In the case of advertisements delivered as MPEG-2 programs, the content provider (e.g., transport stream supplier, network operator or the like) may want to specify which advertisements are stored and viewed, based on the attributes of the Data Stream Player (e.g., location of the player 200, user preferences, user demographics, etc.) A content provider control function 295 may be used to communicate with the content provider for this purpose.

[0037] As show in FIG. 2, the Data Stream Retrieval function 210 retrieves and stores the selected program data stream (including annotation packets) from the input transport stream. It may make use of the annotation packets to replace erroneous packets. At a time specified by the Data Stream Player user or the content provider, the Program Player 230 extracts a selected program data stream from storage, determines the original transport stream rate of the program, and clocks packets out at that rate. This is accomplished using information contained in the annotation packets to regenerate the null packets of the original transport stream, thereby allowing existing hardware and/or software to decode and display the selected program. The Program Decryption function 240 may also be employed to decrypt the program after it is generated by the Program Player 230, allowing the program to be stored in its encrypted form until the appropriate entitlements are provided by the content provider, e.g., using known access control techniques. This allows sensitive material to be securely delivered prior to the time of its intended use.

[0038] Note that the Data Stream Generator and Data Stream Player described herein may be combined to provide an efficient method for storage and playback of programs obtained from single and multi-program transport streams.

[0039] FIG. 3 depicts an example of a transport stream prior to and after processing by the DSG 100. An instantaneous bit rate versus time is shown. A total instantaneous bit rate prior to a data insertion time 335 is the sum of bit rates for a component 305 that includes program-specific information, system information, other programs, etc., real-time program data 310 (e.g., including audio and video), and null and other nonessential data 330. After the data insertion time 335, the total instantaneous bit rate is the sum of bit rates for the data component 305, real-time program data 310, and the inserted non-real-time program data 340. Note that the real-time program data may transition between successive events (e.g., programs, commercials, news announcements, and the like).

[0040] In this example, the video stream for the real-time program(s) is depicted as a variable bit rate (VBR) stream, with peaks and valleys in the instantaneous bit rate that occur based on factors such as picture complexity, picture coding type and the like. However, the invention may also be used where the real-time program(s) have a fixed bit rate allocation.

[0041] Moreover, although only one non-real-time data stream is shown, the DSG may deliver multiple substantially concurrent non-real-time data streams in its output. Additionally, these data streams may have fixed and/or variable bit rates.

[0042] Also, note that the figure is not to scale, as there are typically a number of real-time channels in a TS, such that the region 305 consumes a larger proportion of the TS's bandwidth than indicated. However, the invention is suitable for use when there is only one real-time channel in a TS.

[0043] FIG. 4 illustrates the transport streams and data provided at various points in FIG. 1 in accordance with the invention. As shown in FIG. 1, the DGS 100 replaces MPEG-2 null and other nonessential packets received via input “a” with processed program data received via input “a” and/or input “c”. As mentioned, the Nonessential and Null Packet Extraction and Annotation Packet Generator function 130 replaces nonessential and null packets with annotation packets, and provides the resulting processed stream to the storage device 132 and/or the Data Stream Insertion function 140, which replaces null packets received at input “a” with this processed stream data.

[0044] As an example, FIG. 4 illustrates the transport streams and data provided at points “a”, “b”, “c”, and “d” of FIG. 1. The data 400 represents the real-time TS at “a” or “c”, e.g., for a first event (e.g., a program, commercial, etc.) The data 410 represents non-real-time insertion data for the first event which is formed from processing the data 400. The data 420 represents a real-time TS at “a”, e.g., for a second event. The data 430 represents a real-time TS at “b” which is formed from inserting the packets of the data 410 in place of null packets of the data 420.

[0045] Each packet includes an identifier indicating whether it is a video packet “V”, an audio packet “A”, a null and/or nonessential packet “N/N”, a null packet “NU”, or an annotation packet “AP”. Additionally, the subscripts indicate a transport stream, and a packet sequence for that type of packet in the TS. For example, V_1,2designates a second video packet in a first TS.

[0046] Each packet may be an 188-byte MPEG packet, although other packet formats and sizes may be used.

[0047] The data 410 comprises the video and audio packets from the stream.400, and also provides an annotation packet AP_1,1that describes, and substitutes for, the null and/or nonessential packets N/N_1,1through N/N_1,7. A_1,3is the last packet covered by AP_1,1. That is, in a specific example, the annotation packet provides information regarding any null and/or nonessential packets in a group of, e.g., eighteen packets in the data 400. The data 410 is essentially a condensed version of the data 400.

[0048] A subsequent annotation packet AP_1,2describes a subsequent number of null and/or nonessential packets in a subsequent group of packets of the data 400, starting at N/N_1,8. . . .

[0049] The null packets in the stream 420 are replaced by the video and audio packets, and the annotation packet AP_1,1from the stream 410. In particular, as shown at 430, V_1,1replaces NU_2,1, V_1,2replaces NU_2,2, and so forth. The video and audio packets of the second TS, V_2,1through V_2,5,and A_2,1through A_2,3, respectively, are maintained in the stream 430 at the same position as in the stream 420.

[0050] In FIG. 4, the shaded packets represent the non-real time, insertion packets.

[0051] Each annotation packet (AP) provides information regarding the number of null and/or nonessential packets from the original transport stream that it replaces. It may also contain a timecode which describes the location of the next program access unit (e.g., the next audio or video packet) within the event, allowing the Data Stream Player to identify transport packets indicated to be erroneous (e.g., via the packets' transport_error_indicator bit) and replace them if and when the correct packets are received. As known from the MPEG-2 standard, transport_error_indicator is a one-bit flag. When set to ‘1’, the flag indicates that at least one uncorrectable bit error exists in the associated TS packet.

[0052] The annotation packet may also contain information to facilitate user navigation (e.g., the location of the next I-frame for fast-forward/rewind operation, the location of the next scene in a movie, the start of the next song, etc.).

[0053] The Data Stream Generator may encrypt the non-real-time packets. In one aspect of the invention, packets generated by the Nonessential Packet Extraction and Annotation Packet Generator 130 are encrypted using a key for which authorization data, such as an entitlement management message (EMM), is not distributed to the Data Stream Player until some future time, allowing future concurrent activation of all Data Stream Players.

[0054] An example of the use of annotation packets in accordance with the invention is seen by referring to the streams 400 and 410. Annotation packets contain program-specific information and are conveyed via a unique packet identifier (PID). Each annotation packet describes the number and location of null and/or nonessential packets (i.e., packets not associated with the selected program stream) within the original transport stream. For example, AP_1,1may include information indicating that three null and/or nonessential packets (N/N_1,1through N/N_1,3) are provided prior to the first (next) program packet (A_1,1), that two additional null and/or nonessential packets (N/N_1,4and N/N_1,5) are provided prior to the second next program packet (V_1,5), and that two further null and/or nonessential packets (N/N_1,6and N/N_1,7) are provided prior to the sixth next program packet (V_1,8).

[0055] Annotation packets may also include information to describe the original transport packet rate and identify the location of packets within a program. For example, a time code or codes may be included in the annotation packet to facilitate navigation to user-specified times. Such a time code may also be useful in locating and replacing packets received erroneously (e.g., in applications where the Data Stream is repeated or “carouseled”). Annotation packets may contain all, or a portion, of an Annotation message formatted as a private section using the example syntax shown in Table 1 below. In the tables below, the Mnemonic “uimsbf” stands for “unsigned integer, most significant bit first” and the mnemonic “bslbf” stands for “bit string, left bit first.”

[0056] section_syntax_indicator is set to ‘0’, indicating that private data bytes exist immediately following the section_length field.

[0057] checksum is set as defined by ISO/IEC 13818-6. It is calculated over the entire section.

[0058] The following descriptors are defined for use in the annotation section:

[0059] number_of_null_packets_string is a string of bytes that describe the number of null or nonessential packets (in the original transport stream) that precede each of the service packets following the annotation packet that contains this descriptor and prior to the next annotation packet. Values are represented as follows:

[0060] 0x00-0xFE: 0-254

[0061] 0xFF00-0xFFFE 255-509

[0062] 0xFFFF00-0xFFFFFE 510-764 etc.

[0063] A value of zero indicates that no null or nonessential packets exist. A value of 0xFF is used as an escape code to extend the length of this field to describe null packet sequences in excess of 254 bytes. Each subsequent value of 0xFF describes an additional 255 bytes of null or nonessential packets. For example, a number_of_null_packets_string value of 0x1000FFFF0300FE0A indicates the following:

[0064] The 1^stservice packet following the annotation section is preceded by sixteen null and/or nonessential packets in the original transport stream.

[0065] The 2^ndservice packet is preceded by zero null and/or nonessential packets (i.e., it is contiguous with the 1^stservice packet)

[0066] The 3^rdservice packet is preceded by 513 (255+255+3) null and/or nonessential packets.

[0067] The 4^thservice packet is preceded by zero null and/or nonessential packets (i.e., it is contiguous with the 3^rdservice packet).

[0068] The 5^thservice packet is preceded by 254 null and/or nonessential packets.

[0069] The 6^thservice packet is preceded by 10 null and/or nonessential packets.

[0070] The null_packet_identifier_descriptor may describe up to descriptor_length subsequent service packets. In typical applications (where the occurrence of 255 contiguous null or non-service packets is rare); an annotation packet can describe up to 174 service packets. This represents a transmission and/or storage overhead of 0.57%. This can be seen by noting that the maximum size of a null_packets_string, if contained entirely within one 188-byte annotation packet, is 174 bytes. So, in the best case, one annotation packet may be used to describe the number of N/N packets that precede each of up to 174 packets. The best case means the number of contiguous N/N packets between each service packet is less than 255; this is typically the case. So, one AP can describe up to 174 service packets, and 1/174=0.00575 or 0.57% is the overhead.

[0071] transport_packet_rate is the original transport packet rate in bits per second and is optional. It may be used by the Data Stream Player to control its output transport stream rate.

[0072] It should now be appreciated that the present invention provides a system for extracting programs from single and multiple-program transport streams for subsequent non-real-time (“background”) transmission in a bandwidth-efficient manner. Such “programs” may include anything from targeted commercial advertisements to broadcast full-length movies. A non-real-time program is formed from a real-time transport stream by providing annotation packets that describe the original packet separation (timing) of the program. The non-real-time packets are inserted into a real-time transport stream by replacing null packets therein. At a decoder, the annotation packets are used to maintain the proper spacing between the program packets.

[0073] A system is also provided for delivering pre-encrypted programs in non-real-time. In this case, entitlement control messages (ECMs) may be included; however, entitlement management messages (EMMs) necessary to decrypt a program or programs may not be distributed until some future time specified by the program provider. This allows mass distribution for storage prior to a formal program “release.” After “release” of the content, the decoder, if authorized, is permitted to decrypt the program(s).

[0074] Although the invention has been described in connection with a preferred embodiment, numerous adaptations and modifications may be made thereto without departing from the scope of the invention as set forth in the claims.