Processing coded video

[0002] The present invention relate to processing of coded video signals, particularly but not exclusively MPEG format compressed video. The invention is particularly, but not exclusively, concerned with real-time editing of video signals.

[0003] A problem with transmitting or storing video signals as a compressed or coded video data stream is that it is not normally possible for a decoder to commence decoding a new data stream instantaneously, but there is normally a delay of several frames while the decoder establishes synchronisation. For example with an MPEG data stream, a video sequence is sent as a series of groups of pictures (“GOP”), each GOP comprising an intra-coded frame (“I frame”) and one or more frames predicted from the I frame, either by forward prediction (“P frames”) or by bi-directional prediction (“B frames”). A decoder cannot commence decoding of the sequence until it receives an I frame. In addition, frames are buffered and processed within a decoder, so there can be an appreciable and unpredictable delay between supply of a video data stream to a decoder and output of the sequence of video frames.

[0004] For practical purposes, this presents a problem only when the decoder first receives a new signal; once synchronised, the decoder should continue to produce an output reliably. However, difficulties are encountered when it is desired to switch from one coded video source to another. In particular, a decoder buffer can overflow or underflow if the data stream input to the decoder is simply switched from one source to another.

[0005] The problems associated with switching between coded video sources have been recognised, and techniques have been proposed for switching between compressed bitstreams with minimal output degradation at the point of switching. An example of such a technique is described in our International Application No. WO 97/08898.

[0006] Prior art techniques have generally concentrated on dealing with problems occurring at or about the point of switching, and generally aim to provide a “good compromise” solution.

[0007] In general terms, the inventor has proposed that coded video source material is modified, if necessary, to adjust timing of output of frames from a downstream decoder, and preferably to establish and maintain a desired synchronisation of a downstream decoder. This can be achieved by setting the timing of the video source material (for example by explicitly adjusting values of timing fields contained in the data stream, or by altering the time at which the data is actually output to the decoder, or both) based on a studio reference clock. This can also (additionally or alternatively) be achieved by inserting “synthetic” frames (frames which have been generated and added to the original video source material) within the output data stream, so that a particular frame is displayed at a particular time. It is noted that if decoder synchronization is maintained in this way, the timing of display of particular frames need not be explicitly specified.

[0008] Looked at from another point of view, with the invention, the timing information contained in video source material (this is normally included to enable reliable playback of that material as intended) may be deliberately altered or additional frames may be added. Conventionally, it is not generally considered desirable to tamper with the internal, self-consistent, timing information contained in coded video data. However, this novel basic technique may provide numerous advantages, as will become clear as the description proceeds. Surprisingly, although the invention is in its most general terms not directed specifically at the switching of bitstreams, one advantage is that switching from one video source to another can be facilitated if both sources are synchronised. Another advantage is that playout of coded video material may be more readily controlled, for example for starting or pausing at a particular time.

[0009] According to a first, aspect, the invention provides apparatus for outputting a coded video sequence comprising:

[0010] means for receiving a coded video sequence to be output;

[0011] means for outputting the coded video sequence in a form enabling real-time decoding of the sequence to produce a sequence of pictures;

[0012] means for processing the coded video sequence and/or adjusting the timing of said outputting to produce decoded picture output from a decoder receiving the output sequence at a selected timing.

[0013] In this way, contrary to conventional compressed video sources, the timing information contained within the original sequence, or the actual time of output can be controlled deliberately based on external criteria, rather than in dependence on the timing data contained with the video sequence. This facilitates synchronisation of multiple video sources.

[0014] The video input sequence will typically be received from storage such as a disk, for example as a computer data file, and may be sent over a computer network. However, the sequence may be received in real-time, for example off-air, and may be contained in a multiple transport stream containing several sequences and other data. Where the data is received in real-time, the input data will usually need to be buffered, to enable differences in timing between input and output, and, of course, output cannot occur before the data has been received. As will be appreciated, as the size of the buffer increases, the distinction between receipt of “real-time” data and access of stored sequences diminishes.

[0015] Preferably, the apparatus includes means for receiving a studio reference clock, the processing and/or the timing of the outputting being controlled in dependence on timing information derived from the studio reference clock. This feature may facilitate synchronisation with other apparatus within a studio.

[0016] The studio reference clock is preferably supplied as a relatively high frequency clock counter having a relatively large maximum value, preferably at least 50 kHz and at least about 32 binary bits, “wide” (by which is meant the information content; the clock value may be supplied in another format, such as Binary Coded Decimal, in which case about 9 or 10 nibbles or digits would be required). In a preferred practical implementation, the clock is in a format that is readily or directly convertable to the format of timing fields within the video sequence. For example, the studio clock may be supplied directly as MPEG PCR values.

[0017] The, apparatus may include means for determining an effective clock frequency for the output coded video sequence based on the rate of change of clock values (for example PCR fields within an MPEG video sequence) contained within the output data and means for comparing or synchronising the effective clock frequency to the studio reference clock frequency. This may enable errors in original timing information to be identified and corrected, to achieve playback at a precisely controlled rate.

[0018] The apparatus may also include means for inputting information specifying a desired picture output time of at least one frame or field of a video sequence, said processing and/or adjusting being selected based on the desired picture output time and the studio reference clock so that said at least one frame or field is output from a decoder at the desired picture output time. In this way, the apparatus is able to look ahead, and prepare a video sequence for output at a precisely defined time, which solves a common problem in playing back video sequences live. It has been appreciated that solutions to problems of run-up or lead-in commonly employed in analogue storage systems (recording a 5 second lead-in sequence prior to The first frame required and commencing playing of the sequence 5 seconds before the frame is required) cannot be reliably used with coded sequences, as there is a somewhat more unpredictable delay before a decoder establishes synchronisation.

[0019] One way in which picture output time can be set is by altering one or more values in timing fields contained within the video sequence (for example PCR (programme clock reference) or PTS (presentation time stamp) or DTS (decoding time stamp) or vbv_delay (video buffer variable delay) fields in a MPEG data stream). In place of specifying the output time directly, a measure of the output time may be specified, or a PTS or DTS value may be explicitly specified.

[0020] Since some decoders may not respond to the values explicitly set in all of the PCR, PTS or DTS fields, the apparatus may (additionally or alternatively) adjust the timing at which data output begins to ensure that decoding takes place and is completed at a desired time. This may be achieved by estimating the required delay between transmission of data and commencement of picture output and adjusting data output time based on this estimated delay, relative to the desired picture output time. The delay may include an offset time to take into account delays in a decoder, and also propagation delays in cables and the like in the video signal path, and a variable delay for the frame concerned, for example equal to or based on the vbv_delay value originally specified or calculated. The apparatus may include means for storing characteristic propagation delays for a plurality of different types of decoder, and may also include means for determining propagation delays for a known length of cable. The apparatus may be arranged to set a PTS value to a time in advance of a desired output time, based on a decoder propagation delay.

[0021] Preferably, the apparatus includes means for correlating a studio reference clock time in a first format with a desired output time in a second format. In this way, precise control of timing may be attained whilst facilitating control by a user or by other editing apparatus. Apparatus for converting between formats is independently provided in another aspect of the invention.

[0022] In a further aspect, the invention provides the above mentioned outputting apparatus included in a multiple source video system comprising a plurality of video sequence sources, at least one video sequence source comprising such outputting apparatus arranged to receive a respective coded video sequence and having a decoder coupled to the output thereof to produce a respective decoded video sequence output therefrom; means for supplying a studio reference clock to the or each outputting apparatus; means for receiving editing information specifying timing of changes between video sequence sources; means for supplying a selected video sequence source to a video output channel based on said editing information; and means for supplying desired picture output time information to the or each outputting apparatus based on said editing information.

[0023] In addition to or instead of means for inputting a studio reference clock, the apparatus may include means for generating a sequence of one or more frames to be inserted in the output sequence, the means for processing including means for inserting the generated sequence into the coded video sequence to adjust the timing of picture output of frames of the coded video sequence, the generated sequence being selected to maintain or establish synchronisation of a buffer of a decoder receiving the output sequence. With this feature, it may become possible to effect timing adjustments over longer periods of time, and to maintain decoder synchronisation while a sequence is paused. Other effects, such as slow-motion playback may also be possible, and cutting from one sequence to another may be facilitated. With this feature, the decoder output timing need not be explicitly selected; the maintenance of decoder synchronisation is, for the purposes of this specification, considered to be equivalent to implicit selection of decoder output timing.

[0024] The apparatus may be arranged to insert the generated sequence before a given frame of said coded video sequence, the size of the generated frames and/or the timing information associated with, or the time of transmission of, the generated and/or said given frame being selected so that, on commencement of decoding of said given frame following decoding of the generated sequence, the buffer occupancy of a decoder receiving said output video sequence will be substantially at a desired value. In this way, buffer underflow or overflow problems may be alleviated, and buffer synchronisation established and maintained.

[0025] The generated sequence may be inserted before said given frame and after at least one preceding frame in the coded video sequence, wherein the desired value of buffer occupancy is set to be substantially equal to the buffer occupancy expected if no generated sequence were inserted between the preceding frame and said given frame. This enables buffer synchronisation to be maintained during an interruption.

[0026] It will be appreciated that the invention can effectively provide apparatus capable of outputting a real-time bitstream (a “video pump”) in which synchronisation of the bitstream can be controlled independently of the video data source. Thus, another advantage the invention may provide is that a source of video data, for example a series of computer readable files on a disk or a recorded MPEG transport stream, can be separated, for example over a computer network, from the apparatus for outputting a real-time bitstream. This is provided in a further aspect, in a system for supplying real-time coded video data comprising means for storing the coded video data; a data transmission link for supplying the stored coded video data in response to requests for data; apparatus according to the first aspect arranged to request and receive data across said data transmission link and to output synchronised real-time coded video data therefrom.

[0027] The invention extends to related methods of operation.

[0028] Embodiments of the invention will now be described, by way of example only, with reference to the drawings, in which:

[0029] FIG. 1 is a block diagram of a multiple source studio system embodying The invention;

[0030] FIG. 2 is a block diagram of a video output apparatus employed in the studio system of FIG. 1;

[0031] FIG. 3 shows decoder buffer addressing for decoder initialisation using synthetic bitstream in CBR (constant bit rate) mode;

[0032] FIG. 4 shows decoder buffer addressing for decoder initialisation using synthetic bitstream in using a synthetic bitstream in VBR (variable bit rate) mode;

[0033] FIG. 5 shows decoder buffer addressing for decoder pausing using a synthetic bitstream in VBR mode; and

[0034] FIG. 6 shows decoder buffer addressing for bitstream splicing using a synthetic bitstream.

[0035] In the following discussion, specific reference is made to MPEG sequences; it is to be understood that the various features presented may be applied to other coded video sequences unless otherwise stated. As background we begin by reviewing some particular features of MPEG sequences; this is useful for explaining one particular problem which different preferred features of the embodiments alleviate, and for providing guidance to broader application of the specific features to other video sequences.

[0036] As mentioned above decoding of a new sequence can only commence when a new I frame is completely received; I frames provide entry points to a sequence. Coded frames are of variable length, with I frames normally being much larger than P and B frames. If the data is transmitted over a link whose capacity is sufficient to ensure that the average data rate is enough to supply frames at a desired frame rate, it is likely that an I frame will take several frame periods to transmit, during which time the decoder will have nothing to display. There is a significant but unpredictable delay before initial output of a decoded sequence following supply of a sequence to a decoder. Once synchronisation of the decoder is established, the decoder will continue to operate reliably regularly producing frames. An exception occurs if for some reason the buffer contains insufficient frames to keep the decoder supplied while a frame (for example a large I frame) is being written; the decoder will run out of frames to display and this condition is known as buffer underflow. Another problem is that signal processing within the decoder takes a finite time, which is generally substantially constant for a given decoder, but may vary from decoder to decoder.

[0037] It will be appreciated that the embodiment of the invention may be readily applied to other coded video sequences which have one or more analogous properties, for example, specific entry points at which decoding can commence, variable frame size, timing fields specifying when frames are output, or a need to establish or maintain decoder synchronisation.

[0038] The above summary has concentrated on the data content of a single video programme. In practice, the apparatus may receive computer data files or may receive an MPEG transport stream containing multiple programmes or a single programme, and other header information. Where a transport stream is received, other fields and information (for example PATs, PMTs etc.) may also be processed by the apparatus. Such processing is not, however, germane to the present invention, and for the purposes of the following discussion it will be assumed that the data has been pre-processed and buffered as necessary for correct decoding of an individual programme. Of course, a single apparatus may be capable of processing more than one video sequence in parallel.

[0039] It is noted that the problems mentioned above relating to initial decoder synchronisation can often be tolerated if it is simply required to view a single sequence; the problems become critical in a multiple source video system, for example in a broadcast studio, where frequent changes between video sequence sources are required. Video outputting apparatus embodying the invention will thus be described in the context of a multiple source system, but it is to be appreciated that the apparatus is not limited to such an application.

[0040] Referring to FIG. 1, a multiple source video system comprises a video pump 10asupplying coded real-time video to a decoder 20a, and receiving coded video data from a storage medium 30a. In this embodiment, the video pump 10aand a second video pump 10bsupplying a second decoder 20bare both connected to the storage medium 30a(for example a computer file server) over a computer network, This embodiment also includes a third video pump 10csupplying a third decoder 20cand receiving data from a dedicated data source 30b, and a fourth video pump 10dreceiving “live” coded video via an off-air receiver 32 which includes a demultiplexer for extracting a single programme stream from the received data and buffer 34 having a capacity sufficient to store several minutes (here about 30 minutes) of video data, the fourth video pump supplying a fourth decoder 20d. The outputs of all decoders are connected to a selector 40 which selects an output from one of the decoders; the output may be broadcast (either in analogue or digital form), or may be stored or re-coded. In this embodiment, the decoders produce an output in decoded digital form including information concerning coding decisions based on the original coded video (MOLE format), to enable re-coding with minimum loss of quality. However, other decoding, for example to full analogue may be employed. A further video source 22 producing an uncoded signal of similar format to the decoder output, for example from a studio television camera is also supplied as an input to the selector 40. For synchronisation purposes (explained further below), each video pump receives a timing signal from a studio reference clock 12. The studio reference clock typically supplies a 42 bit SSTC counter value based on a 27 MHz clock, comprising a 33 bit 90 kHz (equal to the clock frequency divided by 300) base value and a 9-bit (modulo 300) extension value clocked at 27 MHz. In this embodiment, the clock 12 also converts the 42 bit studio reference clock times into SMPTE time codes and supplies both along a common bus at regular intervals. Each video pump also receives playout decision information from an edit/playout controller 14. The edit/playout controller typically specifies edit/playout times in SMPTE codes. The SSTC and related SMPT time values can be supplied separately, but preferably the SSTC bus provides information enabling correlating the two. Of course, other variants are possible; for example, the studio clock values may be converted into SMPTE values, and it is not necessary to supply SMPTE values if the corresponding studio clock counter values are supplied by the playout controller.

[0041] Editing or playout decisions may be specified in advance, for example as a play schedule for a period of time (for example “At 9.00 pm and 15.0 seconds, play the video sequence entitled ‘Film A’ commencing at Frame 0 via video pump 10a″), or may be modified or set in real-time. For example, an immediate request to play a sequence may be translated into a request to commence playing in exactly 5 seconds time.

[0042] The editing or playout decision information is sent to both the selector 40 and the video pumps 10. Where a video pump only has access to a single coded video source, for example the video pump load receiving a single buffered off-air video sequence, the editing information need not specify the description of the source; otherwise, the description of the source may be sent to the video pump which may request the appropriate data from the storage medium, or may be sent directly to the storage medium.

[0043] The selector simply switches the output from one source to another at the specified time, and may be entirely conventional. The appropriate video pump determines when the coded video sequence must be played, and how to ensure synchronisation so that the decoder is producing the correct output at the time of the switch. This can be achieved in a number of ways, as will be described in more detail below.

[0044] Referring to FIG. 2, apparatus for outputting coded video from a coded video source (a “video pump”) 10 will now be described. Programme or video data is received from a data input, for example a computer network, and is fed to a network buffer 110 which smooths discontinuities in the time of arrival of the data resulting from server and network latency. The precise method of feeding data to the network buffer is not critical, and forms of itself no part of the present invention; any suitable protocol used in similar computer networks may be used, provided of course that the buffer always contains enough data for the output to be maintained. Typically, the apparatus will be controlled by a computer, which will also control the feeding of data from a network server or disk. The data is fed to a bitstream parser 111, which detects appropriate fields in the video and system layers for re-stamping and for switching to synthetic streams. The data is also fed to a delay 112, which compensates for the delay in the bitstream parser 111. A synthetic bitstream store 113 contains versions of synthetic video bitstreams in an appropriate form. A switch 114 is fed with the data from the delay 112 and from the synthetic bitstream store 113, and selects between these sources, ie between the input and synthetic bitstreams.

[0045] A studio system time clock (SSTC) 120 provides a count which can be inserted in PCR fields such that the decoder will synchronize to studio time. The bitstream emerging from the switch 114 is fed to a unit 121, which re-stamps the fields mentioned above, including in particular the PTS/DTS fields. The bitstream then passes to a unit 122, which re-stamps the PCR fields. This is shown as a separate final block in the bitstream, because there must be a minimum of timing jitter between the insertion of the PCR fields and the reception of these fields by the decoder.

[0046] A controller 123 is fed with commands from a play-out control list and by the local SSTC counter 120. This controller controls the following functions:

[0047] the timing of the switch between synthetic and image bitstreams;

[0048] the re-stamping of PTS/DTS fields as well as the other fields listed previously; and

[0049] the re-stamping of PCR fields.

[0050] The interface hardware 121 and 122 re-stamps PCR and PTS/DTS fields for both synthetic and image bitstreams. The values inserted in the PCR fields are such that the following decoder will have an internal decoder clock count which is synchronized to a similar count in the studio. This enables the presentation/display times of particular decoded frames to be synchronized to precise studio times.

[0051] The PTS/DTS values are re-stamped with appropriate values, as described more fully below, in relation to the decoder clock count (DCC) such that:

[0052] there is continuity of PTS/DTS values between synthetic and image bitstreams;

[0053] the decoder buffer does not overflow or underflow; and

[0054] when the DCC is synchronized to the studio system time clock, then the PTS values correspond to the times at which it is desired to display particular frames of the image sequence (e.g. in accordance with a play-out list).

[0055] Continuity of PTS/DTS values may be set by adding an offset to the values in the original coded video sequence; for example, if the first PCR/PTS/DTS value is modified from a value x (stored in the coded video, based on the original coder counter value) to a value y (based on the studio reference clock), all subsequent values may be modified simply by adding an offset (which may be positive or negative) of (x-y) to the incoming values.

[0056] In addition to setting timing values, the data output rate, or timing of outputting of coded frame data, must be set so that the decoder buffer does not overflow or underflow.

[0057] It is noted that in simplified versions of the apparatus, some features may be omitted; for example, if it is merely desired to synchronise an output clock frequency to a studio reference clock, the means for generating synthetic frames may be omitted. Similarly, if precise timing of the decoder output is not required, the studio reference clock input may be omitted, the apparatus still being capable of generating bitstreams adapted for pre-synchronising of a decoder or pausing or slow-motion playback. Most flexibility is, however, assured if the above features are included together.

[0058] There are a number of ways in which synchronisation can be effected, and these will be explained following a discussion of the function of various timing fields within an MPEG sequence; other fields having related functions may be employed in other coded video sequences.

[0059] Each decoder has its own clock, typically running at 27 MHz, producing a Decoder Clock Count (DCC) value which increments at a similar rate to the PCR values in the original data. Owing to tolerances, there may be a gradual drift between a DCC and the original coder clock values, which might accumulate. By transmitting PCR values at intervals in a coded data stream, a coder in a transmitter can effect and maintain synchronisation of a decoder clock to its own clock, Thus, conventionally. DCC values are set in synchronisation with the original data stream, but have no external applicability. The coder may send with a frame a DTS value specifying the time at which the frame should be extracted from the decoder buffer for decoding and a PTS value specifying the time at which the decoded frame should be displayed For B frames, which are not stored in the receiver, the DTS and PTS are identical; for I and P frames, which are stored in the decoder, the DTS precedes the PTS.

[0060] In the above described apparatus, the PCR values in the original data stream are replaced by values based on the studio reference clock. In this way, the DTS and PTS fields may be used to specify an absolute time. Thus, in principle, display of a frame at a particular time may be effected simply by setting the PIS field to the appropriate time (in practice, this may differ in practice from the actual output time due to output delays in a practical decoder, so the apparatus may include means for storing an offset value to be added to or subtracted from values written into this field).

[0061] A complication is that the PTS and DTS fields are not obligatory for every frame, and, not all decoders may use them, particularly after synchronisation is first established.

[0062] Thus a further parameter, the vbv_delay field may be used; this specifies the delay before a frame, particularly an I frame, can be read from the decoder buffer. It will be realized that the vbv_delay can actually be determined from this more general timing; this time is the difference between the time when the frame is received and the time when the frame is removed from the buffer for decoding. Thus we have

[0063] vbv_delay=DTS−DCC(frame write) (Eqn 1)

[0064] The decoder should use the DTS to determine the vbv_delay if the DTS is available; if there is no DTS available, then it should use the vbv_delay value from the frame header. Thus, the vbv_delay value may not be explicitly specified (in an MPEG sequence, a vbv_delay field containing OxFF throughout signifies no specified value) in the video layer, in which case an effective vbv_delay may be calculated based on the DTS field in the PES layer and the actual time of writing of the frame. Throughout this specification, references to an effective vbv_delay value are intended to encompass any such parameters or combinations of parameters which determine the residency time of a frame within a decoder buffer.

[0065] With the above information, it will be seen that playout of a desired frame at a desired time can in principle be achieved without addition of synthetic frames, simply by setting the appropriate timing field values, and commencing output of the data corresponding to the first I frame of the desired sequence in advance of the desired picture output time by an amount equal to the vbv_delay for that frame plus any additional offset delay. A potential drawback of this method is that the decoder only has one chance to synchronise and then must stabilise immediately; if for some reason the data gets corrupted, the beginning of the sequence will be missed. Thus it is usually preferred to effect synchronisation by pre-pending an initialisation sequence of generated or synthetic frames to the original sequence, as discussed further below. Considerations affecting the choice of synthetic frames will now be discussed.

[0066] Generally speaking, it is preferred that at least some of the generated frames are preferably of relatively low information content (as compared to a typical picture frame), so that writing and reading the frames takes relatively little time. In particular, it is preferred if the average data content of each generated frame is at most equal to the amount of data that can be written in a single frame display period, and preferably substantially less (for example at most half).

[0067] More specifically, in an MPEG sequence, a black I frame, or an I frame containing a simple logo (by which is meant a picture the coding of which requires relatively little data compared to an average real picture scene, preferably an amount of data that can be transmitted in approximately a single frame period or less), or a P frame encoding no motion, may be used. If a static or simply moving logo is transmitted, the initial I frame may contain more data than can be sent in a single frame period, but the average data rate for a sequence containing the I frame and one or more P frames will be below the required rate. In this way, frames can be output at an average rate of one frame per frame period and there is flexibility to insert generated frames for prolonged periods without running into buffer underflow and overflow problems, and without recourse to complex buffer occupancy determining methods. Moreover, output of the generated sequence may make it possible to “buy time” in which to output data for a larger frame. In other words, following outputting of the generated sequence, it may be possible to commence outputting data for the next frame of the original coded video sequence (for example an I frame) containing a large amount of data in advance of the desired display time, while the decoder buffer contains data for several synthetic frames to be output, so that the decoder buffer always contains frames for display. Thus, by controlling the size of the synthetic frames (which will usually be small), and/or by adjusting the time at which the frames are output and/or by adjusting the time values specifying the residency time for the frames in the decoder buffer, a desired buffer occupancy can be achieved.

[0068] Specific examples of synthetic frames which may be inserted, for various purposes, will now be described.

[0069] The relevant properties/parameters of a simple synthetic black I-frame may be summarized as follows:

[0070] Prediction mode Intra

[0071] Frame/Field DCT mode Frame mode (either could be used)

[0072] Coefficient values All zero including DC coefficient (or black I frame)

[0073] Picture size Value taken from associated image sequence

[0074] vbv-delay Value taken from first I frame of following sequence

[0075] elementary stream bit-rate Value taken as max. from associated image sequences

[0076] Temporal Reference Set to zero assuming I frame follows GOP header

[0077] Alternatively the I-frame could be coded as a black I/P field pair

[0078] The relevant parameters to be set for each synthetic P-frame would be as follows:

[0079] Prediction mode Frame prediction

[0080] Motion vector Zero, forward, frame vector

[0081] Frame/Field DCT mode Frame mode (either could be used)

[0082] Differential coefficient values All zero

[0083] Picture size Value taken from associated image sequence

[0084] vbv-delay Value taken from first I frame of following sequence*

[0085] * Not appropriate when pausing a decoder using synthetic P-frames—see below

[0086] elementary stream bit-rate Value taken as max. from associated image sequences

[0087] Temporal Reference Increments mononically for successive coded frames

[0088] For output, as is known, the video elementary stream is first packetised into PES packets. These can have many optional fields in the header of each PES packet. For the synthetic bitstream the PES packet header is preferably as simple as possible with as few optional fields as possible. Also, it would be convenient to contain each coded frame (access unit) within a single PES packet.

[0089] At the PES (Packetised Elementary Stream) layer the following fields in the PES packet headers need, in particular, to be set appropriately;

[0090] PTS/DTS fields

[0091] Stream ID field

[0092] This is a relatively straightforward matter; the considerations affecting PTS and DTS values are discussed generally above, and in more detail in relation to specific synthetic bitstreams below, and the Stream ID field must be set to the same as that of the original video stream.

[0093] Having described suitable basic synthetic frames, some functions achievable using synthetic frames will now be described, together with more details of specific suitable sequences.

[0094] A suitable synthetic bitstream to effect initial decoder synchronisation consists of a black I-frame followed by a sequence of synthetic (null) P-frames. The initial frame need not be black, but may contain a simple logo. To ensure synchronisation, several such sequences (including several I frames) may be transmitted. Although such a sequence may only be transmitted a few frames or a few seconds before start of the desired sequence is required, it is equally possible to transmit such a sequence for prolonged periods of time, for example several hours, to keep a decoder synchronised over a prolonged period of time. In such cases, the decoder may also be “paused” in the synchronised state (as discussed further below). At an appropriate time (once the decoder has locked up) a switch is made to the first I-frame of an image sequence bitstream. For example:

[0095] The synthetic frames are shown in italics. From this example it can be seen that for each synthetic I- and P-frame

[0096] PTS=DTS+T  (Eqn 2)

[0097] where T is the video frame period. As mentioned earlier, B frames are not stored in the decoder and are decoded and output without delay, whereas I and P frames needed to decode B frames are normally output in advance of display time. Thus, in the above sequences, B frames have not been numbered, each B frame in the display sequence corresponds directly to the B frame vertically above in the bitstream sequence.

[0098] Suppose that it is required to display the first I-frame (I₀) of the image sequence at time t_disp(I₀) in relation to the SSTC. Then, the PTS field for this frame should be set to t_disp(I₀) less some offset which is decoder dependent and depends on the time taken for sequential-to-interlace conversion etc. i.e.:

[0099] PTS(I₀)=t_disp(I₀)−offset  (Eqn 3)

[0100] The DTS for this frame will be dependent on the number, n_B, of B-frames following this I frame in the bitstream before the next coded P-frame i.e.

[0101] DTS(I₀)=PTS(I₀)−(n_B+1).T  (Eqn 4)

[0102] where T is the video frame period.

[0103] Assuming that the decoder is synchronised, the time at which data for this frame should start being sent to the decoder is then given by:

[0104] t_trans(I₀)=DTS(I₀)−vbv_delay(I₀)  (Eqn 5)

[0105] i.e. the interface should start feeding out to the decoder the first frame of the image sequence when

[0106] SSTC=t_trans(I₀)=t_{display (}I₀)−offset−(n_B+1).T−vbv_delay (I₀)  (6)

[0107] If we assume that the synthetic bitstream is fed to the decoder at an average rate of one coded frame per frame period, then the first frame of the synthetic bitstream should be fed to the decoder at a time given by:

[0108] SSTC=t_trans(I₀)−n_synth.T  (Eqn 7)

[0109] where n_synthis the number of synthetic frames sent in the initialising synthetic bitstream.

[0110] Video data can be sent to the decoder either as a constant bit-rate (CBR) stream or as a variable bit-rate (VBR) stream. In VBR video streams the total bit-rate of the transport stream may be maintained at a constant rate by sending “null” transport stream packets as required.

[0111] Since the synthetic bitstream only requires a small bit-rate in comparison with real video sequence data, in order to maintain CBR operation, it is necessary to add a large number of stuffing bytes per frame to the coded synthetic frames. These will be written into the decoder buffer and discarded when the data is read from the buffer at a time corresponding to the DTS

[0112] A diagram showing an example of the write and read addresses of the decoder buffer in CBR operation is shown in FIG. 3. As will be appreciated, a buffer is in practice accessed cyclically,with wraparound; when a read or write pointer reaches the end of the buffer (address B, where B is the buffer size), it returns to the beginning. For ease of understanding, the buffer addresses are shown as linearly increasing in FIG. 3 (and the subsequent figures), the real addresses will be equal to the indicated address modulo B (i.e. an address of N.B+x is equivalent to an address of x, where N is an integer and x<B)

[0113] In the example shown in FIG. 3, each coded synthetic frame is padded with stuffing bytes such that each frame takes one frame interval to be fed to the decoder at the CBR bit-rate.

[0114] The vbv-delay for the first synthetic frame is arranged to be equal to the vbv_delay of the first I frame of the image sequence. This allows CBR mode to be maintained.

[0115] As an example of VBR operation we can consider the case where the pre-coded image sequence has a fixed and constant bit-rate but the bit-rate of the synthetic video data fed to the video decoder is allowed to be less than this constant value. In order to avoid decoder buffer under/over-flow problems, a simple approach is to arrange that the synthetic data is fed to the decoder at a rate of one synthetic frame per frame period.

[0116] This is illustrated in FIG. 4. The data for the first synthetic frame is assigned an appropriate DTS value and fed to the decoder buffer at a time (given by Eqn 7) such that its vbv_delay value is equal to the vbv_delay of the first I frame of the sequence. Then the remaining synthetic frames are fed at a rate of one frame per frame period. At time, t_trans(0), the first I frame of the sequence is sent and the vbv_delay value for this frame will be correct. FIG. 4 shows that the decoder buffer does not under-flow.

[0117] Note that, for all the frames (synthetic and image) there are no differences in the vbv_delay values for each frame between the CBR and VBR modes as shown in FIGS. 3 and FIG. 4.

[0118] Synthetic bitstreams can be used to pause a decoder at certain points in the sequence. A suitable synthetic sequence comprises a null series of P-frames. For example, suppose a series of synthetic P frames is inserted before a P-frame as shown below

[0119] Original Sequence

[0120] Sequence with Synthetic P Frames Inserted

[0121] Whilst the decoder is decoding the synthetic P frames, it will continue to display the last decoded “anchor” frame which in this example is frame P₁. Since the B-frames are not stored in the decoder these cannot be repeated. Following the sequence of synthetic P-frames, the image sequence can be continued.

[0122] Simple “rules” which may be followed for inserting such a pausing bitstream are:

[0123] The DTS and “start-time-of-feeding-to-the-decoder” of the first synthetic P-frame are equal to those of the P-frame it is replacing (i.e. vbv_delays are equal).

[0124] For each synthetic P frame, PTS_I=(DTS_i+T) and DTS_I+I=(DTS₁+T)

[0125] The synthetic P-frames are fed to the decoder at a rate of one synthetic P-frame per frame period This ensures that the vbv_delay for the first coded image frame following the synthetic sequence is unaffected by the presence of the synthetic bitstream. This is illustrated in FIG. 5 for VBR-mode operation.

[0126] The PTS and DTS of coded image frames following the synthetic bitstream will need to be incremented by an amount equal to N frame periods, where N is the number of synthetic frames introduced into the bitstream

[0127] A synthetic P-frame sequence can be used in a similar manner for pausing a decoder after an I-frame, thereby providing a means of displaying a still image.

[0128] Where it is required to pause a decoder for a prolonged period of time (for example several hours), particularly where it is not critical to maintain the last frame output, it may be desirable to insert I-frames into the sequence, for example black I frames, to ensure that synchronisation is not inadvertently lost.

[0129] Repeated (regular) insertion of synthetic frames may be used to effect slow-motion playback. It will be appreciated that to effect smooth slow motion playback, account needs to be taken of motion in the surrounding frames, rather than simply repeating the preceding frame, but nonetheless crude reduction in playback speed may be easily effected. When motion is taken into account, it will be appreciated that the size of the synthetic frames may no longer be negligible, but it should still be possible in most cases to generate synthetic P or B frames based on the content of the original sequence and transmit them within a single frame period avoiding buffer overflow/underflow problems.

[0130] Synthetic sequences can also be used to help in the splicing together of different bitstreams using an approach very similar to that described for decoder initialisation and pausing; a splice using this approach can be considered similar to pausing of a decoder and establishing synchronisation to a new bitstream in a combined operation. However, the “rule” of sending one synthetic frame per frame period is not maintained for this application.

[0131] This can provide a simple, less functional but cheaper solution to splicing together MPEG-2 bitstreams thin conventional apparatus. A common requirement could be to switch (in real time) to a second sequence bitstream at some fairly arbitrary point in a first sequence bitstream without causing a downstream decoder any significant disturbance. The main problem here is matching the decoder buffer occupancy at the point of leaving the first sequence to the buffer occupancy required before the first coded frame of the second sequence. If these are not matched then the decoder buffer will over or under flow which will cause a significant disturbance to the decoded image sequence.

[0132] This matching of vbv_delay values can be achieved using synthetic bitstreams.

[0133] Consider the example below:

[0134] It is likely that, in order to avoid decoder buffer underflow for the first I frame of bitstream2, the vbv_delay value for I₀of bitstream 2 will be greater than the vbv_delay value for frame P₂of bitstream1. However, since the synthetic bitstreams contain an almost negligible number of bits, it is possible to transmit these in a short period of time and then advance the time at which the first byte of frame I₀is fed to the decoder. In this way it is possible to achieve the correct vbv_delay value for I₀. This is illustrated in FIG. 6.

[0135] In FIG. 6, the three synthetic frames are fed to the decoder (taking negligible bits). Given that the decoding time of I₀is probably pre-determined according to a display schedule, then the interface should start feeding out to the decoder the first frame of the image sequence at a time given by:

[0136] t_trans(I₀)=t_disp(I₀)−offset−(n_B+1).T−vbv_delay(I₀) as in Eqn 6.

[0137] The following considerations apply in this example:

[0138] the relative values of the PTS and DTS for each frame are maintained since the synthetic frames are introduced in place of a P-frame in sequence 1.

[0139] The PTS/DTS values of sequence 2 need to be have an offset value added to them in order to maintain continuity of these fields across the splice.

[0140] The DTS value and “start-time-of-feeding-to-the-decoder” of the first synthetic frame are equal to those of the P-frame it is replacing in sequence1.

[0141] For each synthetic P frame, PTSi=(DTSi+T) and DTSi+1=(DTSi+T)

[0142] The following points should also be noted:

[0143] The first synthetic frame could equally be a synthetic I frame, which would give black frames between the sequence.

[0144] The minimum number of synthetic frames required through the splice depends on the size of the vbv_delay adjustment required.

[0145] The first frame of sequence2 must be an I frame.

[0146] In addition to the features mentioned above, the video output apparatus can provide other functions. For example, even without the means for generating synthetic frames, synchronisation to a studio reference clock can be effected, as discussed below.

[0147] A stored MPEG video sequence contains a series of PCR fields, each field containing a clock value which increments at the clock rate (typically 27 Mhz) of the original capture apparatus when the sequence is played at the correct bit-rate. The correct bit rate will usually be specified with the stored data, and conventionally a video source simply plays the data out at the specified rate. For example a sequence may have been captured at 4 Mbit/s, and a conventional video source will simply output the data at that rate, based on its own clock source However when played back at 4 Mbit/s, it may be found that the PCR values increment at a rate slightly greater or less than 27 MHz, for example due to inaccuracies in the recording apparatus clock frequency. This can be detected by comparing the rate at which PCR values increase to a studio clock reference frequency, and can be corrected by adjusting the data output rate so that a counter updated at the same rate as the PCR values in the recorded video sequence maintains synchronism with the studio reference clock.

[0148] It will be appreciated that the invention has many applications, and can be extended to applications other than those described above. An example of another application of the invention is the “repair” or editing of corrupted video sequences by re-generating timing field values, or patching synthetic frames in to over-write corrupted data. It will also be appreciated that, whilst described in the context of video processing, the same techniques may be applied to other coded data supplied as a sequence of coded frames; in particular, synthetic coded audio frames and synchronisation of coded (e.g. MPEG) audio may be effected. Specifically, although MPEG format is most commonly employed to transmit data including video data, in an MPEG audio/visual/data sequence containing no video, the invention may be employed to establish synchronisation for the purposes of playing out of audio or data contained within the sequence. The format of the synthetic frames will, of course change; specifically, null frames will usually be employed as padding. However, in such cases, the term video as used throughout the description and claims may be replaced by audio or data as required. Each feature of the description or claims may be independently provided, unless otherwise stated. The appended abstract is incorporated herein by reference.