Data stream processor

[0001] The present invention relates to a data stream processor that performs processing such as encoding or decoding for data, particularly data including video and/or audio information.

[0002] The international standard MPEG2 (Moving Picture Experts Group 2) is one method for compressing audiovisual information. MPEG2 is a method that individually compresses video and audio and synchronizes and multiplexes the compressed video, audio, data, etc. MPEG2 is suitable for storage media, networks, etc. There are two types of MPEG2: MPEG2-PS (Program Stream) for multiplexing and demultiplexing a single program and MPEG2-TS (Transport Stream) for a plurality of programs. MPEG2-TS can handle a plurality of program data as a sequence of data (hereafter referred to as a data stream) and is mainly used for television broadcasting services.

[0003] Data to be compressed by MPEG2-TS includes a Decoding Time Stamp (DTS) and Presentation Time Stamp (PTS) for times of decoding and presenting each frame of video and audio information. Further, compressed data include two types of time information: System Clock Reference (SCR) and Program Clock Reference (PCR). These types of information are used for a decoder to decode compressed data (hereafter simply referred to as decoder). The decoder uses SCR and PCR to correct a System Time Clock (STC) value to an intended value for a coder to compress data (hereafter simply referred to as coder).

[0004] Unless the decoder uses the PCR to correct the STC, the decoder's STC gradually deviates from the coder's system clock. When the decoder's STC precedes the coder's system clock, the decoder starts decoding or reaches the reproduction time before complete arrival of a data stream from the coder, i.e., with a temporary empty state of a buffer that should store data streams including audiovisual data. Namely, the decoder repeats the same video or audio. On the other hand, when the decoder's STC follows the coder's system clock, the decoder causes a delay in timing for decoding or reproduction. A data stream from the coder cannot be stored in the buffer that should store data streams including audiovisual data, resulting in a missing frame in the audiovisual data.

[0005] To prevent repeated audiovisual data or a missing frame on the decoder as mentioned above, the MPEG2-TS standard requires the decoder to provide a PLL (Phase Locked Loop) integrated with the STC for complete correspondence between the coder's system clock and the decoder's STC frequency. The decoder's PLL generates a clock having a frequency completely matching that of the coder based on the difference between a value of the PLL's reference time counter and the PCR embedded in a compressed data stream. Since the PCR is used to correct the STC, the decoder can synchronously reproduce the video and audio at the timing intended on the coder. For MPEG2, the system clock is 27 MHz and an error of up to 810 Hz is allowed.

[0006] Recently, there has been an increasing demand for a capability of transferring audiovisual data encoded at a plurality of remote locations via networks and receiving and decoding a plurality of data streams at a receiving site, for example, to implement a TV conference system, video chatting, etc.

[0007] When prior art is used to satisfy the above-mentioned demand, a plurality of data streams, according to the MPEG2-TS system, are transferred to the decoder. In this case, the decoder requires a plurality of PLLs for correcting the decoder's STC at a timing intended on the coder with respect to all data streams. However, this increases decoder costs. When the respective data streams are output to a plurality of monitors, the plurality of PLLs can correct the reference time for each of the data streams and reproduce the data streams at a plurality of reference frequencies. When a single monitor is used for simultaneously reproducing a plurality of data streams, a single reference frequency must be used for reproduction. All data streams cannot be reproduced according to the corresponding time information.

[0008] The technology for satisfying the above-mentioned demand is disclosed in JPA No. 5499/2001. According to the technology, the decoder is provided with a reference time information selection section that obtains the PCR of each data stream from a plurality of data streams and selects one of the PCRs. The decoder according to the technology uses the PCR selected in the reference time information selection section to correct the STC.

[0009] When this technology is used, the decoder cannot reproduce a data stream other than selected ones at the correct time. In an environment where a jitter or a packet loss occurs for networks, wireless communications, etc., the technology complicates processing of the reference time information selection for decreasing reproduction anomalies due to reference frequency differences. Even when a single data stream is transferred, PCR data is received incompletely due to a jitter or a packet loss on the network. The decoder cannot reproduce data at proper timing.

[0010] This invention provides a data stream coder and decoder capable of allowing the decoder to reproduce data streams transmitted from one or many coders at proper timing. To achieve this, in one aspect, a data stream processor is configured to have a reference frequency control section supplied with time information from outside the apparatus; and a data stream control section to input and output a data stream based on a signal generated from the reference frequency control section.

[0011] In another aspect, the reference frequency control section may be configured to have a reference time information acquisition section supplied with the time information; an oscillator to generate a signal for determining a frequency in the data stream processing section; and an oscillator control section to calculate a difference between time information input to the reference time information acquisition section and an frequency of a signal generated from the oscillator and to control the oscillator based on the calculation result.

[0012] In still another aspect, the data stream processing section may be configured to have a data stream storage section to store an externally supplied data stream; and a data stream processing section to process a data stream stored in the data stream storage section and output the data stream to the outside.

[0013] In yet another aspect, the data stream processing section may be configured to comprise: a data stream demultiplexing section to separate data supplied from the data stream storage section into video encoding information and audio encoding information; a video decoder section to generate video data from the video encoding information supplied from the data stream demultiplexing section; an audio decoder section to generate audio data from the audio encoding information supplied from the data stream demultiplexing section; a video selection/synthesis section to select and synthesize the video data for external output; and an audio selection/synthesis section to select and synthesize the audio data for external output.

[0014] In another aspect, the data stream storage section may be configured to have a video storage section to store an externally supplied video data stream and an audio storage section to store an externally supplied audio data stream; and the data stream processing section may be configured to have a video encoder section to generate video encoding information from the video data, an audio encoder section to generate video encoding information from the audio data, and a stream multiplexing section to multiplex the video encoding information and the audio encoding information and generate a data stream. The time information may be transmitted from an NTP server.

[0015] Other and further objects, features and advantages of the invention will appear more fully from the following description.

[0016] FIG. 1 shows a configuration of a data stream processor according to the present invention;

[0017] FIG. 2 shows a configuration of a data stream decoder according to the present invention;

[0018] FIG. 3 shows another configuration of a data stream coder according to the present invention;

[0019] FIG. 4 shows another configuration of a data stream decoder according to the present invention;

[0020] FIG. 5 shows yet another configuration of a data stream decoder according to the present invention;

[0021] FIG. 6 shows a configuration of a register to store;

[0022] FIG. 7 shows a configuration of the data stream processor according to the present invention connected to a network;

[0023] FIG. 8 shows another configuration of the data stream processor according to the present invention connected to a network;

[0024] FIG. 9 shows another configuration of a data stream coder/decoder according to the present invention; and

[0025] FIG. 10 shows an NTP format.

[0026] FIG. 1 shows a configuration of a data stream processor 1 according to a preferred embodiment of the present invention. The term data stream processor is used herein as a generic name for a data coder, a data decoder, and a data coder/decoder. Data stream processor 1 has a reference frequency control section 11 and a data stream control section 12. Reference frequency control section 11 provides a reference frequency that defines timings for data transmission etc. in the apparatus based on time information supplied from the outside of data stream processor 1. Data stream control section 12 temporarily stores an externally supplied data stream and outputs the data stream based on the reference frequency determined by a signal generated from reference frequency control section 11. Reference frequency control section 11 includes an oscillator 13, a reference time counter 16, an oscillator control section 17, and a reference time information acquisition section 15.

[0027] Reference time counter 16 is incremented synchronously with the reference frequency (27 MHz in this embodiment) defined by a clock signal generated from oscillator 13. Oscillator control section 17 controls oscillator 13 using a difference between a value of reference time counter 16 and the external time information obtained by reference time information acquisition section 15. The NTP (Network Time Protocol) is an example of the time information obtained by reference time information acquisition section 15.

[0028] FIG. 10 is a diagram of an NTP format. The NTP time information is a 2³² Hz (approximately 4 GHz) counter value comprised of the number of seconds (32 bits representing time since Jan. 1, 1900 and the fractional portion (32 bits) of a second. Oscillator control section 17 holds the NTP external time information and a value of reference time counter 16 when the NTP was obtained. Thereafter, oscillator control section 17 compares next obtained NTP time information and a value of reference time counter 16 at that time with the previously stored time information and the value of reference time counter 16 to compute each increase. Further, oscillator control section 17 compares an increase in the NTP time information with an increase in the value of reference time counter 16 and controls oscillator 13 according to the difference.

[0029] Data stream control section 12 includes a data stream storage section 14 to store an externally supplied data stream, and a data stream processing section 18 to process a data stream stored in data stream storage section 14 for output. Data stream processing section 18 externally outputs a processed data stream according to the clock signal generated from oscillator 13.

[0030] FIG. 2 illustrates an embodiment in which data stream processor 1, explained in FIG. 1, is used as a data stream decoder 2. In data decoder 2, data stream processing section 18 of data stream control section 12 has a data stream demultiplexing section 23, a video decoder section 24, an audio decoder section 25, a video selection/synthesis section 26, and an audio selection/synthesis section 27. The configuration of reference frequency control section 11 is the same as that in FIG. 1.

[0031] Data stream storage section 14 stores an externally supplied data stream (MPEG2-TS in this embodiment). Data stream demultiplexing section 23 extracts video encoding information (video stream, PTS, and DTS), and audio encoding information (audio stream, PTS, and DTS) from a data stream supplied from data stream storage section 14. Data stream demultiplexing section 23 outputs the extracted video encoding information to video decoder section 24 and the extracted audio encoding information to audio decoder section 25. In the embodiment, the PCR contained in a data stream is not used for control of the reference frequency.

[0032] Video decoder section 24 stores video encoding information supplied from data stream demultiplexing section 23 in a buffer of video decoder section 24. Thereafter, video decoder section 24 starts decoding video data at the timing when the count value (STC) of reference time counter 16 becomes equal to the DTS contained in the data. The generated video data is output to video selection/synthesis section 26.

[0033] On the other hand, audio decoder section 25 decodes audio encoding information supplied from data stream demultiplexing section 23 at the timing when the STC becomes equal to the DTS contained in the data. The generated audio data is output to audio selection/synthesis section 27.

[0034] Video selection/synthesis section 26 selects video data a user requests to display from a plurality of video data supplied from video decoder section 24. Video selection/synthesis section 26 stores a value added by weighting each video data at a specified location in a video data storage area corresponding to one frame provided for output to a display device such as a monitor, and thus synthesizes a plurality of video data. Thereafter, video selection/synthesis section 26 outputs the synthesized video data as a video signal at the timing when the count value (STC) of reference time counter 16 becomes equal to the PTS contained in the data.

[0035] Similarly, audio selection/synthesis section 27 selects audio data a user requests to display from a plurality of audio data supplied from audio decoder section 25. Audio selection/synthesis section 27 stores a value added by weighting each audio data's frequency level or output level at a specified location in an audio data storage area corresponding to one frame provided for output to an audio device such as a speaker, and thus synthesizes a plurality of audio data.

[0036] FIG. 3 illustrates a configuration for using data stream processor 1 in FIG. 1 as a data stream coder 3. Data stream storage section 14 comprises a video storage section 33 and an audio storage section 34. Data stream processing section 18 comprises a video encoder section 35, an audio encoder section 36, and a data stream multiplexing section 37. Reference frequency control section 11 is the same as that in FIG. 1.

[0037] Video storage section 33 stores an externally input data stream, specifically a video signal, as video data. Audio storage section 34 stores an externally input audio signal as audio data. Video encoder section 35 encodes video data supplied from video storage section 33 to generate video encoding information. Audio encoder section 36 encodes audio data supplied from audio storage section 34 to generate audio encoding information.

[0038] Data stream multiplexing section 37 generates PTS, DTS, and PCR based on a value of reference time counter 16. Data stream multiplexing section 37 generates a data stream from these pieces of time information, video encoding information, and audio encoding information, and outputs the data stream to the outside.

[0039] FIG. 4 shows a configuration of the data stream decoder according to another embodiment. The embodiment corrects the reference frequency even if reference time information acquisition section 15 cannot obtain external time information. It is possible to minimize a phenomenon in which a coder's intended time gradually deviates from the encoder's STC.

[0040] The reference frequency control section 41 differs from reference frequency control section 11 of FIG. 1 in that the former has a reference time information selection section 43. Data stream control section 48 differs from data stream demultiplexing section 18 described with reference to FIG. 2 in the function of a data stream demultiplexing section 44.

[0041] Data stream demultiplexing section 44 extracts the PCR from each of a plurality of input data streams and outputs the PCR to reference time information selection section 43. Reference time information selection section 43 selects either the PCR or time information supplied from reference time information acquisition section 15 and outputs it to oscillator control section 17. When no time information is supplied from reference time information acquisition section 15, reference time information selection section 43 selects the PCR supplied from data stream demultiplexing section 44. The case of no time information supplied means that no external time information is input over the threshold value for a time interval provided in the apparatus. When no external time information is supplied, reference time information selection section 43 selects the PCR.

[0042] According to the embodiment, if reference time information acquisition section 15 cannot obtain the external reference time information for some reason, data stream decoder 4 can select the PCR extracted in data stream demultiplexing section 43. This makes it possible to decrease errors during reproduction to a certain degree.

[0043] FIG. 7 shows an embodiment of a configuration that connects data stream coder 3 and data stream decoder 2 to a network. A network 61 connects with a reference time station 62 in addition to data stream coder 3 and data stream decoder 2. The time information supplied from reference time station 62 is obtained by reference time information acquisition section 15 provided for each of data stream coder 3 and data stream decoder 2. Based on the obtained time information, data stream coder 3 and data stream decoder 2 correct respective reference frequencies so that each of reference frequencies equals the reference frequency of reference time station 62. When a plurality of data stream coders 3 is available as shown in FIG. 7, there occurs no deviation between reference frequencies for data stream coder 3 and data stream decoder 2. This can prevent an error due to deviation of reference frequency while data stream decoder 2 reproduces data.

[0044] As reference base station, it is possible to use an ordinary host computer, a GPS (Global Positioning Satellite), an NTP server, etc. The NTP server is a server used for the technology that synchronizes the time between computers connected to the Internet. A computer connected to the Internet can adjust its own time setting by using NTP servers scattered worldwide. When a server also works as the NTP server, each client can use this server to adjust its own time setting to the same system time as for the NTP server. The NTP technology can keep a time difference between the reference base station and each computer within tens of milliseconds. When an NTP server is used for the reference time station, it is possible to always keep an error between the reference time for the coder and that for the decoder within tens of milliseconds.

[0045] When reference time information acquisition section 15 obtains time information supplied from the NTP server, however, the information may contain an error of tens of milliseconds as mentioned above. In this case, if data stream decoder 2 controls oscillator 13 using a set of time information and the counter value, the reference frequency may not be stabilized.

[0046] FIG. 5 shows an embodiment of data stream decoder 5 for solving the above-mentioned problem. A reference frequency control section 51 differs from reference frequency control section 11 in that the former has a register 52.

[0047] When obtaining external reference time information, reference time information acquisition section 15 registers reference time information and a value of reference time counter 16 to register 52. As shown in FIG. 6, register 52 is configured to store a history of reference time and counter values. Based on this history, reference frequency control section 51 can determine an increasing tendency for the reference time and counter values. Oscillator control section 17 controls oscillator 13 based on the increasing tendency determined from the history of reference time and counter values stored in register 52. For example, oscillator control section 17 finds average values for increased time intervals corresponding to the reference time and the counter value and controls oscillator 13 according to a difference between these average values.

[0048] According to the embodiment, data stream decoder 5 analyzes an increasing tendency of the reference time and counter values to control oscillator 13 based on the history of the reference time and counter values stored in register 52. Consequently, data stream decoder can generate a clock signal with more decreased reference frequency fluctuations.

[0049] FIG. 9 shows a configuration of a data stream coder/decoder 63 according to the present invention. A data stream coder/decoder 63 has reference frequency control section 11 and data stream control section 12. The configuration of reference frequency control section 11 is the same as those in FIGS. 2 and 3. Data stream control section 12 has data stream storage section 14 and data stream processing section 18 in FIG. 2 for decoding; and data stream storage section 14 and data stream processing section 18 in FIG. 3 for encoding. A data stream coder/decoder 63 uses one reference frequency control section 11 to send the reference frequency to encoding portions and decoding portions.

[0050] FIG. 8 shows a system connecting a plurality of data stream coders/decoders shown in FIG. 9 to network 61. This configuration can transfer an audiovisual data stream encoded in the local apparatus. Remotely located apparatuses can interchange a plurality of encoded audiovisual data streams with each other. This makes it possible to decrease occurrences of repetition or loss of audiovisual data in bidirectional communication systems such as a TV conference system, video chatting, etc.

[0051] When a single data stream cannot be transferred from another apparatus, a problem arises according to the prior art if it uses the reference time in that data stream. The system according to the present invention does not cause such a problem because the reference time is externally obtained. Even if one or more data streams are received and are displayed on a display apparatus, the present invention can solve the problem of repeated or missing audiovisual data due to a reference frequency difference between the coder side and the decoder side.

[0052] The foregoing invention has been described in terms of preferred embodiments. However, those skilled, in the art will recognize that many variations of such embodiments exist. Such variations are intended to be within the scope of the present invention and the appended claims.