US 2003/0103243 A1Application
Transmission system
Publication Date:2003-06-05
•12 Claims
•9 Drawing Sheets
Abstract
A proposed transmission system is capable of setting an optimal transmission rate at which high-quality packet transmission can be realized with reduced packet loss. A packet generating part divides encoded media data represented or output at an identical time into packets. A receiving condition information acquiring part acquires receiving condition information about a condition of receiving the packets from an opposing device. A packet sending part, based on the receiving condition information, adjusts intervals at which the packets are sent so that a transmission rate can be varied and performs a sending control of the packets. A receiving condition information generating part receives the packets, generates the receiving condition information, and sends the receiving condition information to the packet sending device.
Metadata
Inventors
- Hideaki Watanabe
- Akira Nakagawa
Application Information
Application Number:US 10/116,928
Filing Date:2002-04-05
Priority Date:2001-11-30
Patent Drawings (9 sheets)
Description
Background of the Invention
[0001] 1. Field of the Invention
[0002] The present invention relates to transmission systems, and more particularly, to a transmission system that transfers packets via a network.
[0003] 2. Description of the Related Art
[0004] There has been considerable activity in the development of transmission technologies about real time multimedia such as moving image data with practical application of multimedia service. Digitized image data has a huge amount of information and is encoded at a bit rate that matches the transmission band before transmission.
[0005] Image encoding typically employs MPEG. In order to simultaneously obtain a desired high compression rate and high picture quality, image encoding uses an intraframe predictive picture (I picture) and an interframe forward predictive picture (P picture) are introduced. A larger amount of information is allotted to the I picture than to the P picture, and is then encoded.
[0006] In transmission of a picture to which a large amount of information is allotted, if the picture is transferred at its encoding rate (fixed), in other words, if the picture is transferred at a transmission rate nearly equal to the encoding rate, it will take a long delay of time to retrieve the picture received at the receiving side. In order to prevent such a delay in retrieval it is required to transmit the picture at a transmission rate higher than the encoding rate.
[0007] However, a problem of loss of packet will arise from a mere increase of the transmission rate from the encoding rate although the starting time of the delay in retrieval can be shortened.
[0008] A packet may be discarded if a router in the network operates with a high load. Loss of packet degrades the transmission quality. If the packets are transferred at an increased transmission rate without any consideration of a transmission delay stemming from a network congestion, loss of packet will increase.
Summary of the Invention
[0009] Taking the above into consideration, it is an object of the present invention to provide a transmission system capable of setting an optimal transmission rate at which loss of packet does not occur and transferring packets with high reliability.
[0010] The above object of the present invention is achieved by a transmission system transferring packets via a network comprising:
[0011] a packet sending device including a packet generating part dividing encoded media data represented or output at an identical time into packets, a receiving condition information acquiring part acquiring receiving condition information about a condition of receiving the packets from an opposing device, and a packet sending part adjusting intervals at which the packets are sent so that a transmission rate can be varied and performing a sending control of the packets; and a packet receiving device including a receiving condition information generating part receiving the packets, generating the receiving condition information, and sending the receiving condition information to the packet sending device.
[0012] The above and other objects, features and advantages of the present invention will become apparent from the following description when taken in conjunction with the accompanying drawings which illustrate preferred embodiments of the present invention by way of example.
Brief Description of the Drawings
[0013] FIG. 1 is a diagram of the principles of a transmission system according to the present invention;
[0014] FIG. 2 is a graph describing a concept of VBV;
[0015] FIG. 3 is a graph describing an increase in delay of time in retrieval start;
[0016] FIG. 4 is a graph showing a condition of a receiving buffer at the time of burst transmission;
[0017] FIG. 5 is a block diagram of a concept of packet communication using a RTP/RTCP-based transmission system of the present invention;
[0018] FIG. 6 is a block diagram of a server;
[0019] FIG. 7 is a view of receiving condition information included in an RR packet; and
[0020] FIG. 8 is a view of a concept of packet transmission interval adjustment control.
Description of the Preferred Embodiments
[0021] A description will now be given of embodiments of the present invention with reference to the accompanying drawings. FIG. 1 shows the principles of a transmission system. A transmission system 1 is made up of a packet sending device 10 and a packet receiving device 20, between which packets are transferred via a network 2. The functions of the packet sending device 10 and the packet receiving device 20 according to the present invention may be implemented in a single transmission device in practice.
[0022] A packet generating part 11 of the packet sending device 10 divides media data (audio/video data: AV data) encoded by MPEG or the like and represented or output at the same time into packets.
[0023] For example, for consecutive video frames respectively represented at times t1 and t2 at the receiving side, the video frame represented at time t1 is divided into five packets, and the video frame represented at time t2 is divided into six packets.
[0024] A receiving condition information acquiring part 12 acquires information about the packet receiving condition from an opposing side (packet receiving device 20). As the receiving condition information, at least one of loss-of-packet information and information about transmission delay is acquired.
[0025] The loss-of-packet information indicates the ratio of packets lost to all the packets received by the packet receiving device 20. The transmission delay information indicates the time of transmission delay it takes the packet sent by the packet sending device 10 to arrive at the packet receiving device 20.
[0026] A packet sending part 13 adjusts the packet transmission intervals based on the receiving condition information, and dynamically sets the transmission rate at which the packets are transferred.
[0027] The packet sending part 13 sets the transmission rate higher than the encoding rate for media data if it does not recognize the occurrence of loss of packet or transmission delay (occurrence means an increasing trend in the present invention), and sets the transmission rate close to the encoding rate if loss of packet or transmission delay is recognized.
[0028] A receiving condition information generating part 21 of the packet receiving device 20 receives packets and generates receiving condition information thereon, sending it to the packet sending device 10.
[0029] Now, a VBR (Variable Bit Rate) and a CBR (Constant Bit Rate) are described. Generally, picture information is encoded in advance of transmission. VBR Encoding is frequently employed in an environment in which data can be read directly from a recording medium at a high rate without passing through a network, such as DVD. In digital broadcasting, CBR encoding is generally employed.
[0030] The VBR is used to control the amount of information that is encoded at a variable bit rate. However, in practice, there is a limit on the average read rate and the peak read rate due to the nature of the recording medium. The average bit rate over a certain time may be equal to a constant amount of information.
[0031] In practice, the CBR does not encode information so that every video frame has the same amount of information, but allows a difference in the amount of information between frames in MPEG. An encoding that guarantees a constant amount of information over hundreds of milliseconds to a few seconds is called CBR although it depends on applications.
[0032] That is, the difference between the VBR and CBR merely depends on whether the interval at which the constant amount of information is guaranteed is long or short. In many cases using the VBR, encoding is carried out so that a constant amount of data is substantially handled over a time range of tens of seconds to a few minutes.
[0033] Consequently, the encoding rate for picture data can be deemed to be constant over a certain time irrespective of whether the VBR or CBR is employed. In the following description, a bit stream encoded at a certain rate is employed.
[0034] Next, a delay in the retrieval starting time is described. First, VBV (Video Buffering Verifier) of MPEG video encoding (MPEG Video Elementary Stream; hereinafter referred to as MPEG Video ES) is described.
[0035] In the MPEG Video ES, conditions required by a hypothetical buffer called VBV must be satisfied, and the encoder encodes data so that the VBV on the decoder side does not underflow or overflow.
[0036] More particularly, on the encoder side, 16-bit information called vbv delay is added to the head of every frame. The above information is used to set the time from buffer input to retrieval in the buffer on the decoder side. The vbv delay enables the decoder to know the frame retrieval timing.
[0037] FIG. 2 shows the concept of VBV. More particularly, this figure shows how the amount of data that occupies the receiving buffer on the decoder side varies with retrieval of frames. The vertical axis of the figure denotes the amount of data that occupies the receiving buffer, and the horizontal axis thereof denotes time.
[0038] The vbv delay of MPEG Video ES is represented as time information on the 90 kHz basis ({fraction (1/90000)} sec basis), and the buffer occupying quantity VBVn for decoding frame n can be obtained as follows:
[0039] When the buffer occupying quantity becomes equal to VBVn by equation (1), data of the frame n is read from the receiving buffer and is processed for retrieval. In FIG. 2 , frame 1 is received and buffered at time t0, and is read at time t3 when the buffer occupying quantity becomes equal to VBV1. Frame 2 is received and buffered at time t1, and is read at time 4. The same operation as described above follows.
[0040] The delays of time necessary to retrieve frames 1 to 3 are respectively delay 1 to delay 3. The slant in the graph depends on the transmission rate.
[0041] The above control guarantees retrieval of successive frames without any overflow or underflow at the receiving buffer. In a case where the frames are transported at the transmission rate equal to the encoding rate (that is, the frames are transported at the minimum rate), if the amount of information per frame is large, the quantity of data to be received up to retrieval increases. This increases the delay in the retrieval start time.
[0042] FIG. 3 shows an increased delay in the retrieval start time, in which the vertical axis denotes the buffer occupying quantity and the horizontal axis denotes time. Frame a is received and buffered at time t0, and is read at time ta when the buffer occupying quantity becomes equal to VBVa. The retrieval start delay of time is delayA.
[0043] A case is now considered where a frame b is transmitted at the same transmission rate (the same slant) as that at which frame a is transmitted in which frame b has a larger amount of information than frame a. Frame b is received and buffered at time t0, and is read at time tb (>ta) when the buffer occupying quantity becomes equal to VBVb (>VBVa). The delay in retrieval start time is delayB. That is, delayB>delayA. As the quantity of information per frame increases, the delay of time in the retrieval start increases.
[0044] As described above, the delay of time in the retrieval start increases for frames that are transmitted at rates nearly equal to the encoding rate and are assigned a large amount of information. The present invention assumes that the transmission rate is higher than the encoding rate. Hereinafter, such a transmission is called burst transmission.
[0045] FIG. 4 shows the condition of the receiving buffer at the time of burst transmission, in which the vertical axis denotes the receiving buffer occupying quantity, and the horizontal axis denotes time. Frame 1 is received and buffered at time t0, and is read at time t12 when the buffer occupying quantity becomes equal to VBV1. Frame 2 is received and buffered at time t11, and is read at time t15. The same operation as described above follows.
[0046] Since the transmission rate is selected so as to be higher than the encoding rate, The slants of the graph for frames 1 to 3 are sharper than those for the corresponding frames shown in FIG. 2 . In other words, the time necessary for buffering at the receiving side can be shortened due to burst-like transmission of frames.
[0047] In FIG. 4 there are some flat portions in which the buffer occupying amount is constant. As to section C in frame 2, all of frame 2 is received at time t13 and is read at time t15. It means that data stays for the section C in the buffer.
[0048] The delays of time in retrieval start for frames 1-3 are respectively delay1a-delay3a. As compared to FIG. 2 , they are delay1>delay1a, delay2>delay2a, delay3>delay3a. The burst transmission reduces the delays of time in retrieval start.
[0049] Simple burst transmission that attempts to reduce the delays of time in retrieval start would increase the ratio of packets lost in the network.
[0050] The present invention primarily employs burst transmission directed to reducing the delays of times in retrieval start, and has an additional structure directed to suppression of occurrence of packets lost by setting the rate of the burst transmission close to the encoding rate if an increase of packets lost is recognized by provisioning using the information concerning the receiving condition sent by the packet receiving device 20. As described above, the transmission rate is variably set so that the optimal transmission rate dependent on the network condition is determined.
[0051] Next, a description will be given of an embodiment of the present invention in which the transmission system 1 of the invention is applied to a system based on RTP (Real-time Transport Protocol/RTCP (Real-Time Control Protocol). FIG. 5 shows the concept of packet communication in the RTP/RTCP-based transmission system of the present invention.
[0052] RTP is a data transfer protocol that is located in the transport layer and operates on UDP and ATM AAL5 used for real-time delivery of a data stream of audio and video.
[0053] RTCP is a control protocol used for evaluating the channel quality in RTP-based packet communication. In order to generate a stream data that conforms to the channel quality, an RTCP packet is periodically transferred between the sending and receiving terminals (an idea that does not require a network device such as a router to control the band guarantee).
[0054] A transmission system 1a is made up of a server 100 (which corresponds to the packet sending device 10), and a client device 200 (which corresponds to the packet receiving device 20). The transmission system 1a sends and receives packets to and from networks 2a and 2b connected via a router 3.
[0055] The server 100 sends an RTP packet including video data to the client device 200. RTCP packets are transferred between the server 100 and the client device 200.
[0056] In the following, an RTCP packet sent to the client device 200 from the server 100 is referred to as an SR (Sender Report) packet, and an RTCP packet sent to the server 100 from the client device 200 is referred to as an RR (Receiver Report) packet.
[0057] Next, the server 100 is described. FIG. 6 is a block diagram of a structure of the server 100, which is made up of a packet generating part 101, an RR packet receiving part 102, a packet sending part 103, and an SR packet sending part 104. The packet generating part 101 is made up of an RTP packet generating part 101a, and an RTP packet queue 101b.
[0058] The RTP packet generating part 101a receives digital video data, and encodes it at CBR according to MPEG4. In MPEG4, one frame of video is called VOP (Video Object Plane), and one VOP is divided into units called video packets, which are then encoded.
[0059] It is free to determine how many units one VOP is divided. When video data conforming to MPEG4 is transported at RTP, an RTP packet is generated in VOP or a video packet boundary (recommended by IETF RFC3016).
[0060] Here, it is assumed that the RTP packet is generated in accordance with the above-mentioned specification. If the packet size of the RTP packet goes beyond MTU (Maximum Transfer Unit: the maximum length of an IP packet) of the network, it is divided into parts equal to or smaller than MTU.
[0061] The RTP packet queue 101b temporarily stores the RTP packet, which is then read by the packet transmission part 103 and is sent to the client device 200.
[0062] The RR packet receiving part 102 acquires (extracts) information about the receiving condition from the RR packet received, and notifies the information about the receiving condition of the packet sending part 103.
[0063] The packet sending part 103 sets the optimal transmission rate from the information about the receiving condition, and sends the RTP packet. The SR packet sending part 104 periodically acquires necessary information from the packet sending part 103, and generates the SR packet, which is then sent to the client device 200.
[0064] FIG. 7 is a view of the information concerning the receiving condition contained in the RR packet. A field SSRC (synchronization source) is an ID of the sender terminal. A field fraction lost is the ratio of packets lost to an expected number of packets received after the immediately previous SR packet or RR packet is sent. A field cumulative number of packets lost is the cumulative number of packets lost after the receiving is started.
[0065] A field extended highest sequence number received is the highest sequence number of the packet received previously. A field interarrival jitter indicates the quantity of jitter at intervals at which packets arrive. A field last SR (LSR) timestamp is the timestamp of the last SR packet. A field delay since last SR (DLSR) is a delay of time from the time of receipt of the last SR packet to the time of sending its RR packet.
[0066] In the server 100, timestamp information and the cumulative number of packets/bytes sent are stored in the SR packet and are then sent. When the server 100 receives the RR packet, it is possible to recognize the conditions in which loss of packets occurs from the fraction lost and the cumulative number of packets lost.
[0067] The RTT (roundtrip time: the time it takes the packet sent to arrive at the destination and receive its response) regarding the transmission delay is computed by RTT(TLSRDLSR) where T denotes the time when the RR packet is received. In the above manner, the network delay can be estimated.
[0068] The server 100 consecutively measures the rate of the packets lost and the network delay on the receiving side, and thus recognizes change of the transfer performance of the network. The present invention optimizes the packet transmission rate using not only the information (fraction lost, cumulative number of packets lost) about the receiving condition with which an increase in packet loss is directly recognizable, but also information (LSR timestamp, DLSR) about the receiving condition used for notice of the transmission delay (it is possible to estimate an increase in packet loss by recognizing an increase in transmission delay).
[0069] A description will now be given of sending interval adjustment control by the packet sending part 103. FIG. 8 shows the concept of the packet sending interval adjustment control. RTP packets P1-Pn are respectively sent at times T1-Tn. D1-Dn are the differences between the adjacent times (time intervals for sending). The packets P1-Pn have quantities I1-In of data.
[0070] The interval time D for sending, the quantity I of packet information, and the target transmission rate B have the following relationship:
[0071] It is now assumed that RTP packet P1 is sent at time T1. In order to make the transmission rate equal to B, RTP packet P2 is sent at time T2 which is D1I1/B spaced apart from time T1. In other words, the sending interval D1 is adjusted so that the bit rate required for I1/D1 becomes equal to the target transmission rate B. By adjusting the sending interval D, it is possible to make the transmission rate close to the target value B.
[0072] When the target value B corresponds to the encoding rate, it is possible to maintain a condition in which data to be sent is always available. If the target value B exceeds the encoding rate, a condition occurs in which no data for transmission is available. This is because a condition in which data equal to vbv delay necessary for decoding of the above-mentioned one frame has been sent.
[0073] If transmission is temporarily interrupted as described above, the computation of the transmission rate does not consider the waiting time, and newly adjusts the transmission rate from the beginning packet when data transmission necessary for decoding of the next frame starts.
[0074] The upper limit Bmax of the transmission rate B is set equal to the maximum transfer band rate of the network, and the lower limit Bmin thereof is set equal to the encoding rate. According to the present invention, the optimal transmission rate B is set so as to meet BminBBmax.
[0075] Next, a description will be given of control of setting of the optimal transmission rate. At the commencement of transmission, the server 100 performs burst transmission. The RR packet receiving part 102 obtains information about the transmission delay and the ratio of packets lost from the received RR packet. The packet sending part 103 adjusts the transmission intervals D(making the intervals wider) and makes the transmission rate of the RTP packet close to the encoding rate when recognizing an increase in the transmission delay or the ratio of packets lost.
[0076] At the commencement of transmission, the transmission rate may be made equal to the encoding rate, and is gradually set in a burst fashion if the information about the receiving condition in the RR packet exhibits that neither transmission delay nor packet loss is recognized (making the intervals narrower).
[0077] If the ratio of packets lost is not improved after burst transmission is ceased, it is recognized that the transmission rate itself exceeds the transmission band. Thus, the RR packet receiving part 102 controls the RTP packet generating part 101a to suppress the quantity of information that occurs.
[0078] If the transmission rate is changed at the sending side, data to be retrieved at the reception side may be temporarily in short or excessive. In such as case, the frame rate is adequately increased or decreased at the receiving side in order to adjust the retrieval rate. This prevents degradation of the quality of retrieval.
[0079] As described above, according to the present invention, the packet transmission intervals are adjusted based on the information about the receiving condition obtained from the opposing side, and the transmission rate is varied without changing the picture quality (without changing the encoding rate). It is therefore possible to realize high-quality packet transfer with loss of packets being reduced. Particularly, the present invention is advantageous to transmission of data at a frame rate as low as a few fps (frames per second).
[0080] As described above, the transmission system of the present invention adjusts the packet transmission interval based on information concerning the receiving condition obtained from the opposing device and dynamically adjusts the transmission rate. It is therefore possible to set the transmission rate optimal to suppression of occurrence of packet loss and realize highly reliable packet transmission.
[0081] The foregoing is considered as illustrative only of the principles of the present invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and applications shown and described, and accordingly, all suitable modifications and equivalents may be regarded as falling within the scope of the invention in the appended claims and their equivalents.
Claims
1. A transmission system transferring packets via a network comprising:
a packet sending device including a packet generating part dividing encoded media data represented or output at an identical time into packets, a receiving condition information acquiring part acquiring receiving condition information about a condition of receiving the packets from an opposing device, and a packet sending part, based on the receiving condition information, adjusting intervals at which the packets are sent so that a transmission rate can be varied and performing a sending control of the packets; and
a packet receiving device including a receiving condition information generating part receiving the packets, generating the receiving condition information, and sending the receiving condition information to the packet sending device.
2. The transmission system according to claim 1, wherein the receiving condition information acquiring part acquires, as the receiving condition information, at least one of packet loss information and transmission delay information.
3. The transmission system according to claim 1, where the packet sending part sets the transmission rate higher than an encoding rate if occurrence of a transmission delay or packet loss is not recognized and sets the transmission rate close to the encoding rate if occurrence of a transmission delay or packet loss is recognized, so that an optimal transmission rate for packet transmission can be set.
4. The transmission system according to claim 1, wherein the packet sending part adjusts transmission intervals D for change of the transmission rate so that a bit rate I/D becomes equal to a target transmission rate B and sends the packets where D denotes packet transmission intervals and I denotes a quantity of packet information.
5. The transmission system according to claim 1, wherein the packet sending part uses an upper limit value of the transmission rate as a maximum transmission band rate of the network, and a lower limit value thereof as an encoding rate.
6. A packet transmission device sending packets via a network comprising:
a packet generating part dividing encoded media data represented or output at an identical time into packets;
a receiving condition information generating part receiving the packets, generating receiving condition information, and sending the receiving condition information to a device of sending side;
a receiving condition information acquiring part acquiring the receiving condition information from a device of receiving side; and
a packet sending part, based on the receiving condition information, adjusting intervals at which the packets are sent so that a transmission rate can be varied and performing a sending control of the packets.
7. A transmission system performing transmission control based on RTP/RTCP comprising:
a server including a packet generating part dividing encoded media data represented or output at an identical time into RTP packets, a receiving condition information acquiring part acquiring receiving condition information about a condition of receiving the RTP packets from an RTCP packet sent by an opposing device, and a packet sending part, based on the receiving condition information, adjusting intervals at which the RTP packets are sent so that a transmission rate can be varied and performing a sending control of the RTP packets;
a client device including a receiving condition information generating part receiving the RTP packets, generating the receiving condition information, and sending the RTCP packet including the receiving condition information to the server.
8. The transmission system according to claim 7, wherein the receiving condition information acquiring part acquires, as the receiving condition information, at least one of packet loss information and transmission delay information.
9. The transmission system according to claim 7, where the packet sending part sets the transmission rate higher than an encoding rate if occurrence of a transmission delay or packet loss is not recognized and sets the transmission rate close to the encoding rate if occurrence of a transmission delay or packet loss is recognized, so that an optimal transmission rate for packet transmission can be set.
10. The transmission system according to claim 7, wherein the packet sending part adjusts transmission intervals D for change of the transmission rate so that a bit rate I/D becomes equal to a target transmission rate B and sends the RTP packets where D denotes RTP packet transmission intervals and I denotes a quantity of RTP packet information.
11. The transmission system according to claim 7, wherein the packet sending part uses an upper limit value of the transmission rate as a maximum transmission band rate of the network, and a lower limit value thereof as an encoding rate.
12. A packet transmission device performing transmission control based on RTP/RTCP comprising:
a packet generating part dividing encoded media data represented or output at an identical time into RTP packets;
a receiving condition information generating part receiving the RTP packets, generating receiving condition information, and sending an RTCP packet including the receiving condition information to a server;
a receiving condition information acquiring part acquiring the receiving condition information from the RTCP packet from a client device; and
a packet sending part, based on the receiving condition information, adjusting intervals at which the RTP packets are sent so that a transmission rate can be varied and performing a sending control of the RTP packets.