High Speed Transport Protocol

This relates to transmission protocols and more particularly to transmission protocols in a packet transmission environment.

Advances in data transmission and switching over the last decade are promising deployment of communication systems with raw bandwidth and switching speeds that are an order of magnitude higher than the current systems. Optical fibers, for example, allow transmission of tens of gigabits/sec over several kilometers without repeaters. Switch fabrics that can switch bit-streams of more than hundreds of megabits/sec have already been prototyped. One such system has been described, for example, by A. Huang and S. Knauer in "Starlite: a Wideband Digital Switch," Proceedings of Globecom 84, December 1984, pp 125-127. However, the fruits of these efforts have not yet been realized in internetworking of diverse high speed networks, and have not yet been realized in delivery of high end-to-end bandwidth to applications within an operating system. Ideally, any single user connected to a packet network should be able to transmit at the peak bandwidth of the channel, once access is obtained. In practice, however, the obtainable end-to-end throughput is only a small fraction of the transmission bandwidth, particularly at high speeds. This throughput limitation comes from a variety of factors, including protocol processing in the networks layers, buffer congestion, and flow control mechanisms, as well as from various interfaces that transfer data from the network to a process in the host.

Reliable communication can be achieved with very little protocol processing if the network is perfect. It is clear that the limitations due to protocol processing come about because the protocol has to overcome network deficiencies such as bit errors large and varying packet delays in the network, packet loss due to congestion, out-of-order delivery of packets and overflow of buffers at various nodes in the network. Recently, there has been considerable interest in transport protocols for high speed networks. Broadly, higher speed can be achieved by a combination of three means. First, one can assume that the network has fewer deficiencies and therefore the protocol has to correct for fewer network problems. An example of this is the Packet Stream Protocol (PSP) which assumes a virtual circuit network and, therefore, packets are never received out of order. Second, higher speeds can be obtained by implementing some of the protocol processing in hardware. Third, one can invent new protocols that are better suited for high speed networks. The latter is the primary thrust of the instant invention.

To illustrate where the bottlenecks in present day transport protocols come from, it is instructive to consider a protocol such as the one based on "Go back N method" of error and flow control over a datagram network. This protocol is described, for example, by D. Comer in Internetworking with TCP/IP, Prentice Hall, 1988. Such a protocol may be in use, for example, over a network which has a large bandwidth--such as 1 Gbit/sec--and large latency--such as 60 msec (which is approximately the round trip delay for a terrestrial network between New York and San Francisco). In such a network, the states of the two communicating ends (i.e. Transmitter and Receiver) are usually out of synchronization due to the round trip propagation delay. Therefore, any change in the state of the transmitter or the receiver can be made known to the other only after a certain amount of delay. For a 1 Gbit/sec network, there can be 30 Mbits in transit from the transmitter to the receiver. Therefore, in the Go back N method of error control, if there is either a transmission error, a packet loss, long delay in delivery of a packet, or delivery of a packet out of sequence, 60 Mbits may have to be retransmitted. Similarly, if the buffer at the receiver overflows, the overflowed packets can be recovered only after 60 Mbits are retransmitted. This causes significant loss of throughput due to retransmissions. In addition, if some of the control messages from the receiver to the transmitter are received in error, elaborate error recovery mechanisms are necessary because the receiver has no way of knowing whether the transmitter has received the message correctly. Such difficulties are usually handled by timers which go off after a certain amount of waiting, determined by the round trip propagation delay. In most datagram networks this delay varies widely and is difficult to estimate. Thus, while such protocols were deemed reasonable two decades ago, when only low bandwidth was available at a considerable expense and the latency of the network was not large (due to small geographical size of most of the networks), their suitability for the future is questionable. This is due to loss of throughput and the large amount of protocol processing that results from using economized control messages that contain only changes in certain states. Moreover, the large number of control messages, and states, and the dependence on round trip propagation delay in some of the protocol processing make it difficult to parallelize the protocol processing.

Our protocol overcomes many of these difficulties without restricting the network, by using a small portion of the bandwidth for periodic exchange of states.

The new protocol exchanges control messages that contain all of the relevant state information between the transmitter and the receiver on a frequent, routine and periodic basis, independent of any significant event that may have taken place in the receiver or the transmitter. The periodic rate of state exchange depends on the activity on the logical channel. This is in sharp contrast to all the current protocols in which only changes in the state are exchanged whenever significant events take place (such as detected loss of a packet, overflow of a buffer). The instant approach greatly simplifies the protocol processing by removing the elaborate error recovery procedures and makes it easy to parallelize the protocol processing, which further improves the performance. In addition, in order to handle datagrams without losing throughput in high speed and high latency networks, we employ selective retransmission. Selective retransmission procedures typically require large tables and complex processing. To keep processing within reasonable limits, we use the concept of blocking. A group of packets (typically, 16) is called a block. The receiver acknowledges blocks, not individual packets. If one packet in a block is delivered incorrectly, the entire block is retransmitted. This enables us to make throughput almost independent of the variations in round trip delay, while keeping the processing within reasonable limits.

FIG. 1 describes the receiver's control packet;

FIG. 2 describes the transmitter's control packet;

FIG. 3 and FIG. 3a describe the transmitter's data packet arrangement;

FIG. 4 presents the flow chart for transmitter process T.sub.1 ;

FIG. 5 presents the flow chart for transmitter process T.sub.2 ;

FIG. 6 presents the flow chart for transmitter process T.sub.3 ; and

FIG. 7 presents the flow chart for receiver process R.sub.1.

As indicated above, our protocol is directed to packet oriented systems. In such systems, there is a transmitter that transmits packets to a designated receiver, a receiver that receives the packets destined to it, and a communications channel. The communications channel may have dedicated connections in a network or an interconnection of networks.

In accordance with the principles of our invention, the transmitter sends data packets to the receiver in blocks, or groups, of packets. At least as often as necessary, the transmitter also sends a control packet to the receiver. The control packet contains all of the information about the transmitter that is relevant to the receiver. For example, the transmitter sends to the receiver a measure of the number of packets that are in the transmission queue. In one embodiment of our invention, the transmitter sends its control packets periodically. The repetition rate is dependent of the level of activity of the transmitter.

Also in accordance with the principles of our invention, the receiver periodically sends a control, or a state, packet to the transmitter. The receiver's control packet provides the transmitter with all of the information about the state of the receiver that is relevant to the transmitter.

The transmitter receives the receiver's control packet and analyzes the data. If it finds that a particular block had been received with an error (or not received at all), and if a certain wait indicator permits, then it retransmits the block. At that point the wait indicator is set to prevent further retransmissions until the retransmitted block has sufficient time to be received and sufficient time to acknowledge the reception. In other words, the wait indicator is set to at least cover the round trip delay (RTD).

The information about packets that have been received correctly and incorrectly is kept in the transmitter within a table (LUP). That table is periodically updated so that only the proper blocks get retransmitted, and so that continuous block sequences that have been send and acknowledged as having been received properly can be deleted from the table, to keep the table relatively current.

By virtue of our protocol, the various operations that need to be performed in the transmitter and the receiver have all of the necessary information in the transmitter and the receiver, respectively. Therefore, those operations can be performed without interaction between the transmitter and the receiver. Furthermore, even the operations within the transmitter and the receiver are independent of each other and, accordingly, these operations can be performed in parallel.

Packets that carry data are identified by a sequence number and are numbered modulo 2.sup.32. A group of packets (typically a power of 2, such as 16) is called a block. The sequence number of a block that contains 16 packets is the number resulting from dropping the lowest 4 bits from the sequence number of the first packet in the block.

The protocol provides for error recovery, sequenced delivery, flow control, and multiplexing/demultiplexing. It has three modes of operation. Mode 0 has no error and no flow control. It is suited for virtual circuit networks in which quick interaction (e.g. terminals connected to a host) is desired. Mode 1 has no error control but does have flow control. Many real time applications, such as packetized voice or real-time monitoring of remote sensor data, can be supported by this. Mode 2 has both error and flow control. It is useful for large file transfers.

Multiplexing/demultiplexing is done in all three modes. Traffic from/to many users is multiplexed/(demultiplexed) by the protocol. A logical connection is established between two users once the connection establishment phase is completed successfully. The local connection is maintained by a logical connection identifier field (LCI) in each packet that flows from the transmitter to the receiver (and vice versa).

The description that follows assumes that an initial connection phase of the receiver to the transmitter is accomplished via the standard three-way "handshake" in which the following parameters are negotiated: mode of communication, bandwidth, packet size, block size, and buffer required at the receiver. This "handshake" standard is described by A. S. Tannenbaum in Computer Networks, Second Edition, 1988. In addition, it is expected that an estimate of round trip delay is available during the connection set-up.

Each logical connection has its own prenegotiated buffer at the receiver. This buffer is arranged according to the packet sequence numbers and random update/access to any sequence numbered location within its range is possible. This buffer can typically store blocks that the transmitter would send during one roundtrip delay (L seconds). Its size should be slightly larger than (RTD.times.negotiated bandwidth) bits, where RTD is the round trip delay. Thus, as the receiver accepts valid packets, it stores them according to their sequence in the buffer memory and it keeps a record of the largest sequence number (LW.sub.r) of a packet that has been received properly and which is characterized by no packets of a lower sequence number that have not been received properly. Described differently, the receiver's buffer keeps packets in ascending sequence number order. First are the sequence numbers of blocks that have been received properly but have not yet been read by the host that is connected to the receiver's protocol interface. Associated with these sequence numbers, the buffer contains, of course, the received blocks of packets. Next comes a sequence number that corresponds to a block that has not been received, or has been received with an error (i.e. a packet or packets in the block are in error). That sequence number is LW.sub.r +1. Thereafter come the sequence numbers of other blocks that have been received properly or improperly. The total number of packets kept is not more than the receiver's buffer.

The host that is connected to the receiver's interface protocol also uses the LW.sub.r number. Specifically, the host is restricted to read from the buffer only up to LW.sub.r. In the case where there are no missing blocks, LW.sub.r is set to the sequence number of the last block in the buffer.

FIG. 1 depicts the control, or state, packet that flows from the receiver to the transmitter. In a sense, this packet is the primary transmission control packet because it serves the acknowledgement function. The LCI field in FIG. 1 identifies the Logical Connection that ties a particular transmitter to a specific receiver. The unique identifier in LCI is assigned to the logical connection during the connection establishment phase. The second field in FIG. 1 defines k, which is the interval between two control packet transmissions expressed in units of T.sub.IN. T.sub.IN is the minimum time between two state transmissions. It is a predefined parameter. The next field in FIG. 1, LOB, is the List of Outstanding Blocks field. It corresponds to the received blocks about which the control packet reports to the transmitter. More specifically, the LOB field is encoded as a bit map, with each bit representing the reception status of a corresponding block of packets. The first bit indicates whether the block identified by LW.sub.r was received correctly (1), or incorrectly (0). The second bit indicates whether the block having the sequence number LW.sub.r +1 was received correctly (1), or incorrectly (0), etc. As indicated above, a block is said to have been received incorrectly when at least one packet in the block was not correctly received.

One way to look at the LOB field is that the number of bits in the LOB field is related to the level of redundancy that is desired. In one embodiment, we employ an LOB field of L-bits, where L is the number of blocks in transit. When the interval between successive receiver control blocks is one tenth the round-trip delay, use of L-bits in the LOB block represents a ten-fold redundancy. Higher or lower redundancies are, of course, possible. Of course, the receiver's buffer size (in terms of blocks of packets) must be greater than the number of bits in LOB. It also should be pointed out that when there are no missing blocks in the receiver and LW.sub.r is equal to largest block sequence, then there is only one active bit in the LOB field.

The next field in FIG. 1 is the buffer-available field. Through this field, the receiver informs the transmitter of the amount of buffer that is available at the receiver. When the buffer-available figure is below a predetermined level, the transmitter has the option of ceasing transmission or, at least, lowering the rate of transmission.

The last field in FIG. 1 is the error-check field. This field contains an error checking and correcting code that permits the transmitter to determine that the receiver's control packet was received correctly.

FIG. 2 describes the control packet sent by the transmitter. The first field in FIG. 2 is the LCI field. This field may differ from the LCI field of FIG. 1 in that this LCI field must include a bit that identifies whether the transmitted packet is a data packet or a control packet. Of course, this additional bit can be included in the FIG. 1 LCI field to make the two fields identical in format.

The second field in FIG. 2 is the k field, and it is identical to the corresponding field in FIG. 1.

The third field in FIG. 2 is the UW.sub.t field. It carries the block sequence number of the last data block transmitted immediately prior to the transmission of the transmitter's control block. The penultimate field in FIG. 2 identifies the number of packets queued for transmission. This number may be large, indicating that the transmitter is heavily burdened with data, or it may be very small--perhaps less than the number of packets to fill a complete block. In both instances, this number is used by the receiver to help it respond properly to the transmitter's state. The number of blocks queued for transmission can also help the receiver to decide whether to accept connection to another transmitter.

The last field in FIG. 2, as in FIG. 1, is the error check field which permits the receiver to verify that the transmitter's control packet was received accurately.

FIG. 3 and FIG. 3a depict the structure of the data transmission by showing a sequence of transmitted blocks and by further showing the first packet in the first block. That packet, as all other packets, contains a sequence number, the user data, and an error check field. The sequence number is the aforementioned modulo 2.sup.32 number. The error check field in FIG. 3a is a 16 bit field. Advantageously, it can contain a standard CRC (cyclic redundancy code). The error check information may come from the lower layer if such error detection is done at that level.

As indicated above, the field k specifies the number of control packet transmissions within one round trip delay. The value of k is negotiated at the connection establishment time but can vary with the activity on the logical connection. That is, when a logical connection is set up, k is set to a constant such as 32 (typically a power of 2), when the logical connection becomes inactive, the frequency of data transmission is reduced by a factor of 2. Further reductions follow if the transmitter remains inactive and, in this manner a number of consecutive frequency reductions can be made; but not below some predetermined frequency. For example, a reduction factor of 8 may be the largest acceptable frequency reduction factor. When the logical connection becomes active again, the frequency jumps back to the original level, i.e. the value of k is reduced to the original constant.

As indicated earlier, our protocol is well partitioned from a processing standpoint at both the transmitter and the receiver. The transmitter requires three processes (T.sub.1, T.sub.2, and T.sub.3) and the receiver requires four processes (R.sub.1, R.sub.2, R.sub.3, and R.sub.4). There is very little interaction between these processes.

Transmitter process T.sub.1 determines whether there is any block that needs retransmission. For that determination, the transmitter maintains an LUP table. An entry in the LUP contains the sequence number of an outstanding block (seq #), its retransmission count (count), and a bit indicating whether it has been successfully acknowledged (ack). When a block is transmitted, the retransmission count for that block is set to (C+2), where C is equal to RTD/T.sub.IN. Thereafter, it is decremented by k every time the transmitter receives the state packet from the receiver. The constant 2 is added to C to insure that the time-out period is slightly larger than the round-trip delay. Since the round trip delay can vary, one would want to make sure that a block does not have to be retransmitted simply because the acknowledgement arrives a little late. A block is scheduled for retransmission only if it is not yet acknowledged and its retransmission count is 0.

Process T.sub.1 examines the LUP table and, based on its contents, it picks either a new block or a previously transmitted block and schedules it for transmission. After transmitting the block, the process makes appropriate updates to the LUP table; to wit it sets the count to C+2.

FIG. 4 presents a flow diagram of process T.sub.1. Step 10 determines the mode of operation. When the mode is 2, indicating that error control is in effect, control passes to step 20 which determines whether the LOB field in the received control packet reveals a need to retransmit a block. If it does, control passes to step 30 which retransmits the block and sets a "busy" indicator to 0. This indicator is used in process T.sub.3 to control the frequency of control packet transmissions. When the LOB field does not reveal a need to retransmit a block, control passes to step 40. Control also passes to step 40 when the indication at step 10 is that mode is not equal to 2. At step 40, the mode is again queried. When it is not 0, control passes to step 50 which determines whether space is available at the receiver's buffer. If space is available, or when step 40 determines that the mode is equal to 0, control passes to step 60 where a new block is transmitted and the "busy" indicator is set to 0.

Process T.sub.2, which updates the LUP table, is presented in FIG. 5. Step 70 in FIG. 5 captures the sequence number, LW.sub.r, and the buffer available information from the control block sent by the receiver. Step 80 determines the mode of operation. When the mode of operation is 2, indicating that error control is in effect, control passes to step 90 which updates the LUP table with the information contained in the LOB field.

Process T.sub.3 transmits the state of the transmitter at regular intervals, as described above. The output of process T.sub.3 is multiplexed with the output of process T.sub.1. In case of conflict, state packets get priority over data packets.

FIG. 6 describes the transmitter process T.sub.3. The main loop in the process T.sub.3 is depicted by step 100, which basically is a timer that determines the expiration of k intervals. When the step determines that it is time to send a control packet, control passes to step 110. Step 110 determines whether a control packet was received from the receiver within a predetermined time interval (LIM seconds). Should it occur that a control packet had not been received from the receiver within the LIM seconds, the system moves to state DISC which disconnects operations. Otherwise, the transmitter's control packet is sent by step 120 and control passes to step 130. Step 130 determines the mode of operation. When the mode of operation is 2, step 140 ascertains the status of the "busy" variable set by process T.sub.1. When busy equals 0, step 150 sets k to its original value and sets "busy" to 1. When "busy" equals 1, step 160 sets k to the minimum of 2k or a constant (such as 8 times the original value of k) and also sets "busy" to 1. Control returns thereafter to step 100 to determine the next time a control packet is to be sent in accordance with the new value of k.

It is assumed that the data packets received from the network are first checked for errors by the link layer (or a front-end of the transport layer) and if there are errors in a packet, it is simply dropped; otherwise it is delivered to process R.sub.1. A variable "busy" is set in the receiver process R.sub.1 to 0 when a packet arrives. This variable is used in a similar manner as the variable "busy" in the transmitter.

FIG. 7 illustrates the receiver process R.sub.1. Step 170 directs operations by determining the mode. When mode is 0, the received packet is immediately forwarded to the host by step 180. When the mode is 1, packet is stored in the receiver's buffer by step 190. When the mode is 2, control passes to step 200. Step 200 determines whether the sequence number of the received packet is not greater than LW.sub.r. In such a case, the packet is one that had been received accurately before but was retransmitted anyway. This packet is ignored. When the sequence number is greater than LW.sub.r, control passes to 210 where the packet is stored in the receiver's buffer and, as necessary, information is stored concerning the block of packets to permit updating of LW.sub.r.

Process R.sub.2 updates the LW.sub.r variable and runs a timer on the interval between state transmissions by the transmitter. When it is detected that the transmitter has not sent a control packet within a predetermined time interval, the system enters the DISC state and ceases operations.

Process R.sub.3 develops the control packet that is sent by the receiver to the transmitter. It basically follows the flow chart of process T.sub.3 (FIG. 6).

Process R.sub.4 delivers fully acknowledged and sequenced packets to the receiver host on demand from the host. It should be noted that while the unit of data transmission over the network is a block containing many packets (e.g. 16), the delivery to the receiver host is in terms of packets.