US 6,894,974 B1Grant

Method, apparatus, media, and signals for controlling packet transmission rate from a packet source

Issue Date:2005-05-17

•21 Claims

•10 Drawing Sheets

Abstract

A method, apparatus, computer readable media and signal embodied in a carrier wave for controlling the rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network involves adjusting an acknowledgement rate at which acknowledgement signals are sent to the source, in response to the difference between target queue occupancy and measured queue occupancy of a queue buffer in which data packets from the source are received.

Metadata

Assignee

Nortel Networks Limited

Inventors

James Aweva
Michel Ouellette
Delfin Y. Montuno

Application Information

Application Number:US 09/566,391

Filing Date:2000-05-08

Priority Date:2000-05-08

Art Unit:2666

Patent Drawings (10 sheets)

Description

Background of the Invention

[0001] 1. Field of the Invention

[0002] The present invention relates to packet network communications, and more particularly to method, apparatus, media, and signals for controlling a packet transmission rate from a packet source to reduce queue congestion.

[0003] 2. Description of Related Art

[0004] The rapid growth of applications and services over the Internet has made the performance of the Transmission Control Protocol (TCP) a major issue in data networking. TCP is an end-to-end transport protocol that is designed to work with any underlying network technology. The TCP protocol makes no assumption on how the network processes data it sends, and has its own data recovery and flow control features. The basic TCP does not rely on any explicit feedback from the network to detect congestion. Instead, it relies on packet loss as the only means of signaling network congestion. Every time a TCP data source detects a packet loss, it assumes that the loss is due to buffer overflow somewhere in the network. The source then reduces its data transmission rate to alleviate the congestion that could have caused the packet loss.

[0005] Specifically, congestion control and error recovery are implemented by a dynamic window in TCP. The source expects to receive an acknowledgment from the destination for every packet it transmits. The size of the dynamic window determines the number of unacknowledged packets that can be present in the network at any given time. When a packet is acknowledged, the window size is increased (up to a certain maximum), and when a packet is lost, the window size is decreased, forcing the source data rate to decrease.

[0006] In a high-latency network environment, the dynamic window flow control mechanism of TCP may not be very effective because it relies on packet loss to signal congestion, instead of avoiding congestion and buffer overflow. Furthermore most implementations use coarse timers, with 200-500 ms granularity, for example, which results in delayed response. In general, by the time the source starts decreasing its data rate because of packet loss, the network has already been overly congested. Ideally, it would be desirable for the source to react to congestion before it occurs rather than acting when it is too late.

[0007] By incorporating efficient and faster congestion indication mechanisms in the end-to-end TCP feedback control loop, the performance of TCP can be improved significantly. Providing such mechanisms by modifying or changing the senders or receiver's TCP stack is an option. However, this may not be a feasible solution in the near term since the large installed base of TCP end-user implementations have to be modified or changed. In addition, only upgraded end-users get the performance improvements. What is needed are solutions which ensure compatibility with current TCP end-user implementations, expedite market acceptance, and meet current data networking requirements.

[0008] A number of mechanisms have been proposed to use a network node as a possible place to detect congestion and take the necessary control decisions to limit the impact of congestion on network performance with these requirements in mind. Some mechanisms are based on implicit feedback and only indirectly influence TCP behavior, e.g., Random Early Detection (RED) which relies on packet dropping for congestion control.

[0009] ACK pacing is a more direct technique for regulating the TCP source rate by controlling the rate of arrival of acknowledgments to the TCP source. ACK pacing is analogous to the storage of permits in the well-known leaky-bucket scheme. In this scheme TCP acknowledgments can be viewed as permits that allow the destination (and also a network element through a slower pacing interval) to request packets from the source. The ACKs arrive at the network node according to TCP dynamics and are drained out at a configured rate. ACK pacing at a network node eliminates the need to change or modify the sender or receiver's TCP stack.

[0010] Studies have shown that by appropriately regulating the inter-ACK spacing, an intermediate network node can essentially reduce queue buildups and buffer requirements, TCP timeouts and packet loss while still achieving high link utilization. A number of algorithms have been proposed for determining the proper ACK spacing in network nodes. Some of these algorithms include techniques to infer when a TCP connection is in slow-start versus congestion avoidance, so that the desired spacing between ACKs can be adjusted accordingly to provide more spacing in the slow-start phase, for example. Most of the existing ACK pacing algorithms do not offer optimal behavior because it is difficult to design an algorithm that can determine the proper ACK spacing for a wide range of network conditions. In addition, it will be very difficult, if not impossible, to infer at a network node the (slow-start or congestion avoidance) phases of the aggregate TCP traffic mix.

Summary of the Invention

[0011] The present invention addresses the above need by providing a method of controlling the rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network. In a broad sense the method comprises adjusting an acknowledgement rate at which acknowledgement signals are sent to the source, in response to the difference between target queue occupancy and measured queue occupancy of a queue buffer in which data packets from the source are received.

[0012] Adjusting the acknowledgement rate may be done such that the difference between the target queue occupancy and the measured queue occupancy is minimized.

[0013] The method may involve producing a difference value representing the difference between the target queue occupancy value and the measured queue occupancy value and filtering successively produced difference values to produce a final difference value.

[0014] Filtering may involve producing the difference value as an exponentially weighted moving average of the successively produced difference values.

[0015] Preferably the method involves transmitting acknowledgement signals at a rate determined as a function of a present rate and the difference value. This rate value may be produced according to the relation:
r(n)=r(n−1)+αe(n)
where:

- [0016] r(n) is the rate value at time interval n;
- [0017] r(n−1) is a rate value immediately preceding the rate value at time interval n;
- [0018] α is a control gain value; and
- [0019] e(n) is a difference value representing the difference between the target queue occupancy and the actual queue occupancy.

[0020] Preferably, bounds are imposed to limit the acknowledgement rate.

[0021] In accordance with another aspect of the invention, there is provided a computer readable medium for providing instructions for directing a processor to carry out a method of controlling the rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network, the instructions being operable to direct a processor to adjust an acknowledgement rate at which acknowledgement signals are sent to the source, in response to the difference between target queue occupancy and measured queue occupancy of a queue buffer in which data packets from said source are received.

[0022] In accordance with another aspect of the invention, there is provided a signal embodied in a carrier wave having a segment for directing a processor to adjust an acknowledgement rate at which acknowledgement signals are sent to a source, in response to the difference between target queue occupancy and measured queue occupancy of a queue buffer in which data packets from said source are received.

[0023] In accordance with another aspect of the invention, there is provided an apparatus for controlling the rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network. The apparatus comprises provisions for detecting queue occupancy of a queue buffer in which data packets from the source are received and provisions for adjusting an acknowledgement rate at which acknowledgement signals are sent to the source, in response to the difference between target queue occupancy and measured queue occupancy of a queue buffer in which data packets from said source are received.

[0024] In accordance with another aspect of the invention, there is provided an apparatus for controlling the rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network. The apparatus comprises a queue monitor for producing signals indicative of queue occupancy of a queue buffer in which data packets from said source are received and an acknowledgement rate control signal generator for generating a rate signal in response to the difference between target queue occupancy and measured queue occupancy of a queue buffer in which data packets from said source are received, for adjusting an acknowledgement rate at which acknowledgement signals are sent to the source.

[0025] Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying figures.

Brief Description of the Drawings

[0026] In drawings which illustrate embodiments of the invention,

[0027] FIG. 1 is a schematic representation of a network employing a network element configured with an apparatus according to a first embodiment of the invention;

[0028] FIG. 2 is a schematic block diagram of the network element and apparatus according to the first embodiment of the invention;

[0029] FIG. 3 is a block diagram of the apparatus according to the first embodiment of the invention;

[0030] FIG. 4 is a queue monitor routine implemented by a block of instructions stored in the apparatus shown in FIG. 3;

[0031] FIG. 5 is a flowchart of a rate determiner routine implemented by a block of instructions stored in the apparatus shown in FIG. 3;

[0032] FIG. 6 is a flowchart of a threshold smoother routine implemented by a block of instructions stored in the apparatus shown in FIG. 3;

[0033] FIG. 7 is a flowchart of an acknowledgement schedule monitor routine implemented by instructions stored in the apparatus shown in FIG. 3;

[0034] FIG. 8 is a flowchart of an acknowledgement redundancy controller routine implemented by a block of codes stored in the apparatus shown in FIG. 3; and

[0035] FIG. 9 is a schematic representation of a multiple class network element according to a second embodiment of the invention.

Detailed Description

[0036] As shown generally at 10 in FIG. 1, a packet network according to a first embodiment of the invention includes a first data transmitter 12, a network element 14 and a first data receiver 16. In general, the transmitter 12 transmits data packets in a forward direction to the network element 14, which, in turn, transmits the data packets to the receiver 16. It will be appreciated that there may be a plurality of network elements between a plurality of transmitters and a plurality of receivers, however, for simplicity only one of each is shown.

[0037] In this embodiment, the data packets transmitted by the transmitter 12 are transmitted as “forward” data packets 18, which are communicated in a forward direction i.e. from the transmitter 12 to the receiver 16. In this specification, the term “packet” is applied broadly, and contemplates any quantum of data, such as a block, a frame, a datagram, a cell, a word, a byte, or a bit, for example. In general, a transmitter 12-receiver 16 pair that exchanges packets via one or more network elements 14 is called a connection.

[0038] The first transmitter 12 may be any device capable of transmitting data packets on a network, for example a telephone, a computer, a terminal, a video camera, an appliance with embedded logic or processor circuitry, or more generally any telecommunication or telephony device. Additionally, the transmitter 12 may include a receiver 23 for receiving data packets from the network 10.

[0039] The receiver 16 may be any device capable of receiving data packets from a network, for example a telephone, a computer, a terminal, a video receiver, an appliance with embedded logic or processor circuitry, or more generally any telecommunication or telephony device. The receiver 16 includes a receive buffer 24 for receiving forward data packets 18 for use at the receiver 16. Additionally, the receiver 16 has a transmitter 26 for transmitting data packets on the network 10.

[0040] When the receiver 16 receives a forward data packet 18, it engages its transmitter 26 to transmit an acknowledgement signal in the form of an acknowledgement packet, in a reverse direction for receipt by the transmitter 12 via the network element 14 associated with the connection. Generally, an acknowledgement signal is a special reverse data packet transmitted in the reverse direction, i.e. from the receiver 16 to the transmitter 12, and includes a specific pattern of bits that identifies it as an acknowledgement signal.

[0041] The transmitter 12 includes a transmission buffer 22 for queuing forward data packets 18 prior to transmission. The number of forward data packets 18 which are transmitted from the transmission buffer 22 in a transmission is determined by a sliding window called a “congestion window” maintained by a processor at the transmitter 12 and operating on the transmission buffer 22. Each time the transmitter 12 receives an acknowledgement packet from the receiver 16, the congestion window advances, permitting the transmitter 12 to transmit a new set of forward data packets 18 onto the network 10. The size of the congestion window determines the number of forward data packets 18 transmitted from the transmitter 12 and the rate at which acknowledgement packets are received determines the rate at which sets of forward data packets are transmitted and hence determines the data packet transmission rate from the transmitter 12.

[0042] Referring to FIG. 2, a network element 14 according to the first embodiment of the invention is shown in greater detail and includes an apparatus 38 for controlling the rate at which data packets are transmitted from a source which in this embodiment is the transmitter 12. The apparatus works with sources which are responsive to acknowledgement signals transmitted from the receiver 16. To do this, the apparatus 38 is in communication with a forward data queue buffer 40 and an acknowledgement packet processor shown generally at 42, of the network element 14.

[0043] Effectively, the apparatus 38 adapts an acknowledgement pacing rate r at which acknowledgement packets are transmitted to the transmitter 12 from the network element 14, in response to the traffic load at the network element by increasing the rate r to make aggressive use of the forward data queue buffer 40 when a measured queue parameter is less than a target value and by decreasing the rate r to reduce the burden on the forward data queue buffer 40 when the measured queue parameter is greater than the target value. The measured queue parameter may be queue occupancy, for example. The goal is to adapt r to react to the traffic dynamics and to keep the queue parameter close to the target value in order to minimize packet losses and TCP timeouts, and improve data throughput.

[0044] The forward data queue buffer 40 is a first-in first-out buffer implemented in random access memory (not shown) of the network element 14. A processor (not shown) in the network element 14 defines and maintains input and output pointers defining memory locations at which new forward data packets are to be loaded into the forward data queue buffer 40 and at which forward data packets are to be taken from the forward data queue buffer 40 for transmission to the TCP destination or receiver 16. The forward data queue buffer 40 has a buffer size B representing the number of data packets which can be held in the buffer at any given time. The selection of buffer size depends on buffer dimensioning and traffic engineering and influences packet losses and system utilization level. Typically B is selected to be at least equal to one bandwidth-delay product worth of data.

[0045] The acknowledgement signal processor 42 is also part of the network element and in this embodiment, it includes a detector 44 for detecting a conventional data packet or an acknowledgement signal which, in this embodiment, is an acknowledgement data packet, produced by the receiver 16 in response to receipt of a forward data packet from the transmitter 12. On detection of an acknowledgement packet, such packet is forwarded to an acknowledgement signal queue 46 from which an acknowledgement signal transmitter 48 receives packet signals and transmits them through a multiplexer 50 to the first data transmitter 12, in response to a signal representing an acknowledgement token received from the apparatus 38.

[0046] If the received packet is a conventional data packet, the detector 44 forwards the packet to a reverse data buffer 52 which provides such packets to the multiplexer 50, also for transmission to the transmitter 12.

[0047] Acknowledgement packet identification is required to separate acknowledgement packet flow from normal data traffic. To identify an acknowledgement packet, the detector checks an ACK bit in a packet received from the receiver 16 to see if the bit is set. It is also possible that acknowledgement information might be carried (piggybacked) in a data packet traveling in the reverse direction. Acknowledgement information can be separated from the data packet by first clearing the ACK bit in the data packet and then a new ACK packet can be generated and stored in the acknowledgement signal queue 46, with acknowledgement information copied from the data packet.

[0048] In one sense, the apparatus 38 monitors the forward data queue buffer 40 and determines when to send an acknowledgement token signal to the acknowledgement signal transmitter 48 to cause an acknowledgement packet to be transmitted from the network element 14 in response to forward data queue occupancy at the forward data queue buffer 40 of the network element 14.

[0049] Referring to FIG. 3, to do this the apparatus 38 comprises a processor circuit shown generally at 60. In this embodiment, the processor circuit 60 comprises a processor 62 in communication with random access memory (RAM) 64, an input/output 110 interface 66, and program memory 68. Optionally and preferably, the processor circuit 60 further comprises a communications interface 70 and a media interface 72.

[0050] The I/O interface 66 preferably has a q-threshold input 74, a q-size input 76 and optionally has an acknowledgement buffer size input 78. It also has an acknowledgement rate control signal output 80 and optionally has an acknowledgement removal signal output 82.

[0051] The q-threshold input 74 may be in communication with a processor (not shown) of the network element 14, to receive a value representing a q-threshold level (T) therefrom. The q-size input 76 is in communication with the forward data queue buffer 40 shown in FIG. 2, and is operable to receive values representing the input and output pointers of the forward data queue buffer 40. The acknowledgement buffer size input 78 is operable to receive values representing the input and output pointers of the acknowledgement signal queue 46. The acknowledgement rate control signal output 80 is in communication with the acknowledgement signal transmitter 48 and provides acknowledgement rate control signals representing tokens for controlling the acknowledgement signal transmitter 48 to cause it to transmit an acknowledgement signal from the acknowledgement signal queue 46 through the multiplexer 50 to the transmitter 12. The acknowledgement removal signal output 82 is operable to provide an acknowledgement removal signal to the acknowledgement signal queue 46 to facilitate removal of acknowledgement packets therefrom.

[0052] The program memory 68 is preferably loaded with blocks of program instructions for directing the processor 62 to accomplish various functions used in carrying out the method according to this embodiment of the invention. Such blocks of program instructions may be provided to the program memory 68 by causing the processor to read such instructions from a memory medium such as a CD-ROM, floppy diskette or other processor readable medium in communication with the media interface 72. The media interface may include a floppy disk drive for this purpose, for example. Alternatively, the instructions may be provided in a signal embodied in a carrier wave received at the communications interface 70, for example, the signal having various segments for providing various functional blocks of instructions.

[0053] In this embodiment, the program memory 68 is loaded with program instructions including a q monitor block 84, an ack rate signal generator block 86 and an optional ack redundancy controller block 88.

Q Monitor

[0054] The q monitor block 84 directs the processor circuit 60 to measure the queue size, and produce a signal indicative of queue occupancy. Referring to FIG. 4, to do this, in this embodiment, the q monitor block 84 includes a block of instructions 90 which directs the processor circuit 60 to act as a sampler for sampling the forward data queue buffer 40 to determine values for the input and output address pointers of the forward data queue buffer 40. Block 92 then directs the processor to determine a queue parameter, which may include a measure of the actual queue occupancy or a measure of the rate at which the queue occupancy is changing, the queue parameter being derived from the input and output address pointers of the forward data queue. Actual queue occupancy is determined by taking the difference between the input and output address pointers of the queue buffer 40. The difference represents the number of packets in the queue, that is, the measured queue occupancy.

[0055] Block 94 directs the processor to store the queue parameter for later use.

Ack Rate Signal Generator

[0056] The ack rate signal generator block 86 includes blocks of instructions which direct the processor to implement a rate determiner 96, an ack schedule monitor 98 and an optional q-threshold smoother 100. The rate determiner is shown generally at 96 in FIG. 5.

[0057] Referring to FIG. 5, the rate determiner 96 begins with a first block 110 which directs the processor to call the q monitor block 84, shown in FIG. 4, to acquire a new value for the queue parameter. Having obtained this new value, block 112 directs the processor to determine a difference value representing the difference between a target queue occupancy value and the measured queue occupancy value.

[0058] The target queue occupancy value used in determining this difference is a stored value derived from the q-threshold input 74 of the I/O interface 66 shown in FIG. 3. Recall that the value received at the q-threshold input 74 may be provided by the processor of the network element and thus may vary in time because of system operation needs. To prevent acknowledgement token signals from being produced at an erratic rate, the value received at the q-threshold input 74 may be filtered by a threshold smoothing routine as shown at 114 in FIG. 6. Referring to FIG. 6, the threshold smoothing routine 114 essentially limits the rate of change of the threshold value T received at the q-threshold input 74. Thus the routine directs the processor to act as a ramp unit or rate limiter. Effectively, when the threshold value T is within a certain range −a≦T≦a the threshold smoother directs the processor to provide a value y which follows the value at the q-threshold input 74. An integrator action in the routine effectively causes the output value y to be identical to the input value T in the steady state. However, the integrator action is applied to an amplitude limited input signal T, hence the rate of change of the output y will be limited to the bounds imposed on the input value T. Hence the smoothing action of the threshold smoother may be described by the following equations:

\frac{ⅆ y}{ⅆ t} = sat (x) = sat (T - y)

in continuous-time domain and

\begin{matrix} Δ y (n) = y (n) - y (n - 1) = sat (T - y (n - 1)) \\ y (n) = y (n - 1) + sat (T - y (n - 1)) \end{matrix}}

in discrete-time domain.

[0059] The amplitude limiter or saturation sat(e) is defined as

sat (e) = {\begin{matrix} - a, & x \leq - a \\ x, & \langle x \rangle < a \\ a, & x \geq a \end{matrix}

- [0060] where the limit a is a ramping step size and can be defined as a small fraction of a threshold step size ΔT, i.e., α=g·ΔT, 0<g<1, if ΔT is large enough to require a smoothing of T.

[0061] Smoothing may not be required for small ΔT. In FIG. 6, y is the smoothed threshold input to the computations of the new acknowledgement rate.

[0062] In this embodiment, the smoothing of the value T is implemented in discrete time steps by the routine shown at 114 in FIG. 6, which begins with a first block 116 which initializes the value y to 0. Block 118 then directs the processor to determine whether the following expression is non-zero:
x=T−y

[0063] If so, then block 120 directs the processor to determine a value for y according to the relation:
y←y+sat(x)
where:

sat (e) = {\begin{matrix} - a, & x \leq - a \\ x, & \langle x \rangle < a \\ a, & x \geq a \end{matrix}

[0064] Block 122 then directs the processor to store the value y for later use in calculations and then block 124 directs the processor to determine whether or not it is time to recompute. If so, then the process is repeated beginning at block 118. Otherwise, the processor waits until it is time to recompute.

[0065] If at block 118, the value x is equal to zero, there is no need to recalculate a value for y and the processor is directed to block 124 to wait for the next time to recompute.

[0066] Preferably, the processor 62 shown in FIG. 3 is fast enough to ensure that the time interval between each computation of a y value is smaller than the time interval between any changes to the threshold value T.

[0067] Referring back to FIG. 5, block 126 directs the processor to calculate an instantaneous difference value according to the relation:
e(n)=T(n)−q(n) (in units of 64 bytes)
where:

- [0068] e(n) is the instantaneous difference value at time n.
- [0069] T(n) is a target queue occupancy value at time n or a smoothed target queue occupancy value y(n), if smoothing is applied.
- [0070] q(n) is the queue parameter value at time n.

[0071] Block 128 then directs the processor to filter the instantaneous difference value with a low pass filter routine. Low pass filtering is desirable because network traffic can be very bursty, hence the measured queue occupancy can also be bursty, resulting in a bursty instantaneous difference value.

[0072] In this embodiment the low pass filter routine effectively provides a discrete-time first-order low-pass filter with gain 0<β<1. The value β controls the reaction speed of the filter. As β decreases the ê curve becomes smoother. The value β should be selected small enough to eliminate the noisy fluctuations, and large enough to discover the changes in traffic dynamics. A suggested value is β=0.002.

[0073] The filtered difference value is given by
ê(n)=(1−β)ê(n−1)+βe(n).

[0074] The filtered difference value appears as an exponentially weighted moving average (EWMA) of the error signal and can be expressed as

\begin{matrix} \hat{e} (n) = {(1 - β)}^{n} \hat{e} (0) + β \sum_{i = 0}^{n - 1} {(1 - β)}^{i} e (n - i) \\ = β \sum_{i = 0}^{n - 1} {(1 - β)}^{i} e (n - i), if \hat{e} (0) = 0 \end{matrix}

[0075] Still referring to FIG. 5, after the filtered difference value has been calculated, block 130 directs the processor to compute a new acknowledgment rate value. In this embodiment, this may be done by a first block 132 which directs the processor to calculate a new rate value and block 134 which directs the processor to bound the new rate value between predefined limits.

[0076] Effectively, these blocks direct the processor to incrementally change the acknowledgement pacing rate r in proportion to the filtered error signal. These blocks implement a simple recursive or integral control scheme by calculating a new rate value r(n) according to the relation:
r(n)=r(n−1)+αê(n) (in ACKs/sec)

- [0077] where α is a control gain.

[0078] Effectively the value α controls the reaction and stability of the control system implemented by the apparatus 38 and the transmitter 12. As the gain value α increases, the stronger the change in the acknowledgement pacing rate r to a variation in ê. The gain value α must be large enough to cause a change in the acknowledgement pacing rate to a change in ê, but not too large to avoid the emergence of undesired oscillations.

[0079] The basic recursion r(n)=r(n−1)+αe(n) implements a standard integral or summation control scheme since Δr(n)=r(n)−r(n−1)=αe(n) or

r (n) = α \sum_{i = 0}^{n} e (i),

in discrete time (and dr(t)/dt=αe(t) or

r (t) = α \int_{0}^{1} e (τ) ⅆ τ,

in continuous-time).

[0080] The bounding of the new ack rate as provided by block 134 places minimum and maximum limits on the value r(n) according to the relation:

r (n) = {[r (n - 1) + α \hat{e} (n)]}_{r_{\min}}^{r_{\max}} (in ACKs / \sec)

- [0081] where r_minand r_maxare, respectively, lower and upper bounds on r.

[0082] The upper bound r_maxcan be set equal to a forward data buffer service rate C(n) (i.e., the allocated capacity), and r_minto C(n)/3, for example, or any suitable value r_min>0.

[0083] The limits must be selected carefully, however, too hard a limitation will impose an unnecessary limitation on the control action, and too weak a limitation will cause windup of the rate value, due to the integrator action in the method of calculation. A suitable initial value for r(0) may be C(n)/2, for example.

[0084] For network traffic control with a wide range of operating conditions, it may happen that there is a persistent control error (or mismatch) between the control target value T and the measured queue occupancy q. This persistent error may be due to rate constraints at certain network nodes. For example, the errors may be due to rate constrained traffic source(s), rate constraints due to downstream bottlenecks, or rate reduction due to upstream and/or downstream packet losses. Bounding the rate value reduces the effects of these uncontrollable sources of error.

[0085] Still referring to FIG. 5, after producing a new rate value, block 136 directs the processor to determine whether or not it is time to resample the queue pointers and if so, directs the processor back to block 110. If not, the processor waits until it is time to resample.

[0086] Thus, the rate determiner directs the processor to use information from the queue to determine a rate at which acknowledgement packets are to be transmitted to the transmitter 12.

Acknowledgement Schedule Monitor

[0087] Concurrently running with the rate determiner is the acknowledgement schedule monitor shown at 140 in FIG. 7, which directs the processor to use the current rate value r to produce a new departure time value representing a next departure time at which an acknowledgement packet is to be transmitted by the acknowledgement signal transmitter 48 shown in FIG. 2. To do this, the ACK schedule monitor 140 maintains a head-of-line (HOL) tag, HOLTag. This tag (or timestamp) represents the next time at which the ACK transmitter should send an acknowledgement packet to the TCP transmitter 12. Block 142 produces a HOLTag value according to the relation:

HOLTag \leftarrow HOLTag + TransTime + \frac{1}{r (n)}, (in seconds)

Where:

- [0088] TransTime is the time taken to transmit the last ACK packet (TransTime=0 if there is no waiting ACK packet); and
- [0089] r(n) is the currently calculated rate value produced by the rate determiner shown in FIG. 5.

[0090] The value HOLTag is initialized to a current time value on receiving a first ACK packet in the acknowledgement signal queue 46. The current time value may be provided by a realtime clock running at the processor, for example. The HOLTag value is updated even in the absence of waiting ACK packets in the acknowledgement signal queue.

[0091] Block 144 directs the processor to determine whether a current time value TNow is greater than or equal to the current HOLTag value and if so block 146 directs the processor to send an acknowledgement token signal to the acknowledgement signal transmitter 48 shown in FIG. 2, to cause the acknowledgement signal transmitter 48 to transmit an acknowledgement packet from the acknowledgement signal queue 46 to the transmitter 12. The processor then is directed back to block 142 to calculate a new HOLTag value.

[0092] If the current time value TNow is not greater than or equal to the current HOLTag value the processor waits until it is.

[0093] To control the number of ACK packets queued in the ACK buffer on the reverse link (if needed), an ACK filtering mechanism can be used. This is provided by the optional ack redundancy controller block 88 shown in FIG. 3. Use of the ack redundancy controller block 88 is just a pre-cautionary measure to prevent the acknowledgement signal queue 46 from overflowing if ever the acknowledgement rate r gets too low. Use of the ACK redundancy controller exploits the cumulative nature of TCP acknowledgement packets to filter redundant acknowledgement packets from the queue when a new acknowledgement packet arrives for the same connection. Effectively some or all of the “earlier” acknowledgement packets are removed from the queue and discarded. To do this, referring to FIG. 8, the ack redundancy controller begins with block 180 which directs the processor to determine whether or not the number of acknowledgement packets in the acknowledgement signal queue 46 is greater than a predefined number. The number of packets in the acknowledgement signal queue 46 is found by reading the acknowledgement buffer size input 78 of the I/O interface 66 shown in FIG. 3 and the signal received at this input may be generated by the processor (not shown) of the network element.

[0094] Next, blocks 182 and 184 direct the processor to send signals from the acknowledgement removal signal output 82 to the network element processor to cause it to scan the acknowledgement signal queue 46 for redundant acknowledgement packets and to remove such packets therefrom. Since the transmitter 12 increments its congestion window based on the number of acknowledgement packets received and not on how many bytes are acknowledged with each acknowledgement packet, congestion window growth at the transmitter could be much slower. Thus, when a new acknowledgement packet is received at the acknowledgement signal queue 46 and there is a danger of overflow as indicated by signals received at the acknowledgement buffer size input 78, a number of the acknowledgement packets belonging to the same connection can be removed by directing the processor to produce signals at the acknowledgement removal signal output 82 of the I/O interface 66 to cause the network element processor to remove such acknowledgement packets.

[0095] In addition, ACK-first scheduling can also be employed in the reverse link to give acknowledgement packets strictly higher priority than data packets since the smaller acknowledgement packets, when sent infrequently, will have little impact on the performance of the reverse data transfer, but their timely transmission is desired to sustain good performance for forward data transfer.

[0096] In control system terms, referring to FIG. 2, in this embodiment the apparatus 38 effectively samples and determines a measured queue size q every Δt units of time, and provides a new ACK pacing rate r every Δt units of time. Thus, Δt is the sampling/control interval for the system and represents the time interval required for taking measurements and making computations.

[0097] In the method described herein, since the objective is to control the queue size to be close to T, a sampling rate that allows overflows (i.e., queue sizes much greater than T) and underflows (i.e., queue sizes much lower than T) to be observed is more appropriate. Thus, a sampling interval that permits accumulation of a number of packets not greater than T is reasonable. To ease implementations at high-speeds and reduce processing overhead, Δt can be higher, e.g., Δt=10 packets or any suitable value.

[0098] The value T received at the q-threshold input 74 denotes the target queue occupancy (or threshold). This value sets the buffer utilization level and average queuing delay, since the queue evolves around this value. There is flexibility in the selection of T. Typically, T could be one-half of the buffer capacity B, i.e., T=B/2. T can even be varied from time to time, if required, such as where a user desires to change operating conditions such as user delays, packet loss rates, etc. The value T is, as a result, typically piece-wise constant with changes occurring less frequently.

[0099] Effectively, the goal of the controller is to adapt r so that the magnitude of the error signal
e(n)=T−q(n) (in units of 64 bytes)
is kept to a minimum. To simplify system implementation and dimensioning, T and q(n) can be specified in any units (e.g., bytes) and e(n) measured in units of 64 bytes. Note that a typical ACK packet is 64 bytes long.

[0100] The ACK pacing method and apparatus described above can be extended to handle per-class or flow queuing scenarios. Merely using a fair queuing scheduler, without appropriate queue management, does not necessarily lead to improved performance. This is because per-class (or per-flow) scheduling algorithms by themselves do nothing to control the overall queue size of individual queues. Per-flow scheduling can allocate bandwidth fairly among backlogged connections and can provide low queuing delays. However, for TCP packets, fair bandwidth allocation does not necessarily result in fair bandwidth usage. Some form of active queue management is needed to control the overall queue sizes, so that arriving bursts of packets can be accommodated without dropping packets. The method described above can be adapted to control the queue sizes for each individual flow or class, so that they do not experience unnecessarily high delays.

[0101] Also, to accommodate non-responsive or ill-behaved traffic such as User Datagram Protocol (UDP) traffic in the method and apparatus described herein, TCP traffic can be isolated from UDP traffic. UDP and TCP traffic flows can each be queued in a number of (per-class) queues which can be served according to a number of queuing disciplines (e.g., weighted round-robin, weighted fair queuing, etc.). The rates of UDP flows can be shaped using a leaky-bucket or similar scheme. The criteria for classifying the input traffic flows can be based on source/destination IP addresses, TCP and UDP port numbers, the Type of Service (TOS) field (or DS field), and any higher level IP packet attributes.

[0102] Referring to FIG. 9, a system according to a second embodiment of the invention is shown generally at 150 in which an ack pacing apparatus 38 as described in connection with the first embodiment of the invention is used in conjunction with a network element having a packet classifier 152, a plurality of forward data queue buffers 154 and a packet scheduler 156 in the forward direction and an acknowledgement scheduler 158 a plurality of acknowledgement buffers 160 and a packet classifier 162 in the reverse direction.

[0103] The packet classifier 152 applies a policy for mapping incoming traffic into different traffic classes. A wide range of traffic classes can be supported depending on the implementation complexity desired. The packet classifier 152 in the forward direction maps traffic into per-class forward data queue buffers 154 and the packet classifier 162 in the reverse direction maps ACK packets to their corresponding reverse acknowledgement buffers 160.

[0104] The forward packet scheduler 156 applies a policy for discriminating how traffic associated with different traffic classes is managed and scheduled for transmission to the receiver 16. The acknowledgement scheduler 158 in the reverse direction schedules each ACK queue according to its allowed ACK pacing rate, as determined by the ACK pacing apparatus 38.

[0105] Each queuing class may be controlled by a separate ACK pacing apparatus 38, or a single unit capable of performing individual calculations for each rate to be calculated.

[0106] In the case where a single controller is used, the ACK pacing apparatus 38 maintains a timestamp for each ACK buffer of the plurality of acknowledgement buffers 160. Each buffer has an associated HOL tag HOLTag, for each class i=1,2, . . . ,K computed from the per-class token generation rate r_i(n). The lower bound r_min,iand upper bound r_max,ion each ACK pacing rate r_i(n), must be set based on the allocated bandwidth of each queue i, e.g., r_max,i=C_i(n), where C_i(n) is the allocated bandwidth of queue i at time n. The HOLTag represents the next time at which an ACK packet should be taken from the associated queue and transmitted to the transmitter 12.

[0107] The ACK pacing apparatus 38 continually monitors the values of the HOLTags. Whenever the current time TNow becomes greater than or equal to HOLTag_i, the corresponding ACK packet from class i is scheduled for transmission. The eligible ACK packet is transmitted and the HOLTag_iis updated according to the relation:

{HOLTag}_{i} \leftarrow {HOLTag}_{i} + TransTime + \frac{1}{r_{i} (n)} (in seconds)

[0108] When conflicts occur in the per-class scheduling times (i.e., when two or more classes are eligible for transmission at the same time instant, or in other words when two or more classes have the same HOLTag value), the conflicting classes are served in a round-robin fashion with their associated HOLTags updated as follows:

{HOLTag}_{i} \leftarrow {TActual}_{i} + \frac{1}{r_{i} (n)} (in seconds)

where TActual_iis the actual time that class i is scheduled for transmission.

[0109] In summary, the ACK pacing scheme described herein causes ACK packets to be transmitted to the transmitter 12 in response to network load dynamics. The method described herein achieves this without having to know explicitly when a TCP source is in slow-start or congestion avoidance modes, which is desirable since the desired spacing between ACKs is likely to be different in the two modes, for example. The method described herein permits a network element to actively reduce congestion (in any packet queue) by controlling the rate at which end systems communicate. As the network element tends toward congestion, the packet flows are rate shaped by slowing down the return of ACK packets so that the end-to-end communication slows down towards the allocated rate.

[0110] While specific embodiments of the invention have been described and illustrated, such embodiments should be considered illustrative of the invention only and not as limiting the invention as construed in accordance with the accompanying claims.

Claims

1. A method of controlling a rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network, the method comprising adjusting an acknowledgement rate at which acknowledgement signals are sent to said source, in response to a difference between a target queue occupancy and a measured queue occupancy of a queue buffer in which data packets from said source are received.

2. The method of claim 1 further comprising adjusting said acknowledgement rate such that the difference between said target queue occupancy and said measured queue occupancy is minimized.

3. The method of claim 2 further comprising producing a difference value representing said difference between said target queue occupancy and said measured queue occupancy.

4. The method of claim 3 further comprising filtering successively produced difference values to produce said difference value.

5. The method of claim 4 wherein filtering comprises producing said difference value as an exponentially weighted moving average of said successively produced difference values.

6. The method of claim 3 further comprising transmitting acknowledgement signals at a rate determined as a function of a present rate and said difference value.

7. The method of claim 6 further comprising producing a rate value representing a rate at which said acknowledgement signals are to be transmitted to said source, said rate value being produced according to the following relation:

r(n)=r(n−1)+αe(n)

where:

r(n) is the rate value at time interval n;

r(n−1) is a rate value immediately preceding the rate value at time interval n;

α is a control gain value; and

e(n) is said difference value representing said difference between said target queue occupancy and said measured queue occupancy.

8. The method of claim 1 further comprising imposing bounds on said acknowledgement rate.

9. A computer readable medium for providing instructions for directing a processor to carry out a method of controlling a rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network, the instructions being operable to direct a processor to adjust an acknowledgement rate at which acknowledgement signals are sent to said source, in response to a difference between a target queue occupancy and a measured queue occupancy of a queue buffer in which data packets from said source are received.

10. A computer-readable signal encoded with codes for directing a processor to adjust an acknowledgement rate at which acknowledgement signals are sent to a source, in response to a difference between a target queue occupancy and a measured queue occupancy of a queue buffer in which data packets from said source are received.

11. An apparatus for controlling a rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network, the apparatus comprising:

a) means for detecting queue occupancy of a queue buffer in which data packets from said source are received; and

b) means for adjusting an acknowledgement rate at which acknowledgement signals are sent to said source, in response to a difference between a target queue occupancy and a measured queue occupancy of said queue buffer.

12. An apparatus for controlling a rate at which data packets are transmitted from a source responsive to acknowledgement signals in a packet network, the apparatus comprising:

a) a queue monitor for producing signals indicative of queue conditions of a queue buffer in which data packets from said source are received; and

b) an acknowledgement rate signal generator for generating a rate signal in response to a difference between a target queue occupancy and a measured queue occupancy of said queue buffer in which data packets from said source are received, for adjusting an acknowledgement rate at which acknowledgement signals are sent to said source.

13. The apparatus of claim 12 wherein said queue monitor is operable to produce a signal indicative of said measured queue occupancy.

14. The apparatus of claim 13 wherein said acknowledgement rate signal generator comprises a processor for generating said rate signal in response to said difference between said target queue occupancy and said measured queue occupancy.

15. The apparatus of claim 14 wherein said processor is programmed to adjust said acknowledgement rate such that said difference between said target queue occupancy and said measured queue occupancy is minimized.

16. The apparatus of claim 15 wherein said processor is programmed to produce a difference value representing said difference between said target queue occupancy and said measured queue occupancy.

17. The apparatus of claim 16 further comprising a filter for filtering a plurality of said difference values to produce a filtered difference value.

18. The apparatus of claim 17 wherein said filter comprises said processor and said processor is programmed to produce said difference value as an exponentially weighted moving average of successively produced difference values.

19. The apparatus of claim 16 wherein said processor is programmed to produce said rate signal as a function of a present rate and said difference value.

20. The apparatus of claim 19 wherein said processor is programmed to produce said rate signal to represent a rate value produced according to the following relation: