Time-Wheel ATM Cell Scheduling

1. Field of the Invention

The invention relates generally to cell scheduling in Asynchronous Transfer Mode (ATM) networks, and more specifically to time-wheel cell scheduling in ATM networks.

2. Background Art

Reference is now made to FIG. 1 of the drawings, showing a simple prior art time-wheel scheduler. A number of queues 10 are organized in a round queue of queues 12. Each queue 10 contains zero or more Virtual Channel (VC) entries 14 to be transmitted. Two time pointers are employed to control the operation of the scheduler: a time pointer 16 and a transmission pointer 18. The time pointer 16 advances clockwise from one queue 10 to the next queue 10 every cell time, indicating that all VC entries 14 in the queues 10 behind it are to be transmitted as soon as possible. The transmission pointer 18 either lags behind or steps together with time pointer 16, indicating which queue 10 is currently being serviced. Scheduling is typically done by an external manager (not shown in the drawings) which is responsible for placing VC entries 14 in queues 10.

Simple time-wheel cell schedulers, as the one depicted in FIG. 1, have several limitations: first, complex computations are required every time transmission pointer 18 reaches a new queue, even if the queue is empty. As a result of the need for calculation, the total number of queues that can be implemented is also limited. Further, the limitation on the number of queues also limits the minimum bandwidth and the bandwidth resolution that can be assigned to a VC. Any changes to the number or the attributes of the VCs require complex calculations to be redone to determine the fixed entries in the queues.

There is a need for an easy-to-implement method for constant bit rate (CBR) and variable bit rate (VBR) cell scheduling with high resolution of bandwidth and the ability to support a large number of VCs.

Accordingly, the present invention provides an easy to implement method for CBR and VBR cell scheduling with high resolution of bandwidth and able to support a large number of VCs.

The invention further eliminates the need for complex computations every time the transmission pointer reaches a new queue, even if the queue is empty.

In addition, the invention eliminates the need for complex calculations in case changes are made to the number or the attributes of the VCs.

To achieve the foregoing and other objectives, and in accordance with the purposes of the present invention, a time-wheel ATM cell scheduler is provided that allows for high resolution of bandwidth and a large number of VCs, with an automatic rescheduling scheme which eliminates the need for complex recalculations.

The scheduler includes a number of rotating queues, one or more stationary queues and means to transfer VC entries between rotating and stationary queues. The scheduler operates by progressively bringing in contact each of the rotating queues with the stationary queue. All VC entries in the rotating queue are then transferred to the stationary queue and one entry is transmitted on the physical channel every cell-time (if the entry exists).

Immediately after transmission the entry is rescheduled on an upcoming rotating queue according to a step determined by the VC's [m,i] attributes (m cells in i cell-times). If m/i is not an integer, its fractional part is stored in an accumulation register. The next rotating queue then is brought into contact with the stationary queue and its entries are added to the stationary queue. Another entry is then transmitted and rescheduled on an upcoming queue. When the accumulation register overflows, the step with which the cell just transmitted is rescheduled is incremented by one.

As a result, no global computations are required if either new VCs are added or the properties of existing VCs are modified. In addition, the scheduler is able to handle any number of VCs as long as m/i is less than or equal to the number of queues.

As the queues become increasingly oversubscribed, cells may end up waiting in the stationary queue longer than their desired step. To solve this problem, a timestamp is added to each VC entry when it is placed in the appropriate upcoming rotating queue. When the cell is rescheduled, its timestamp is compared to the current time and, if the difference is greater than its computed step, it is rescheduled to the stationary queue.

As a result, the bandwidth of different VCs are adjusted as the channel becomes increasingly oversubscribed: VCs with higher bandwidths are rescheduled to the stationary queue more often, thus reducing their bandwidths, while VCs with lower bandwidths are rescheduled to the stationary queue less often, thus maintaining their assigned bandwidths. Accordingly, the scheduler can handle variations in both the number and properties of VCs without need for recalculation.

The accompanying drawings incorporated in and forming part of the specification, illustrate several aspects of the present invention and, together with the description, serve to explain principles of the invention. In the drawings:

FIG. 1 is a perspective view of a simple prior art time-wheel cell scheduler.

FIG. 2 is a perspective view of a time-wheel cell scheduler employing an automatic rescheduling scheme according to one embodiment of the present invention.

FIG. 3 is a perspective view of another embodiment of the present invention.

FIGS. 4A and 4B are partial views of the stationary queue component of FIG. 3.

FIG. 5 is a block diagram of an embodiment of the present invention.

FIGS. 6A-6D are two-dimensional graphs representing the effects of over-subscription on the bandwidth of virtual channels.

FIGS. 7-10 are examples of the operation of the present invention, illustrating perspective views of the preferred embodiment of FIG. 3 at successive times in the operation of the invention.

FIGS. 11A-11H and 12A-12P are pictorial representations of the operation of the present invention.

FIG. 13 is a two-dimensional graph representing the relationship between a minimum guaranteed bandwidth and a maximum allowed bandwidth in a multiple scheduler embodiment of the present invention.

FIG. 14 is a perspective view of a multiple scheduler embodiment of the present invention.

FIG. 15 is a pictorial representation of a "doubly leaky bucket."

FIGS. 16-17 are pictorial representations of multiple scheduler embodiments of the present invention.

FIG. 18 is a flowchart diagram of the processing of VC entries.

FIG. 19 is a block diagram of an integrated circuit for scheduling of ATM cells, according to one embodiment of the invention.

FIG. 20 is a block diagram of an ATM switch according to one embodiment of the invention.

FIG. 2 depicts an improvement over the simple prior art scheduler of FIG. 1, and employs an automatic rescheduling scheme rather than having fixed entries in the queues. The operation of this scheduler is somewhat analogous to that of the simple scheduler of FIG. 1, except that when a VC is serviced, it is removed from its present queue 10 and placed in an upcoming queue 10 according to some predefined fixed step. The step determines the bandwidth assigned to the VC. As long as no two VCs are placed on the same queue, changes in the number or properties of other VCs will have no effect on the bandwidth of that VC. The minimum bandwidth supported, however, is still equal to the bandwidth of the channel divided by the number of queues and the discrete bandwidths supported are equal to the minimum bandwidth multiplied by the step size. Calculations are still required every time the transmission pointer reaches a new queue, even if the queue is empty.

Reference is now made to FIG. 3 of the drawings depicting the preferred embodiment of the present invention. As in the background art depicted in FIG. 1, a time wheel 12 is constructed, consisting of a round-robin queue of queues 10. VC entries 14 scheduled for transmission are placed in queues 10. Time wheel 12 rotates clockwise, one step every ATM cell time. A stationary queue 20, is attached to location 0 on time wheel 12. Whenever a rotating queue 10 reaches location 0 on time wheel 12, its entries 14 are added to stationary queue 20. Once every cell time a VC entry 14 in stationary queue 20 is taken for transmission and immediately rescheduled, by being placed in an upcoming queue 10.

The step used in placing the VC entry into an upcoming queue is calculated according to two measures: (1) channel assigned bandwidth; and (2) effect of other VCs.

The bandwidth assigned to a VC is defined by two parameters i and m, as i cells in m cell-times. The bandwidth assigned to a VC is thus equal to the bandwidth of the entire physical channel multiplied by the ratio i over m of the parameters defining the VC. Ideally, cells from a virtual channel should be spread evenly in the physical channel and cells should be emitted every m/i cell times. Thus, if m/i is an integer the step will simply be m/i. If, however, m/i is not an integer its integer part is taken in the initial step and its fractional part is added to accumulation register 22, shown in FIG. 5. In FIG. 5, time wheel 12 is shown as a cylindrical shape formed by rotating queues 10. Each rotating queue contains a head pointer 25 and a tail pointer 27, pointing to the head and the tail of queue of VC entries 14. Each VC entry 14, comprises a step 21, which is a real number equal to m/i, an accumulation register 22 and a pointer 29 to the next VC entry 14 in the rotating queue 10. When accumulation register 22 overflows, the step size is increased. The step generation is more formally described in Table 1, where A is the value stored in accumulation register 22 and S is the value stored in step 21, i.e. m/i.

When there is more than one VC entry 14 in the stationary queue 20, VCs are serviced one per each cell time. Thus, the time that each VC entry has waited in the stationary queue must be taken into account in calculating the step size. This is done by adding a timestamp 23 to VC entries 14, as shown in FIG. 5. When the VC is serviced, its timestamp is compared to the current time and the difference is subtracted from the step size. If the difference is larger than or equal to the step size, the VC entry 14 is placed back on stationary queue 20, as shown in FIG. 4B, otherwise it is placed on an upcoming rotating queue 10 according to its step size, as shown in FIG. 4A.

This process is more formally described in Table 2.

FIG. 18 is a flowchart diagram further describing the process of Table 2. In step 60, it is determined whether there are any VC entries scheduled on the stationary queue. If there are, the first of these VC entries is transmitted in step 62. The interval with which the the VC entry is to be rescheduled is then generated in step 64. In step 66, the VC entry is rescheduled. Finally, in step 68 the queues are rotated and the process repeats.

Reference is now made to FIGS. 6A-6D, showing graphs of VC bandwidth allocation at successive points in time. As the physical channel is oversubscribed, i.e. the sum of the bandwidths of all VCs exceeds the bandwidth of the physical channel, the entries of the VC with the highest bandwidth 24 will be constantly placed back on the stationary queue, effectively reducing its bandwidth. The process will then be repeated for the VC with the next highest bandwidth 26. If oversubscription continues, eventually all VCs will share the same effective bandwidth.

FIGS. 7-10 show examples of the functioning of the invention for different VC configurations.

FIG. 7 shows the simple case of a single VC entry 14 on time-wheel 12 with m and i parameters equal to 3 and 1 respectively. At time=0, VC entry 14 is in rotating queue 10, one position removed from stationary queue 20. At time=1, rotating queue 10 reaches stationary queue 20, its contents are transferred to stationary queue 20, VC entry 14 is transmitted on transmission line 30 and rescheduled onto rotating queue 10 three positions removed from stationary queue 20. At time=2, time wheel 12 rotates bringing rotating queue 10, on which VC entry 14 is placed, two positions removed from stationary queue 20.

FIG. 8 illustrates a single VC entry 14 on time wheel 12, with m and i parameters equal to 3 and 2, respectively. At time=0, VC entry 14 is in rotating queue 10, one position removed from stationary queue 20. At time=1, rotating queue 10 reaches stationary queue 20, its contents are transferred to stationary queue 20, VC entry 14 is transmitted on transmission line 30 and rescheduled onto rotating queue 10 one position removed from stationary queue 20. Since the step for m/i=3/2 would be 1.5, 0.5 is stored in accumulation register 22. At time=2, rotating queue 10 reaches stationary queue 20, its contents are transferred to stationary queue 20, VC entry 14 is transmitted on transmission line 30 and rescheduled onto rotating queue 10 two positions removed from stationary queue 20, the 0.5 stored in accumulation register 22 is added to step m/i=1.5.

FIG. 9 shows a more complicated example with two VC entries 14 with m and i parameters equal to 2 and 1 and 3 and 2 respectively. At time=0, VC entry 14 with m/i term=3/2 on stationary queue 20 is transmitted on transmission line 30 and rescheduled onto rotating queue 10 one position removed from stationary queue 20. Since the step for m/i=3/2 would be 1.5, 0.5 is stored in accumulation register 22. At time=1, rotating queue 10 reaches stationary queue 10 and its contents are transferred onto stationary queue 20. VC entry 14 with m/i term=2/1 on stationary queue 20 is transmitted on transmission line 30 and rescheduled to the end of stationary queue 20, since time stamp 23 is equal to step m/i=2. At time=2, VC entry 14 with m/i term=3/2 is transmitted on transmission line 30 and rescheduled at the end of stationary queue 20, since between the current time and time stamp 23 is greater than the step size, computed by adding the value stored into accumulation register 22, 0.5, to m/i=1.5. At time=3, VC entry 14 with m/i term=2/1 on stationary queue 20 is transmitted on transmission line 30 and rescheduled to the end of stationary queue 20, since the difference between the current time and time stamp 23 is equal to step size m/i=2.

FIG. 10 shows a more complicated example with two VC entries 14 with m and i parameters equal to 3 and 2 and 4 and 3 respectively. At time=0, VC entry 14 with m/i term=4/3 on stationary queue 20 is transmitted on transmission line 30 and rescheduled onto rotating queue 10 one position removed from stationary queue 20. Since the step for m/i=4/3 would be 1.34, 0.34 is stored in accumulation register 22. At time=1, rotating queue 10 reaches stationary queue 10 and its contents are transferred onto stationary queue 20. VC entry 14 with m/i term=3/2 on stationary queue 20 is transmitted on transmission line 30 and rescheduled onto rotating queue 10 one position removed from stationary queue 20. Since the step for m/i=3/2 would be 1.5, 0.5 is stored in accumulation register 22. At time=2, VC entry 14 with m/i term=4/3 is transmitted on transmission line 30 and rescheduled at the end of stationary queue 20, since the difference between the current time and time stamp 23 is equal to the step size, computed by adding the value stored into accumulation register 23, 0.5, to m/i=1.5. At time=3, VC entry 14 with m/i term=2/1 on stationary queue 20 is transmitted on transmission line 30 and rescheduled to the end of stationary queue 20, since the difference between the current time and time stamp 23 is equal to the step size m/i=2. At time=4, VC entry 14 with m/i term=4/3 is transmitted on transmission line 30 and rescheduled at the end of stationary queue 20, since the difference between the current time and time stamp 23 is greater than the step size, computed by adding the value stored into accumulation register 23, 0.5, to m/i=1.5. At time=5, VC entry 14 with m/i term=2/1 on stationary queue 20 is transmitted on transmission line 30 and rescheduled to the end of stationary queue 20, since the difference between the current time and time stamp 23 is greater than the step size m/i=2.

FIGS. 11A-11H offer a pictorial, step-by-step representation of the operation of a single scheduler with m=3 and i=2. At time=0 (FIG. 11A), a single VC entry 14 is queued on rotating queue 31 at position 32 on time wheel 35, while stationary queue 33 is empty and no entries are present on transmission line 30. At time=1 (FIG. 11B), VC entry 14 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35. At time=2 (FIG. 11C), VC entry 14 is again moved to stationary queue 33, transmitted and rescheduled on rotating queue 31 at position 34 on time wheel 35, rather than at position 32 because accumulation register 22 has now overflowed. At time=3 (FIG. 11D), then, we find the queues in the same state as at time=0 and the same steps as at times 0, 1 and 2 are repeated at times 3,4 and 5 respectively. Thus, transmission line 30 at time=7 (FIG. H) shows a pattern of transmission of two VC entries every three cell-times as expected.

FIGS. 12A-12P offer a pictorial, step-by-step representation of the operation of a single scheduler with two VC entries 14 with m and i terms equal to 3 and 2 and 9 and 2 respectively. At time=0 (FIG. 12A), two VC entries 36 and 38 are queued on rotating queue 31 at position 32 on time wheel 35, while stationary queue 33 is empty and no entries are present on transmission line 30. At time=1 (FIG. 12B), VC entry 36 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35. At time=2 (FIG. 12C), VC entry 36 is moved to stationary queue 33 and VC entry 38 is transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 42 on time wheel 35. At time=3 (FIG. 12D), VC entry 36 is transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35. At time=4 (FIG. 12E), VC entry 36 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35. At time=5 (FIG. 12F), VC entry 36 moved to stationary queue 33 and VC entry 38 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 44 on time wheel 35. At time=6 (FIG. 12G), VC entry 36 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35. At time=7 (FIG. 12H), VC entry 36 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35. At time=8 (FIG. 12I), VC entry 36 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 34 on time wheel 35. At time=9 (FIG. 12J), VC entries 38 and 36 are both moved to rotating queue 31 at position 32 on time wheel 35. At time=10 (FIG. 12K), VC entry 36 is moved to stationary queue 33 and VC entry 38 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 40 on time wheel 35. At time=11 (FIG. 12L), VC entry 36 is transmitted on transmission line 30 and rescheduled on stationary queue 33, since the time elapsed since it was last serviced is equal to its step. At time=12 (FIG. 12M), VC entry 36 is transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 34 on time wheel 35. At time=13 (FIG. 12N), VC entry 36 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35. At time=14 (FIG. 12O), VC entry 36 is moved to stationary queue 33 and VC entry 38 is moved to stationary queue 33, transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 44 on time wheel 35. At time=15 (FIG. 12P), VC entry 36 is transmitted on transmission line 30 and rescheduled on rotating queue 31 at position 32 on time wheel 35.

Finally, multiple schedulers may be used in parallel to implement particular arbitration/priority schemes. For example, one might want to assign to a virtual channel a minimum guaranteed bandwidth, but at the same time limit the maximum bandwidth that can be taken by this VC at any particular time. FIG. 13 illustrates the relationship between the minimum guaranteed bandwidth 50 and the maximum allowed bandwidth 52. This can be accomplished with two schedulers working in parallel. FIG. 14 depicts two schedulers according to the preferred embodiment of FIG. 3: scheduler 54, with m and i attributes equal to the minimum guaranteed bandwidth 50 and scheduler 56, with m and i terms equal to the difference between the maximum allowed bandwidth 52 and the minimum guaranteed bandwidth 50. The desired bandwidth allocation is achieved by giving scheduler 56 priority over scheduler 58.

Another way of describing this scheme is offered by the "doubly leaky bucket" of FIG. 15. Bucket 70 is filled with liquid 72. Faucets 74 and 76 allow liquid 72 to drain into funnel 78 at different rates. If faucet 74 is set to a rate corresponding to the minimum guaranteed bandwidth 50 and faucet 76 is set to a rate corresponding to the difference between the maximum allowed bandwidth 52 and the minimum guaranteed bandwidth 50, liquid 72 will drain into funnel 78 in a fashion analogous to that in which VC entries are processed by schedulers 56 and 58 in FIG. 14.

In addition, several lower bandwidth channels may be multiplexed onto a single, higher bandwidth channel by using multiple schedulers. In FIG. 16, scheduler 80 is used to divide bandwidth among several low bandwidth channels and scheduler 82 is used to assign the appropriate bandwidth to each channel's VCs.

Furthermore, several VCs may be carried over a single virtual path (VP). In FIG. 17, scheduler 84 is used to provide the appropriate bandwidth to several VPs, while scheduler 86 is used to provide appropriate bandwidth to the VCs in one of the VPs.

A block diagram of an integrated circuit for ATM cell scheduling according to one embodiment of the invention is shown in FIG. 19. ATM cell scheduler 90 comprises a time-wheel scheduler 92, as well as a weighted round robin/strict priority scheduler 94. Weighted round robin schedulers are described copending, commonly assigned U.S. patent application Ser. No. 08/843,108, entitled "METHOD FOR WEIGHTED FAIR QUEUEING FOR ATM CELL SCHEDULING" by Alexander Joffe, which is incorporated herein by reference in its entirety.

ATM cell scheduler 90, further includes a subport scheduling interface module 95, a queue rate descriptor memory interface module 96, an OVC descriptor memory interface 97 and a queuing controller interface 98. A listing of signals received and transmitted by ATM cell scheduler 90 and their corresponding pins is provided in Appendix A.

ATM cell scheduler 90 interfaces with Per-VC circuit 100 via OVC descriptor memory interface 97. Per-VC circuit 100 includes storage for data structures representing VC cells. By exchanging signals with Per-VC circuit 100 according to a predetermined protocol, ATM cell scheduler 90 controls the scheduling of ATM cells queued on the ATM switch of FIG. 20. FIG. 20 illustrates the structure of an ATM switch according to the present invention. Switch controllers 120 interface through switch interface 110 with per-VC circuits 100. Per-VC circuits 100 exchange signals with schedulers 90 to perform the scheduling of ATM cells for transmission. The ATM switch of FIG. 20 is described in U.S. patent application Ser. No. 08/706,104, entitled "CELL QUEUING IN ATM SWITCHES" by Alexander Joffe, Ari Birger and Pravat Mishra, which is incorporated herein by reference in its entirety.