Method and apparatus for adaptive routing in packet networks

[0002] The invention concerns adaptive routing in packet networks, in order to reduce network congestion, and to optimize selected parameters, such as throughput and end-to-end packet delays.

[0003] A simplified network will be explained, in order to provide background regarding basic concepts and terminology.

[0004] FIG. 1 illustrates a network containing nodes A-F and links L1-L9. The nodes are physical stations, which contain one, or more, computers, and associated equipment. For example, a university's computer system may act as a node. The nodes receive data from other nodes, or from external sources (not shown). The nodes may process the data, but in the present context, their most important function is to store the data temporarily, and forward the data to other nodes.

[0005] Links are data transmission channels, used to carry data between nodes. For example, a common telephone line represents one exampletype of link; a satellite communication link and an optical fiber link represents two others.

[0006] A common characteristic of networks is that every node is not necessarily connected to every other node. For example, in FIG. 1, node A is not connected to node F. If these two nodes wish to transmit data to each other, they must find a path, such as the combination of links L1+L7 or the combination of links L2+L6.

[0007] However, not all paths are equally desirable. One reason is that each link imposes a time delay, and these time delays are not uniform. FIG. 2A provides an example of time delays for the network of FIG. 1, and indicates that the combination L1+L7, mentioned in the previous paragraph, represents a time delay of 6 units, while the combination of L2+L6 represents a longer time delay of 7 units.

[0008] In many cases, it is preferred to utilize the path imposing the smallest overall time delay. Well-known algorithms, such as the Dijkstra algortithm, are available for ascertaining these shortest-time paths, for all pairs of nodes in a given network. FIG. 2B provides an example of a table of shortest-time paths for the network of FIG. 1. Some spaces are left blank because they represent redundancy: for example, the time from B to C is the same as that from C to B.

[0009] One mode of utilizing such tables resides in “next-hop” routing. In next-hop routing, each node does not refer to the entire contents of the shortest-path table, but, instead, utilizes an abbreviated table, termed a “routing table,” which is derived from the shortest-path table.

[0010] FIG. 3 illustrates the next-hop routing table 2 used by node F. If node F wishes to transmit data to node A, node F routes the data to node B, as indicated. Node B, according to its own next-hop routing table (not shown), then routes the data to node A. The indirect path is F-to-B-to-A, using the link pair L7+L1 of FIG. 2A, consistent with the shortest-path table of FIG. 2B, for the node-pair (F, A).

[0011] One problem with next-hop routing, as just described, is that, as a matter of probability, the links having the shortest delay times are those most likely to appear in the shortest-time table of FIG. 2B, and thus most likely to be used in the routing tables. Consequently, under conditions of high data traffic, the nodes at the ends of these links can become congested. When congestion occurs, data traffic at these nodes can become stalled temporarily.

[0012] The congestion, in effect, causes the transit times shown in FIG. 2A to increase. However, the nodes continue to route their packets along the same paths, despite these increases, because the routing tables do not change: they are static. Static routing tables do not accommodate network congestion well, because they do not take into account the variable nature of network traffic.

[0013] In one form of the invention, data packets are routed in a network according to next-hop routing tables, which are derived based on measured packet delays and link utilization between nodes. The tables are updated as the delays or utilization change.

[0014] FIG. 1 illustrates a generic network.

[0015] FIG. 2A illustrates transit times of the network of FIG. 1, and FIG. 2B illustrates a shortest-time routing table for the network.

[0016] FIG. 3 illustrates a next-hop routing table for node F, which was derived from the shortest-time routing table of FIG. 2B.

[0017] FIG. 4 illustrates the use of two next-hop routing tables in a network.

[0018] FIG. 5 illustrates the scheduling process and flow charts implemented by the invention.

[0019] FIG. 6 illustrates flow charts and transmission of packets, once scheduled, by the invention.

[0020] FIG. 7 illustrates flow charts and the updating of the next-hop routing tables, which occurs when link costs change sufficiently.

[0021] FIG. 8A illustrates a network used in a simulation of the invention, compared with other types of routing.

[0022] FIG. 8B illustrates results of the simulation.

[0023] FIG. 9 illustrates computer code used by the invention.

[0024] The invention applies to packet networks, and preferably to those based on the IP (Internet Protocol) protocol. IP packet networks are in widespread use today, for example, in the INTERNET, in corporate IP networks (also called intranets), and in Internet Service Provider (ISP) networks.

[0025] Information is transmitted across the network in the form of packets. The packet is the minimum unit of information subject to transport and control in a packet network. Packets are variable in size: different packets can contain different numbers of bytes.

[0026] The invention handles packets of two different types, depending upon the Type of Service (TOS) provided to the packet.

[0027] The service provided to TOS-1 packets is designed to reduce, or minimize, the time delay experienced by the packet during transmission from its origin to its destination. TOS-1 packets will generally be of small size.

[0028] The service provided to TOS-2 packets is not designed to necessarily minimize time delay, because such packets are not so sensitive to such delay. They are probably, but not necessarily, of large size.

[0029] The application, or computer program, generating the packets, such as an electronic mail program, inserts tags into the packets which specify the TOS requested.

[0030] Each packet, within the network, may be assigned to an “originating node.” The originating node may create the packet, or may receive the packet from a source outside the network, such as from a person operating a microcomputer.

[0031] The originating node may retain the packet, if the packet is destined for the originating node, or the originating node may transfer the packet to a receiving node. The receiving node may retain the packet, or transfer the packet to a third node. This transfer process continues until the packet reaches its destination.

[0032] Each packet contains identifiers, which identify (1) the packet's destination node and, as stated above, (2) the packet type (i.e., TOS-1 or TOS-2). For each packet type, each node stores a routing table: one routing table is used for TOS-1 packets, and the other is used for TOS-2 packets. Two different routing tables are used because it is desired to find separate paths, if possible, for both packet types to satisfy their different requirements regarding end-to-end delay. As an example, FIG. 4, right side, indicates the two routing tables derived for node F.

[0033] When the originating node receives, or creates, a packet, the originating node first uses a routing table to identify the link on which the packet will be transmitted. Then, the node places the packet into one of two queues assigned to that link, based on the packet-type identifier. That is, TOS-1 packets are placed into a TOS-1 queue, and TOS-2 packets are placed into a TOS-2 queue.

[0034] As a specific example, three links are connected to node A in FIG. 4. A pair of queues is assigned to each link. Each pair contains a queue for TOS-1 packets, and a queue for TOS-2 packets.

[0035] The queues are each emptied in a first-in-first-out (FIFO) manner. The originating node (1) withdraws a single packet from the TOS-1 queue, (2) reads the destination node from the packet, and (3) transmits the packet to another node, based on the originating node's TOS-1 routing table.

[0036] The originating node then (1) withdraws a single packet from the TOS-2 queue, (2) reads the destination node from the packet, and (3) transmits the packet to another node, based on the originating node's TOS-2 routing table.

[0037] The originating node then returns to the TOS-1 queue, and repeats the process.

[0038] Each node which receives a packet from the originating node then becomes an originating node, and executes the same procedures just described, using two queues and two routing tables (unless, of course, that node is the destination of the packet, in which case the node retains the packet).

[0039] As explained in the Background of the Invention, the effective transit times between nodes can change, in response to network traffic, for example. To accommodate these changes, the invention monitors the changes in transit times. When the changes become sufficiently great, the invention adjusts the routing tables. (As will be seen, the invention monitors a more complex quantity than the transit times discussed in the Background. However, for this present Overview, the fact of adjustment of the routing tables is emphasized, rather than the particular quantities involved in the adjustment.)

[0040] The preceding Overview allows the following summary:

[0041] 1. Each packet is assigned one of two packet types, TOS-1 or TOS-2.

[0042] 2. Each node maintains a pair of queues for each output link of the node. In each pair, one queue is used for TOS-1 packets, and the other is used for TOS-2 packets.

[0043] 3. Each node maintains two routing tables, one for each packet type.

[0044] 4. Each node processes the queues by

[0045] a) withdrawing a single packet from one queue, and transmitting the packet to another node, based on the routing table for the service type of that queue;

[0046] b) withdrawing a single packet from the other queue, and transmitting the packet to another node, based on the routing table for the service type of that queue; and

[0047] c) repeating (a) and (b).

[0048] This queue-processing is termed “scheduling.”

[0049] 5. The invention monitors the transit times over the links, and modifies the routing tables when the transit times change significantly.

[0050] Under the invention, two classes of service are provided: type-1 (TOS-1) service is provided for urgent packets, which are also called delay-sensitive packets, and type-2 (TOS-2) service is provided for less urgent packets, which are also called throughput-sensitive packets. In type-1 packets, the number of bytes is expected to be “small.” In type-2 packets, the number is expected to be “large.”

[0051] The dividing line between “small” and “large” will depend upon different circumstances, but, in general, small packets typically contain 64 to 512 bytes, while large packets contain more than 512 bytes.

[0052] The originator of a packet tags the packets with a label indicating the type of service requested. For example, the originator may be an electronic mail system, located within, or without, the network. The electronic mail system generates packets. A header in each packet contains a field, which contains a tag designating the service-type (i.e., type-1 or type-2) requested for the packets.

[0053] Therefore, incoming packets to the network are divided into two types: (1) delay-sensitive packets and (2) non-delay-sensitive packets. Each packet indicates its respective packet-type.

[0054] In the Overview, “transit times” between nodes were considered. However, the invention utilizes a more complex parameter than the physical transit times across the links, and this parameter will be termed a “cost.”

[0055] One example of a cost is the time delay experienced by a packet between (a) an input port P1 in FIG. 1 of node A and (b) an input port P2 of node E. This time delay will depend not only on the physical transit time across link L3, but also on processing delays occurring at node A. For example, if node A is congested with traffic, the cost may be greater than if node A were idle.

[0056] The particular cost just described is a single example of a cost. The term “cost” is generic, and represents a parameter, or metric, which is to be minimized (or maximized). For example, cost can represent monetary cost, time delay, data flow rate, or other parameter to be minimized (or maximized). Further, costs are assigned to links, even though the characteristics of the link do not determine the cost exclusively, but behavior of nodes may also affect the cost.

[0057] Still further, more than one cost can be assigned to each link: each type of service can incur a different cost on a given link. To explain the latter, an analogy is perhaps appropriate. A type-1 packet can be analogized to a postcard, and type-2 packets can be analogized to pages of a book. It is likely that the postcard can be packed into a single packet, and that the book will require numerous packets.

[0058] However, it is likely that the postcard-packet will be not completely filled. On the other hand, it is likely that every packet in the book-packet will be completely filled, with the exception of a single last packet, which will hold the final paragraphs of the book's contents.

[0059] Thus, it is not surprising that the average cost, per packet, will be different, for the two cases. The average postcard-packet, which is partially filled, will experience a lower cost, because less data, per packet, is transferred. The average book-packet will experience a higher cost, because more data, per packet, is transferred.

[0060] Independent of the preceding reasons, the invention itself creates a difference in costs for the two classes. This difference will be explained in greater detail below but, in brief, the dual-queue scheduling procedure itself can create a difference in costs for the two packet types. As a simple example, consider two cases.

[0061] Case 1. Assume that a given node receives (a) five type-2 packets, and (b) five type-1 packets. Assume that all packets will be routed to the same next-hop, so that all packets are placed into the same pair of queues.

[0062] Using the dual-queue scheduling process, that node will schedule these packets by interleaving the five type-1 packets between the five type-2 packets. The scheduling sequence will be:

[0063] 1-2-1-2-1-2-1-2-1-2.

[0064] With this scheduling sequence, the type-2 packet will not be transmitted until almost all type-1 packets have been transmitted. Thus, the presence of the type-l packets delays the type-2 packet.

[0065] Case 2. Consider the same assumptions, and assume that the node receives the same type-2 packet, but no type-1 packets. The scheduling sequence will be

[0066] 2-2-2-2-2.

[0067] No delay is imposed because of the presence of any type-1 packets, as in Case 1.

[0068] Therefore, for these, and other reasons, the average cost, per packet, will be different for the two types of service. (Of course, in some cases, the average costs may be identical.)

[0069] Each node maintains two routing tables, one for each service type. FIG. 4 provides an example. The left side of the Figure shows two costs for each link, with one cost being given for each type of service. The details of computation of these costs will be explained below. Using these costs, two next-hop routing tables are developed for each node, one table for each service type. The right side of the Figure indicates two such routing tables for node F.

[0070] The costs are computed according to equations given below. The variable “LL” represents cost.

[0071] Utilization Variable, U

[0072] In computing the costs, each node computes an average utilization, U, for each output link connected to it. For example, node A in FIG. 1 will compute three U's, one for each of link L1, L2, and L3.

[0073] Average utilization is the fraction of time that the scheduler is busy transmitting packets on a link. From another point of view, utilization is the fraction of the scheduler's capacity which is actually used. At each node, one scheduler is provided for each output link, and its pair of queues.

[0074] As a numerical example of utilization, assume a link scheduler having a capacity of one million bytes per second, which serves two incoming packet flows, from two respective queues. The first flow contains a constant flow of 1,000 packets per second, with a fixed packet size of 128 bytes. The second flow contains a random packet flow rate, with a fixed packet size of 256 bytes.

[0075] Assume that, during a particular 3-second interval, the second flow is 2,000 packets per second. The utilization of the link is the fraction of time, during these 3 seconds, that the scheduler is busy. Mathematically, the utilization is

[0076] (total bytes processed)/(scheduler capacity for time of process)

[0077] or

[0078] (1,000×128×3+2,000×256×3)/(3×1,000,000)=0.43.

[0079] Delay Variables, D

[0080] An average delay D1 is computed for type-1 packets, for each link. This average delay includes queuing, scheduling, and transmission delays. This delay, in effect, measures the time delay a packet experiences in travelling from the input port of one node, such as port P1 in FIG. 1, to the input port of a neighboring node, such as P2. Numerous approaches, known in the art, exist for measuring these delays.

[0081] In one approach to measuring D1, each node can periodically insert a sample packet into its type-1 queue. The sample packet contains a time-stamp, which indicates the exact time of the insertion. This node sends the sample packet to each of the nodes listed in its next-hop routing table.

[0082] Each receiving node (1) reads the time-stamp, (2) compares the time-stamp with present absolute real time, and (3) ascertains the time delay, D1. Each receiving node reports the time delay D1 to all other nodes.

[0083] A second average delay, D2, is computed for all packets, namely, both type-1 and type-2. This delay is computed in the same manner as D1, with the exception, of course, that all packets are considered, not just type-1 packets, as in D1.

[0084] Using these variables, the link costs are computed according to the following equations: $\begin{array}{cc}{\mathrm{LL1}}_{\mathrm{ik}}\left(t+\epsilon \right)=\left[\begin{array}{cc}1+\mathrm{INTEGER}\left\{\left(1-\alpha \right)Q_D\left[{\mathrm{D1}}_{\mathrm{ik}}\left(t\right)\right]+\alpha {\mathrm{LL1}}_{\mathrm{ik}}\left(t\right)\right\}& \mathrm{IF}\left|\frac{{\mathrm{LL1}}_{\mathrm{ik}}\left(t+\epsilon \right)-{\mathrm{LL1}}_{\mathrm{ik}}\left(t\right)}{{\mathrm{LL1}}_{\mathrm{ik}}\left(t\right)}\right|\ge 0.75\\ \mathrm{LL1}\left(t\right)& \mathrm{otherwise}\end{array}\right]& \mathrm{Equation}1\\ {\mathrm{LL2}}_{\mathrm{ik}}\left(t+\epsilon \right)=\left[\begin{array}{cc}1+\mathrm{INTEGER}\left\{\left(1-\beta \right)Q_U\left[U_{\mathrm{ik}}\left(t\right)\right]+\beta {\mathrm{LL2}}_{\mathrm{ik}}\left(t\right)\right\}& \mathrm{IF}\left|\frac{{\mathrm{LL2}}_{\mathrm{ik}}\left(t+\epsilon \right)-{\mathrm{LL2}}_{\mathrm{ik}}\left(t\right)}{{\mathrm{LL2}}_{\mathrm{ik}}\left(t\right)}\right|\ge 0.15\\ \mathrm{LL2}\left(t\right)& \mathrm{otherwise}\end{array}\right]& \mathrm{Equation}2\end{array}$

[0085] In these expressions

[0086] 0≦α≦1.0

[0087] Q_D(x)=1+INTEGER(97.5x)

[0088] Q_v(x)=1+INTEGER[x10³]

[0089] These costs are used to compute the routing tables, preferably using the well-known Dijkstra algorithm. For each node, this algorithm computes the shortest path from that node to every other node in the network. That is, if the network contains the given node and 8 other nodes, then eight shortest paths will be computed. “Shortest” means smallest cost.

[0090] Several significant features of these computations are the following.

[0091] 1. Each “LL” refers to the cost associated with a link, with LL1 referring to the cost for type-1 packets, and LL2 referring to the cost of type-2 packets. The subscripts “ik” refer to the end-nodes of the link which is being computed. For example, in FIG. 1, link L1 (not to be confused with the variable LL1 in the equations above) would appear as LL_ABin the equations above. of course, for convenience in writing computer code, the nodes may be indicated by numbers, rather than letters, so that numerical subscripts can be used.

[0092] 2. The “epsilon” ε in the equations refers to the time interval occurring between updates of the costs, and preferably lies in the renge of 5 seconds to 30 minutes.

[0093] 3. Because of the rightmost IF-statement in Equation 1, LL1 is updated only if LL1 has changed by roughly 75 percent, or more. A similar comment applies to LL2, but the percentage is roughly 15 percent. Thus, a larger change in cost LL1 is required to induce an update in LL1, compared with LL2. “Change” refers to either an increase, or decrease, because the absolute value of the changed variable is used in the equations.

[0094] 4. A phenomenon known as “oscillation” can occur in network routing. For example, as stated above, links having the lowest costs will tend to be the most popular, and can appear most often in the routing tables. However, if the cost of one of these popular links increases drastically, then other links will become popular, and will replace the previously popular links in the routing tables. These changes cause the lowest-cost paths to shift in position, or oscillate.

[0095] The variables alpha, α, and beta, β, in the expressions above act as weights, or filters, to suppress oscillation. That is, in Equation 1, the updated cost LL1 depends on the previous D1 and on the previous LL1. But the relative weights given to the previous D1 and the previous LL1 depend on alpha, as Equation 1 indicates, thereby reducing oscillation.

[0096] It has been found that when alpha, α, and beta, β, are both in a range between approximately 0.7 and 1.0, that high oscillations occur. On the other hand, when alpha and beta are both near zero, convergence of the processes computing the costs becomes very slow. Consequently, alpha is preferably held between 0.25 and 0.3, while beta is fixed at 0.25.

[0097] 5. The invention utilizes load sharing. As background, load sharing will be first explained. It is possible that, in the shortest-time table of FIG. 2B, no single lowest-cost path exists between an origination node and a destination node. Multiple paths, of equal cost, may exist. Further, even if a lowest-cost path exists, other paths may exist which are only slightly higher in cost, and thus may be considered interchangeable with the strictly lowest-cost path.

[0098] These factors, and others, allow “load sharing,” in which several paths are used simultaneously, in transmitting packets between an originating node and the destination node. In a sense, “parallel” lowest-cost paths exist between pairs of nodes. Since parallel paths exist, some routing tables will contain two next hops for a given final destination.

[0099] FIG. 4 provides an example. Dashed block 3 is an addendum to the type-2 routing table. This routing table (plus addendum) indicates that, for a path between an originating node F and a final destination node E, two next-hops are suitable: nodes D and B. These nodes D and B can share the load.

[0100] The invention implements load sharing, by defining two additional parameters, namely, UMAX and Z. Whenever a utilization variable, U, for a given link exceeds UMAX, the Dijkstra algorithm is instructed to consider additional paths for each destination reachable by that link, for both packet types (ie, types 1 and 2). To be considered, these additional paths must not differ in cost by more than a percentage Z. These additional paths are used for load sharing.

[0101] In addition, a type of hysteresis is imposed. Load sharing continues until the particular U falls below (UMAX−3Z). That is, the variable Z is trebled, and subtracted from UMAX, to determine the point at which to terminate load sharing. This hysteresis tends to reduce oscillations.

[0102] Restating the preceding:

[0103] 1. When U for a given link exceeds UMAX, then paths are added to the lowest-cost path, provided the added paths are not more expensive than the lowest-cost path by Z.

[0104] 2. The additional paths are used until the U falls below (UMAX−3Z), at which time the lowest-cost path is used exclusively.

[0105] 6. In the equations above, “x” is given in milliseconds. The variable “x” is a generic variable, used to define the quantization functions QD and QU.

[0106] 7. A variable D2 was defined, but not used in the equations given above (only D1 was used). This variable D2, together with the U's, D1's, LL1's, and LL2's, are reported to administrators of the network, and are used for solution of long-term network design problems. The variable D2 is not used in deriving the next-hop routing tables.

[0107] 8. Examination of Equations 1 and 2 indicates that the primary dependent variables which change are L1, L2, D1, and U. Variable D2 is not a dependent variable.

[0108] One view of this dependence is that, for type-1 packets, each updated cost LL1(t+ε) depends on (1) previous delay D1 (for type-1 packets) and (2) the previous LL1.

[0109] In contrast, for type-2 packets, the updated cost LL2 (t+ε) depends on (1) utilization U, but not D1, and (2) the previous L2.

[0110] Restated, each updated cost depends on its previous value, and one other factor: previous delay D1 for cost L1, and previous utilization U for cost L2. Thus,

[0111] Changes in delays D1 influence the updated routing tables for type-1 packets, and

[0112] Changes in utilizations U influence the updated routing tables for type-2 packets, but changes in delays D1 do not.

[0113] It is a known fact that, at high data traffic conditions, queuing delays (or scheduling delays) tend to be the dominant factor in link delays, rather than transmission delays. The invention exploits this fact by two measures. One, it reduces the queuing delays for type-1 packets. That is, type-1 packets are “short,” so that their waiting-times in their type-1 queues is “small,” compared with a single-queue system.

[0114] The second measure is that the routing tables for type-1 packets are based on transit delays D1 (and other variables). Based on these two measures, type-1 packets spend less time in their queues, and are routed along paths having the lowest end-to-end delays.

[0115] In contrast, type-2 packets (the “less urgent” ones) are routed along the least-utilized paths.

[0116] FIGS. 5-8 illustrate a flow chart of logic undertaken by the invention. FIG. 5 illustrates operation of the scheduler used by the invention. A node 50 contains a computer COMP which runs the scheduling process.

[0117] Scheduling

[0118] Incoming packets 55 are received by the node 50. Each dashed box indicates one packet. Each packet contains one, or more, packets, as indicated. The packets contain identifiers which indicate the final destination and the packet type, as indicated.

[0119] As block 60 indicates, the packets are allocated into queues, according to the packet type, namely, type-1 or type-2. Arrows 65 indicate this allocation process. As block 70 indicates, the scheduler withdraws the packets from queue 1, then queue 2, then queue 1, and so on. Row 75 of packets indicates the interleaved packets. The symbols “FIRST,” “SECOND,” etc., indicate the order of withdrawing of the packets in the FIFO processing of the queues.

[0120] Routing

[0121] FIG. 6 illustrates the routing process which follows scheduling. As block 80 indicates, the type-1 packets are sent to nodes, based on the type-1 routing table of node 50. The routing table implements next-hop routing. As block 85 indicates, the type-2 packets are sent to nodes, based on the type-2 routing table of node 50. Arrows 90 indicate this routing.

[0122] It should be remembered that the destinations stated in the row of packets 75 (e.g., A, B, C, etc.) do not necessarily coincide with the next hop, to which the packets are sent, unless that next hop happens to be the destination.

[0123] Each receiving node repeats the scheduling and routing processes just described, for the packets received.

[0124] Updating of Routing Tables

[0125] FIG. 7 illustrates the update procedures undertaken by the invention. Block 100 indicates that the delays, D, utilizations, U, and costs, L, are periodically updated. This updating utilizes the equations given above.

[0126] Block 110 indicates that both routing tables for all nodes are periodically re-computed, using the updated variables. Block 115 indicates that selected data is transmitted to a manager of the system. This manager can reside at one of the nodes. This selected data is used for general management purposes.

[0127] Summarizing the updating in another manner, the algorithm of the invention updates the link costs. After this updating, the invention utilizes the Dijkstra algorithm to update the routing tables. The updated routing tables may contain new shortest-paths, because of changes in the link costs.

[0128] Numerous approaches can be taken in implementing the steps shown in FIG. 7. For example, one single computer can ascertain all the data of block 100, perform the computations of block 110, and distribute the routing tables which are produced.

[0129] Alternately, one, or more, computers can be assigned to ascertain the data of block 100. These computers can distribute the data to all other computers, which independently compute their own routing tables. However, in this case, it is preferred that all computers utilize identical algorithms, so that the routing tables will be identical.

[0130] FIG. 8 illustrates results of a computer simulation of the invention's behavior, compared with the behavior of three other systems. FIG. 8A illustrates the network used, which contains eight nodes N. The three heavy links L1-L3 have a speed of 45 Mbits/sec, while the others have a speed of 1.5 Mbits/ sec. All are bi-directional.

[0131] The three other systems tested are the following:

[0132] 1. STATIC. Link costs are fixed. A single type of service is provided, and thus a single cost per link is used.

[0133] 2. ANTOS. Link costs are updated, but a single cost is used per link.

[0134] 3. AFIFO. Link costs are updated. Two types of service are provided, and thus two costs per link are used. Load sharing is not used. The costs are based on average delay for all packets, rather than for type-1 packets exclusively. (In contrast, as Equation 1 indicates, the invention updates costs based on the delay D1 for type-1 packets exclusively, and ignores delays of type-2 packets.)

[0135] FIG. 8B illustrates results of the simulation, in which an average delay, D, for all packets, and an average end-to-end delay, D1, for type-1 packets, D, are plotted against a parameter R. R is the ratio of (type-1 traffic/type-2 traffic).

[0136] Traffic sources were modeled by two-state Markov processes, which generate packet trains in the ON state. The generation rate was constrained to maintain R in the range from 1 to 5, as the Figure indicates. Type-1 packets were, on average, 128 bytes long, while type-2 packets were, on average, 2,000 bytes long. The Figure indicates that, in all cases, the invention provided superior delay performance.

[0137] FIG. 9 illustrates computer code which implemented the simulation. The “Trigger” module calls a “tick” routine, which computes the delay, d, and the utilization, u. Then, an “AQUA” module computes the link costs L1 and L2.

[0138] Numerous substitutions and modifications can be undertaken without departing from the true spirit and scope of the invention. What is desired to be secured by Letters Patent is the invention as defined in the following claims.