Flow-level demultiplexing within routers

[0002] U.S. patent applications entitled “Supertrunking for Packet Switching” and “Scheduling Technique for Delayed Queue Service” both by Almulhem et al, both filed on the same day as the present application, and both assigned to the assignee of the present application, disclose and claim subject matter related to that of the present invention and are herein incorporated by reference.

[0003] This invention relates generally to data routing systems and more specifically to data routing systems over multiple physical links.

[0004] The following paragraphs give definitions of terms used throughout this document.

[0005] Physical link: a single point-to-point (PPP) serial transmission link between two nodes in the network (such as between two routers or between a router and a host machine). The implementation of a serial link may take various forms such as an optical fibre or a wavelength segment on an optical fibre, among other options.

[0006] Physical input/output port: the input/output port of the router that supports one physical link.

[0007] Logical link: a point-to-point traffic path between two routers that is composed of multiple physical links and appears from a routing point of view to be one link.

[0008] Logical input/output port: the collection of physical input/output ports that support the physical links of a logical link.

[0009] Supertrunk: the aggregation of physical links into larger, logical links.

[0010] Transmission Control Protocol (TCP): a library of routines that applications can use when they need reliable network communications with another computer. TCP is responsible for verifying the correct delivery of data from client to server. It adds support to detect errors or lost data and to trigger reconstruction until the data is correctly and completely received.

[0011] Internet Protocol (IP): a library of routines that TCP calls on, but which is also available to applications that do not use TCP. IP is responsible for transporting packets of data from node to node. It forwards each packet based on a four-byte destination address (the IP address).

[0012] There has been an incredible increase in demand for bandwidth within communication routing systems over the past few years. This increase is particularly pronounced when considering the increase in data networking information transferred within these systems directly associated with the expanding popularity of the Internet. Soon the traffic rates needed between router pairs will be higher than the serial link transmission technology available. Currently, the highest transmission rate is 9.6 Gb/s, (on a single wavelength) but 2.4 Gb/s is much more commonly available. Purchasers of routers are already demanding 2.4 Gb/s links and it is expected that within a short time, some routes will require multiple physical links.

[0013] There are other reasons why multi-link routes are attractive. In situations where routers are clustered in close physical proximity, the use of multiple links might allow the interconnect to be multiple low cost links rather than single high cost connections. Another reason is that the application of the multi-link approach might also be a fast way to provide higher rate ports on existing routers. Yet another reason is that the use of multiple links allows more granularity of growth than the large steps in the transmission network and so may allow savings in bandwidth costs. Finally, another reason is that multiple links can allow for redundancy to cover link failure without requiring the spare link to cover the whole bandwidth of the route.

[0014] When using multiple links between two routers, it is a requirement that the total bandwidth be used efficiently. That is to say, the traffic offered must be spread over all available links, hereinafter referred to as load balancing. It would not be acceptable to have one link under utilized while traffic is queued on another. This suggests that packets from any source can be delivered over any link to any destination. In fact, because of the bursting nature of the traffic, allocating links statically to particular sources or destinations would result in inefficient use of the total available bandwidth.

[0015] When data traffic flows are spread over multiple links, successive packets from a particular flow (for example, a TCP connection between two IP hosts) can travel over different lengths and may arrive at the destination out of order. The variability of delay can be caused by different path lengths or different congestion levels on the paths, as well as the normal indeterminacy introduced by queuing and scheduling. The TCP can accommodate some mis-ordering of packets, but there is a problem if too much mis-ordering occurs on a connection where the transmitter is using the fast retransmission protocol.

[0016] This mis-ordering of packets forces routers receiving these packets to comprise a sorting function that is capable of re-ordering the packets into the proper order. This sorting function is likely to be relatively complex and cause a delay in the delivery of data flows to the destination. As well, if the receiving router is not setup for receiving packets of a single flow along multiple physical links and therefore does not have a sorting function, significant errors would occur within the transmission, delaying it further.

[0017] Therefore, a router is required that will allow the use of multiple physical links for transmitting data traffic without requiring re-ordering of packets or any other special functions at the receiving router. This router must ensure that the order of the packets during transmission is maintained, but still allow for efficient load balancing between physical links.

[0018] It is an object of the present invention to overcome the disadvantages of the prior art and, in particular, to provide an apparatus and method for increasing the efficiency of data communications.

[0019] According to a first aspect, the present invention provides a forwarding node capable of operation within a router that transfers digital data with a plurality of other routers within a packet routing system, the forwarding node comprising: a load balancing device that, when input with individual packets of a data flow, reads a logical output port corresponding to the data flow, assigns the data flow to a physical output port within the logical output port based upon physical link criteria, and outputs the individual packets to their assigned physical output port.

[0020] According to a second aspect, the present invention provides a router capable of operation within a packet routing system that transfers digital data between a plurality of the routers, the router comprising: a route controller; a rotator space switch; at least one first forwarding node, coupled to both the route controller and the rotator space switch, comprising a load balancing device that, when input with individual packets of a data flow, reads a logical output port corresponding to the data flow, assigns the data flow to a physical output port within the logical output port based upon physical link criteria, and outputs the individual packets to their assigned physical output port; and at least one second forwarding node, coupled to both the route controller and the rotator space switch, that is arranged to operate as a physical output port for outputting packets to at least one transmission apparatus.

[0021] According to a third aspect, the present invention provides a packet routing system that transfers digital data between at least one first router and at least one second router, the packet routing system comprising: the first router comprising: a route controller; a rotator space switch; at least one first forwarding node, coupled to both the route controller and the rotator space switch, comprising a load balancing device that, when input with individual packets of a data flow, reads a logical output port corresponding to the data flow, assigns the data flow to a physical output port within the logical output port based upon physical link criteria, and outputs the individual packets to their assigned physical output port; and at least one second forwarding node, coupled to both the route controller and the rotator space switch, that is arranged to operate as a physical output port for outputting packets to at least one transmission apparatus; a plurality of transmission apparatuses coupled between the first and second routers; and the second router comprising: a second route controller; a second rotator space switch; at least one third forwarding node, coupled to both the second route controller and the second rotator space switch, that is arranged to operate as a physical input port for receiving packets from at least one transmission apparatus; and at least one fourth forwarding node, coupled to both the second route controller and the second rotator space switch, that, when input with packets from the third forwarding node, outputs a data flow corresponding to the inputted packets.

[0022] According to a fourth aspect, the present invention provides in a packet routing system comprising a plurality of routers, a method of transferring digital data between the routers comprising the steps of: inputting individual packets of a data flow into a first router; reading a logical output port corresponding to the data flow of the individual packets; assigning each of the data flows to a physical output port within the logical output port based upon physical link criteria; transmitting each of the packets, via their assigned physical output ports, to a second router; and outputting a data flow corresponding to the packets received at the second router.

[0023] The preferred embodiment of the present invention is described with reference to the following figures, in which:

[0024] FIGS. 1aand 1bare simplified diagrams illustrating logical and physical routing links respectively;

[0025] FIG. 2 is a block diagram of a routing system according to an example application of the preferred embodiment of the present invention;

[0026] FIG. 3 is a block diagram of an Internet Protocol (IP) Forwarding (IPF) node used in FIG. 2;

[0027] FIG. 4 illustrates a Packet Forwarding Header (PFH) according to preferred embodiment of the present invention;

[0028] FIG. 5 is a flow chart illustrating steps performed by the Ingress Port Controller (IPC) according to the preferred embodiment of the present invention; and

[0029] FIG. 6 illustrates a Rotator Information Unit (RIU) header according to a preferred embodiment of the present invention.

[0030] FIGS. 1aand 1billustrate example logical and physical routing systems respectively that depict communication links from a first router 102 to a second router 104. Both the first and second routers 102, 104 are connected to a plurality of other communication links 106, 108 respectively. Within FIG. 1a, a logical link 110 is shown coupled between the routers 102, 104. In this particular example, the logical link is a 3.5 Gb/s connection. FIG. 1billustrates a series of physical links 112, that make up a supertrunk, coupled between the routers 102, 104 that together are equivalent to the logical link 110. The supertrunk 112, in this example, comprises four physical links that when combined total the 3.5 Gb/s of the logical link 110.

[0031] Each of the physical links 112 depicted in FIG. 1bmay be implemented with a series of connections within a series of different networks. Despite appearing within FIG. 1bas being directly connected and physically local, it should be understood that physical links 112 are typically quite complex with the routers 102, 104 possibly a large distance from each other.

[0032] FIG. 2 is a block diagram of a routing system between the first router 102 and the second router 104 illustrating an example application of the preferred embodiment of the present invention. Each router 102, 104, according to the preferred embodiment, comprises a rotator space switch 202, a route controller 204, and a plurality of transport nodes 206. The routers 102, 104 depicted in FIG. 2 each comprise two transport nodes, though this is not meant to limit the scope of the present invention. The rotator space switch 202 and the route controller 204 are independently coupled to each transport node 206, although, within FIG. 2, the route controller 204 is only shown to be coupled to one transport node 206 via signal bus 208.

[0033] A person skilled in the art would understand that the operation of the rotator space switch 202 and the route controller 204 would be specific to a particular company's implementation. For example, one such rotator space switch is disclosed in U.S. Pat. No. 5,745,486 entitled “High Capacity ATM Switch” by Beshai et al and assigned to the assignee of the present invention. No modifications are made in the route controller 204 and the rotator space switch 202 from prior art devices within the preferred embodiment of the present invention.

[0034] Each transport node 206 comprises a plurality of Internet Protocol Forwarding (IPF) nodes 210, 212, 214, 216, 218. In FIG. 2, each transport node 206 comprises four IPF nodes, though this is not meant to limit the scope of the present invention. In fact, the routers 102, 104, in an alternative embodiment, do not comprise transport nodes, but instead have a plurality of IPF nodes, coupled independently to the route controller 204 and the rotator space switch 202, that are not within transport nodes.

[0035] The IPF nodes, within the system depicted in FIG. 2, include an input IPF node 210, a plurality of supertrunk output IPF nodes 212 representing a single logical output port, a plurality of supertrunk input IPF nodes 214 representing a single logical input port, an output IPF node 216, and a plurality of other IPF nodes 218 which are unrelated to the description herein below of an example application of the preferred embodiment of the present invention. Each IPF node, according to the preferred embodiment, is identical. Hence, any one of the IPF nodes may, in practice, be an input IPF node, a supertrunk output IPF node, a supertrunk input IPF node, or an output IPF node.

[0036] As depicted in FIG. 2, the input IPF node 210 is within a transport node of the first router 102 and is input with a TCP input signal (TCPIN) via line 220. The input IPF node 210 is coupled to the rotator space switch 202 of the first router 102 via line 222.

[0037] The plurality of supertrunk output IPF nodes 212 are spread across the transport nodes 206 of the first router 102. In this example, there are four supertrunk output IPF nodes 212 with two in each transport node 206 of the first router 102, though other configurations are possible. The supertrunk output IPF nodes 212 are each independently coupled to the rotator space switch 202 of the first router 102 via lines 224 and, via the supertrunk 112, to the second router 104. The supertrunk 112 comprises physical links 226a, 226b, 226c, 226dbetween the supertrunk output IPF nodes 212 (the physical output ports) within the first router 102 and the supertrunk input IPF nodes 214 (the physical input ports) within the second router 104.

[0038] The supertrunk input IPF nodes 214 are spread across the two transport nodes 206 of the second router 104. In this example, there are four supertrunk input IPF nodes 214, three in one transport node 206 and one in the other. This illustrated configuration is not meant to limit the scope of the present invention and it should be recognized that the number of supertrunk input IPF nodes 214 does not have to equal the number of supertrunk output IPF nodes 212. A single supertrunk output IPF node 212 could be coupled via a plurality of physical links to a plurality of supertrunk input IPF nodes 214 and a plurality of supertrunk output IPF nodes 212 could be coupled to the same supertrunk input IPF node 214. Each of the supertrunk input IPF nodes 214 are independently coupled to the rotator space switch 202 of the second router 104 via lines 228.

[0039] The output IPF node 216 is within the transport node 206 of the second router 104 that has only one supertrunk input IPF node 214, but it is recognized that the output IPF node 216 could be located in any transport node 206 of the second router 104. The output IPF node 216 is coupled to the rotator space switch 202 of the second router 104 via line 230 and outputs a TCP output signal (TCPOUT) via line 232.

[0040] The other IPF nodes 218 are simply shown within FIG. 2 to illustrate that other operations are being performed concurrently with the example application described herein below. Each of the other IPF nodes 218 is coupled to the rotator space switches 202 of the respective router 102, 104 via lines 234.

[0041] The supertrunking operation of the routing system of FIG. 2, according to the preferred embodiment of the present invention, is now described by example. The TCPIN, which is a series of packets that comprise a data flow (a TCP over IP stream in the preferred embodiment), is input to the input IPF node 210 packet by packet. The input IPF node 210 attaches a Packet Forwarding Header (PFH), described in detail herein below, and a Rotator Information Unit (RIU) header to each packet. Referring also to FIG. 6, the RIU header 602 comprises a physical output port indicator 604 that corresponds to a particular physical output port 212 determined, with use of a load balancing function also described in detail herein below, for the particular data flow in which the packet is comprised.

[0042] Each data flow, split into individual packets with heir respective PFH and RIU header, is then forwarded to its respective supertrunk output IPF node 212 via the line 222, the rotator space switch 202 of the first router 102, and the lines 224. The supertrunk output IPF node 212 when sequentially input with the packets of a data flow then proceeds to remove the PFH and RIU header from each of the packets and outputs the packets of the data flow to a physical link 112 between the first and second routers 102, 104.

[0043] The data flows are received packet by packet at the supertrunk input IPF nodes 214 and forwarded to the output IPF node 216 via the lines 228, the rotator space switch 202 of the second router 104, and the line 230. The output IPF node 216 outputs TCPOUT which, in this case, is a series of packets that comprise a data flow consistent with the data flow of TCPIN.

[0044] The IPF nodes of FIG. 2 according to the preferred embodiment of the present invention are now described in detail with reference to FIG. 3. In this preferred embodiment, all of the IPF nodes are identical and therefore, all of the IPF nodes contain the load balancing function. As depicted in FIG. 3, each IPF node comprises an ingress block 302, an egress block 304, a Space Switch Interface (SSI) 306, an Embedded Processor Interface (EPI) 308, and a Router Monitor (RMON) 310. The ingress block 302 comprises an Ingress Convergence Processor (ICP) 312 coupled in series with an Ingress Filter and Mapper (IFM) 314, and an Ingress Port Controller (IPC) 316 further coupled to the SSI 306. The egress block 304 comprises an Egress Packet Queuer (EPQ) 318 coupled to the SSI 306 and further coupled in series with an Egress Packet Scheduler (EPS) 320, and an Egress Convergence Processor (ECP) 322. It is noted that according to the preferred embodiment, all of these components are discrete devices, though in alternative embodiments some or all of these components could be combined or implemented within software.

[0045] The EPI 308, in the preferred embodiment, is coupled to the components of both the ingress and egress blocks 302, 304 along with the SSI 306 and the RMON 310. The RMON 310 is further coupled to at least one component in each of the ingress and egress blocks 302, 304. In the preferred embodiment, the RMON 310 is coupled to the outputs of the IPC 316 and the EPS 320. The EPI 308 is the local processor for the IPF node that performs numerous processor related functions. The RMON 310, on the other hand, is utilized as a central point for all counters within the IPF node.

[0046] The following is a simplified description of the operation within the devices of the IPF node-during ingress and egress functions according to the preferred embodiment of the present invention. It is noted that the devices of FIG. 3, with the exception of the IPC 316, are not modified in the preferred embodiment of the present invention and one skilled in the art would understand their operations.

[0047] During the receiving of incoming data packets of a data flow, the ICP 312 terminates the ingress physical and link layer interfaces, verifies the link layer, IP headers, and formatting, maps the incoming packets to a Receive Context Identifier (RCI) that is a generic tag to uniquely identify the logical interface that a packet arrived on, and encapsulates each of the individual packets with a PFH. As described herein below, not all the contents of the PFH are inserted by the ICP 312, but bytes are reserved for the entire PFH.

[0048] The PFH, with reference to FIG. 4, comprises four 32 bit words. The first word 402 comprises a 4 bit Start Of Packet Offset (SOP_OFFS) signal that indicates the position of the first word of the packet to be sent to the link layer for segmentation, an 8 bit Virtual Private Network Identifier (VPN_ID) signal that identifies the virtual private network or virtual router interface that the packet arrived on, and the 20 bit RCI. The second word 404 comprises 32 bits of Packet Forwarding Flags (PF_FLAGS) that are a collection of generic status and control flags. The third word 406 comprises 32 reserved bits not utilized in the preferred embodiment of the present invention. The fourth word 408 comprises a 24 bit Egress Context Identifier (ECI) that is the result of a forwarding look-up function and is somewhat analogous to a flow identifier, and an 8 bit PFH Integrity Check (CRC). The contents of the first word 402 of the PFH are determined and inserted by the ICP 312 while the contents of the second word 404 can be set or modified by each of the blocks of the IPF node. The fourth word 408 is reserved space by the ICP 312, but the contents of the fourth word 408 are determined and inserted by another device as will described herein below.

[0049] The outputted packets from the ICP 312 are input to the IFM 314 which maps the incoming packets to the logical ECI, inserts the ECI into the PFH, and filters packets based on a configured policy. The IPC 316 then receives the packets, performs the load balancing function, as will be described herein below, and encapsulates each of the packets with a RIU header which maps the packet to physical output ports determined by the load balancing function. Within the present invention, all the packets of a particular data flow are mapped to the same physical output port.

[0050] Subsequently, the packets are sequentially input to the SSI 306 which is the interface between the rotator space switch 202 and the particular IPF of FIG. 2. The SSI 306 then outputs the packets through the rotator space switch to the particular physical output port chosen by the load balancing function for the particular data flow.

[0051] At the physical output port, the packet is input to the SSI 306 and forwarded, via the EPQ 318 and the EPS 320, to the ECP 322. The ECP 322 subsequently removes the PFH and RIU header from each of the packets and forwards the packets, via a physical link, to a physical input port; that being an IPF node within the router of the destination output IPF node.

[0052] Within the preferred embodiment of the present invention, the packets are received at the physical input port and input to the corresponding ICP 312. The ICP 312 and the IFM 314 perform similar functions as described herein above, such as attaching a PFH to each of the packets, and sequentially forwarding the packets to the IPC 316. The load balancing function within the IPC 316 is not performed on the packets of the data flow since the output IPF node for the flow is an IPF node within the same router. The IPC 316 simply attaches an RIU header, which maps the packet to its output IPF node, to each of the packets and forwards them, via the SSI 306 and the rotator space switch, to the SSI 306 of the output IPF node.

[0053] The packet is forwarded by the SSI 306 of the output IPF node to the EPQ 318. The EPQ 318 reassembles packet segments and notifies the EPS 320 of the availability of packets to be scheduled. Since all packets of a particular data flow are routed via the same physical output port, the packets remain in the proper order. Hence, each packet that is reassembled becomes available for scheduling without the need for a re-ordering function. Subsequently, the EPS 320 maps the packets to scheduling queues and outputs the packets to the ECP 322 which removes the PFH and RIU header and performs L2 and PHY encapsulation functions as required.

[0054] The detailed description of the load balancing function, according to the preferred embodiment of the present invention, performed within the IPC 316 is now described with reference to FIG. 5. Firstly, as depicted at step 502, the IPC 316 receives a packet from the IFM 314. Next, the IPC 316 determines, at step 504, the logical output port corresponding to the input packet. This is done, in the preferred embodiment, by reading the ECI within the PFH that was inserted by the IFM 314 after a look-up function was completed.

[0055] Subsequently at step 506, a flow identifier is determined for the packet. The flow identifier is a unique identifier for the data flow, preferably a TCP over IP stream. One characteristic that defines a data flow is the ECI in which the packet has been assigned. In the preferred embodiment, the data flows are TCP streams which have specific start and end packets. The ECI of the packet is used in combination with specific TCP control fields to determine an index into a look-up table which will hold the flow identifier for the data flow. After the arrival of a start packet for a particular data flow, a new flow identifier is assigned in the look-up table to the particular data flow. Subsequent packets of the same flow, prior to the end packet being received, are assigned the identical flow identifier. When the end packet of the particular TCP flow is received, the flow is terminated and the corresponding flow identifier is removed from the look-up table so it can be reused by a new TCP flow. In the preferred embodiment, a flow identifier is also removed from the look-up table after a predetermined refresh period has expired that indicates that no further packets of that particular data flow are still to arrive.

[0056] The next step within the load balancing function, as depicted at step 508, is to determine the physical output port for the packet to traverse. If the packet is not the first of its data flow to be routed, the look-up table described above for the flow identifier would also indicate the physical output port for the particular packet. If the packet is the first packet of its data flow, the load balancing function determines the physical output port for its data flow and saves this determined physical output port selection within the look-up table for further packets of the same data flow to utilize. This determination of the physical output port is the key to the effectiveness of the load balancing function and can be done in numerous different ways.

[0057] According to the preferred embodiment of the present invention, a Link Available Bandwidth (LAB) in units of bytes/sec is generated within the load balancing function for each of the available physical output ports. This is determined with use of the following formula:

[0058] LAB=(LBW)−(IR)*(APS)

[0059] where, LBW is the Link Bandwidth in units of bytes/sec, IR is the Input Rate in units of packets/sec, and APS is the Average Packet Size. The LBW is known for each available physical output port while the IR and the APS are generated, within the preferred embodiment, by statistical means described herein below.

[0060] To calculate, according to the preferred embodiment, the IR of a particular physical output port, an exponentially weighted moving average (EWMA) technique is used that generates a continuously adjusting long term average of the number of packets per second being routed through the particular physical output port. To calculate, according to the preferred embodiment, the APS for all the physical output ports, an EWMA technique is also used that generates a long term average of the size of the packets being sent. In an alternative embodiment, the APS is calculated once using a large amount of data and subsequently is constant.

[0061] In the preferred embodiment, this calculation of LABs for each physical output port is done within each IPC 316, though in an alternative embodiment these calculations are done in a central location within the router, such as the route controller, and the calculated LABs are bused to the individual IPF nodes.

[0062] The size of the LABs indicate the availability of bandwidth for the incoming data flow and therefore, to make load balancing optimal, the current data flow is directed to the physical output port with the largest LAB. The packet is directed to a particular physical output port, according to the preferred embodiment, by writing to the RIU header attached to each of the packets of the particular data flow. Finally, at step 510, the packets are individually output from the IPC 316 and directed to the physical output port assigned to their corresponding data flow while maintaining the proper order.

[0063] The key advantage of this load balancing of data flows among multiple physical links that comprise a supertrunk is that the user gains the advantages of using a supertrunk while not requiring the router that receives the packets of the data flow to re-order them due to differences in delays on the particular physical links. This means that traditional routers without supertrunk technology can be utilized as receiving routers according to the preferred embodiment of the present invention with no special treatment required for the supertrunked packets compared to regular non-supertrunked packets.

[0064] Alternative embodiments of the present invention may use different methods to determine the physical output port for a data flow compared to that disclosed herein for the preferred embodiment. One such alternative technique is disclosed within the previously referenced U.S. patent application entitled “Supertrunking for Packet Switching” by Almulhem et al, previously incorporated by reference. This method of determining physical output ports is specific to packet-level load balancing, but one skilled in the art would understand that it could be modified for flow-level load balancing as described herein.

[0065] One skilled in the art would recognize that the data flows being input to the router could be TCP flows or more specifically, IP streams that comprise a series of IP packets. In this case, the logical link for the data flow can be read from the IP header within each individual packet.

[0066] Persons skilled in the art will appreciate that there are alternative implementations and modifications possible to use an apparatus similar to that described above to provide a routing system over multiple physical links, and that the above implementation is only an illustration of this embodiment of the invention. The scope of the invention, therefore, is only to be limited by the claims appended hereto.