Method and apparatus for multipath processing

[0001] This application is entitled to the benefit of provisional U.S. Patent Application Ser. No. 60/385,867, filed Jun. 4, 2002, which is incorporated herein by reference.

[0002] The invention relates generally to the field of networking and more specifically to a method and apparatus for processing multipath data packets.

[0003] Multipath routing is a router feature that allows a stream of packets going to a given destination to be split across multiple paths. Routers that do not support multipath routing only remember a single path to any given destination. As a result, if a link failure occurs along that path, network traffic to that destination may be stopped completely. Conventional routers that support multipath routing, on the other hand, remember multiple paths to any given destination. In the event of a failure that impacts one of the paths in a multipath arrangement, traffic can be shunted over to the surviving paths, thus reducing the impact of link failures. Multipath routing also allows the routers to balance traffic loads across multiple paths.

[0004] FIG. 1 illustrates a small network where multipath routing is supported. As shown, Router A has one connection to Router B and one connection to Router C. Router B and Router C each have a connection to Router D. To reach Router D, data packets from Router A can be routed via Router B and Router C. FIG. 2 illustrates the contents of Router A's routing table and shows the “cost” associated with each path. If the two paths have the same cost, Router A is indifferent to either path and may split traffic equally between both paths. If the path via Router B and the path via Router C have unequal bandwidths or “costs,” Router A may direct more traffic to the one connection with a lower cost than the one with a higher cost.

[0005] Multipath load balancing can be performed on a “per packet” basis or a “per destination” basis. Balancing multipath traffic on a per packet basis means that packets bound for the same destination are distributed to different interfaces. For example, with reference to FIGS. 1 and 2, Router A can route half of the data packets bound for destination IP address 192.68.253.5 via Router B and half of the data packets bound for the same destination IP address via Router C. Balancing traffic on a per-packet basis is typically not too resource intensive. However, some data may be received late, or out of sequence, which can cause application problems.

[0006] Multipath load balancing on a per destination basis means that packets bound for the same destination will be forwarded via the same path. For example, with reference to FIG. 1, Router A can route all data packets bound for destination IP address 192.68.253.5 via Router B while routing all data packets bound for destination IP address 192.68.253.2 via Router C. Balancing traffic on a per-destination basis alleviates some potential problems caused by per-packet load balancing. One drawback of balancing traffic on a per-destination basis, however, is that it is more resource intensive than load balancing on a per-packet basis. A significant amount of memory is used in conventional routers to cache route information such that packets bound for the same destination can be routed via the same path. Another drawback is that it is difficult to control the percentage of traffic routed through a particular path.

[0007] In view of the foregoing, what is needed is a router that can handle multipath routing and load balancing in a resource-efficient manner. What is also needed is a method of handling multipath load balancing in a resource-efficient manner.

[0008] An embodiment of the invention is a multipath processing method that allows multipath packets to be distributed across multiple paths using a hash function and a modulo-arithmetic-based mapping function. In one embodiment, the multipath processing method is implemented as part of a packet classification engine of a network node.

[0009] The packet classification engine accesses a Multipath Weight Vector Table that includes a plurality of Multipath Weight Vectors. Each of the Multipath Weight Vectors is predefined by users and/or router software and is associated with an index. The Multipath Weight Vectors describe a number of available paths for routing multipath packets and the desired relative traffic distribution for each of the available paths. In one embodiment, the packet classification engine allows traffic to be partitioned in fractions of 1/M, where M represents a maximum number of available paths for a given destination. In that embodiment, the Multipath Weight Vectors are each M-bits wide.

[0010] The packet classification engine examines one or more fields of the header (e.g., destination address fields, source address fields) of a data packet to be routed. Using this information, the packet classification engine generates an index and retrieves a corresponding Multipath Weight Vector from the Multipath Weight Vector Table. The packet classification engine also generates a value representative of at least part of the header information. In one embodiment, the representative value is a hash value generated using a hash function and “non-random” header fields such as source and destination application port numbers, source and destination addresses and flow identifiers. The hash values thus generated will be the same for packets that share the same header information. In another embodiment, the representative value is generated using a hash function and “random” header fields such as the CRC field. In that embodiment, the resulting hash values will be random (or appear to be random) and will differ from one packet to another even if those packets share the same non-random header information. In yet another embodiment, the packet classification engine generates a random or pseudo-random value without regard to any fields of the data packet.

[0011] The packet classification engine also generates a path index based on the Multipath Weight Vector and the hash value. In one embodiment, a packet processor receives the path index and the forwarding index from the packet classification engine and uses the received information to select an appropriate one of the multiple paths for routing the data packet. In another embodiment, the packet processor receives values representative of other header information from the packet classification engine and uses these values with the path index to select an appropriate path for routing the data packet.

[0012] According to one embodiment, the packet classification engine determines a number of paths available for routing a packet and the desired relative traffic distribution for each available path solely from the Multipath Weight Vectors. In one embodiment, the number of “1” bits in a Multipath Weight Vector indicates the number of available paths configured for a given packet flow. The number of bit positions between a particular “1” bit and a closest less significant “1” bit divided by a weight denominator (which is the bit position of the most significant “1” bit plus one) indicates the desired relative traffic load for a path that is associated with that particular “1” bit.

[0013] In one embodiment, the Multipath Weight Vector Table is stored within an off-chip Static Random Access Memory (SRAM) that is accessible by the packet classification engine. Specific information of the available paths is stored in an off-chip Dynamic Random Access Memory (DRAM) that is accessible by the packet processor.

[0014] Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.

[0024] Embodiments of the invention can be implemented within a network node that handles traffic in discrete units, often referred to as datagrams, data packets or packets. In one embodiment, the network node can be an Ethernet switch/router that forwards traffic using OSI (Open Systems Interconnect) Layer 2, Layer 3, and/or Layer 4 header information. The network node may have a distributed architecture. That is, the network node may include line cards that are coupled to each other via a switch matrix and each line card is partially responsible for processing and forwarding the datagrams. Furthermore, the network node supports network protocols such as Ethernet, ATM, and Frame Relay. Although embodiments of the invention can be implemented within an Ethernet-based switch/router with a distributed architecture, the invention can be implemented within network nodes with other types of architecture as well.

[0025] According to one embodiment of the invention, the network node is capable of performing hardware-based multipath load balancing. Multipath load balancing herein refers to a network node function that allows a packet stream to be split across multiple paths according to predetermined link-specific relative traffic loads. In this embodiment, the network node sorts incoming packets by predetermined header fields using a hash function, and directs the packets to a given link or port based on the value of that hash function, a modulo-based mapping function and link-specific relative traffic load information embedded in the Multipath Weight Vectors.

[0026] FIG. 3 is a block diagram illustrating a portion of a network node 100 capable of multipath routing in accordance with an embodiment of the invention. As shown, the network node 100 includes a Packet Processor 110 and a Packet Classification Engine 120. A main function of the Packet Classification Engine 120 is to classify the incoming packets, and a main function of the Packet Processor 110 is to decide where a packet is to be routed based on its classification. In this embodiment, the Packet Processor 110 is coupled to a Forwarding Table 115, which contains information for forwarding packets such as the port number to which packets matching certain criteria should be sent. The Packet Classification Engine 120 is coupled to a Forwarding Index Lookup Table 130, which contains Forwarding Indices, and a Multipath Weight Vector Table 125, which contains Multipath Weight Vectors. In one embodiment, the Multipath Weight Vector Table 125 is stored within an off-chip Static Random Access Memory (SRAM) that is accessible by the Packet Classification Engine 120. The Forwarding Table 115 is stored in an off-chip Dynamic Random Access Memory (DRAM) that is accessible by the Packet Processor 110. The Forwarding Index Lookup Table 130 is stored within a Content Addressable Memory that is accessible by the Packet Classification Engine 120. In one embodiment, the Packet Processor 110 and the Packet Classification Engine 120 are implemented as ASICs (Application Specific Integrated Circuits). In another embodiment, the Packet Processor 110 and the Packet Classification Engine 120 are implemented in the same ASIC.

[0027] In operation, when the network node receives a packet, the Packet Processor 110 reads its header information and transmits a request, or REQ message, containing relevant portions of the header information to the Packet Classification Engine 120. The Packet Classification Engine 120 then generates a Forwarding Index based on the received information and accesses a Multipath Weight Vector Table 125 to retrieve an appropriate Multipath Weight Vector. In one embodiment the Packet Classification Engine 120 uses the Multipath Weight Vector, as well as a hash function and a modulo-based function, to generate a Path Index. The Packet Classification Engine 120 then forms a response, or RESP message, with the Forwarding Index and the Path Index and sends RESP message to the Packet Processor 110. A more detailed description of how the Packet Classification Engine 120 generates the Forwarding Index and the Path Index is provided further below.

[0028] Upon receiving the RESP message, the Packet Processor 110 accesses the Forwarding Table 115 to retrieve forwarding information that corresponds to the RESP message's Forwarding Index and the Path Index. Using the retrieved Path Information, the Packet Processor 110 then generates control signals for a Switch Matrix (not shown) of the network node 100 to direct the packet to an interface (e.g., a port number) specified by the forwarding information.

[0029] FIG. 4 depicts some contents of the Forwarding Table 115 according to an embodiment of the present invention. Each entry of the Forwarding Table 115 includes a Forwarding Index, a Path Index and a port number. The forwarding information contained in the Forwarding Table 115 indicates how a packet is to be routed. For instance, as shown in FIG. 4, if the Packet Classification Engine 120 generates a Forwarding Index “01” and a Path Index “3” using header information of a packet, the Packet Processor 110 will direct that packet to Port 15.

[0030] In one embodiment, the Forwarding Table 115 can be programmed with information specified by the user or network administrator. In some embodiments, contents of the Forwarding Table 115 can include information generated by software that dynamically calculates routes based on a current network status.

[0031] Attention now turns to the generation of Forwarding Indices and Path Indices by the Packet Classification Engine 120. FIG. 5 is a block diagram depicting portions of the Packet Classification Engine 120 in accordance with an embodiment of the present invention. In particular, the Packet Classification Engine 120 includes a HASH/CRC block 210, a CAM lookup block 220, a Multipath SRAM Controller 230, a Path Index Calculation block 240, a FIFO (First-in-first-out) buffer 215, and a RESP Assembly block 250. Various components shown in FIG. 5 do not necessarily represent circuit blocks of the Packet Classification Engine 120. Rather, the components shown in FIG. 5 represent functions performed by the Packet Classification Engine 120 and such functions can be accomplished through hardware, software and/or a combination of hardware and software.

[0032] With reference still to FIG. 5, when the Packet Classification Engine 120 receives a REQ message from the Packet Processor 110, the HASH/CRC block 210 examines one or more fields of the header information (e.g., destination address field, source address field, flow identifier, source application port number, destination application port number, CRC field) contained in the REQ message. Using this information and a predetermined hash function, the HASH/CRC block 210 generates a hash value and provides it to the FIFO 215. In the illustrated embodiment, the FIFO 215 is used because the Packet Classification Engine 120 may take extra clock cycles to retrieve a corresponding Multipath Weight Vector. Therefore, the FIFO 215 has a number of stages so that the hash value and the corresponding Multipath Weight Vector can be provided concurrently to the Path Index Calculation block 240.

[0033] In one embodiment, the HASH/CRC block 210 obtains a 32-bit CRC (Cyclic Redundancy Check) value from the header of the packet, and XOR's the upper and lower 16-bits together to provide a 16-bit hash value. It should be noted that other hash functions are used in other embodiments of the present invention.

[0034] With reference still to FIG. 5, after the hash value is determined or concurrently with the generation of the hash value, the CAM lookup block 220 uses one or more fields of the header information (e.g., destination address field, source address field) contained in the REQ message to perform a look up on the Forwarding Index Lookup Table 130 and to retrieve a corresponding Forwarding Index. Some contents of the Forwarding Index Lookup Table 130 according to an embodiment of the invention are shown in FIG. 6. In this embodiment, each Forwarding Index is associated with a source address and a destination address. In other embodiments, each Forwarding Index can be associated with these and/or other fields. The contents of the Forwarding Index Lookup Table 130 can be programmed information specified by the user or network administrator. In some embodiments, contents of the Forwarding Index Lookup Table 130 can include information generated by software that dynamically calculates routes based on a current network status.

[0035] With reference again to FIG. 5, the Forwarding Index retrieved by the CAM lookup block 220 is used by the Multipath SRAM Controller 230 to retrieve a Multipath Weight Vector from the Multipath Weight Vector Table 125. Contents of the Multiple Weight Vector Table 125 according to an embodiment of the invention are shown in FIG. 7. In the embodiment of FIG. 7, each Forwarding Index is associated with a Multiple Weight Vector. In one embodiment, the Packet Classification Engine 120 allows traffic to be partitioned in fractions of 1/M, where M represents a maximum number of available paths for a given destination. In this embodiment, the Multipath Weight Vectors are each M-bits wide. In one particular implementation, M equals sixteen and the Multipath Weight Vectors are each 16-bits wide. The Multipath Weight Vectors stored in the Multipath Weight Vector Table 125 can be predefined by users and/or generated automatically by software running on the network node.

[0036] According to the present embodiment, the Multipath Weight Vectors of the invention describe a number of available paths for routing data packets that are bound for the same destination and the desired relative traffic distribution for each of the available paths. In one embodiment, the number of “1” bits in a Multipath Weight Vector indicates the number of available paths configured for a given packet flow. The number of bit positions between a particular “1” bit and a closest less significant “1” bit divided by a weight denominator indicates the desired relative traffic load for a path that is associated with that particular “1” bit. The weight denominator, in this embodiment, is the bit position of the most significant “1” bit plus one.

[0037] Note that in the embodiment described above, the Multipath Weight Vectors use “1” bits as “on” bits to indicate the number of available paths and the desired relative traffic distribution. In another embodiment, the Multipath Weight Vectors use a different notation to describe the number of available paths and the desired relative traffic distribution. In that embodiment, the “0” bits are used as “on” bits. That is, the number of “0” bits indicates the number of available paths configured for a given packet flow, and the number of bit positions between a particular “0” bit and a closest less significant “0” bit divided by a weight denominator indicates a desired relative traffic load for a path. In some other embodiments, the Multipath Weight Vectors may use some other encoding schemes to indicate the number of available paths and the desired relative traffic distribution.

[0038] An example illustrating how a Multipath Weight Vector describes the number of available paths and the desired relative traffic load is as follows. Suppose there is a Multipath Weight Vector 0000_—0000_—1001_—0100. Since there are three “1” bits in the vector, there are three available paths for routing a data packet to the same destination. Further, the bit position of the most significant “1” bit of the vector is “7”. Thus, the weight denominator of the vector is “8”. Additionally, since the difference between the bit positions of the most significant “1” bit and the second most significant “1” bit is three, approximately ⅜ of the traffic should be directed to one of the available paths. The difference between the bit positions of the second most significant “1” bit and the third most significant “1” bit is two. Thus, approximately 2/8 of the traffic should be directed to a second one of the available paths. Finally, the remaining approximately ⅜ of the traffic should be directed to the third one of the available paths.

[0039] Some additional examples illustrating how a Multipath Weight Vector describes the number of available paths to a given destination and the relative traffic load of each path are given in Table 1 below. Table 1

[0040]

TABLE 1
Multipath Weight Vector	Desired Relative Traffic Load
0000_0001_0000_0001	1/9, 8/9 (Path #0 listed first)
0000_1000_1000_1000	1/3, 1/3, 1/3
0000_0000_0000_0111	1/3, 1/3, 1/3
1000_0000_1000_0000	1/2, 1/2
0000_1100_1101_0110	2/12, 1/12, 2/12, 2/12, 1/12, 3/12, 1/12
0000_0111_1111_1111	1/11, 1/11, 1/11, 1/11, . . .

[0041] Note that the Multipath Weight Vectors of the invention are compact. As shown in the examples above, a vector merely 16-bits long can completely describe the number of paths available for a given destination and the desired relative traffic load for each of the path. An efficient implementation of multipath load balancing without consuming a significant amount of memory and hardware resources is therefore possible with the use of the Multipath Weight Vector format of the invention.

[0042] With reference again to FIG. 5, the Multipath SRAM Controller 230 retrieves a Multipath Weight Vector according to the information generated by the CAM lookup block 220, and sends the retrieved Multipath Weight Vector to the Path Index Calculation block 240. Recall that the Path Index Calculation block 240 also receives from the FIFO 215 a hash value generated by the HASH/CRC block 210. The Path Index Calculation block 240 uses the Multipath Weight Vector and the hash value to generate a Path Index, which the Path Index Calculation block 240 sends to the RESP Assembly block 250. The RESP Assembly block 250, which also receives the Forwarding Index from the Multipath SRAM Controller 230, combines the Forwarding Index and the Path Index to generate a RESP message. The RESP Assembly 250 then sends the RESP message back to the Packet Processor 110.

[0043] Recall that the Packet Processor 110, when given a RESP message including a Forwarding Index and a Path Index, will be able to retrieve appropriate forwarding information, such as a port number to which the packet should be sent, from the Forwarding Table 115.

[0044] Referring now to FIG. 8, there is shown a flow diagram for a process of generating a Path Index in accordance with an embodiment of the invention. At step 810, a value N, which is the modulo of the hash value (generated by HASH/CRC block 210) and a Weight Denominator of the Multipath Weight Vector (retrieved by Multipath SRAM Controller 230). The Weight Denominator, in this embodiment, is the bit position of the most significant “1” bit of the Multipath Weight Vector plus one.

[0045] At step 820 of FIG. 8, the Path Index is determined by counting the number of “1” bits in the Multipath Weight Vector that are less significant than the N-th bit of the Multipath Weight Vector.

[0046] The following example illustrates the process shown in FIG. 8. Suppose a hash value for a packet is 0000_—0000_—0110_—1101 (109 in decimal), and suppose a corresponding Multipath Weight Vector is 0000_—0000_—1001_—0100. The Weight Denominator of this Multipath Weight Vector is “8”, and the modulo N is 109 mod 8, or 5. The number of “1” bits that are less significant than the 5th least significant bit of the Multipath Weight Vector is two. The Path Index for this hash value and Multipath Weight Vector therefore also equals two. In this embodiment, a modulo-based mapping function is used to generate a Path Index because modulo logic is simple to implement in hardware and does not incur significant hardware overhead. In other embodiments, modulo arithmetic and/or other types of mathematical or algorithmic functions can be used to map the hash values to appropriate ones of the available paths.

[0047] The following example illustrates how a random hash value affects the generation of the Path Index. Suppose the Multipath Weight Vector is 0000_—0000_—1001_—0100, and suppose the HASH/CRC block 210 uses header fields containing random or pseudo-random values. The resulting hash values will vary randomly (or pseudo-randomly) from packet to packet even if the packets are bound for the same destination. Consequently the modulo values will vary randomly (or pseudo-randomly) from 0 to 7. Under the process outlined in FIG. 8, modulo values 5, 6, 7 map the Path Index “2”, modulo values 3 and 4 map to Path Index “1”, and modulo values 0, 1 and 2 map to Path Index “0”. That is, given a large enough number of samples, three times out of eight the Path Index will be “2”, two times out of eight the Path Index will be “1”, and three times out of eight the Path Index will be “0”. As a result, traffic is load-balanced according to the relative traffic load information embedded within the Multipath Weight Vector.

[0048] In other embodiments where the HASH/CRC block 210 uses header fields containing non-random values (e.g., destination address fields, flow identifiers and application port numbers), the resulting hash values will likely be the same if the packets are bound for the same destination or share common attributes. In these embodiments, load balancing can be achieved when an appropriate Multipath Weight Vector with a desired relative traffic load profile matching the profile of the packet stream is used.

[0049] FIG. 9 illustrates portions of a Packet Classification Engine 300 according to another embodiment of the invention where the Path Index can be generated based on a random or pseudo-random value that is not derived from a data packet header. As shown, the Packet Classification Engine 300 includes a Pseudo-Random Number Generator 260 and multiplexer 270. The multiplexer 270 receives a pseudo-random number from the Pseudo-Random Number Generator 260 or a hash value from FIFO 215 according to a control signal. Upon receiving appropriate control signals, the multiplexer 270 can select the outputs from the Pseudo-Random Number Generator 260 or the outputs from the HASH/CRC block 210. When the outputs of the Pseudo-Random Number Generator 260 are selected, the Path Index Calculation block 240 will use a pseudo-random number and a Multipath Weight Vector to generate the Path Index. The Path Index thus generated will therefore vary randomly (or pseudo-randomly) within the range prescribed by the Multipath Weight Vector. When the multiplexer 270 selects outputs from the HASH/CRC block 210 and the FIFO 215, the Path Index Calculation block 240 will use the hash value and a Multipath Weight Vector to generate the Path Index.

[0050] In another embodiment, a true random number generator, such as one that samples and processes an external entropy source, is used for generating truly random values. However, because a true random number generator is difficult to implement, a pseudo-random number generator is preferred.

[0051] It shall be appreciated that not all methods steps described must be performed, nor must they be performed in the order stated.

[0052] Although specific embodiments of the invention have been described and illustrated, the invention is not to be limited to the specific forms or arrangements of parts as described and illustrated herein. The invention is limited only by the claims.