US 8,477,597 B2Grant

Method and system for resilient routing reconfiguration

Issue Date:2013-07-02

•42 Claims

•12 Drawing Sheets

Abstract

A method and system for resilient routing reconfiguration, which provides provably congestion-free rerouting under a large number of failure scenarios and achieves efficiency and flexibility in accommodating different performance requirements, is disclosed. The method of embodiments comprises an offline precomputation phase and an online reconfiguration phase. Embodiments of the present invention also provide a number of useful functionalities, including (i) coping with traffic variations, (ii) supporting realistic failure scenarios, (iii) accommodating prioritized traffic with different protection levels, and (iv) balancing the trade-off between performance and resilience.

Metadata

Inventors

Yin Zhang
Lili Qiu

Application Information

Application Number:US 12/787,411

Filing Date:2010-05-26

Priority Date:2009-05-27

Art Unit:2462

Classifications

IPC:

G01R31/08

Patent Drawings (12 sheets)

Description

Related Applications

[0001] The present application claims the priority benefits of Zhang et al., U.S. Provisional Patent Application No. 61/181,604 titled, “RESILIENT ROUTING RECONFIGURATION (R3)”, filed on May 27, 2009, which is hereby incorporated by reference in its entirety.

Field of the Invention

[0002] This invention relates generally to Internet Protocol (IP) network operations. More particularly, the invention relates to a method and system for resilient routing reconfiguration (R3), a protection scheme for routing in an IP network that can avoid congestion and offer predictable performance under one or a series of link failures.

Background of the Invention

[0003] Network resiliency, defined as the ability of an IP network to recover quickly and smoothly from one or a series of failures or disruptions, is becoming increasingly important in the operation of modern IP networks. Recent large-scale deployment of delay- and loss-sensitive services such as VPN and IPTV impose stringent requirements on the tolerable duration and level of disruptions on IP traffic. In a recent survey of major network carriers including AT&T, BT, and NTT, Telemark concludes that “The 3 elements which carriers are most concerned about when deploying communication services are network reliability, network usability and network fault processing capabilities” (See Telemark, “Telemark survey,” http://www.telemarkservices.com/, 2006). All three relate to network resiliency.

[0004] Unfortunately, the current techniques for fault processing to achieve resiliency are still far from ideal. Consider fast-rerouting (FRR) (See M. Shand and S. Bryant, “IP fast reroute framework,” IETF Internet-Draft, draft-ietf-rtgwg-ipfrr-framework-06.txt, 2007), the major currently deployed technique to handle network failures. As a major tier-1 ISP pointed out at Multi-Protocol Label Switching (MPLS) World Congress 2007, there are major practical challenges when using FRR in its business core network (See N. So and H. Huang, “Building a highly adaptive, resilient, and scalable MPLS backbone,” http://www.wandl.com/html/support/papersNerizonBusiness WANDL MPLS2007.pdf, 2007):

[0005] (a) Complexity: “the existing FRR bandwidth and preemption design quickly becomes too complicated when multiple FRR paths are set up to account for multiple failures;”

[0006] (b) Congestion: “multiple network element failure can cause domino effect on FRR reroute due to preemption which magnifies the problem and causes network instability;”

[0007] (c) No performance predictability: “service provider loses performance predictability due to the massive amount of combinations and permutations of the reroute scenarios.”

[0008] The importance of network resiliency has attracted major attention in the research community. Many mechanisms have been proposed to quickly detour around failed network devices (See, P. Francois, C. Filsfils, J. Evans, and O. Bonaventure, “Achieving sub-second IGP convergence in large IP networks,” ACM Computer Communication Review, 35(3):35-44, 2005 (Francois et al. 2005), G. Iannaccone, C. Chuah, S. Bhattacharyya, and C. Diot, “Feasibility of IP restoration in a tier-1 backbone,” IEEE Network Magazine, 18(2):13-19, 2004 (Iannaccone et al. 2004), M. Motiwala, M. Elmore, N. Feamster, and S. Vempala, “Path splicing,” Proc. ACM SIGCOMM, 2008 (Motiwala et al. 2008), J. P. Vasseur, M. Pickavet, and P. Demeester, “Network Recovery: Protection and Restoration of Optical, SONET-SDH, and MPLS,” Morgan Kaufmann, 2004 (Vasseur et al. 2004)). The focus of these studies, however, was mainly on reachability only (i.e., minimizing the duration in which routes are not available to a set of destinations). Hence, they do not address the aforementioned practical challenges, in particular on congestion and performance predictability.

[0009] It is crucial to consider congestion and performance predictability when recovering from failures. Since the overall network capacity is reduced after failures, if the remaining network resources are not efficiently utilized, serious congestion may occur. As observed in a measurement study on a major IP backbone (See S. Iyer, S. Bhattacharyya, N. Taft, and C. Diot, “An approach to alleviate link overload as observed on an IP backbone,” Proc. IEEE INFOCOM, April 2003 (Iyer et al. 2003)), network congestion is mostly caused by traffic that has been rerouted due to link failures. Meanwhile, it has been shown that focusing only on reachability may lead to long periods of serious congestion and thus violation of service level agreements (SLAs).

[0010] However, it may be challenging to derive a routing protection scheme to offer performance predictability and avoid congestion. The main difficulty may lie in the vast number of failure scenarios, which grows exponentially with the number of links to be considered. Consider a tier-1 ISP network with 500 links, and assume that the network would like to find a routing protection plan to protect 3 simultaneous link failures. The number of such scenarios exceeds 20 million! Despite much progress on intra-domain traffic engineering, optimizing the routing simultaneously for just a few hundred network topologies is already beyond the means of any existing technique. As a result, existing routing protection schemes have to either focus exclusively on reachability (hoping that congestion does not occur), or consider only a single link failure (which is insufficient as SLAs become ever more demanding).

[0011] Therefore, there is a need for a method and system for deriving a routing protection scheme to provide predictable performance and avoid congestion under one or a series of failures in an IP network.

Brief Summary of the Invention

[0012] The problems and challenges outlined above may at least in part be addressed by a system and method for Resilient Routing Reconfiguration (R3) as described herein.

[0013] The following presents a simplified summary of the disclosure in order to provide a basic understanding to the reader. This summary is not an exhaustive or limiting overview of the disclosure. The summary is not provided to identify key and, or critical elements of the invention, delineate the scope of the invention, or limit the scope of the invention in any way. Its sole purpose is to present some of the concepts disclosed in a simplified form, as an introduction to the more detailed description that is presented later.

[0014] Embodiments of the present invention provide a general method for covering all possible failure scenarios with a compact set of linear constraints on the amounts of traffic that should be rerouted. Specifically, when F links fail, the traffic originally routed through each failed link has to be rerouted by the remaining network. While the amount of rerouted traffic for a failed link depends on the specific failure scenario, it is always upper bounded by the capacity of the failed link (so long as the routing before the failure is congestion-free). Therefore, by creating a virtual demand for every link in the network (whose volume is equal to its link capacity) and taking the convex combination of all such virtual demands, the method of embodiments can cover the entire space of rerouted traffic under all possible combinations of F link failures. Since the convex hull of virtual demands can be represented as a compact set of linear constraints, linear programming duality can be used to efficiently optimize routing over the entire set. In this way, the method of embodiments eliminates the needs for enumerating failure scenarios by converting topology uncertainty (due to failures) into uncertainty in rerouted traffic, which may be easier to cope with.

[0015] Embodiments of the present invention further provide a routing protection method and system that comprises an offline precomputation phase and an online reconfiguration phase. The offline phase precomputes routing for the actual demand plus the virtual demand on the original network topology; while the online reconfiguration phase responds to failures using a simple rescaling procedure, which converts the offline precomputed routing into a protection routing that does not traverse any failed links. A unique feature of the method of embodiments is that it is (i) provably congestion-free under multiple link failures, and (ii) provably optimal for single-link failure scenarios.

[0016] Embodiments of the present invention also provide several methods that extend R3 to cope with (i) traffic variations, (ii) realistic failure scenarios, (iii) prioritized traffic with different protection levels, and (iv) the tradeoff between performance and resilience.

[0017] R3 protection may be implemented using MPLS-fr (a simple extension of MPLS), while the base routing may use either OSPF or MPLS. Evaluation using Emulab and simulation based on real Internet topologies and traffic traces show that R3 can achieve near-optimal performance and is at least 50% better than existing schemes such as OSPF recovergence, OSPF with CSPF fast rerouting, FCP, and Path Splicing.

Brief Description of the Drawings

[0018] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of an embodiment of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure may be practiced without these specific details. The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

[0019] FIG. 1 is a simple example illustrating the failure response of MPLS-fr and the use of label stacking. Specifically, FIG. 1A shows that under normal condition, router R1 routes flows to R3 through R2; FIG. 1B shows that after link (R1,R2) fails, R4 and R5 carry protection traffic by label stacking.

[0020] FIG. 2 illustrates the time series of worst-case normalized traffic intensity with one failure during a given day for US-ISP.

[0021] FIG. 3 summarizes the overall performance with one failure during the entire evaluation period (which lasts seven days) for US-ISP.

[0022] FIG. 4A and FIG. 4B summarize the performance under two-link or three-link failures during peak hour for US-ISP.

[0023] FIG. 5A and FIG. 5B summarize the performance under two-link or three-link failures for SBC.

[0024] FIG. 6A and FIG. 6B summarize the performance under two-link or three-link failures for Level 3.

[0025] FIG. 7A-7C show the normalized bottleneck traffic intensities for the three classes of traffic under R3 with and without priority.

[0026] FIG. 8 shows the benefit of penalty envelope by comparing the performance of four algorithms: R3 without penalty envelope, OSPF, R3 with penalty envelope, and optimal.

[0027] FIG. 9 shows the effect of base routing on the performance under one-link failures (FIG. 9A) and two-link failures (FIG. 9B).

[0028] FIG. 10A-10C summarizes the network performance using R3 Linux implementation under multiple link failures.

[0029] FIG. 11 plots the real-time RTT of a flow between Denver and Los Angeles during the Emulab experiment process.

[0030] FIG. 12 shows the performance improvement of R3 over OSPF reconvergence protection using Emulab experiment.

Detailed Description of the Invention

[0031] Embodiments of the present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the inventions are shown. Indeed, embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like numbers refer to like elements throughout.

[0032] The present invention broadly discloses a method and system for Resilient Routing Reconfiguration (R3), a routing protection scheme that is (i) provably congestion-free under a wide range of failure scenarios, (ii) efficient in terms of router processing overhead and memory requirement, (iii) flexible in accommodating diverse performance requirements (e.g., different traffic protection levels), and (iv) robust to traffic variations and topology failures. Here by “congestion-free”, it means that all traffic demands (except those demands that have lost reachability due to network partition) are routed without creating any link overload. This is a much stronger guarantee than providing reachability alone (as existing protection such as FRR).

[0033] At the heart of the approach of embodiments described herein is a general method for covering all possible failure scenarios with a compact set of linear constraints on the amounts of traffic that should be rerouted. Specifically, when F links fail, the traffic originally routed through each failed link has to be rerouted by the remaining network. While the amount of rerouted traffic for a failed link depends on the specific failure scenario, it is always upper bounded by the capacity of the failed link (so long as the routing before the failure is congestion-free). Therefore, by creating a virtual demand for every link in the network (whose volume is equal to its link capacity) and taking the convex combination of all such virtual demands, the method of embodiments can cover the entire space of rerouted traffic under all possible combinations of F link failures. Since the convex hull of virtual demands can be represented as a compact set of linear constraints, linear programming duality can be leveraged to efficiently optimize routing over the entire set. In this way, the method of embodiments eliminates the needs for enumerating failure scenarios by converting topology uncertainty (due to failures) into uncertainty in rerouted traffic, which may be easier to cope with.

[0034] Since the virtual demands are upper bounds of the rerouted traffic, the method of embodiments described herein guarantees that if a routing is congestion-free over the virtual demand set, it yields a link protection scheme that is congestion-free under all possible failure scenarios. The converse is also true for single-link failures: if there is a link protection scheme that can guarantee no congestion for all single-link failure scenarios, then there must be a routing that is congestion-free over the entire virtual demand set. Therefore, the seemingly wasteful replacement of rerouted traffic with link capacities is actually efficient.

[0035] Based on the foregoing general method of embodiments for converting topology uncertainty to traffic uncertain, embodiments of the present invention further provide a routing protection scheme called Resilient Routing Reconfiguration (R3), which comprises an offline precomputation phase and an online reconfiguration phase. During the offline phase, according to one embodiment, R3 computes routing for the actual demand plus the virtual demand on the original network topology. During the online reconfiguration phase, according to one embodiment, R3 responds to failures using a simple rescaling procedure, which converts the offline precomputed routing into a protection routing that does not traverse any failed links. A unique feature of R3 as described herein is that it is (i) provably congestion free under multiple link failures, and (ii) provably optimal for single-link failure scenarios.

[0036] Embodiments of the present invention further provide methods for extending R3 to handle (i) traffic variations, (ii) realistic failure scenarios, (iii) prioritized traffic with different protection levels, and (iv) the tradeoff between performance and resilience.

[0037] R3 protection may be implemented using MPLS-fr (a simple extension of MPLS), while the base routing can use either OSPF or MPLS. Emulab evaluation and simulation based on real Internet topologies and traffic traces show that R3 can achieve near-optimal performance and is at least 50% better than existing routing protection schemes, such as OSPF recovergence, OSPF with CSPF fast rerouting, FCP (See K. Lakshminarayanan, M. Caesar, M. Rangan, T. Anderson, S. Shenker, and I. Stoica, “Achieving convergence-free routing using failure-carrying packets,” Proc. ACM SIGCOMM, August 2007 (Lakshminarayanan et al. 2007)), and Path Splicing (See M. Motiwala, M. Elmore, N. Feamster, and S. Vempala, “Path splicing,” Proc. ACM SIGCOMM, 2008 (Motiwala et al. 2008)).

[0038] To help the reader of this disclosure better understand the present invention, the remainder of this detailed description will be organized as follows. Section 1 formulates the problem of resilient routing protection, and provides an overview of the basic approach of embodiments for converting topology uncertainty into traffic uncertainty. Section 2 provides a detailed description of the basic R3 protection scheme (comprising an offline precomputation phase and an online reconfiguration phase) and its theoretical guarantees. Section 3 describes a prototype Linux implementation of R3. Section 4 evaluates the performance of R3 using both simulations and testbed experiments. Section 5 summarizes the related works. Section 6 concludes.

1. Overview

[0039] A traditional traffic engineering algorithm computes an effective base routing r that optimizes a network metric, such as minimizing congestion cost or maximum link utilization (See, B. Fortz, J. Rexford, and M. Thorup, “Traffic engineering with traditional IP routing protocols,” IEEE Communication Magazine, October 2002 (Fortz et al. 2002), B. Fortz and M. Thorup, “Internet traffic engineering by optimizing OSPF weights,” Proc. IEEE INFOCOM, March 2000 (Fortz et al. 2000), M. Roughan, M. Thorup, and Y. Zhang, “Traffic engineering with estimated traffic matrices,” Proc. Internet Measurement Conference (IMC), October 2003 (Roughan et al. 2003), H. Wang, H. Xie, L. Qiu, Y. R. Yang, Y. Zhang, and A. Greenberg, “COPE: Traffic engineering in dynamic networks,” Proc. ACM SIGCOMM, 2006 (Wang et al. 2006)). Then a protection routing p may be derived from r, for example, through fast rerouting (FRR). However, this traditional approach may result in serious network congestion and performance unpredictability under failures. The remainder of this section first formally defines the problem of resilient routing and explains why it is challenging before presenting the key ideas of R3. Notations: Let G=(V, E) be an IP network under consideration, where V is the set of routers in the network, and E is the set of network links connecting the routers. Let d be the traffic matrix between the routers in V, where d_abis the traffic from router a to router b. Let c_eor c_ijdenote the capacity of a directed link e=(i, j) from router i to router j. i is referred to as the source node of link e and j its tail node. To define routing precisely, the flow representation of routing will be used (See, D. Applegate and E. Cohen, “Making intra-domain routing robust to changing and uncertain traffic demands: Understanding fundamental tradeoffs,” Proc. ACM SIGCOMM, August 2003 (Applegate et al. 2003), D. Bertsekas and R. Gallager, “Data Networks,” Prentice-Hall, 1992 (Bertsekas et al. 1992)). Formally, a flow representation of a routing r is specified by a set of values {r_ab(e)|a, bεV, eεE}, where r_ab(e) or r_ab(i, j) specifies the fraction of traffic for the origin-destination (OD) pair a→b that is routed over the link e=(i, j). For actual traffic d_abof the OD pair a→b, the contribution of this traffic to the load on link e is d_abr_ab(e). For {r_ab(e)} to be a valid routing for a given OD pair a≠b, it should satisfy the following conditions:

[0040]

\begin{matrix} [R 1] \forall i \neq a, b : \sum_{(i, j) \in E} r_{ab} (i, j) = \sum_{(j, i) \in E} T_{ab} (j, i); [R 2] \sum_{(a, i) \in E} r_{ab} (a, i) = 1; [R 3] \forall (i, a) ⋐ E : r_{ab} (i, a) = 0; [R 4] \forall e \in E : 0 \leq r_{ab} (e) \leq 1. & (1) \end{matrix}

The first condition indicates flow conservation at any intermediate nodes. The second condition specifies that all traffic from a source should be routed. The third condition prevents traffic from returning to the source. Finally, according to the definition of r_ab(e), it is between 0 and 1. Problem formulation: The following basic formulation of resilient routing is considered in this disclosure. In Section 2.4, several useful extensions to the basic formulation are described.

[0041] Definition 1 (Resilient Routing) The problem of resilient routing is to design an effective base routing r and protection routing p for traffic matrix d to ensure that the network is congestion free (i.e., the maximum link utilization stays below 100%) under all possible failure scenarios involving up to F failed links. The base routing r can also be given as an input (e.g., by OSPF), in which case only the protection routing p needs to be designed.

[0042] Multiple protection routing schemes are possible in practice. To minimize disruption, it is common to only consider protection routing that changes the route of an OD pair when the OD pair traverses a failed link. Among this class of routing reconfiguration techniques, link-based protection is the most widely used and implemented. Thus, the methods of embodiments described herein focus on link-based protection. However, the present invention can easily extend to path-based protection, which can be viewed as a special case of link-based protection in an overlay topology. In link-based protection, the source node of a failed link reroutes the traffic originally passing through a failed link along a detour route to reach the tail node of the link. As a result, the protection routing p only needs to be defined for each link that requires protection. In contrast, the base routing r defines routing for each OD pair.

[0043] Challenge in coping with topology uncertainty: Due to the frequency of failures, the delay in failure recovery (See, Iyer et al. 2003, Lakshminarayanan et al. 2007) and the increasingly stringent SLA for network services, it is essential for resilient routing to avoid congestion under multiple link failures overlapping in time. This requires the design of resilient routing to explicitly consider all possible failure scenarios. One natural approach to resilient routing is to enumerate all failure scenarios and derive a routing that works well for all these scenarios. However, this approach faces serious scalability and efficiency issues. Suppose a network with |E| links needs to handle up to F link failures. Then there will be

[0044]

\sum_{i = 1}^{F} (\begin{matrix} \langle E \rangle \\ i \end{matrix})

failure scenarios, which result in prohibitive computation and configuration cost even for a small number of failures. On the other hand, in order to guarantee congestion-free, it is imperative to protect against all of the

[0045]

\sum_{i = 1}^{F} (\begin{matrix} \langle E \rangle \\ i \end{matrix})

scenarios, since a skipped scenario may arise in practice and cause network congestion and violation of SLA. Therefore, fundamental challenges in achieving resilient routing involve (i) efficient computation of protection routing that is provably congestion-free even under multiple failures and (ii) simple re-configuration in response to failures.

[0046] From topology uncertainty to traffic uncertainty: The key idea of R3 is to convert topology uncertainty (due to the number of failure scenarios) into traffic uncertainty that captures the different traffic demands that need to be rerouted under different failure scenarios.

[0047] Specifically, suppose the routing protection scheme is required to protect against up to F arbitrary link failures. Under link-based protection, the rest of the network needs to carry traffic previously carried by the failed links. It is easy to see that the rerouted traffic is upper bounded by the capacity of each failed link (as long as no link is fully utilized under the base routing r). Therefore, every link in the network can induce a virtual demand equal to the link capacity. The convex combination of all such virtual demands should cover the entire space of rerouted traffic. Formally, each link eεE is associated a virtual demand variable x_e. A rerouting virtual demand set X_Fis then formed as

[0048]

\begin{matrix} X_{F} \overset{Δ}{=} {x ❘ 0 \leq \frac{x_{e}}{c_{e}} \leq 1 (\forall e \in E), \sum_{e \in E} \frac{x_{e}}{c_{e}} \leq F} . & (2) \end{matrix}

[0049] For any failure scenario that involves up to F link failures, it can be shown that the traffic that needs to be rerouted always belongs to set X_F. Therefore, X_Frepresents an envelope (i.e., superset) of the rerouted traffic under all possible failure scenarios.

[0050] Instead of trying to optimize routing for the fixed traffic matrix d on a variable topology under all possible failure scenarios, the methods of embodiments described herein try to find a routing that works well for the entire demand set d+X_Fbut on the fixed original topology, where d+X_F custom character

{d+x|xεX_F} denotes the sum of the actual demand d and the set of virtual demands X_F. In this way, the methods of embodiments convert topology uncertainty into traffic uncertainty.

[0051] At the first glance, converting topology uncertainty into traffic uncertainty makes the problem more challenging, because the number of failure scenarios is at least finite, whereas d+X_Fmay contain an infinite number of traffic matrices. However, the rerouting virtual demand set X_Fcan be represented using a compact set of linear constraints (in Equation 2). By applying linear programming duality, the optimal base routing r and protection routing p for the entire demand set d X_Fcan be found without enumerating traffic matrices.

[0052] Another potential concern is that the definition of the rerouting virtual demand set X_Fappears rather wasteful. When links e₁, . . . , e_Ffail, the corresponding virtual demands in X_Fcan be as large as x_e_i=c_e_i(i=1, . . . , F). That is, the rerouted traffic on failed link e_iis replaced with a virtual demand equal to the link capacity c_e_i. Interestingly, it will be proved in Section 2.3 that the seemingly wasteful replacement of rerouted traffic with link capacities is necessary at least for F=1. Specifically, if there exists a link-based protection routing that guarantees no congestion for all single-link failure scenarios, then there must exist a routing that is congestion-free over the entire virtual demand set d+X_F.

[0053] R3 overview: R3 is designed based on the preceding insight. In one embodiment, R3 comprises the following two main phases:

- [0054] Offline precomputation. According to one embodiment, during the offline precomputation phase, R3 computes routing r (if not given) for traffic matrix d and routing p for rerouting virtual demand set X_Fto minimize the maximum link utilization on the original network topology over the combined demand set d+X_F. The optimization is made efficient by leveraging linear programming duality, a technique also exploited by recent research on traffic oblivious routing (See, Applegate et al. 2003, Wang et al. 2006).
- [0055] Online reconfiguration. According to one embodiment, during the online reconfiguration phase, after a failure, R3 applies a simple procedure called rescaling to convert p (which is defined on the original network topology and thus may involve the failed link) into a protection routing that does not traverse any failed link and thus can be used to reroute traffic on the failed links. The rescaling procedure is efficient and can be applied in real-time with little computation and memory overhead.

[0056] A unique feature of the methods of embodiments described herein is that R3 can provide several provable theoretical guarantees. In particular, R3 guarantees no congestion under a wide range of failure scenarios involving multiple link failures. As a result, it provides stronger guarantee than simple reachability. Moreover, the conversion from topology uncertainty into traffic uncertainty is efficient in that the seemingly wasteful replacement of rerouted traffic with link capacity is indeed necessary for single-link failure scenarios. Finally, the online reconfiguration procedure is independent of the order in which the failed links are detected. So routers can apply R3 independently even when the failed links are detected in different orders.

[0057] Embodiments of the present invention also provide a number of useful extensions to R3 for (i) coping with traffic variations, (ii) supporting realistic failure scenarios, (iii) accommodating prioritized traffic with different protection levels, and (iv) balancing the trade-off between performance and resilience.

2. Resilient Routing Reconfiguration (R3)

[0058] This section presents the detailed design of R3. Specifically, Section 2.1 describes offline precomputation, and Section 2.2 describes online reconfiguration. Section 2.3 proves several theoretical guarantees of R3. Section 2.4 introduces several useful extensions to R3.

2.1 Offline Precomputation

[0059] Problem formulation: The goal of offline precomputation is to find routing r for traffic matrix d and routing p for rerouting virtual demand set X_Fdefined in (2) to minimize the maximum link utilization (MLU) over demand set d+X_F. This can be formulated as the optimization problem shown in (3). The objective is to minimize MLU over the entire network. Constraint [C1] ensures that r and p are valid routing, i.e., they both satisfy routing constraints (1). Constraint [C2] enforces all links have utilization below MLU.

[0060]

\begin{matrix} {minimize}_{(r, p)} MLU subject to : [C 1] r = {r_{ab} (e) ❘ a, b \in V, e \in E} is a routing; p = {p_{ℓ} (e) ❘ ℓ, e \in E} is a routing; [C 2] \forall x \in X_{F}, \forall e \in E : \frac{\sum_{a, b \in V} d_{ab} r_{ab} (e) + \sum_{l \in E} x_{l} p_{l} (e)}{c_{e}} \leq MLU . & (3) \end{matrix}

[0061] Note that p is defined for each link whereas r is defined for each OD pair. Also note that when r is pre-determined (e.g., by OSPF), r becomes an input to the optimization in (3) instead of being an optimization variable.

[0062] Solution strategy: The main challenge in solving (3) is that there is a constraint [C2] for every element x belonging to the rerouting virtual demand set X_F. Since X_Fhas an infinite number of elements, the number of constraints becomes infinite. Fortunately, linear programming duality can be applied to convert (3) into an equivalent, simpler linear program with a polynomial number of constraints as follows.

[0063] First, constraint [C2] in (3) is equivalent to:

[0064]

\begin{matrix} \forall e \in E : \frac{\sum_{a, b \in V} d_{ab} r_{ab} (e) + ML (p, e)}{c_{e}} \leq MLU, & (4) \end{matrix}

where ML(p, e) is the maximum load on e for ∀xεX_F, and thus is the optimal objective of the following problem:

[0065]

\begin{matrix} \begin{matrix} {maximize}_{x} & \sum_{l \in E} p_{l} (e) x_{l} \\ subject to : & {\begin{matrix} \forall ℓ \in E : x_{ℓ} / c_{ℓ} \leq 1 \\ \sum_{ℓ \in E} x_{ℓ} / c_{ℓ} \leq F . \end{matrix} \end{matrix} & (5) \end{matrix}

Here (5) is a linear program when p is a fixed input. From linear programming duality, the optimal objective of (5), ML(p, e), is no more than a given upper bound UB if and only if there exist dual multipliers π_e(lεE) and λ_esuch that:

[0066]

\begin{matrix} \sum_{ℓ \in E} π_{e} (ℓ) + λ_{e} F \leq UB; \forall ℓ \in E : \frac{π_{e} (ℓ) + λ_{e}}{c_{ℓ}} \geq p_{ℓ} (e); \forall ℓ \in E : π_{e} (ℓ) \geq 0; λ_{e} \geq 0. & (6) \end{matrix}

Here π_e(l) is the dual multiplier for constraint x_l/c_l≦1, λ_eis the dual multiplier for Σ_lx_l/c_l≦F, and the subscript e indicates that (5) computes the maximum load on link e.

[0067] Since all of the constraints in (6) are linear, (4) can be converted into a set of linear constraints by substituting ML(p, e) with Σ_lεEπ_e(l)+λ_eF and incorporating (6). It can be shown that the original problem (3) then becomes the following equivalent linear program, which can be solved using commercial software such as cplex.

[0068]

\begin{matrix} {minimize}_{(r, p, π, λ)} MLU subject to : {\begin{matrix} r = {r_{ab} (e) ❘ a, b \in V, e \in E} is a routing; \\ p = {p_{l} (e) ❘ ℓ, e \in E} is a routing; \\ \forall e \in E : \frac{\sum_{a, b \in V} d_{ab} r_{ab} + \sum_{l \in E} π_{e} (l) + λ_{e} F}{c_{e}} \leq MLU; \\ \forall e, l \in E : \frac{π_{e} (l) + λ_{e}}{c_{l}} \geq p_{l} (e); \\ \forall e, l \in E : π_{e} (l) \geq 0; \\ \forall e \in E : λ_{e} \geq 0 \end{matrix} & (7) \end{matrix}

[0069] Note that in (7) the base routing r and the protection routing p are jointly optimized. It is also possible to only optimize the protection routing p and have the base routing r as a fixed input. To do so, it suffices to eliminate r as an optimization variable in (7), yielding:

[0070]

\begin{matrix} {minimize}_{(p, π, λ)} MLU subject to : {\begin{matrix} p = {p_{l} (e) ❘ l, e \in E} is a routing; \\ \forall e \in E : \frac{\sum_{a, b \in V} d_{ab} r_{ab} + \sum_{l \in E} π_{e} (l) + λ_{e} F}{c_{e}} \leq MLU; \\ \forall e, l \in E : \frac{π_{e} (l) + λ_{e}}{c_{l}} \geq p_{l} (e); \\ \forall e, l \in E : π_{e} (l) \geq 0; \\ \forall e \in E : λ_{e} \geq 0. \end{matrix} & (7) \end{matrix}

[0071] Complexity: Linear program (7) has O(|V|²·|E|+|E|²) variables and O(|V|³+|E|²) constraints. Even if the problem to solve only needs to find r to minimize the MLU for fixed traffic matrix d, routing constraints (1) already have O(|V|²·|E|) variables and O(|V|³) constraints. In most networks, |E|²≦|V|³. So (7) only causes moderate increase in the size of the linear program. Finally, note that linear programming duality has also been exploited in recent research on oblivious routing (See, Applegate et al. 2003, Wang et al. 2006). However, oblivious routing requires O(|V|·|E|²) constraints, which is much higher than (7).

2.2 Online Reconfiguration

[0072] After the failure of link e is detected, two main tasks are performed by online reconfiguration. First, the source router of e needs to immediately reroute the traffic originally traversing e through a detour route. Second, in preparation for additional link failures, every router needs to adjust r and p so that no demand traverses the failed link e.

[0073] Fast rerouting of traffic on the failed link: After link e fails, the source router of e immediately uses p to derive a detour route ξ_eto reroute the traffic that traverses e before it fails, according to one embodiment. Note that p_e={p_e(l)|lεE} cannot be directly used as the detour route ξ_e, because p_eis defined on the original topology and may assign non-zero traffic to e (i.e., p_e(e)>0). Fortunately, {p_e(l)|l≠e} already satisfies routing constraints [R1], [R3] and [R4] in (1). To convert it into a valid detour route ξ_e, it suffices to perform the following simple re-scaling to ensure that all traffic originally traversing e is rerouted (thus satisfying [R2]).

[0074]

\begin{matrix} ξ_{c} (ℓ) = \frac{p_{n} (ℓ)}{1 p_{e} (e)} (\forall ℓ \in E \ {e}) . & (8) \end{matrix}

[0075] As an example, consider a simple network with 4 parallel links e₁, e₂, e₃, and e₄. Suppose the protection routing p for virtual demand e₁specifies that p_e1(e₁)=0.1, p_e1(e₂)=0.2, p_e1(e₃)=0.3, and p_e1(e₄)=0.4. After e₁fails, the original traffic traversing e₁needs to be rerouted through e₂, e₃, and e₄in proportion to p_e1(e₂), p_e1(e₃), and p_e1(e₄). To ensure all such traffic is rerouted, the methods of embodiments scale up all of the fractions by a factor of

[0076]

1 / (1 - p_{e 1} (e_{1})) = 1 / 0.9, yielding ξ_{e 1} (e_{2}) = \frac{2}{9}, ξ_{e 1} (e_{3}) = \frac{3}{9}, and ξ_{e 1} (e_{4}) = \frac{4}{9} .

[0077] Note that when p_e(e)=1, it suffices to simply set ξ_e(l)=0. As shown later, under the condition of Theorem, p_e(e)=1 implies that there is no (actual or virtual) demand that traverses link e (and thus needs to be rerouted).

[0078] Adjusting r and p to exclude the failed link: In preparation for additional link failures, R3 adjusts r and p to ensure that no (actual or virtual) demand traverses the failed link e, according to one embodiment. This can be achieved by moving the original traffic allocation on link e to the detour route ξ_e. Specifically, let E′=E\{e} and G′=(V, E′). The updated base routing r′ is defined as:
r′_ab(l)=r_ab(l)+r_ab(e)·ξ_e(l), ∀(a,b)εV, ∀lεE′,
where r_ab(l) is the original allocation on link for OD pair a→b, and r_ab(e)·ξ_e(l) gives the increase due to using ξ_eto reroute the original allocation on the failed link (i.e., r_ab(e)). Similarly, the updated protection routing p′ is defined as:
p′_uv(l)−p_uv(l)+p_uv(e)·ξ_e(l), ∀(u,v)εE′, ∀lεE′.

[0079] Efficiency: All of the operations in online reconfiguration are simple and thus highly efficient. Specifically, computing ξ_efrom p only requires simple rescaling of {p_e(l)}. Even such simple rescaling can be avoided if the methods of embodiments directly store p_e(e) and

[0080]

ξ_{e} = {\frac{p_{e} (ℓ)}{1 - p_{e} (e)} ❘ ℓ \neq e}

instead of {p_e(l)|lεE}. Meanwhile, updating r′_ab(e) and p′_uv(l) is also extremely simple and is only required for demands with non-zero traffic allocation on the failed link (i.e., r_ab(e)>0 and p_uv(e)>0). Note that R3 does not require all routers to finish updating their r and p before it recovers from the failed link e—the recovery reaches full effect as soon as the source router of e starts rerouting traffic through the detour route ξ_e.
3.3 Theoretical Guarantees of R3

[0081] Sufficient condition for congestion-free guarantee: A key feature of R3 is that it can provide provable congestion-free guarantee under all possible failure scenarios as long as the optimal MLU in (7) is below 1. More formally:

[0082] Theorem 1 Let X_Fbe the rerouting virtual demand set with up to F link failures, as defined in (2). If offline precomputation (Section 2.1) finds routing r and p such that the MLU for the entire demand set d+X_Fis no larger than 1 on the original topology G=(V, E), then online reconfiguration (Section 2.2) guarantees that the MLU for the real traffic matrix d and the rerouted traffic is no larger than 1 under any failure scenario with up to F failed links.

[0083] Proof: Let e be the first failed link. Let E′=E\{e}. Let r′ and p′ be the updated routing after online reconfiguration. Let X_F-1be the rerouting virtual demand set with up to F-1 failures in E′. Below it will be shown that r′ and p′ guarantee that the MLU for demand set d+X_F-1is no larger than 1 on the new topology G′=(V, E′). Consider any lεE′ and xεX_F-1. Let L (d, x, r′, p′, l) be the load on link l coming from real traffic d and virtual demand x using base routing r′ and protection routing p′. The following holds:

[0084]

\begin{matrix} \begin{matrix} L (d, x, r^{'}, p^{'}, ℓ) = \sum_{a, b \in V} d_{ab} r_{ab}^{'} (ℓ) + \sum_{(u, v) \in E} {x_{uv} (p^{'})}_{uv} (ℓ) \\ = \sum_{a, b \in V} d_{ab} (r_{ab} (ℓ) + r_{ab} (e) ξ_{e} (ℓ)) + \\ \sum_{(u, v) \in E} x_{uv} (p_{uv} (ℓ) + p_{uv} (e) ξ_{e} (ℓ)) \\ = L (d, x, r, p, ℓ) + L (d, x, r, p, e) \cdot \frac{p_{e} (ℓ)}{1 - p_{e} (e)} \end{matrix} & (9) \end{matrix}

[0085] Given xεX_F-1, yεX_Fcan be obtained by adding a virtual demand for the failed link e to x. That is, y_e=c_eand y_uv=x_uvfor ∀(u, v)εE′. Since r and p guarantee no congestion for d+X_F, the following holds:
c_l≧L(d,y,r,p,l)=L(d,x,r,p,l)+c_e·p_e(l) (10)
c_e≧L(d,y,r,p,e)=L(d,x,r,p,e)+c_e·p_e(e) (11)

[0086] From (11) and when p_e(e)<1, it follows that:
c_e≧L(d,x,r,p,e)/(1−p_e(e)). (12)

[0087] Substituting c_ein (10) with the R.H.S. of (12), the following is obtained:

[0088]

\begin{matrix} c_{ℓ} \geq L (d, x, r, p, ℓ) + L (d, x, r, p, e) \frac{p_{e} (ℓ)}{1 - p_{e} (e)} & (13) \end{matrix}

[0089] Combining (9) and (13), it is clear that c_l≧L(d, x, r′, p′, l) (for ∀lεE′). Note that this also holds when p_e(e)=1. In this case, under the assumption that MLU≦1, no other actual or virtual demand traverses e and thus needs to be rerouted. So R3 can simply set ξ_e(l)=0 and L(d, x, r′, p′, l)=L(d, x, r, p, l)≦c_l. Therefore, r′ and p′ guarantees that the MLU for d+X_F-1on G′=(V, E′) is no larger than 1. Consequently, r′ guarantee that the MLU for d is no larger than 1. By induction, it then follows that d is congestion-free for any failure scenario with up to F failed links. Note that depending on the value of F and the connectivity of G, it may not be possible to find r and p that meet the sufficient condition. For example, if there exist F failures that partition the network, then it is impossible to find r and p to ensure that the MLU is no larger than 1 for the entire demand set d+X_F. Interestingly, the evaluation results in Section 4 show that when such scenario occur, the online reconfiguration of R3 can automatically remove those demands that have lost reachability due to the partition of the network (by setting ξ_e(l)=0 when p_e(e)=1). Moreover, by choosing r and p that minimize the MLU over the entire demand set d+X_F, R3 is able to achieve much lower MLU than existing methods.

[0090] Necessary condition for single-link failures: A potential concern on the resilient routing is that it may be rather wasteful, as it requires the routing to maintain MLU within 1 when routing both real traffic and rerouting virtual demand up to the link capacity. However, it is actually more economical than it seems. In particular, Theorem shows that the requirement in Theorem is tight for single-link failures (i.e., F=1). Evaluation results in Section 4 will further show it is efficient under general failure scenarios.

[0091] Theorem 2 Let X₁be the rerouting virtual demand set for single-link failures, as defined in (2). If there exists base routing r and link-based protection routing p* such that for all cases of single-link failures, the MLU (due to both regular traffic and rerouted traffic) is no larger than 1 and there is no traffic loss, then there exists p such that with r and p, d+X₁can be routed without creating any congestion.

[0092] Proof: Let L(d, r, e)=Σ_a,bε∀d_abr_ab(e) be the load on link e due to real traffic d and base routing r. Then p can be explicitly constructed as follows.

[0093]

\begin{matrix} \forall e, l \in E : p_{e} (l) = {\begin{matrix} 1 - \frac{L (d, r, e)}{c_{e}}, & if l = e; \\ p_{e}^{*} (l) \cdot \frac{L (d, r, e)}{c_{e}}, & otherwise . \end{matrix} & (14) \end{matrix}

[0094] The analysis below will show that the resulted routing p together with the base routing r ensures that there is no congestion for demand set d+X₁. According to linear programming, the MLU for routing (r,p) over the entire demand set d+X₁will be reached at an extreme point of d+X₁, which corresponds to having a single x_e/c_e=1 and all the other x_l/c_l=0 (∀l≠e). It is easy to see that for ∀l≠e, the following holds:

[0095]

\begin{matrix} L (d, x, r, p, l) = L (d, r, l) + x_{e} p_{e} (l) \\ = L (d, r, l) + c_{e} \frac{L (d, r, e)}{c_{e}} p_{e}^{*} (l) \\ = L (d, r, l) + L (d, r, e) p_{e}^{*} (l) . \end{matrix}

[0096] That is, L(d, x, r, p, l) is the same as the link load on l when protection routing p*_eis used to reroute traffic traversing the failed link e, which is no larger than c_lby assumption. Meanwhile, it is easy to see that:

[0097]

\begin{matrix} L (d, x, r, p, e) = L (d, r, e) + x_{e} p_{e} (e) \\ = L (d, r, e) + c_{e} (1 - \frac{L (d, r, e)}{c_{e}}) \\ = c_{e} . \end{matrix}

Therefore, the MLU is no larger than 1 under routing (r, p) and the entire demand set d+X₁.

[0098] Order independent online reconfiguration: In case multiple link failures occur close in time, it is possible that different routers may detect these failures in different order. Theorem ensures that the online reconfiguration procedure in Section 2.2 will eventually result in the same routing as long as different routers eventually discover the same set of failed links. In other words, the order in which the failures are detected does not affect the final routing. This is useful because different routers can then apply R3 in a distributed, independent fashion without requiring any central controller to synchronize their routing states.

[0099] Theorem 3 The online reconfiguration procedure is order independent. That is, any permutation of failure sequence e₁, e₂, . . . , e_nalways result in the same routing after applying online reconfiguration for each failed link.

[0100] Proof: It suffices to show that routing after two topology changes is the same regardless of their order, because it will immediately follow by induction that that the routing after more than 2 topology change events, say e_i1, e_i2, . . . , e_in, can be converted to e₁, e₂, . . . , e_n, by swapping two topology changes at a time, and that such swapping results in the same routing due to the order independence of two topology changes.

[0101] To prove that routing after two topology changes is the same regardless of their order, let f(d, e)′ denote the routing after e₀changes, and f(d, e)″ denote the routing after e₀and e₁change. Then f(d, e₀)″=0 and f(d, e₁)″=0 since after e₀and e₁failures no traffic is routed through these two links. Therefore f(d, e) is symmetric when e=e₀or e=e₁. Next, the goal is to show that f(d, e) is symmetric when e≠e₀and e≠e₁. The update rule ensures that f(d, e)′=f(d, e)+f(d, e₀)×f(e₀, e)/(1−f (e₀, e₀)). Similarly, the following holds:

[0102]

\begin{matrix} {f (d, e)}^{″} = {f (d, e)}^{'} + {f (d, e_{1})}^{'} \times {f (e_{1}, e)}^{'} / (1 - {f (e_{1}, e_{1})}^{'}) \\ = f (d, e) + f (d, e_{0}) \times f (e_{0}, e) / (1 - f (e_{0}, e_{0})) + \\ \frac{[f (d, e_{1}) + f (d, e_{0}) \times f (e_{0}, e_{1}) / (1 - f (e_{0}, e_{0}))]}{1 - f (e_{1}, e_{1}) - f (e_{1}, e_{0}) \times f (e_{0}, e_{1}) / (1 - f (e_{0}, e_{0}))} \times \\ \frac{[f (e_{1}, e) + f (e_{1}, e_{0}) \times f (e_{0}, e) / (1 - f (e_{0}, e_{0}))]}{1 - f (e_{1}, e_{1}) - f (e_{1}, e_{0}) \times f (e_{0}, e_{1}) / (1 - f (e_{0}, e_{0}))} \end{matrix}

Let a00=1−f(e₀, e₀), a11=1−f(e₁, e₁), b01=f(e₀, e₁), and b10=f(e₁, e₀). It can be shown that:

[0103]

\begin{matrix} {f (d, e)}^{″} = f (d, e) + f (d, e_{0}) \times f (e_{0}, e) / a 00 + \\ \frac{[\begin{matrix} f (d, e_{1}) + \\ f (d, e_{0}) \times b 01 / a 00 \end{matrix}] \times [\begin{matrix} f (e_{1}, e) + \\ b 10 \times f (e_{0}, e) / a 00 \end{matrix}]}{a 11 - b 10 \times b 01 / a 00} \\ = f (d, e) + f (d, e_{0}) \times f (e_{0}, e) / a 00 + \\ \frac{[\begin{matrix} a 00 \times f (d, e_{1}) + \\ f (d, e_{0}) \times b 01 \end{matrix}] \times [f (e_{1}, e) + b 10 \times f (e_{0}, e) / a 00]}{a 00 \times a 11 - b 10 \times b 01} \\ = f (d, e) + f (d, e_{0}) \times f (e_{0}, e) / a 00 + \\ \frac{\begin{matrix} a 00 \times f (d, e_{1}) \times f (e_{1}, e) + \\ f (d, e_{0}) \times f (e_{0}, e) \times b 01 \times b 10 / a 00 \end{matrix}}{a 00 \times a 11 - b 10 \times b 01} + \\ \frac{b 10 \times f (d, e_{1}) \times f (e_{0}, e) + b 01 \times f (d, e_{0}) \times f (e_{1}, e)}{a 00 \times a 11 - b 10 \times b 01} \\ = f (d, e) + \\ \frac{\begin{matrix} a 00 \times f (d, e_{1}) \times f (e_{1}, e) + \\ f (d, e_{0}) \times f (e_{0}, e) \times b 01 \times b 10 / a 00 \end{matrix}}{a 00 \times a 11 - b 10 \times b 01} + \\ \frac{(a 00 \times a 11 - b 10 \times b 01) \times f (d, e_{0}) \times f (e_{0}, e) / a 00}{a 00 \times a 11 - b 10 \times b 01} + \\ \frac{b 10 \times f (d, e_{1}) \times f (e_{0}, e) + b 01 \times f (d, e_{0}) \times f (e_{1}, e)}{a 00 \times a 11 - b 10 \times b 01} \\ = f (d, e) + \\ \frac{a 00 \times f (d, e_{1}) \times f (e_{1}, e) - a 11 \times f (d, e_{0}) \times f (e_{0}, e)}{a 00 \times a 11 - b 10 \times b 01} + \\ \frac{b 10 \times f (d, e_{1}) \times f (e_{0}, e) + b 01 \times f (d, e_{0}) \times f (e_{1}, e)}{a 00 \times a 11 - b 10 \times b 01} \end{matrix}

Therefore f(d, e)″−f(d, e) is symmetric with respect to e₀and e₁.
2.4 R3 Extensions

[0104] Handling traffic variations: So far only a fixed traffic matrix d is considered. In practice, traffic varies over time. To accommodate such variations, according to one embodiment, a traffic engineering system may collect a set of traffic matrices {d₁, . . . , d_H} and use their convex combination to cover the space of common traffic patterns (e.g., see, Y. Zhang and Z. Ge, “Finding critical traffic matrices,” Proc. DSN '05, 2005 (Zhang et al. 2005a), Roughan et al. 2003, Wang et al. 2006). That is, the fixed traffic matrix d is replaced with the convex hull of {d₁, . . . , d_H}:
D custom character

{d|d=Σ_h=1^Ht_hd_h, Σ_h=1^Ht_h=1, t_h≧0(∀h)}

[0105] Constraint [C2] in (3) then becomes:

[0106]

\begin{matrix} \forall d \in D, \forall x \in X_{F}, \forall e \in E : \frac{\sum_{a, b \in V} d_{a b} r_{a b} (e) + \sum_{l \in E} x_{l} p_{l} (e)}{c_{e}} \leq MLU & (15) \end{matrix}

[0107] As in Section 2.1, linear programming duality can be applied to convert (15) into a set of linear constraints.

[0108] Handling realistic failure scenarios: So far R3 protection has considered arbitrary K link failures. Embodiments of the present invention provide a method to take into account of structure in realistic failure scenarios and classify failure events into the following two classes:

- [0109] Shared Risk Link Group (SRLG). A SRLG consists of a set of links that are disconnected simultaneously. For example, due to sharing of lower layer physical components (e.g., optical switch), multiple IP layer links may always fail together. Another example is the high-bandwidth composite links, in which a single member link down will cause all links in the composite link to be shut down. Let F_SRLGbe the set consisting of all SRLGs. Each element in F_SRLGconsists of a set of links.
- [0110] Maintenance Link Group (MLG). A network operator may shut down a set of links in the same maintenance operation. Let F_MLGbe the set consisting of all MLG events. Each element F_MLGconsists of a set of links.

[0111] To capture these failure characteristics, the method of embodiments introduces an indicator variable I_f, where I_f=1 if and only if the basic event set f is down. Then (5) is changed to (16), where the first constraint limits the maximum number of concurrent SRLGs, the second constraint expresses the fact that maintenance is carefully scheduled so that at most one MLG undergoes maintenance at any instance of time, and the last constraint encodes the fact that the rerouting traffic for a link is upperbounded by whether the link belongs to any SRLG or MLG. Linear programming duality can then be applied in a similar way to compute resilient routing.

[0112]

\begin{matrix} \begin{matrix} {maximize}_{x} \sum_{l \in E} p_{l} (e) x_{l} \\ subject to : \\ {\begin{matrix} \sum_{f \in ℱ_{𝒮ℛℒ𝒢}} I_{f} \leq K \\ \sum_{f \in ℱ_{ℳℒ𝒢}} I_{f} \leq 1 \\ \forall e \in E : \frac{x_{e}}{c_{e}} \leq 1 \\ \forall e \in E : \frac{x_{e}}{c_{e}} \leq \sum_{f \in ℱ_{𝒮ℛℒ𝒢} : e \in f} I_{f} + \sum_{f \in ℱ_{ℳℒ𝒢} : e \in f} I_{f} \end{matrix} \end{matrix} & (16) \end{matrix}

[0113] Supporting prioritized resilient routing: So far, R3 considers all traffic requires equal protection. Operational networks increasingly provide different levels of SLAs for different classes of traffic. For example, some traffic has more stringent SLA requirement and require tolerating more overlapping link failures. An SLA requirement can be translated into the number of overlapping link failures to tolerate. Embodiments of the present invention provide a method to extend R3 to enable prioritized resilient routing by associating traffic demand with a protection level, where protection level i indicates tolerating up to i link failures.

[0114] Let F_ibe the number of link failures that traffic with protection level i should tolerate. Let dⁱbe the total traffic demands that require protection level i or higher. Let X_Fibe the rerouting virtual demand set with up to F_ifailures. Then the goal of R3 is to find (r, p) such that for any i, the network has no congestion for the entire demand set dⁱ+X_Fi. To achieve this goal, according to one embodiment, it suffices to simply replace [C2] in (3) with (17), which can again be converted into linear constraints by applying linear programming duality.

[0115]

\begin{matrix} \forall i, \forall x^{i} \in X_{F_{i}}, \forall e \in E : \frac{\sum_{a, b \in V} d_{a b}^{i} r_{a b} (e) + \sum_{l \in E} x_{l}^{i} p_{l} (e)}{c_{e}} \leq MLU . & (17) \end{matrix}

[0116] As an example, consider a network has three classes (VPNs) of traffic, with decreasing priority levels: IPFR (IP-enabled Frame Relay) d_F, PNT (Private Network Transport) d_P, and general IP d_I. Then the algorithm computes d¹=d_F+d_P+d_I, indicating that all three classes of traffic should survive any single-link failure scenario; d²=d_F+d_P, indicating that IPFR and PNT must be protected by up to two-link failures, and d³=d_F, indicating that IPFR demand should be protected up to three links fail. This essentially means that resilient routing should carry d¹+X₁, d²+X₂, and d³+X₃, where X_idenotes the rerouting virtual demand set with up to i link failures.

[0117] Trade-off between performance and resilience: A potential concern about optimizing performance for failures is that good performance after failures may come at the expense of poor performance when there are no failures. To address this issue, R3 can bound MLU under no failures to be close to the optimal. This can be achieved by adding additional constraints, called a penalty envelop, to the previous optimization problem:
Σ_a,bεVd_abr_ab(e)≦MLU_opt×β,
where MLU_optis MLU under optimal routing and β≧1 is an operator-specified input that controls how far the normal-case performance is away from the optimal. With these constraints, the methods of embodiment not only optimize performance under failures but also ensure acceptable performance under normal conditions. β is a tunable parameter. A small β improves the normal-case performance at the cost of degrading the performance after failures by reducing the feasible solution space over which the optimization takes place.

3. R3 Linux Implementation

[0118] To evaluate the feasibility and effectiveness of R3 in real settings, a prototype of R3 is implemented in Linux (kernel version 2.6.25). This section describes the R3 implementation.

3.1 Overview

[0119] A key challenge in implementing R3 protection routing is its flow-based representation of p, because current routers do not readily support such a routing scheme.

[0120] One way to address the issue is to convert a flow-based routing to a path-based routing, e.g., using the flow decomposition technique (See H. Wang, Y. R. Yang, P. H. Liu, J. Wang, A. Gerber, and A. Greenberg, “Reliability as an interdomain service,” Proc. ACM SIGCOMM, August 2007 (Wang et al. 2007)). A path-based routing can then be implemented using MPLS. A problem of this approach is that after each failure the protection routing should be rescaled and the rescaled protection routing may decompose to new sets of paths, which have to be signaled and setup.

[0121] Due to this problem, embodiments of the present invention provide a more efficient implementation. According to the embodiments described herein, MPLS is chosen as the base mechanism since it is widely supported by all major router vendors. A flow-based routing using MPLS, called MPLS-fr, is implemented. MPLS-fr involves a simple modification to MPLS and can be easily implemented by router vendors. For wider interoperability, R3 may also be implemented using traditional MPLS, but with larger overhead.

3.2 MPLS-fr

[0122] Forwarding Data Structure: In standard MPLS, when an MPLS packet with label l arrives at a router, the router looks up the label l in a table named incoming label mapping (ILM), which may map the label to a forward (FWD) instruction. The FWD contains a next-hop label forwarding entry (NHLFE), which specifies the outgoing interface for packets with the incoming label.

[0123] MPLS-fr extends MPLS forwarding information base (FIB) data structure to allow multiple NHLFE entries in a FWD instruction. Furthermore, each NHLFE has a next-hop splitting ratio. Thus, after looking up the label of an incoming packet in ILM, the router selects one of the NHLFE entries contained in the FWD according to the splitting ratios.

[0124] Implementing Next-hop Splitting Ratios: Consider the implementation of the protection routing for link (a, b). Let l_abbe the label representing (a, b). For all traffic at router i with label l_ab, router i should split the traffic so that the fraction of traffic to neighbor j is

[0125]

\frac{P_{a b} (i, j)}{\sum_{j^{'}, (i, j^{'}) \in E, (i, j^{'}) \neq (a, b)} p_{a b} (i, j^{'})} .

[0126] One straightforward approach of implementing splitting is random splitting. However, this may cause packets of the same TCP flow to follow different routes, which will generate out-of-order packets and degrade TCP performance. To avoid unnecessary packet reordering, packets belonging to the same TCP flow should be routed consistently. This is achieved using hashing, according to one preferred embodiment. The hash function should satisfy two requirements: (i) The hash of the packets belonging to the same flow should be equal at the same router. (ii) The hash of a flow at different routers should be independent of each other (i.e., the input to the hash should include router ID in addition to flow identification fields). If the hash value is only determined by the flow, the probability distribution of the hash values might be “skewed” on some routers. For example, for flow ab, if router i only forwards the packets with hash values between 40 and 64 to router j, then router j may never see packets in flow ab with hash values less than 40.

[0127] To meet these two requirements, according to one embodiment, the implementation uses a hash function that takes as input both the flow fields in the packet header (Source IP Address, Destination IP Address, Source Port, Destination Port) and a 96-bit router-dependent private number based on router ID. The output of the hash function is a 6-bit integer.

3.3 Routing Reconfiguration Using MPLS-fr and Label Stacking

[0128] With MPLS-fr support, resilient routing reconfiguration may be implemented easily. In an implementation of the methods of embodiment, a central server performs precomputation of protection routing p, establishes a label for each protected link, signals of MPLS-fr setup, and distributes p. The central server can be integrated with Routing Control Platform (See N. Feamster, H. Balakrishnan, J. Rexford, A. Shaikh, and K. van der Merwe, “The case for separating routing from routers,” Proc. ACM SIGCOMM 2004 FDNA Workshop, September 2004 (Feamster et al. 2004)) or Path Computation Element (PCE) (See A. Farrel, J.-P. Vasseur, and J. Ash, “A Path Computation Element (PCE)-based Architecture,” RFC 4655, August 2006 (Farrel et al. 2006)). Online reconfiguration is distributed, and conducted by each router locally. It has three components: failure detection and notification, failure response, and protection routing update. Below details on each component will be given.

[0129] Failure Detection and Notification: Link failure may be detected using layer 2 interface monitoring, according to one embodiment. Upon a local failure event, a notification is generated and flooded to all other routers in the network through ICMP packets with type 42. In operational networks, failure detection and notification can be made more efficient using the deployed network management infrastructure. For example, SRLG failure can be detected by risk modeling algorithm based on network monitoring (See R. R. Kompella, J. Yates, A. Greenberg, and A. C. Snoeren, “IP fault localization via risk modeling,” Proc. NSDI, 2005 (Kompella et al. 2005)). The detection could be conservative (e.g., if any link in a SRLG down, assume all links in SRLG down). Also, the operator can issue preparation notifications to all routers before starting a MLG maintenance operation.

[0130] Failure Response: After a failure is detected, MPLS-fr for the detected failure is activated by label stacking FIG. 1 is a simple example illustrating the failure response. An IP packet of flow (S1,D1) reaches router R1. R1 looks up the packet using the base forwarding table and decides that the next-hop for this packet is R2. Normally, the packet follows the base routing and is sent to R2.

[0131] If link (R1,R2) fails, R1 activates the protection routing for (R1,R2), looks up the protection label 200 for link (R1,R2) in ILM, and pushes label 200 onto the MPLS stack of the packet. The lookup in ILM indicates that the next-hop neighbor is R4, so R1 forwards the packet to R4. When the packet reaches router R4, R4 looks up the ILM for the incoming label 200. For the protection label 200, R4 has two NHLFEs: 40% of the flows to R2, and 60% to R5. Assume that the hash of flow (S1,D1) on R4 selects R2, then R4 forwards the packet to R2. Similarly, protection traffic for flow (S2,D2) through R4 can be carried by R5. At R2, the protection label of the packet will be popped. The packet will be forwarded to R3 following the remaining base routing of OD pair (R1,R3). When the network recovers from a failure event, the base routing is immediately re-activated and the protection routing is disabled.

[0132] Protection Routing Update: After a failure, each router needs to update the protection routing (i.e., reconfiguring next-hop splitting ratios) for other protected links. To facilitate local update, each router stores p in its RIB (routing information base), according to one embodiment. The resulting storage requirement is O(|E|²). Considering backbone routers already maintain the network topology information (e.g., in Link State Database), this additional storage overhead may be acceptable.

[0133] Due to the order independence of rescaling, when multiple failures happen in a network, different routers can perform rescaling on its local copy of p. When all routers are notified of all failures, the routers will have a consistent protection routing p. During the transition process, different routers may have inconsistent p, which may lead to transit loops. If transit loops are of concern, techniques, such as failure-carry packets (FCP) (See Lakshminarayanan et al. 2007), can be integrated with R3.

4. Evaluations

[0134] This section evaluates R3 using both real experiments and extensive simulations based on realistic network topologies and traffic traces.

4.1 Evaluation Methodology

[0135] Network Topology: The simulations use the PoP-level topology of a large tier-1 ISP network, called US-ISP. In addition, the simulations also use PoP-level topologies of three IP networks, Level-3, SBC, and UUNet (2003), as inferred by RocketFuel (N. Spring, R. Mahajan, and D. Wetherall, “Rocketfuel: An ISP topology mapping engine,” Available from http://www.cs.washington.edu/research/networking/rocketfuel/). The backbones of these networks are obtained by recursively merging the leaf nodes of the topologies with their parents until no nodes have degree one. OC192 is used as the capacity for links in the RocketFuel topologies. A topology generator GT-ITM is also used to generate a large backbone topology. For experimental results, the Abilene backbone topology (2006) is created on Emulab (See B. White, J. Lepreau, L. Stoller, R. Ricci, S. Guruprasad, M. Newbold, M. Hibler, C. Barb, and A. Joglekar, “An integrated experimental environment for distributed systems and networks,” Proc. OSDI, December 2002). The link capacities are scaled down to be 100 Mbps. The link delay is configured to be measured values. Table 1 summarizes the used topologies. The data for US-ISP are not shown due to privacy concerns.

[0136]

TABLE 1

Summary of network topologies used.
	Network	Aggregation level	# Nodes	# D-Links

	Abilene	router-level	11	28
	Level-3	PoP-level	17	72
	SBC	PoP-level	19	70
	UUNet	PoP-level	47	336
	Generated	router-level	100	460
	US-ISP	PoP-level	—	—

[0137] Traffic: Real hourly traffic demand matrices are obtained from US-ISP for a one-week period. For RocketFuel topologies, the gravity model (See Y. Zhang, M. Roughan, C. Lund, and D. L. Donoho, “An information-theoretic approach to traffic matrix estimation,” Proc. ACM SIGCOMM, August 2003 (Zhang et al. 2003)) is used to generate synthetic traffic demands. To generate realistic traffic during the experiments on Emulab, the Abilene traffic matrix is extracted from measurement data and the values are scaled down. Then traffic for each OD pair is generated at the rate encoded in the traffic matrix. CAIDA Anonymized 2008 Internet traces is used for real-time IP packet generation.

[0138] Failure Scenarios: To evaluate the performance under failures, the evaluation enumerates all possible single- and two-link failures, and randomly samples around 1100 scenarios of three- and four-link failures. Random sampling is used for three- and four-link changes due to the large number of all possible such failures. This sampling is only needed for quantifying the performance under failures and not required for computing protection routing, since R3 does not require enumeration of failure scenarios. In addition, for US-ISP, real maintenance link groups (i.e., the sets of links that were under maintenance together) are obtained for a 6-month period, and each maintenance link group as is treated as a single failure event.

[0139] Performance Metrics: For simulation results, two performance metrics are used: (1) bottleneck traffic intensity, and (2) performance ratio. Bottleneck traffic intensity measures network congestion. The performance ratio of an algorithm is defined as the ratio between the bottleneck traffic intensity of the algorithm and that of optimal routing, under the same network topology and traffic demand, and measures how far the algorithm is from being optimal under the given network topology and traffic demand. It is always no less than 1, and a higher value indicates that the performance of the algorithm is farther away from the optimal. The evaluation further examines the router storage overhead and the efficiency of resilient routing reconfiguration using measurement data from Emulab experiments.

[0140] Algorithms: The following two base routing schemes are considered:

- [0141] OSPF: This is widely used in IP/MPLS networks for traffic engineering. For US-ISP, the IGP weight optimization technique in Fortz et al. 2000 is used to compute a set of optimized weights for each day during the evaluation period based on the 24 traffic demand matrices of that day.
- [0142] MPLS-fr: The base routing is computed using the algorithms in Section 2.

[0143] In some cases, the evaluation also shows the optimal bottleneck traffic intensity that can be achieved using flow-based routing. This is the optimal routing for each given network topology and traffic. It is unrealistic to implement, because it requires complete change of routing after the network topology and/or traffic changes, and could cause large disruption to network traffic. It is used only as a lower bound for evaluating the performance of other algorithms and to compute the performance ratios.

[0144] The following protection algorithms are considered:

- [0145] CSPF-detour: This algorithm is widely used in fast rerouting. The bypass routing for a set of failed links is computed using OSPF algorithm with the failed links removed. The implementation of the bypass routing is generally based on standard MPLS.
- [0146] OSPF reconvergence (recon): In this algorithm, the OSPF routing protocol is allowed to re-compute routing for every changed topology.
- [0147] Failure-Carrying Packet (FCP): This is the algorithm as described in Lakshminarayanan et al. 2007. In this algorithm, individual data packet keeps track of topology changes that have been encountered by the packet, and the packet is routed along the OSPF path in the current snapshot of topology.
- [0148] Path Splicing (PathSplice): This algorithm is proposed in Motiwala et al. 2008. The evaluation computes k=10 slices with a=0, b=3 and
  Weight(a,b,i,j)=(degree(i)+degree(j))/degree_max,
  where degree_maxis the maximal node degree of the network. When forwarding traffic, if a router detects the outgoing link for a destination unavailable, it detours the traffic destined to this destination through other connected slices using uniform splitting.
- [0149] R3: The protection routing is computed using the algorithms in Section 2.
- [0150] Flow-based optimal link detour routing (opt): This is the optimal link detour routing for each given traffic and failure scenario. Specifically, for each failure scenario f, this scheme computes an optimal protection plan (i.e., a rerouting for each link in f). Since the detour routing varies according to each failure scenario, it is challenging to implement in a scalable way. Its performance is considered as a lower bound of practical protection algorithms.
  4.2 Simulation Results

[0151] US-ISP: To preserve confidentiality of US-ISP, the absolute traffic intensity on the bottleneck link is not reported. Instead, normalized bottleneck traffic intensity is reported. Specifically, for each interval in the trace, the evaluation computes the bottleneck traffic intensity using optimal flow-based routing when there is no failure. The evaluation then normalizes the traffic intensity during different intervals by the highest bottleneck traffic intensity observed in the trace.

[0152] Single failure: Initially, only one failure event (SRLG or MLG) is introduced. At each interval, the network topology deviates from the base topology by only one failure event. The worst case performance upon all possible single failure events is identified. The normalized traffic intensity on the bottleneck link is then reported. FIG. 2 shows the results. For clarity, the figure zooms in to a one-day time frame during the evaluation period; thus, there are 24 intervals. The following observations can be made. First, R3 based protection (MPLS-fr+R3 and OSPF+R3) performs close to the optimal, and achieves performance similar to flow-based optimal link detour routing on top of OSPF (OSPF+opt). However, flow-based optimal link detour (opt) requires the computation of optimal protection routing for each individual topology-change scenario, whereas R3 achieves similar performance with only a single protection routing and a simple, light-weight routing reconfiguration. Second, comparing the two R3 schemes, MPLS-fr+R3 performs better than OSPF+R3 (see intervals 40 to 48). This is expected since OSPF is less flexible than MPLS. Third, without a good protection scheme, OSPF+recon, OSPF+CSPF-detour, and FCP all lead to higher levels of normalized traffic intensity. In the early part of the day, their traffic intensity can be as high as 3 times that of the other routing protection schemes (˜5 vs. ˜1.5). Fourth, starting from interval number 49, FCP starts to have better performance than OSPF+recon, OSPF+CSPF-detour. But its traffic intensity in the later part of the day can still be as high as 2 times (e.g., during interval number 60) that of MPLS-fr+R3, OSPF+R3 and OSPF+opt. Finally, by rerouting traffic to multiple slices in a “best effort” fashion, PathSplice leads to less congestion and achieves much better performance than other existing protection algorithms, though it is still less efficient than R3 based algorithms.

[0153] The previous evaluation shows the effectiveness of R3 during one day. The next evaluation summarizes the overall performance during the entire evaluation period (which lasts seven days). FIG. 3 shows the performance ratio versus the time interval sorted based on the performance ratio. The following observations can be made. First, MPLS-fr+R3, OSPF+R3, and OSPF+opt consistently perform within 30% of the optimal throughout the entire evaluation period. Second, OSPF+recon, OSPF+CSPF-detour, PathSplice, and FCP all cause significant performance penalty. The performance of OSPF+recon, OSPF+CSPF-detour, and FCP can be 260% higher than optimal. PathSplice performs better, but it still can be 100% higher than the optimal while R3 based schemes are within 30%. Thus, the traffic intensity of PathSplicing can be 54% higher than R3.

[0154] Multiple failure events: Next, multiple failure events are introduced in US-ISP. For clarity of presentation, the evaluation fixes the interval (a peak hour) and evaluates the failure events. The results are reported for two failures and sampled three failures. Only sampled three failures are reported because there are too many failure scenarios to enumerate; thus, random sampling is used. FIG. 4 shows the performance ratio versus the scenario sorted based on the performance ratio. To make it easier to read, the y-axis of FIG. 4 is truncated at the value of 4. It is evident that under two and three failures, MPLS-fr+R3 and OSPF+R3 continue to significantly out-perform OSPF+recon, OSPF+CSPF-detour, FCP, and PathSplice. From FIG. 4A, it can be seen that OSPF+recon, OSPF+CSPF-detour, FCP and PathSplice can cause bottleneck traffic intensity to be more than 3.7 times of the optimal for two failures. This is 94% higher than the highest of MPLS-fr+R3 and OSPF+R3 (they reach around 1.9). For three failures, OSPF+recon, OSPF+CSPF-detour, FCP, and PathSplice reach at least 2.4 times of optimal, while MPLS-fr+R3 and OSPF+R3 reach only 1.6; thus they are at least 50% higher than R3 based protection.

[0155] Summary: For US-ISP, R3 based schemes consistently achieve better performance than OSPF+recon, FCP, and PathSplice, outperforming them by at least 50% in all scenarios and much higher in some scenarios.

[0156] Rocketfuel Topologies: Next, the evaluation uses the RocketFuel topologies. For each RocketFuel topology, the evaluation randomly generates one traffic matrix using gravity model. Due to lack of SRLG information, the evaluation generates all two-link failures and randomly samples 1100 three-link failures. The evaluation then compares the performance of different algorithms under these failures.

[0157] FIG. 5 and FIG. 6 show the performance ratios for the SBC network and Level 3, respectively. These two topologies are chosen because they give representative results among the Rocketfuel topologies. From these figures, the following observations can be made. First, on SBC, MPLS-fr+R3, with its jointly optimized base routing and protection routing, significantly out-performs all OSPF based algorithms, including OSPF+opt. This demonstrates the advantage of joint optimization of base routing and protection routing. Second, on Level 3, MPLS-fr+R3 and OSPF+R3 have very similar performance, and consistently out-perform other OSPF based algorithms, except for OSPF+opt. In fact, on Level 3, OSPF+opt performs very close to optimal and slightly better than MPLS-fr+R3 and OSPF+R3. Recall that it is substantially more expensive to implement OSPF+opt, this indicates that on networks with very good OSPF routings, R3 on top of OSPF can be used to achieve most of the gains of R3 while retaining the simplicity of OSPF routing.

[0158] Prioritized R3: Prioritized R3 is evaluated using three classes of traffic with different priorities. Specifically, the evaluation extracts traffic of IPFR and PNT from the US-ISP backbone traffic in a peak interval. The evaluation then subtracts these two types of traffic from the total traffic and treats the remaining traffic as IP. The traffic volumes of the two VPNs are rescaled for confidentiality. For prioritized R3, the protection levels of IPFR, PNT, and IP are set to four failures, two failures, and one failure, respectively. For general R3, all traffic is protected against one failure. The evaluation then reports results for all single failures, 100 worst-case two-failure scenarios, and sampled four-failure scenarios.

[0159] FIG. 7 shows the normalized bottleneck traffic intensities for the three classes of traffic under R3 with and without priority. The following observations can be made. First, both prioritized and general R3 provide congestion-free rerouting under single failures. Comparing the performance between prioritized and general R3, it is evident that IP traffic has lower bottleneck traffic intensity under prioritized R3 than under general R3, while the bottleneck traffic intensities of PNT and IPFR under prioritized R3 are slightly higher than under general R3. The reason for the latter is because even though IP traffic has lower priority than PNT and IPFR under multiple failures, prioritized R3 can give IP better treatment under single failures as long as PNT and IPFR traffic are well protected, which is the case (i.e., the bottleneck traffic intensities of PNT and IPFR are always smaller than 0.4 under single failures). Second, under two-link failures, prioritized R3 guarantees congestion-free for IPFR and PNT, whereas IPFR and PNT experience congestion under general R3. The bottleneck traffic intensities of IP traffic is higher under prioritized R3 than under general R3, which is inevitable due to the limited resources. Third, under four-link failures, IPFR incurs no congestion using prioritized R3, whereas all traffic experience congestion using general R3. Even PNT, which is protected up to two-link failures, experience lower traffic intensities under prioritized R3 than under general R3. As expected, IP traffic experiences congestion under both general and prioritized R3 during four-link failures. These results demonstrate that prioritized R3 is effective in providing differentiated treatment to different traffic.

[0160] Penalty Envelope: In R3 formulation, a penalty envelope can be introduced on normal case MLU. The goal is to balance the need between being robust to topology changes and being optimal when there are no topology changes. To motivate the importance of this technique, the evaluation below examines the network performance under no topology changes. FIG. 8 shows the performance of four algorithms: R3 without penalty envelope, OSPF, R3 with penalty envelope, and optimal. The evaluation picks a time period when OSPF performs particularly well with optimized IGP weights. The following observations can be made. First, adding the penalty envelope significantly improves normal case performance. The 10% penalty envelope is effective and R3 performs within the envelope during normal operations. Second, R3 without penalty envelope can lead to significant performance penalty in normal case. Its normalized traffic intensity sometimes goes as high as 200% of the optimal and may perform even worse than OSPF. This is because R3 without penalty envelope optimizes exclusively for the performance under failures and only enforces no congestion during normal network topology and traffic.

[0161] Robustness on Base Routing: The previous evaluation shows that R3, which jointly optimizes base routing and protection routing, out-performs OSPF+R3. So a better base routing leads to better overall performance. To further understand the impact of base routing, the following evaluation is conducted. Specifically, two versions of OSPF are used as the base routing: (i) OSPFInvCap+R3 and (ii) OSPF+R3, where in the former the IGP weights of the base routing is inverse proportional to link capacity and in the latter IGP weights are optimized. As shown in FIG. 9, R3 based on OSPFInvCap is significantly worse than R3 based on an optimized OSPF routing. These results further demonstrate the importance of base routing.

4.3 Implementation Results

[0162] The evaluation below examines the efficiency of the Linux implementation of R3.

[0163] Offline Computation Complexity: To evaluate the computation complexity of R3, R3 offline precompuation is run for the 8 topologies with different failure guarantees. All the computation is done using a single Linux machine with commodity hardware configuration (2.33 GHz CPU, 4 GB memory). Table 2 summarizes the results. It shows that the precomputation phase takes no more than half an hour for major ISP backbones. It also shows that the computation time does not change significantly with an increasing number of protected link failures due to the compact representation of failure scenarios.

[0164]

TABLE 2

R3 Offline Precomputation Time (seconds).
Network/#
failures	1	2	3	4	5	6

Abilene	0.3	0.30	0.30	0.32	0.33	0.29
Level-3	1.80	1.97	2.56	2.71	2.46	2.43
SBC	1.46	1.76	1.75	1.76	1.92	1.91
UUNet	1010	572	1067	810	864	720
Generated	1388	929	1971	2001	1675	2131
US-ISP	21.3	21.9	21.4	20.1	22.1	21.8

[0165] Storage and MPLS Overhead: One concern about R3 protection implementation based on MPLS-fr is router storage overhead (i.e., FIB and RIB size), given that routers need to maintain the protection labels for all protected links and store local copies of the protection routing p. To evaluate the storage overhead, for a given topology, R3MPLS-fr protection is run assuming that all backbone links are protected except the stub links which cannot be bypassed. The evaluation measures the ILM table size, the NHLFE table size, the FIB size, and the RIB size per router. Table 3 summarizes the results for 6 topologies. It shows that all of these 6 network topologies can be protected by R3 with modest FIBs (<267 KB) and RIBs (<20 MB).

[0166]

TABLE 3

Router storage overhead of R3 implementation.
Network	# ILM	# NHLFE	FIB memory	RIB storage

Abilene	28	71	<9	KB	<83	KB
Level-3	72	304	<36	KB	<535	KB
SBC	70	257	<31	KB	<503	KB
UUNet	336	2402	<267	KB	<11	MB
Generated	460	2116	<251	KB	<20	MB
US-ISP	—	—	<39	KB	<656	KB

[0167] A related overhead is R3 usage of MPLS labels. Recall that the number of MPLS labels used by MPLS-fr for protection routing is bounded by the number of links in the network. Since many routers can support at least tens of thousands of MPLS labels, the number of MPLS labels used in protection routing may not be an issue.

[0168] Effective Resilient Routing Reconfiguration: Next, the evaluation examines the effectiveness of protection routing. The evaluation generates failure scenarios by disconnecting three links (Houston-Kansans, Chicago-Indianapolis, Sunnyvale-Denver) sequentially on the emulated Abilene topology (each link is two directed links). After failing one link, there is about one-minute delay before failing the next link. During the evaluation, bursty traffic is generated to allow measurement of the traffic throughput between every OD pair, the traffic intensity on each link, and the aggregated loss rate at each egress router (the traffic matrix encodes the expected outgoing traffic).

[0169] As shown in FIG. 10, the R3 Linux implementation successfully reroutes traffic without overloading any link. From FIG. 10B, it is clear that despite three failed links, the bottleneck traffic intensity is always within 0.37. FIG. 11 further plots the real-time RTT of a flow between Denver and Los Angeles during the test process. The figure clearly depicts the three-step increases of RTT, due to the three link failures. Clearly, the R3 protection routing implementation achieves smooth and efficient routing protection.

[0170] To appreciate the effectiveness of R3, the same failure scenario is tested using OSPF reconvergence protection. FIG. 12 compares the traffic intensity by OSPF+recon vs MPLS-fr+R3. Using OSPF, the traffic intensity on the link between Washington and Atlanta (link index 28) reaches as high as 1.07 (instantaneous rate). Due to congestion, it can be seen from the trace that the throughput for the OD pair New York City to Indianapolis drop by up to 32.6% using OSPF+recon.

5. Related Work

[0171] The work described herein builds on a body of existing work that can be classified into two categories: (i) routing under failures and (ii) routing under variable traffic.

[0172] Routing under failures: Many of recent studies focus on minimizing the duration of disruption due to failures (See, K. Kar, M. S. Kodialam, and T. V. Lakshman, “Routing restorable bandwidth guaranteed connections using maximum 2-route flows,” IEEE/ACM Transactions on Networking, 11(5):772-781, 2003 (Kar et al. 2003), M. Kodialam and T. V. Lakshman, “Dynamic routing of locally restorable bandwidth guaranteed tunnels using aggregated link usage information,” Proc. IEEE INFOCOM, April 2001 (Kodialam et al. 2001), M. Kodialam, T. V. Lakshman, and S. Sengupta, “A simple traffic independent scheme for enabling restoration oblivious routing of resilient connections,” Proc. IEEE INFOCOM, April 2004 (Kodialam et al. 2004), M. S. Kodialam and T. V. Lakshman, “Dynamic routing of restorable bandwidth-guaranteed tunnels using aggregated network resource usage information,” IEEE/ACM Transactions on Networking, 11(3):399-410, 2003 (Kodialam et al. 2003), A. Li, P. Francois, and X. Yang, “On improving the efficiency and manageability of NotVia,” Proc. CoNEXT, December 2007 (Li et al. 2007), M. Shand and S. Bryant, “IP fast reroute framework,” IETF Internet-Draft, draft-ietf-rtgwg-ipfrr-framework-06.txt, 2007 (Shand et al. 2007), Atlas et al. 2007, Lakshminarayanan et al. 2007, Motiwala et al. 2008). These techniques precompute protection and quickly reroute traffic upon detecting failures (and before routing convergence) (See V. Sharma, B. M. Crane, S. Makam, K. Owens, C. Huang, F. Hellstrand, J. Weil, L. Andersson, B. Jamoussi, B. Cain, S. Civanlar, and A. Chiu, “Framework for MPLS-Based Recovery,” RFC 3469, February 2003 (Sharma et al. 2003)). However, they do not provide performance predictability or avoid congestion. As shown in Section 4, they may lead to serious congestion and thus violation of service level agreements. Meanwhile there are also significant studies on optimizing performance under failures. Fortz and Thorup have studied optimization of OSPF/IS-IS weights under failures (See B. Fortz and M. Thorup, “Robust optimization of OSPF/IS-IS weights. In Proc. INOC, October 2003 (Fortz et al. 2003)). However, it is a heuristics based approach and does not provide performance guarantee or avoidance of congestion. In MATE (See A. Elwalid, C. Jin, S. Low, and I. Widjaja, “MATE: MPLS adaptive traffic engineering,” Proc. IEEE INFOCOM, April 2001 (Elwalid et al. 2001)) and TeXCP (See S. Kandula, D. Katabi, B. Davie, and A. Charny, “Walking the tightrope: Responsive yet stable traffic engineering,” Proc. ACM SIGCOMM, August 2005 (Kandula et al. 2005)), the authors study how to react to instantaneous traffic load and redistribute traffic on alternate links or paths. Many previous studies achieve optimal performance by re-optimizing routing after each failure (e.g., MPLS routing (See Wang et al. 2006)). A major advantage of these approaches is that the new routing is computed specifically for the new topology. Thus, the new routing can efficiently utilize the remaining network resources and provide certain guarantees (e.g., how close the rerouting response compared with the optimal (See D. Applegate, L. Breslau, and E. Cohen, “Coping with network failures: Routing strategies for optimal demand oblivious restoration,” Proc. ACM SIGMETRICS, June 2004 (Applegate et al. 2004))). A drawback of these approaches, however, is their slow response time. Re-optimization from scratch for the new topology can be computationally expensive. In addition, the new routing could be very different from the existing one and thus take substantial delay in installation and convergence. This can cause significant service disruption because of operation errors, forwarding loops and packet loss during long convergence process. As a result, network operators are highly reluctant to completely change their routing. Instead, they prefer simple routing reconfiguration. They completely re-optimize only periodically or after a major change, instead of after each topology failure. The only work that optimizes routing simultaneously for different topologies is Applegate et al. 2004, but it requires enumeration of all possible topologies after failures and faces scalability issues under multiple failures.

[0173] Routing under variable traffic demand: High variability in Internet traffic has motivated researchers to design robust traffic engineering that works well under variable traffic. One class of algorithms (See, S. Agarwal, A. Nucci, and S. Bhattacharyya, “Measuring the shared fate of IGP engineering and interdomain traffic,” Proc. ICNP, November 2005 (Agarwal et al. 2005), C. Zhang, Z. Ge, J. Kurose, Y. Liu, and D. Towsley, “Optimal routing with multiple traffic matrices: Tradeoff between average case and worst case performance,” Proc. ICNP, November 2005 (Zhang et al. 2005b), Elwalid et al. 2001, Kandula et al. 2005, Roughan et al. 2003] maintains a history of observed traffic demand matrices, and optimizes for the representative traffic demand matrices. Another class of algorithms is oblivious routing (See, L. G. Valiant, “A scheme for fast parallel communication,” SIAM Journal on Computing, 11(7):350-361, 1982 (Valiant 1982), Applegate et al. 2004, Applegate et al. 2003, Kodialam et al. 2001), which optimizes the worst-case performance over all possible traffic demands. More recently, Wang et al. 2007 further combined oblivious routing with prediction-based optimization to provide good performance under typical demands while guaranteeing the worst-case performance. These works focus on traffic variability and do not consider topology variability.

6. Conclusions

[0174] As described herein, the present invention provides a method and system for R3, the first routing scheme that provides congestion-free guarantee under a large number of failure scenarios. R3 is also flexible and can handle traffic variations, realistic failure scenarios, prioritized traffic, and the tradeoff between performance and resilience. A full implementation of R3 is carried out on Linux using MPLS-fr. Real experiments and extensive simulations using realistic network topologies and traffic clearly demonstrate the feasibility and effectiveness of the methods of embodiments.

[0175] As described above and as will be appreciated by one skilled in the art, embodiments of the present invention may be configured as a system, method or apparatus. Accordingly, embodiments of the present invention may be comprised of various means including entirely of hardware, entirely of software, or any combination of software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. Any suitable computer-readable storage medium may be utilized including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0176] Embodiments of the present invention have been described above with reference to block diagrams and flowchart illustrations of methods, apparatuses (i.e., systems) and computer program products. It will be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by various means including computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions which execute on the computer or other programmable data processing apparatus create a means for implementing the functions specified in the flowchart block or blocks.

[0177] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0178] Accordingly, blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.

[0179] Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these embodiments of the invention pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the embodiments of the invention are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe exemplary embodiments in the context of certain exemplary combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

What is claimed is:

1. A method for rerouting traffic in an Internet Protocol (IP) network, the method comprising:

creating a virtual demand for links in said IP network;

precomputing a routing for an actual demand plus said virtual demand for said links in said IP network on an original network topology prior to detecting any failures in said links in said IP network in order to minimize a maximum link utilization over said IP network after an occurrence of link failures;

detecting one or more links in said IP network that have failed; and

converting, by a processor, said precomputed routing into a routing that does not traverse said detected one or more failed links in said IP network in response to detecting said one or more failed links in said IP network, wherein said routing that does not traverse said detected one or more failed links is guaranteed to be congestion free provided that said maximum link utilization is less than one.

2. The method as recited in claim 1 further comprising:

rerouting traffic on said detected one or more failed links to said routing that does not traverse said detected one or more failed links in said IP network.

3. The method as recited in claim 2, wherein said rerouted traffic is congestion free.

4. The method as recited in claim 1 further comprising:

performing a convex combination of said virtual demand for said links in said IP network.

5. The method as recited in claim 4 further comprising:

representing said convex combination of said virtual demand for said links in said IP network as a set of linear constraints.

6. The method as recited in claim 5 further comprising:

implementing programming duality on said set of linear constraints.

7. The method as recited in claim 1, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling traffic variations and topology variations.

8. The method as recited in claim 7, wherein said precomputing performed in connection with said handling of traffic variations and topology variations comprises precomputing said routing for a convex combination of multiple actual demands plus said virtual demand for said links in said IP network.

9. The method as recited in claim 1, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling realistic failure scenarios.

10. The method as recited in claim 9, wherein said handling of realistic failure scenarios comprises adding constraints that encode knowledge that a single composite failure or a maintenance event simultaneously disables multiple links.

11. The method as recited in claim 1, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling prioritized traffic with different protection levels.

12. The method as recited in claim 11, wherein said precomputing performed in connection with said handling of prioritized traffic with different protection levels comprises precomputing said routing such that, as long as a number of link failures is less than or equal to a specified protection level associated with a traffic priority, all traffic with equal or higher priorities can be routed by said routing without having congestion.

13. The method as recited in claim 1, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with achieving tradeoff between performance under normal and failure conditions.

14. The method as recited in claim 13, wherein said achieving tradeoff between performance under normal and failure conditions comprises bounding a ratio between a performance under a derived protection routing and a performance under an optimal routing under failures.

15. A non-transitory computer readable storage medium storing a computer program product for rerouting traffic in an Internet Protocol (IP) network, the computer program product comprising the programming instructions for:

creating a virtual demand for links in said IP network;

detecting one or more links in said IP network that have failed; and

converting said precomputed routing into a routing that does not traverse said detected one or more failed links in said IP network in response to detecting said one or more failed links in said IP network, wherein said routing that does not traverse said detected one or more failed links is guaranteed to be congestion free provided that said maximum link utilization is less than one.

16. The computer program product as recited in claim 15 further comprising the programming instructions for:

rerouting traffic on said detected one or more failed links to said routing that does not traverse said detected one or more failed links in said IP network.

17. The computer program product as recited in claim 16, wherein said rerouted traffic is congestion free.

18. The computer program product as recited in claim 15 further comprising the programming instructions for:

performing a convex combination of said virtual demand for said links in said IP network.

19. The computer program product as recited in claim 18 further comprising the programming instructions for:

representing said convex combination of said virtual demand for said links in said IP network as a set of linear constraints.

20. The computer program product as recited in claim 19 further comprising the programming instructions for:

implementing programming duality on said set of linear constraints.

21. The computer program product as recited in claim 15, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling traffic variations and topology variations.

22. The computer program product as recited in claim 21, wherein said precomputing performed in connection with said handling of traffic variations and topology variations comprises precomputing said routing for a convex combination of multiple actual demands plus said virtual demand for said links in said IP network.

23. The computer program product as recited in claim 15, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling realistic failure scenarios.

24. The computer program product as recited in claim 23, wherein said handling of realistic failure scenarios comprises adding constraints that encode knowledge that a single composite failure or a maintenance event simultaneously disables multiple links.

25. The computer program product as recited in claim 15, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling prioritized traffic with different protection levels.

26. The computer program product as recited in claim 25, wherein said precomputing performed in connection with said handling of prioritized traffic with different protection levels comprises precomputing said routing such that, as long as a number of link failures is less than or equal to a specified protection level associated with a traffic priority, all traffic with equal or higher priorities can be routed by said routing without having congestion.

27. The computer program product as recited in claim 15, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with achieving tradeoff between performance under normal and failure conditions.

28. The computer program product as recited in claim 27, wherein said achieving tradeoff between performance under normal and failure conditions comprises bounding a ratio between a performance under a derived protection routing and a performance under an optimal routing under failures.

29. A system, comprising:

a memory unit for storing a computer program for rerouting traffic in an Internet Protocol (IP) network; and

a processor coupled to said memory unit, wherein said processor, responsive to said computer program, comprises:

circuitry for creating a virtual demand for links in said IP network;

circuitry for precomputing a routing for an actual demand plus said virtual demand for said links in said IP network on an original network topology prior to detecting any failures in said links in said IP network in order to minimize a maximum link utilization over said IP network after an occurrence of link failures;

circuitry for detecting one or more links in said IP network that have failed; and

circuitry for converting said precomputed routing into a routing that does not traverse said detected one or more failed links in said IP network in response to detecting said one or more failed links in said IP network, wherein said routing that does not traverse said detected one or more failed links is guaranteed to be congestion free provided that said maximum link utilization is less than one.

30. The system as recited in claim 29, wherein said processor further comprises:

circuitry for rerouting traffic on said detected one or more failed links to said routing that does not traverse said detected one or more failed links in said IP network.

31. The system as recited in claim 30, wherein said rerouted traffic is congestion free.

32. The system as recited in claim 29, wherein said processor further comprises:

circuitry for performing a convex combination of said virtual demand for said links in said IP network.

33. The system as recited in claim 32, wherein said processor further comprises:

circuitry for representing said convex combination of said virtual demand for said links in said IP network as a set of linear constraints.

34. The system as recited in claim 33, wherein said processor further comprises:

circuitry for implementing programming duality on said set of linear constraints.

35. The system as recited in claim 29, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling traffic variations and topology variations.

36. The system as recited in claim 35, wherein said precomputing performed in connection with said handling of traffic variations and topology variations comprises precomputing said routing for a convex combination of multiple actual demands plus said virtual demand for said links in said IP network.

37. The system as recited in claim 29, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling realistic failure scenarios.

38. The system as recited in claim 37, wherein said handling of realistic failure scenarios comprises adding constraints that encode knowledge that a single composite failure or a maintenance event simultaneously disables multiple links.

39. The system as recited in claim 29, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with handling prioritized traffic with different protection levels.

40. The system as recited in claim 39, wherein said precomputing performed in connection with said handling of prioritized traffic with different protection levels comprises precomputing said routing such that, as long as a number of link failures is less than or equal to a specified protection level associated with a traffic priority, all traffic with equal or higher priorities can be routed by said routing without having congestion.

41. The system as recited in claim 29, wherein said precomputing, said detecting one or more links in said IP network that have failed and said converting are performed in connection with achieving tradeoff between performance under normal and failure conditions.

42. The system as recited in claim 41, wherein said achieving tradeoff between performance under normal and failure conditions comprises bounding a ratio between a performance under a derived protection routing and a performance under an optimal routing under failures.

Patent Citations (7)

Patent	Date	Inventor	Cited By
US7388842(B1)	2008-06-01	Applegate et al.	Applicant
US2002/0133756(A1)	2002-09-01	Jain	Examiner
US2002/0163682(A1)	2002-11-01	Su et al.	Examiner
US2004/0196787(A1)	2004-10-01	Wang et al.	Examiner
US2005/0220026(A1)	2005-10-01	Dziong et al.	Applicant
US2006/0215666(A1)	2006-09-01	Shepherd et al.	Applicant
US2010/0124930(A1)	2010-05-01	Andrews et al.	Examiner

Non-Patent Literature (25)

D. Applegate, L. Breslau, and E. Cohen. Coping with network failures: Routing strategies for optimal demand oblivious restoration. In Proceedings of ACM SIGMETRICS, Jun. 2004.Applicant
D. Applegate and E. Cohen. Making intra-domain routing robust to changing and uncertain traffic demands: Understanding fundamental tradeoffs. In Proceedings of ACM SIGCOMM, Aug. 2003.Applicant
A. Atlas and Z. Zinin. Basic specification for IP Fast-Reroute: loop-free alternates. IETF Internet-Draft, draft-ietf-rtgwg-ipfrr-spec-base-10.txt, 2007.Applicant
A. Elwalid, C. Jin, S. Low, and I. Widjaja. MATE: MPLS adaptive traffic engineering. In Proceedings of IEEE INFOCOM, Apr. 2001.Applicant
A. Farrel, J. P. Vasseur, and J. Ash. A Path Computation Element (PCE)-based Architecture, RFC 4655, Aug. 2006.Applicant
C. Zhang, Z. Ge, J. Kurose, Y. Liu, and D. Towsley. Optimal routing with multiple traffic matrices: Tradeoff between average case and worst case performance. In Proceedings of ICNP, Nov. 2005.Applicant
B. Fortz, J. Rexford, and M. Thorup. Traffic engineering with traditional IP routing protocols. IEEE Communication Magazine, Oct. 2002.Applicant
B. Fortz and M. Thorup. Internet traffic engineering by optimizing OSPF weights. In Proceedings of IEEE INFOCOM, Mar. 2000.Applicant
B. Fortz and M. Thorup. Robust optimization of OSPF/IS-IS weights. In Proceedings of INOC, Oct. 2003.Applicant
P. Francois, C. Fillsfills, J. Evans, and O. Bonaventure. Achieving sub-second IGP convergence in large IP networks. ACM Computer Communication Review, 35(3):35-44, 2005.Applicant
G. Iannaccone, C. Chuah, S. Bhattacharyya, and C. Diot. Feasibility of IP restoration in a tier-1 backbone. IEEE Network Magazine, 18(2):13-19, 2004.Applicant
S. Iyer, S. Bhattacharyya, N. Taft, and C. Diot. An approach to alleviate link overload as observed on an IP backbone. In Proceedings of IEEE INFOCOM, Apr. 2003.Applicant
S. Kandula, D. Katabi, B. Davie, and A. Charny. Walking the tightrope: Responsive yet stable traffic engineering. In Proceedings of ACM SIGCOMM, Aug. 2005.Applicant
K. Kar, M. S. Kodialam, and T. V. Lakshman. Routing restorable bandwidth guaranteed connections using maximum 2-route flows. IEEE/ACM Transactions on Networking, 11(5):772-781, 2003.Applicant
M. Kodialam and T. V. Lakshman. Dynamic routing of locally restorable bandwidth guaranteed tunnels using aggregated link usage information. In Proceedings of IEEE INFOCOM, Apr. 2001.Applicant
M. Kodialam, T. V. Lakshman, and S. Sengupta. Efficient and robust routing of highly variable traffic. In Proceedings of HotNets-III, Nov. 2004.Applicant
M. Kodialam, T. V. Lakshman, and S. Sengupta. A simple traffic independent scheme for enabling restoration oblivious routing of resilient connections. In Proceedings of IEEE INFOCOM, Apr. 2004.Applicant
M. Kodialam, T. V. Lakshman. Dynamic routing of restorable bandwidth-guaranteed tunnels using aggregated network resource usage information. IEEE/ACM Transactions on Networking, 11(3):399-410, 2003.Applicant
K. Lakshminarayanan, M. Caesar, M. Rangan, T. Anderson, S. Shenker, and I. Stoica. Achieving convergence-free routing using failure-carrying packets. In Proceedings of ACM SIGCOMM, Aug. 2007.Applicant
M. Motiwala, M. Elmore, N. Feamster, and S. Vempala. Path splicing. In Proceedings of ACM SIGCOMM, 2008.Applicant
M. Roughan, M. Thorup, and Y. Zhang. Traffic engineering with estimated traffic matrices. In Proceedings of IMC, Oct. 2003.Applicant
M. Shand and S. Bryant. IP fast reroute framework. IETF Internet-Draft, draft-ietf-rtgwg-ipfrr-framework-06.txt, 2007.Applicant
V. Sharma, B. M. Crane, S. Makam, K. Owens, C. Huang, F. Hellstrand, J. Weil, L. Andersson, B. Jamoussi, B. Cain, S. Civanlar, and A. Chiu. Framework for MPLS-Based Recovery. RFC 3469, Feb. 2003.Applicant
H. Wang, H. Xie, L. Qiu, Y. R. Yang, Y. Zhang, and A. Greenberg. COPE: Traffic engineering in dynamic networks. In Proceedings of ACM SIGCOMM, Aug. 2006.Applicant
International Preliminary Report on Patentability for International Application No. PCT/US2010/036141 dated Nov. 29, 2011.Applicant

Abstract