System, apparatus and method for providing aggregated network connections

[0001] The present invention relates generally to communications and, in particular, to aggregating dissimilar communications links such as Cable, DSL, T1, or Metro Ethernet.

[0002] While the capacity of network connections has increased since the introduction of dial up, high speed connectivity is not ubiquitous in all regions. Even with the addition of various connection options such as DSL (Digital Subscriber Line) and their variants (VDSL, etc) as well as Cable based broadband, the majority of consumers are unable to use the theoretical maximum throughput. The inability to achieve theoretical maximum throughput from a circuit is a result of the inversely proportional relationship between throughput and distance over various types of network connections such as Cable, DSL.

[0003] Network connection aggregation is a method by which multiple network connections are grouped to provide a single logical aggregated network connection. This provides higher throughput for network applications with the added benefit of high availability/redundancy. Each network connection in the aggregate provides a percentage of the bandwidth to the logical network communication link independent of the characteristics (latency, bandwidth capacity, provider and provider architecture) of the underlying network communication link.

[0004] Although prior ITU recommendations G.998.1 (ATM-based multi-pair bonding) G.998.2 (Ethernet-based multi-pair bonding) and G.998.3 (Multi-pair bonding using time-division inverse multiplexing) speak of various bonding techniques applying to DSL links, there is generally no discussion regarding an architectural approach to aggregating dissimilar network connections to provide a single (logical) aggregated network connection. There are no provisions made for independence from physical and logical characteristics of the aggregated network connections or for high availability and lossless network failure within the aggregated connection.

[0005] U.S. Pat. No. 6,621,859 to Bell (“Bell”) teaches the idea of combining different network connections to provide bi-directional data communication technology. The combined cable and DSL modem disclosed by Bell employs unidirectional data transmission over the existing cable plant for downstream data communication, and unidirectional DSL data transmission over the existing wire pairs employed by the telecommunications network in a single data communications loop. Bell requires private network connections between an internet service provider and a client side cable and DSL provider. Bell also does not teach splitting sessions and distributing the data packets (i.e. aggregating) over different network connections.

[0006] U.S. patent application 20040213386 to Gupta (“Gupta”) discloses a communications device, method and service that integrate broadband and dial-up internet access. A modem is provided that combines both broadband and dial-up connectivity in a single device. The modem includes an auto-sense procedure for switching between the two types of connections. Gupta, however, serves a single connection at any given time. Gupta does not teach splitting of sessions into data packets and distribution of data packets over different, aggregated, network connections.

[0007] U.S. patent application 20080080371 to Liu (“Liu”) teaches a macro-spatial network multiplexing. Liu discloses dividing data from a particular application to be transmitted over different network connections. A communication module can transmit signals to/from a plurality of base stations. In this way a plurality of network connections may be leveraged for the client. Liu teaches that packets formed from the data are transmitted via a first network connection and a second network connection such that the packets transmitted via the first network connection do not match the packets transmitted over the second network connection, thereby improving bandwidth available to the client. Liu also teaches that a determination is made as to which of a plurality of network connection devices are to be used to transmit the data based on one or more considerations. For example the determination may be made based on a quality of service (QoS) specified for a particular data, available bandwidth of network connection, cost, etc. The packets include headers that identify a particular order which may be used to reorder data included in the packets after transmittal. However, Liu does not disclose bonding having bandwidth aggregation with diversity. The apparatus of Liu is restricted to directing traffic among networks and allowing traffic to be distributed amongst different networks.

[0008] U.S. patent application 20060176900 to Liu (“Liu 2”) discloses a de-centralized architecture for bonding communication links, such as ATM-based bonding of DSL lines such as Asymmetric DSL (ADSL), Very high speed DSL (VDSL), or more generally any DSL (XDSL) lines. Liu 2, however, does not teach aggregating cable, ADSL, T1, wireless or dialup.

[0009] U.S. patent application 20080075111 to Feng (“Feng”) discloses a bonding device to allow transmission of a single or unified data stream over two or more different telecommunication links. The two or more telecommunication links are DSL. Feng does not teach aggregating cable and DSL.

[0010] U.S. Pat. No. 5,809,070 to Krishnan (“Krishnan”) teaches methods and apparatus for achieving high data transmission rates using multiple, simultaneous, parallel connections. Two or more modems are coupled to a host computer. An operating system component controls the modems either as independent communications channels, or for high speed data transmissions, and “combines,” or “bonds” multiple Public Switched Telephone Network-type connections together to create a high speed data link. This is accomplished by splitting a high speed data stream into multiple lower speed data streams and transmitting each lower speed data stream via a low speed modem and PSTN connection. Krishnan does not disclose a method to enable aggregating network connections with dissimilar characteristics and does not disclose a method for aggregating connections using an always-on technique.

[0011] Therefore, there remains a need for a connection aggregation technology that provides enhanced high availability mechanisms coupled with high throughput while maintaining independence and transparency to the underlying network connection characteristics whether logical, such as latency, transport protocol or physical, such as provider architecture, transport mechanism, etc.

[0012] In one aspect of the present invention, an apparatus is provided for managing the transfer of communication traffic over one or more diverse network connections is provided, the apparatus comprising a network aggregation means operable to configure the one or more diverse network connections as an aggregated group defining an aggregated connection having a transfer rate defined by an aggregate group transfer rate associated with the available transfer rates of the one or more diverse network connections, the network aggregation means being further operable to transfer communication traffic between a further connection and the aggregated connection.

[0013] In another aspect of the present invention, a method is provided for managing the transfer of communication traffic over one or more diverse network connections is provided, the method comprising the steps of: (a) by means of a network aggregation device, configuring the one or more diverse network connections as an aggregated group thereby defining an aggregated connection having a transfer rate defined by an aggregate group transfer rate associated with the available transfer rates of the one or more diverse network connections; and (b) transferring communication traffic between a further network connection and the aggregated connection.

[0014] In a still other aspect of the invention a system is provided, the system comprising: one or more components configured to manage the transfer of communication traffic over one or more diverse network connections, the one or more components defining, including or being linked to a network aggregation means operable to configure the one or more diverse network connections as an aggregated group defining an aggregated connection having a transfer rate defined by an aggregate group transfer rate associated with the available transfer rates of the one or more diverse network connections, the network aggregation means being further operable to transfer communication traffic between a further connection and the aggregated connection.

[0020] In one aspect, the invention provides a system and architecture for aggregating multiple network access connections from similar or diverse carriers to create a new aggregated connection that accommodates greater speed and high availability characteristics.

[0021] The architecture of the present invention is best understood as a centralized architecture for aggregating network connections, broadband or otherwise. Diverse network connections are aggregated into a virtual (logical) connection that provides higher throughput as well as independence of the network characteristics of the constituent (physical) network. Aggregation may be performed to a given Customer Premises Equipment terminal.

[0022] For instance, in one example of the implementation of the present invention a Metro Ethernet 10 Mbps (E10) link and a T1 (DS1) link are aggregated in accordance with the invention as described below, in order to provide higher fault tolerance and improved access speeds. The aggregation of diverse carriers in accordance with the present invention extends to any broadband network connection including Digital Subscriber Line (DSL) communications links, Data over Cable Service Interface Specification (DOCSIS), Integrated Services Digital Network, Multi protocol Label Switching, Asynchronous Transfer Mode (ATM), and Ethernet, etc.

[0023] According to one aspect of the invention, an apparatus is provided for managing transfer of communication traffic over diverse network connections aggregated into a single autonomous connection, independent of the various underlying network connections. The apparatus may include a network aggregation device and an aggregation engine. The network aggregation device may be adapted to configure a plurality of network connections, which transfers communication traffic between a further network connection and the plurality of network connections, as an aggregated group for providing a transfer rate on the further communication link, and to allocate to the aggregate group a rate of transfer equal to the total available transfer rate of the underlying networks. The aggregation engine may be adapted to manage the distribution of communication traffic received both to and from a plurality of network connections, establishing newly formed aggregated network connections. The aggregation engine may be implemented in software for execution by a processor or in hardware, in a manner that is known to those skilled in the art.

[0024] In accordance with this aspect of the present invention, a plurality of diverse network connections may be aggregated to create an aggregated network connection. The diversity of the network connections may be a result of diversity in provider networks due to the usage of different equipment vendors, network architectures/topologies, internal routing protocols, transmission media and even routing policies. These diversities may lead to different network connections with different latencies and/or jitter on the network connection. Also, variation within transmission paths in a single provider network may lead to latency and/or jitter variations within a network connection.

[0025] Latency and jitter typically affect all data communication across the network connection. Latency, as is known to those skilled in the art, is the round-trip time for a transmission occurring end-to-end on a network connection. Jitter, as is known to those skilled in the art, is the variance in latency on a network connection for the same data flow. High latency and jitter typically have a direct and significant impact on application performance and bandwidth. Applications such as VOIP, and video delivery are typically highly sensitive to jitter and latency increases and can degrade as they increase.

[0026] Transparent aggregation of a plurality of network connections in an aggregated network connection requires the management of data transmitted over the aggregated connection by the aggregation engine and received from the aggregation traffic termination engine. In one aspect of the present invention, transparent aggregation does not require any configuration by a network provider. The aggregation engine and the aggregation traffic termination engine may manage data transmission such that the variable path speeds and latencies on the plurality of network connections do not affect the application data transmitted over the aggregated network connection. The network aggregation engine and the aggregation traffic termination engine may handle sequencing and segmentation of the data transmitted through the aggregated connection to transparently deliver application data through the aggregated connection with minimal possible delay while ensuring the ordered delivery of application data.

[0027] In one aspect of the invention, the network aggregation engine provides a newly aggregated network connection with a capacity equal to the sum of the configured maximum throughput of the network connections.

[0028] The aggregation engine and an aggregation traffic termination engine (further explained below) handle the segmentation of packets as required in confirmation with architectural specifications such as Maximum Segment Size (MSS) and Maximum Transmission Unit of the underlying network connections. The network aggregation device is operable to handle assignment of sequence identifiers to packets transmitted through the aggregated network connection for the purpose of maintaining the ordering of transmitted data units over the aggregated network connection.

[0029] In a further aspect of the invention, the network connection device includes or is linked to a connection termination device, and a plurality of fixed or hot swappable transceivers for transmitting communication traffic on respective sets of network connections, for the purpose of configuring a plurality of network connections as an aggregated connection or the management of multiple aggregated network connections and providing access to the aggregated network connection for any network communications traversing the device.

[0030] In the present specification, routing protocols or route selection mechanisms described are intended only to provide an example but not to limit the scope of the invention in any manner.

[0031] FIG. 1 is a block diagram of a communication device incorporating a particular embodiment of the invention, demonstrating the device acting as a client.

[0032] As shown in FIG. 1, the network element/network aggregation device (also referred to in this disclosure simply as the “device” or the “network aggregation device”) 23 includes (in this particular embodiment shown for illustration) a network connection termination module 25 that includes representative transceiver interfaces 14, 15 and 16. Each transceiver interface 14, 15 and 16 represents an interface to a physical communication medium through which communications may be established to network connections.

[0033] A possible implementation of the network aggregation device may use a single or multiple chassis with slots for multiple network connection termination modules and multiple network aggregation engine modules. The multiple network connection termination modules may be grouped by protocol specific or medium specific transceiver/interfaces.

[0034] The network aggregation engine 11 may handle the configuration of the network aggregation device and all related interactions with external inputs. A device configuration store 24 may provide persistent data storage for device configuration information such as a network aggregation policy.

[0035] The network aggregation engine 11 may handle queries from external sources, such as configuration parameters a network management protocol such as Simple Network Management Protocol, for example. The interface 10 may be a protocol agent and may provide for communication with a Network Management System (NMS) or operator system for configuration of the aggregation engine by the definition of an aggregation policy. Control and management information may be transferred between the network aggregation device 23 and the NMS or operator system through the interface 10 via any available or specifically designated network connection 19, 20, 21 and 17 through any transceiver interface 14, 15 and 16.

[0036] In accordance with an aspect of the present invention, multiple network connections may be combined to form an aggregated network connection 22, as disclosed in further detail herein. Each individual network connection may be configured with a maximum communication traffic rate, which could be expressed as a bit rate in bits per second.

[0037] The network aggregation engine 11 may be implemented in software for execution by a processor in the network aggregation device 23, or in hardware such as by means of a Field Programmable Gate Array (FPGA) or other integrated circuit, or some combination thereof. The network aggregation engine 11 may be implemented in a distributed manner by distributing aggregation engine intelligence to the network connection termination module 25, in a manner that is known.

[0038] The network aggregation engine 11 may receive traffic from client network connection device 18 through a network connection 17 provided through a transceiver interface 16. The client network connection device 18 may be any device including, without limitation, a router, switch, or media converter that is capable of providing termination for a single or multiple client nodes, where nodes are any devices capable of connecting to a network irrespective of protocol or interface specificity. In various embodiments, traffic may be received over multiple network connections through a single or multiple transceiver interfaces. The network aggregation engine 11 may accept all traffic from the client network connection, may provide encapsulation and segmentation services for the traffic for transmission through the aggregated network connection 22, and may transmit it over any of the network connections 19, 20 and 21 through any of the transceiver interfaces 14, 15 and 16. The network aggregation engine 11 may handle segmentation in a manner that avoids the fragmentation of aggregated communication traffic received through the client network connection device 18, when transmission occurs over the aggregated network connection 22 through any of the network connections 19, 20 and 21, by ensuring that the length of a packet/frame transmitted over any of the network connections 19, 20 and 21 is less than or equal to the configured or detected frame length for the respective connections in the aggregated network connection 22.

[0039] The network aggregation engine 11 may poll the state of network connections 19, 20 and 21, for example as per configured intervals stored in the device configuration store 24, to ensure that all network connections configured in an aggregated group are within configured acceptable tolerances. If a network connection 19, 20, and 21 exceeds acceptable tolerance values for any of the polled parameters, the network aggregation engine 11 may remove the network connection 19, 20, and 21 from within the aggregated network connection 22 without removing it from the polled network connections list. By leaving the removed network connection 19, 20, and 21 in the polled network connection list, the network aggregation engine 11 may aggregate the network connection into the aggregated network connection 22 once it has come back within acceptable tolerance values. This may ensure that a network connection may change states between residing in an aggregated network connection 22 or not, without the intervention of an external system or input. The network aggregation engine 11 may handle notifications to all end points configured within the device configuration store 24 with internal events such as changes in network connection state, threshold violations on configured thresholds for any number of configurable variables for any object within or connected to the network aggregation device 23. The network aggregation engine 12 may also handle events such as changes in the state of a network connection 19, 20, and 21 included in the aggregated connection, changes in latency of a network connection included in the aggregated network connection 22, scheduling changes, event logging, and other events.

[0040] FIG. 2 is a block diagram of a communication device incorporating a particular embodiment of the invention, demonstrating the device acting as a server/concentrator.

[0041] The network aggregation engine 11 may provide access to a network aggregation policy database 36 which stores configuration information related to the various aggregated network connections that terminate on the aggregated network connection device 28. The network aggregation termination device 28 may be implemented in such a manner that each aggregated network connection defined in the network aggregation policy database 36 is handled by its own virtual instance, the use of which enables termination of each aggregated network connection from multiple customer premises equipment (CPE).

[0042] FIG. 3 is a block diagram of a communication network incorporating a particular embodiment of the invention, demonstrating the function of the device acting as a client/CPE and server/concentrator.

[0043] In accordance with a particular embodiment of the invention, aggregated network connections 70, 71 and 72 may be built by network aggregation devices 63, 64 and 65, which terminate to a single aggregated network connection termination device 61 through network connections 66 and 68 as their endpoint. The aggregated network connection termination device 61 may access external communications networks through network connections 66 and 68 to access external/remote network resource 69. Access to external communications networks may be provided by the aggregated network connection termination device 61 by using either network connection 66 or 68 through the use of a routing protocol, such as Border Gateway Protocol (BGP), Open Shortest Path (OPS), or through the use of simpler mechanisms such as load sharing over multiple static routes within the communication network 74 that acts as the valid next-hop for the aggregated network connection termination device 61.

[0044] Aggregated network connections 70, 71 and 72 may provide access to client network nodes 67 connected to the network aggregation devices 63, 64 and 65 through the aggregated network connections 70, 71 and 72 to communications networks 74 accessible by the aggregated network connection termination device 61.

[0045] A client network node 67 may request data provided by an external/remote network resource 69 accessible through a communication network 74. This request for the external/remote network resource may be routed over the network connection 73 providing access from the client network node 67 over the aggregated network connection 70 to its end point which is the aggregated network connection termination device 61. This may be done through the communication network 74 through the network connection 66 into the aggregated network connection termination device 61. Any data sent by the external/remote network resource 69 may be routed back through the aggregated network connection termination device.

[0046] A particular embodiment of this invention may use the Internet as the communication network 74 referenced in FIG. 3. However, the communication network 74 may alternatively be built by multiple sub-networks created through the use of multiple network aggregation devices 63, 64 and 65 with aggregated network connection termination device 61 end points through multiple network connections 66 and 68.

[0047] A further aspect of the invention relates to the provisions of high availability over the aggregated network connection by the network aggregation engine 11. FIG. 4 illustrates a method of providing redundancy and increased throughput through a plurality of network connections in an aggregated network connection. The method 90 may begin with a step of configuring a plurality of network connections 91 through the creation of a network aggregation policy to form 92 the aggregated network connection. The aggregated network connection may be initialized as per the network aggregation policy. Control connections may be created 93 for the plurality of network connections configured as part of the aggregated connection to allow the aggregation engine 11 to manage the membership of a network connection within the aggregated connection. The network aggregation engine 11 may accept packets for transmission 94 over the aggregated network connection 22. The network aggregation engine 11 may choose a network connection 95 among the group of network connections configured 91 in the aggregate in the stored aggregation policy for transmission of the current packet being transmitted. The choice of network connection for transmission of the current packet may be specified within the aggregation policy and may take into account data provided by the control connection built at 94.

[0048] According to one embodiment of the invention, a non-responsive network connection may be easily detected when using latency and packet loss as a measure. The mechanism for detecting 96 and adapting to 97 the network connection change within an aggregated network connection may be implemented within the data transmission routine in the aggregation engine 11 or as a separate process in parallel to the transmission routine in the aggregation engine 11 to allow for further flexibility in provisions redundancy within the aggregated network connection.

[0049] Since this may occur on a per packet basis as opposed to on a per stream basis, a single non-responsive network connection may not affect the aggregated network connection and may allow data transmission to continue regardless of the individual states of network connections so long as a single network connection within the aggregated network connection is available for data transmission.