Method and apparatus for allowing proprietary forwarding elements to interoperate with standard control elements in an open architecture for network devices

[0001] 1. Field of the Invention

[0002] The present invention relates to the field of computer systems, and more particularly, to a method and apparatus for allowing proprietary forwarding elements, such as special purpose networking hardware, to interoperate with standard control elements, generally implemented in software, in an open network architecture.

[0003] 2. Background Information

[0004] In recent years, a trend has emerged in the networking hardware industry. Devices such as routers and switches have begun to evolve from monolithic, highly customized and integrated designs into aggregations of discrete, modularized components. The modularization of networking device architecture has begun to enable faster cycles of innovation and development in networking hardware and software by breaking the tight integration dependencies between the various components that make up a complex device, such as a router or switch. More recently, a further trend has emerged in the appearance of programmable ASICs (application-specific integrated circuits) or network processors. These devices allow for off-loading of application-layer or network-level packet processing from general purpose processors or servers to switches that include application-aware packet classification and processing capabilities. For example, a switch may not only perform routing lookup and forwarding in the hardware, but may also be able to perform functions of an application-level proxy and network or transport layer address translation. In order to take advantage of, and accelerate these trends, the use of a horizontal open networking architecture (see, for example, FIG. 2) seeks to standardize a set of APIs (application program interfaces) and protocols for separating the functionality typically associated with packet forwarding. Such functionality is usually implemented in special purpose hardware, which may be referred to as “forwarding elements”. The functionality associated with network signaling and control, typically implemented in software on general purpose processing architectures, may be referred to as “control elements”. The separation and standardization of networking device functionality, along with using control element and forwarding element components advocated by the horizontal open networking architecture, allows proprietary designs to continue to emerge while retaining the characteristics of openness, and further enabling innovation and rapid product development.

[0005] The separation of control and forwarding elements requires that a set of interfaces be developed for each component type. These interfaces expose the functionality of components to each other, allowing them to be integrated together in a working whole. For example, the functionality of application-aware switching capability in forwarding elements must be exposed to the control element via an open interface so that the control element can “program” the switch to off-load packet processing or forwarding functions in the data path. One way to expose such functionality is to define an abstract interface that describes switch capabilities in a generic manner. The control element can use such an abstract interface to configure or manipulate the switching functions using abstract commands/operations to achieve the desired effect, and the forwarding element must translate these abstract commands into hardware-specific tasks. The hardware-specific capability thus abstracted can cover a wide range of hardware capabilities, including ASIC-specific formatting of forwarding tables, packet queuing parameters, and per-packet processing actions such as encryption, marking, or address translation.

[0006] Exposure of the capabilities of the forwarding and control elements via interfaces has certain drawbacks. In particular, interface specifications typically reflect underlying implementations. This is generally true in the case of forwarding elements, which tend to be cost-sensitive, highly task-specific devices. In such devices, reflecting the underlying functionality in a highly abstract manner requires extra processing power (and thus cost) on the part of the forwarding element. The cost-sensitivity of these devices tends to discourage the use of abstract interfaces, instead leading to highly specific interfaces that closely match the actual implementation of each forwarding element. When building proprietary systems, where each component may be designed and built by the same manufacturer, this exposure is acceptable, because confidential information about the underlying implementation is kept within the company. However, when components from different vendors are integrated, particularly when using an open architecture, this exposure becomes problematic in that it can make visible information about the forwarding element architecture and capabilities that would not otherwise be available.

[0007] In an embodiment of the present invention, a computer system allows proprietary forwarding elements to interoperate with standard control elements in an open network architecture. The system includes a forwarding element that is adapted to perform data forwarding functions in a computer network. A control element is adapted to perform network signaling and control functions in the computer network. The control element is adapted to generate a standardized data set for configuring the forwarding element. An interconnecting element operatively connects the forwarding element to the control element. A forwarding element plugin is integrated with the control element for receiving the standardized data set from the control element, translating the standardized data set into a specialized data set, and transmitting the specialized data set to the forwarding element to configure the forwarding element. The specialized data set is utilized to configure the forwarding element for performing data forwarding in the computer network.

[0012] FIG. 1 illustrates an example of a vertical proprietary networking architecture of a prior art networking device. In the networking box level hardware 100 example provided in FIG. 1, an equipment vendor typically uses a proprietary network operating system (NOS) 120 with its proprietary interfaces at the bottom of the networking box level hardware 100 (interfaces to underlying ASICs (application-specific integrated circuits), switch fabric, or bus 110) and the top of the networking box level hardware 100 (interfaces that expose hardware state and functionality so that network services and applications can be written 130). Therefore, third party independent software vendors (ISVs) or independent hardware vendors (IHVs) cannot provide value-added software or hardware solutions without first having access to these private interfaces. For example, an independent software vendor may have a “best-of-the-breed” firewall or intrusion detection software, but the independent software vendor cannot bring it to market without either investing in a complete, vertical solution, or getting cooperation from an established network equipment vendor. In the latter case, the independent software vendor must work with multiple vendors and their multiple proprietary interfaces to port its application to individual platforms. Similarly, an independent hardware vendor with the next-generation forwarding hardware must either invest in a complete, vertical solution, or wait for an established vendor to adopt its hardware solution. Moreover, even an established hardware vendor typically has many different platforms that use different network operating systems and architectures. Thus, introducing a new hardware or software functionality requires substantial efforts and investment. In summary, the vertical integrated architecture shown in FIG. 1 stifles innovation and makes it difficult to exploit new hardware and software capabilities.

[0013] The three main components 110, 120, 130 of the networking architecture shown in FIG. 1, though inter-dependent, perform functions that are largely independent of each other. At the bottom is the forwarding hardware 110 that operates in the data-forwarding plane and is responsible for per-packeting processing and handling. In the middle is the network operating system 120 that is mainly responsible for operations in the control plane and runs routing, signaling, and network control protocols. The network operating system 120 also dictates the forwarding behavior of the underlying hardware 110 by manipulating forwarding tables, per-flow QoS (quality of service) tables, and access control lists for the forwarding interfaces. At the top is a set of applications and/or services 130 that mainly perform network management and control functions. In some cases, some of these applications may also perform per-packet processing. For example, an application-level proxy may handle each data packet that matches a particular description. However, in most cases, these applications mainly influence the data forwarding decisions by “programming” the forwarding hardware appropriately.

[0014] FIG. 2 illustrates a horizontal open networking architecture according to an embodiment of the present invention. The chief aim of a horizontal open networking architecture as in FIG. 2 is to standardize the interfaces among the components shown in FIG. 1 so that innovation at each level can accelerate and can facilitate independent hardware vendors to bring new capabilities to the market as soon as possible. The horizontal open networking architecture 200 of FIG. 2 preferably has three main components—the forwarding element 210, the control element 230, and the network services/applications 250—as well as two open interfaces 220, 240. The horizontal open networking architecture 200 essentially separates the control plane from the data forwarding plane. The control element 230 handles all of the control functions, including routing and signaling protocols, whereas the forwarding element 210 is responsible for forwarding and processing data packets.

[0015] Consider, for example, a L3 switch implemented using the horizontal open networking architecture 200 of FIG. 2. In such a switch, the control element 230 runs the necessary routing protocols (RIP (routing information protocol), OSPF (open shortest path first), etc.) and downloads the forwarding tables to the switching fabric or interfaces so that the forwarding silicon 212 (of the forwarding element 210) can forward the data packets at wire-speed. The forwarding silicon is “programmed” to deliver all control traffic (IGMP (internet group multicast protocol) queries, RSVP (“please reply”) packets, and routing updates) to the control element 230.

[0016] The interface 220 between the control and forwarding elements is designed such that either element can be replaced without affecting the other. The networking middleware 214 (of the forwarding element 210) is responsible for converting the generalized configuration and control information received via the interface 220 into the specific structures necessary to properly manipulate the state of the forwarding silicon.

[0017] The control element 230 and interface 240 exposed at the top of the forwarding element 210 performs a similar function, providing an open, standardized manner of building value-added networking services and applications. The interface 240 abstracts out and exposes the hardware-specific functionality so that these services can exploit the hardware capabilities to implement services such as policy-based networking, security (internal or external firewalls, proxies, intrusions detection, etc.), QoS (quality of service), and even functions such as load-balancing for web servers. For example, a network service may use the interface to download a policy rule that specifies the action to be taken based on a stateful inspection of packets in the forwarding path within the forwarding element 210. The control element 230 takes care of converting the abstract rule representation into a concise set of commands and configuration information to download to the forwarding element 210.

[0018] In addition to providing a standardized method of building value-added services and applications, the interface 240 enables the services to access control or management states (e.g., device-specific configuration, route/path management, topology discovery and database) so that they can provide network-wide services. More complex network management tools and services, such as event-triggered management, traffic management, and policy-based networking could be built on top of this interface.

[0019] The clear-cut functional separation of device functionality and the open nature of these interfaces 220, 240 allows the horizontal open networking architecture 200 to be realized in a variety of configurations. Just like in a traditional switch/router, a control CPU in the box can host the control element 230 and services 250 with the forwarding element 210 and control element 230 connected by a bus. Alternatively, in the case of a chassis with a high-speed backplane and several data forwarding planes, the control element 230 may reside in a separate plane plugged into the same backplane. In this configuration, each of the network services 250 may run on a separate service appliance that also plugs into the same backplane yielding a distributed architecture. In yet another distributed configuration, the control and forwarding elements 230, 210, respectively, may not be physically co-located at all, but, instead, separated by a network (Ethernet, ATM VC (asynchronous transfer mode), or other interconnection technology). In this case, a central server hosts the control element 230 and interacts with the forwarding element 210 across the network. The control server itself may be responsible for control of one or more forwarding elements 210. An example configuration consists of a wire closet with eight switches responsible for switching traffic to/from a floor in a large building. Under the horizontal open networking architecture 200 shown in FIG. 2, a single control element 230 running on a dedicated server may be responsible for providing the control functionality for the eight switches.

[0020] Another example of a distributed configuration is a control element 230 within a service provider cloud that is responsible for controlling one or more forwarding elements 210 residing at customer premises. ADSL (asymmetric digital subscriber line) modems with the ability to route and differentiate among different traffic streams may be controlled remotely from a service provider using the horizontal open networking architecture 200 shown in FIG. 2, allowing the service provider to offer simplified management and value-added services to its customers.

[0021] Finally, the horizontal open networking architecture 200 also allows network services 250 to be hosted separately from the control element 230. In this configuration, the service uses a “remoteable” version of the open interface 240 at the top to interact with the control element 230.

[0022] FIG. 3 illustrates the main components of a horizontal open networking architecture according to an embodiment of the present invention. The horizontal open networking architecture 200 of FIG. 3 consists of four major components: (1) the network services API (application program interface) 240, the control element 230, the forwarding element 210, and the connect API 220. The forwarding element 210 and the control element 230 may be connected by a bus, a high-speed backplane, or a network link. The connect API 220 represents the standard interface between the two elements 210, 230 and hides the details of interaction across these interconnections. In the case of a network link, the connect API 220 is implemented using a standardized wire-format protocol.

[0023] The network services API 240 allows the horizontal open networking architecture 200 and platform to be visible to external parties, such as third-party independent software vendors. The network services API 240 makes the hardware-specific functionality of the data forwarding elements 210 (such as switches, routers, and network interface cards (NICs)) available to the application programmers in a uniform, hardware-independent manner. Third party independent software vendors that write network-aware applications, such as VoIP (voice-over internet protocol) gateways, intrusion detection, application-specific proxies, and VPN (virtual private network) servers, use the network service API 240 to control/modify the behavior of the data forwarding path in both the data and control planes. For example, in the case of a L3/L4 switch with the ability to filter packets based on pre-specified packet filters, an H.323 (H.323, Packet-Based Multimedia Communication Systems, International Telecommunication Union, February 1998) proxy uses the network services API 240 to direct the forwarding elements 210 to intercept and forward H.323-related packets to itself for further processing. However, in the case of a L4/L7 switch with the capability of stateful inspection of data packets at wire-speed, an intrusion detection application could install a policy rule in the forwarding element 210 that specifies the classification filter and the associated actions for examining session state and identifying an “intrusion signature”. In the horizontal open networking architecture 200 according to an embodiment of the present invention, the control element 230 hosts the network services API 240 implementation and hides the details of translating the network services API 240 calls to appropriate message passing and invocations of hardware-specific calls at the forwarding element 210.

[0024] The forwarding element 210 of the horizontal open networking architecture 200 according to the present invention handles the data or packet forwarding functions. Examples of forwarding elements 210 include switches (including a Layer 2 bridge), routers, and network adapters (network interface cards). A forwarding element 210 typically has one or more network interfaces and is responsible for handling incoming packets and forwarding them over one or more outgoing interfaces. The forwarding element 210 may include a variety of capabilities, including L2/L3 forwarding, packet classification, and filtering based on L4 (or higher) layer information in packets, data encryption, and policy enforcement. In the horizontal open networking architecture 200 according to the present invention, the forwarding element 210 relies on the control element 230 to handle the usual management and control functions, such as routing and signaling protocols, interface to value-added applications, etc. The forwarding element 210 exports its capabilities and specific functions to the control element 230 with the use of the connect API 220.

[0025] Within the forwarding element 210, there may be a forwarding element manager 212 adapted to handling interactions with the control element 230, making the hardware services, such as packet forwarding, QoS (quality of service), monitoring, access control, and policy enforcement, visible to the control element 230. For example, the forwarding element manager 212 may use the services of a forwarding element connection manager 214 to establish communication with the control element 230 and accept packet-forwarding tables from the control element 230. The forwarding element connection manager 214 within the forwarding element 210 may be adapted to handle all of the details of communication with the control element 230, including discovering and binding with a particular control element 230 over a specific transport medium and handling ongoing interconnection issues and data exchange.

[0026] The control element 230 of the horizontal open networking architecture 200 according to the present invention preferably has three main components: (1) the control element manager 232; (2) the control element connection manager 234, and the forwarding element plugin API and sample plugin 236. The control element manager 232 represents the “brains” of the horizontal open networking architecture 200 in the sense that it implements most of the control plane functionality, supports the network services API 240, and handles the interactions with the many different kinds of forwarding elements 210 through a forwarding element-specific plugin API and sample plugin 236. The control element connection manager 234 corresponds to the forwarding element connection manager 214, and it handles the details of communication with controlled forwarding elements 210, including the specifics of dealing with the particular transport mediums.

[0027] The control element 230 performs a variety of control functions on behalf of the forwarding elements 210 within its control. For instance, it runs routing protocols on their behalf. For an external observer, the control element 230 and its forwarding elements 210 look just like a traditional router. Given a set of ports on one or more forwarding elements 210, the control element 230 can aggregate them and represent them as a single, virtual router. The control element 230 takes care of generating routing protocol exchanges based on the observed state of the interfaces of forwarding elements 210. Furthermore, the control element 230 may create and represent more than one virtual router at a time by aggregating ports on the forwarding elements into different groups where each group looks like a separate router to an external observer. Such a capability is very useful for administrative purposes, as well as to dynamically build forwarding domains for the purpose of access control, etc.

[0028] The horizontal open networking architecture 200 according to the present invention allows for the forwarding element platform-specific functionality to be available to the control element manager 232 and to independent software vendors through the network services API 240. However, the API and its implementation must do two things: (1) convert the API calls into invocations of hardware-specific functionality in the forwarding element 210, and at the same time, (2) allow hardware vendors the ability to differentiate by exposing any hardware-specific features through the API without divulging their intellectual property. An example of the second aspect is a forwarding element platform that allows the application writers to download code for stateful inspection of packet contents. Another example of a forwarding element-specific functionality is the use of a specialized hashing algorithm to precompute forwarding and flow tables for the forwarding element 210. Depending on the ASICs (application-specific integrated circuits) or custom forwarding silicon used, each hardware vendor typically uses its own customized table format and lookup algorithm to facilitate wire-speed forwarding and per-flow queuing. A hardware vendor would be reluctant to expose the internal details of such algorithms to the control element 230. Instead, the vendor might choose to perform this computation on the control element 230, and then download a customized table to the forwarding element 210.

[0029] Therefore, the forwarding element plugin API and sample plugin 236 allows an independent hardware vendor to provide a software module that captures the hardware-specific functionality and “plug-in” that module into the control element manager 232. The control element manager 232, in turn, uses the plugin module 236 to expose the hardware-specific functionality through the network services API 240 and map the network services API calls to hardware-specific functions supported by the module. For example, the forwarding element plugin API and sample plugin 236 allows the vendor to supply a set of extensions to the network services API 240 for custom functionality that is not covered by the predefined functionality exposed in the network services API 240. Also, the plugin capability allows the vendor to supply its own code that can take the forwarding or flow tables prepared by the control element 230 and format them into the form needed by its ASICs before the control element 230 transmits the tables to the forwarding element 210.

[0030] The connect API 220 interface abstracts the functionality of the forwarding elements 210 so that the capabilities of the forwarding elements 210 can be remotely configured and controlled by the control element 230. The connect API 220 defines the standard features that are expected to be found in all types of forwarding elements 210, such as basic monitoring functionality, as well as features expected to be supported by only a subset of the forwarding elements 210 (such as L3 switching, or encryption). The connect API 220 may also include a provision for passing opaque information to forwarding elements 210 in order to allow inclusion of unanticipated or vendor-specific functionality. The connect API 220 may include a capability negotiation mechanism between the control element 230 and forwarding elements 210 to allow discovery of forwarding element features by the control element 230.

[0031] The connect API 220 itself includes the basic communication primitives along with methods for examining and manipulating the status of the underlying transport in a transport independent manner. The connect API 220 may be implemented over a variety of transports, such as PCI (peripheral component interconnect), input/output backplane, Ethernet, or ATM (asynchronous transfer mode). Depending on the transport, a transport-specific module specifies exactly how the connect API UDPs (user datagram protocols) are encapsulated on specific interconnect technologies, as well as how to deal with normal and exception conditions in the operation of the interconnect technology. Two examples of transports include PCI and IP. The IP transport is preferably used over the Ethernet or ATM, and can be implemented using either UDP (user datagram protocol) or TCP (transmission control protocol) as transport protocols. The use of these protocols, however, does not preclude the possibility of using native ATM or Ethernet transport for implementation of the connect API 220.

[0032] FIG. 4 illustrates a diagram of a forwarding element and a control element according to an embodiment of the present invention. As illustrated in FIG. 4, the standard control element 230 includes three major components: (1) the control element 470, which includes the control element manager 232 and the control element connection manager 234 as illustrated in FIG. 3; (2) the opaque forwarding element plugin API 480; and (3) the forwarding element specific plugin 490. The proprietary forwarding element 210 as illustrated in FIG. 4 includes four major components: (1) the forwarding element software and hardware 410; (2) the device-specific forwarding element interface 420; (3) the “BLOB” decapsulator 430; and (4) the abstract forwarding element API 440. The standard control element 230 and the proprietary forwarding element 210 are interconnected by the open forwarding element/control element interconnect 220.

[0033] In utilizing the horizontal open networking architecture 200 of the present invention, a forwarding element vendor includes with the forwarding element hardware device 210 a software plugin 480, 490 for use with the forwarding element hardware device 210. The software plugin, which includes the forwarding element-specific plugin 490 and the opaque forwarding element plugin API 480, is installed onto the standard control element 230 and becomes integrated with the standard control element 230. Because the standard control element 230 utilizes open APIs—such as the open forwarding element/control element interconnect 220 and the abstract forwarding element API 440—there is no overhead of translating abstract forwarding element API calls to the native format of the proprietary forwarding element 210.

[0034] Therefore, the control element 470 is adapted to generate an abstract forwarding element configuration (i.e., a standardized data set) 475 for configuring the proprietary forwarding element 210. The abstract forwarding element configuration 475 is a standardized data set within the horizontal open networking architecture 200 which provides for conformity for data transactions in the development of the products by independent software vendors as well as independent hardware vendors. The abstract forwarding element configuration 475 is preferably passed to the opaque forwarding element plugin API 480, which then passes the abstract forwarding element configuration 475 to the forwarding element-specific plugin 490. The opaque forwarding element plugin API 480 performs the tasks of relaying the abstract forwarding element configuration 475 (the standardized data set) from the control element 470 to the forwarding element-specific plugin 490, and relaying a forwarding element-specific configuration BLOB (binary large object) 495 (the specialized data set) from the forwarding element-specific plugin 490 to the proprietary forwarding element 210.

[0035] The forwarding element-specific plugin 490 provided by the forwarding element vendor is adapted to perform the translation of the abstract forwarding element configuration (the standardized data set) 475 into a specialized data set, preferably, a forwarding element-specific configuration BLOB 495. The forwarding element-specific plugin 490 is preferably a dynamic linkable library (DLL) in binary form. Because the forwarding element-specific plugin 490 executes on the standard control element 230, the need for additional processing power on the proprietary forwarding element 210 is avoided.

[0036] The forwarding element-specific configuration BLOB (specialized data set) 495, contains the forwarding element-specific invocations of functionality that is specific to the proprietary forwarding element 210. By utilizing the forwarding element-specific plugin 490 to translate a standardized data set into a specialized data set, the proprietary forwarding element 210 implementations need not be exposed. Therefore, the proprietary design and architecture information are kept confidential. Because the forwarding element-specific plugin 490 is in binary form, the transformation process from the standardized data set to the specialized data set is essentially hidden from other components, thus protecting the proprietary forwarding element 210 implementation. A tremendous amount of complex reverse engineering would be required in order to determine the translation process from the forwarding element specific plugin DLL file.

[0037] Once the forwarding element-specific configuration BLOB 495 is generated, it is preferably passed back to the opaque forwarding element plugin API 480, which then transmits the BLOB 495 to the proprietary forwarding element 210. The BLOB 495 is preferably passed through the open forwarding element/control element interconnect 220, then to the abstract forwarding element API 440, and then finally to the BLOB decapsulator 430. The BLOB decapsulator 430 takes the BLOB 495 and “decapsulates” it—which is a very simple computational operation—and passes the decapsulated BLOB data directly to the device-specific forwarding element interface 420. The device-specific forwarding element interface 420 takes the information from the decapsulated BLOB data to configure the forwarding element software and hardware 410 to properly operate the proprietary forwarding element 210. In this manner, confidential information about the proprietary forwarding element's architecture that is present in the device-specific forwarding element interface 420 is never exposed to the standard control element 230, allowing independent hardware vendors of the forwarding elements to protect their intellectual property.

[0038] Additionally, encryption may be utilized to further protect the specialized data set that is passed from the standard control element 230 to the proprietary forwarding element 210. The specialized data set (i.e., the forwarding element-specific configuration BLOB 495) may be encrypted once it has been translated from the standardized data set (abstract forwarding element configuration 475). The encrypted specialized data set would be passed to the proprietary forwarding element 210, and then decrypted at the proprietary forwarding element 210. However, the use of encryption adds additional hardware to the proprietary forwarding element 210 in order to perform the decryption process.

[0039] While the description above refers to particular embodiments of the present invention, it will be understood that many modifications may be made without departing from the spirit thereof. The accompanying claims are intended to cover such modifications as would fall within the true scope and spirit of the present invention. The presently disclosed embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims, rather than the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.