US 5,920,566 AGrant
Routing in a Multi-Layer Distributed Network Element
Issue Date:1999-07-06
•20 Claims
•8 Drawing Sheets
Abstract
A multi-layer distributed network element for relaying packets according to known routing protocols. A distributed architecture of multiple subsystems delivers routing at wire-speed performance across subnetworks. Each subsystem includes a forwarding memory and an associated memory and is configured to identify unicast and multicast packets for routing purposes, modify the packets in hardware, including replace VLAN information, and forward the packets to the next hop. The routing decisions are made in the inbound subsystem, and packets are forwarded, if necessary given the network topology, through a separate outbound subsystem.
Metadata
Assignee
- Sun Microsystems, Inc.
Inventors
- Ariel Hendel
- Shimon Muller
- William T. Zaumen
- Louise Yeung
Application Information
Application Number:US 885114&
Filing Date:1997-06-30
Priority Date:1997-06-30
Art Unit:271
Classifications
IPC:
H04J 302
Field of Search:
370340400;401;402;403;404;405;392;389;356;390;230;231825.2
Patent Drawings (8 sheets)
Description
Background
1. Field of the Invention
This invention relates generally to communication systems that couple computers, and more specifically to relaying messages through a network element.
2. Description of Related Art
Communication between computers has become an important aspect of everyday life in both private and business environments. Computers converse with each other based upon a physical medium for transmitting the messages back and forth, and upon a set of rules implemented by electronic hardware attached to and programs running on the computers. These rules, often called protocols, define the orderly transmission and receipt of messages in a network of connected computers.
A local area network (LAN) is the most basic and simplest network that allows communication between a source computer and destination computer. The LAN can be envisioned as a cloud to which computers (also called endstations or end-nodes) that wish to communicate with one another are attached. At least one network element will connect with all of the endstations in the LAN. An example of a simple network element is the repeater which is a physical layer relay that forwards bits. The repeater may have a number of ports, each endstation being attached to one port. The repeater receives bits that may form a packet of data that contains a message from a source endstation, and blindly forwards the packet bit-by-bit. The bits are then received by all other endstations in the LAN, including the destination.
A single LAN, however, may be insufficient to meet the requirements of an organization that has many endstations, because of the limited number of physical connections available to and the limited message handling capability of a single repeater. Thus, because of these physical limitations, the repeater-based approach can support only a limited number of endstations over a limited geographical area.
The capability of computer networks, however, has been extended by connecting different subnetworks to form larger networks that contain thousands of endstations communicating with each other. These LANs can in turn be connected to each other to create even larger enterprise networks, including wide area network (WAN) links.
To facilitate communication between subnetworks in a larger network, more complex electronic hardware and software have been proposed and are currently used in conventional networks. Also, new sets of rules for reliable and orderly communication among those endstations have been defined by various standards based on the principle that the endstations interconnected by suitable network elements define a network hierarchy, where endstations within the same subnetwork have a common classification. A network is thus said to have a topology which defines the features and hierarchical position of nodes and endstations within the network.
The interconnection of endstations through packet switched networks has traditionally followed a peer-to-peer layered architectural abstraction. In such a model, a given layer in a source computer communicates with the same layer of a peer endstation (usually the destination) across the network. By attaching a header to the data unit received from a higher layer, a layer provides services to enable the operation of the layer above it. A received packet will typically have several headers that were added to the original payload by the different layers operating at the source.
There are several layer partitioning schemes in the prior art, such as the Arpanet and the Open Systems Interconnect (OSI) models. The seven layer OSI model used here to describe the invention is a convenient model for mapping the functionality and detailed implementations of other models. Aspects of the Arpanet, however, (now redefined by the Internet Engineering Task Force, or IETF) will also be used in specific implementations of the invention to be discussed below.
The relevant layers for background purposes here are Layer 1 (physical), Layer 2 (data link), and Layer 3 (network), and to a limited extent Layer 4 (transport). A brief summary of the functions associated with these layers follows.
The physical layer transmits unstructured bits of information across a communication link. The repeater is an example of a network element that operates in this layer. The physical layer concerns itself with such issues as the size and shape of connectors, conversion of bits to electrical signals, and bit-level synchronization.
Layer 2 provides for transmission of frames of data and error detection. More importantly, the data link layer as referred to in this invention is typically designed to "bridge," or carry a packet of information across a single hop, i.e., a hop being the journey taken by a packet in going from one node to another. By spending only minimal time processing a received packet before sending the packet to its next destination, the data link layer can forward a packet much faster than the layers above it, which are discussed next. The data link layer provides addressing that may be used to identify a source and a destination between any computers interconnected at or below the data link layer. Examples of Layer 2 bridging protocols include those defined in IEEE 802 such as CSMA/CD, token bus, and token ring (including Fiber Distributed Data Interface, or FDDI).
Similar to Layer 2, Layer 3 also includes the ability to provide addresses of computers that communicate with each other. The network layer, however, also works with topological information about the network hierarchy. The network layer may also be configured to "route" a packet from the source to a destination using the shortest path. Finally, the network layer can control congestion by simply dropping selected packets, which the source might recognize as a request to reduce the packet rate.
Finally, Layer 4, the transport layer, provides an application program such as an electronic mail program with a "port address" which the application can use to interface with Layer 3. A key difference between the transport layer and the lower layers is that a program on the source computer carries a conversation with a similar program on the destination computer, whereas in the lower layers, the protocols are between each computer and its immediate neighbors in the network, where the ultimate source and destination endstations may be separated by a number of intermediate nodes. Examples of Layer 4 and Layer 3 protocols include the Internet suite of protocols such as TCP (Transmission Control Protocol) and IP (Internet Protocol).
Endstations are the source and ultimate destination of a packet, whereas a node refers to an intermediate point between the endstations. A node will typically include a network element which has the capability to receive and forward messages on a packet-by-packet basis.
Generally speaking, the larger and more complex networks typically rely on nodes that have higher layer (Layers 3 and 4) functionalities. A very large network consisting of several smaller subnetworks must typically use a Layer 3 network element known as a router which has knowledge of the topology of the subnetworks.
A router can form and store a topological map of the network around it based upon exchanging information with its neighbors. If a LAN is designed with Layer 3 addressing capability, then routers can be used to forward packets between LANs by taking advantage of the hierarchical routing information available from the endstations. Once a table of endstation addresses and routes has been compiled by the router, packets received by the router can be forwarded after comparing the packet's Layer 3 destination address to an existing and matching entry in the memory.
The router operates by parsing the header of a received packet, making decisions based on a routing table inside the router, and forwarding the packet, with any required header modifications, to the next node or endstation. Thus, the packet will go through several such "hops" before reaching its destination where a hop is defined as the packet traveling from one node or endstation to another node or endstation.
In comparison to routers, bridges are network elements operating in the data link layer (Layer 2) rather than Layer 3. They have the ability to forward a packet based only on the Layer 2 address of the packet's destination, typically called the medium access control (MAC) address. Generally speaking, bridges do not modify the packets. Bridges forward packets in a flat network having no hierarchy without any cooperation from the endstations.
Hybrid forms of network elements also exist, such as brouters and switches. A brouter is a router which can also perform as a bridge. The term switch refers to a network element which is capable of forwarding packets at high speed with functions implemented in hardwired logic as opposed to a general purpose processor executing instructions. Switches come in many flavors, operating at both Layer 2 and Layer 3.
Having discussed the current technology of networking in general, the limitations of such conventional techniques will now be addressed. With an increasing number of users requiring increased bandwidth from existing networks due to multimedia applications to run on the modern day Internet, modern and future networks must be able to support a very high bandwidth and a large number of users. Furthermore, such networks should be able to support multiple traffic types such as voice and video which typically require different service characteristics. Statistical studies show that the network domain, i.e., a group of interconnected LANs, as well as the number of individual endstations connected to each LAN, will grow at a faster rate in the future. Thus, more network bandwidth and more efficient use of resources is needed to meet these requirements.
Building networks using Layer 2 elements such as bridges provides fast packet forwarding between LANs but has no flexibility in traffic isolation, redundant topologies, and end-to-end policies for queuing and access control. Endstations in a subnetwork can invoke conversations based on either Layer 3 or Layer 2 addressing. As bridges forward packets based on only Layer 2 parsing, the provide simple yet speedy forwarding services. However, the bridge does not support the use of high layer handling directives including queuing, priority, and forwarding constraints between endstations in the same subnetwork.
A prior art solution to enhancing bridge-like conversations within a subnetwork relies on a network element that uses a combination of Layer 2 and upper layer headers. In that system, the Layer 3 and Layer 4 information of an initial packet are examined, and a "flow" of packets is predicted and identified using a new Layer 2 entry in the forwarding memory, with a fixed quality of service (QOS). Thereafter, subsequent packets are forwarded at Layer 2 speed (with the fixed QOS) based upon a match of the Layer 2 header with the Layer 2 entry in the forwarding memory. Thus, no entries with Layer 3 and Layer 4 headers are placed in the forwarding memory to identify the flow.
However, consider the scenario where there are two or more programs communicating between the same pair of endstations, such as an electronic mail program and a video conferencing session. If the programs have dissimilar QOS needs, the prior art scheme just presented will not support different QOS characteristics between the same pair of endstations, because the prior art scheme does not consider information in Layer 3 and Layer 4 when forwarding. Thus, there is a need for a network element that is flexible enough to support independent priority requests from applications running on endstations connected to the same subnetwork.
The latter attributes may be met using Layer 3 elements such as routers. But packet forwarding speed is sacrificed in return for the greater intelligence and decision making capability provided by the router. Therefore, networks are often built using a combination of Layer 2 and Layer 3 elements.
The role of the server has multiplied with browser-based applications that use the Internet, thus leading to increasing variation in traffic distribution. When the role of the server was narrowly limited to a file server, for example, the network was designed with the client and the file server in the same subnetwork to avoid router bottlenecks. However, more specialized servers like World Wide Web and video servers are typically not on the client's subnetwork, such that crossing routers is unavoidable. Therefore, the need for packets to traverse routers at higher speeds is crucial. The choice of bridge versus router typically results in a significant trade-off, lower functionality when using bridges, and lower speed when using routers. Furthermore, the service characteristics within a network are no longer homogenous, as the performance of a server becomes location dependent if its traffic patterns involve routers.
Therefore, there is a need for a network element that can handle changing network conditions such as topology and message traffic yet make efficient use of high performance hardware to switch packets based on their Layer 2, Layer 3, and Layer 4 headers. The network element should be able to operate at bridge-like speeds, yet be capable of routing packets across different subnetworks and provide upper layer functionalities such as quality of service.
Summary
The invention is an apparatus and related method for relaying packets by a multi-layer distributed network element according to known routing protocols.
The invention is directed at a multi-layer distributed network element (MLDNE) for receiving and forwarding packets using known routing protocols. The MLDNE has a number of subsystems that are coupled by internal links. Each subsystem has a forwarding memory and associated memory. The memories associate packet header information including addresses with routing information. A subsystem also includes external ports that connect with neighboring nodes and endstations, and internal ports that connect with other subsystems through the internal links.
When a packet is received by a first "inbound" subsystem, the subsystem determines whether the packet should be routed based upon a first header portion, including a Layer 2 destination address of the received packet, matching a Layer 2 address of the MLDNE. If the first header portion of the received packet matches the MLDNE address, then the first subsystem determines, using its forwarding memory, whether a route has been previously determined for a second header portion, including Layer 3 source and destination addresses, of the received packet.
If a type 2 entry in the forwarding memory matches the received packet's second header portion, then a neighbor node's Layer 2 address (found in associated memory) replaces the Layer 2 destination address of the packet. The neighbor node's address was previously stored in the associated memory as part of the routing information associated with the matching type 2 entry. In addition to Quality of Service information, the routing information in the associated memory also identifies the external ports of the inbound subsystem that connect with the neighbor node. If the neighbor node is connected to a subsystem other than the inbound subsystem, the situation would have been recognized at the time the matching type 2 entry was created such that the associated memory would identify the internal port of the inbound subsystem, rather than external port, that connects with the other subsystem to which the neighbor node or endstation is connected.
When the packet is received over the internal link by a second subsystem, the packet is forwarded to the neighbor node in response to the packet's new first header portion matching a type 1 entry in the second forwarding memory. The type 1 entry in the second subsystem contains the address of the neighbor node or endstation and had been created independently of the matching type 2 entry of the inbound subsystem.
After determining that a received packet should be routed, the inbound subsystem also generates a first control signal which indicates to the external port that eventually forwards the packet that a third header portion identifying the packet's source be modified before sending the packet to the neighbor node. A Layer 2 source address of the packet is replaced with a source address associated with the external port. The control signal is also passed over an internal link to the second subsystem if the neighbor node is reachable through that subsystem.
The invention's distributed architecture can also be configured to support routing of multicast packets. Once a multicast routable packet has been identified in the inbound subsystem, a second control signal may be sent across an internal link in response to which the second subsystem performs a type 2 search of the forwarding memory (based on the network layer and higher layer headers of the packet). If a matching type 2 entry is found, then the external ports of the second subsystem check the first control signal (also received from the inbound subsystem) to see if the source address of the packet needs to be replaced, and the packet is then forwarded with the appropriate modifications to its headers. The first control signal may also be received and checked by the external ports of the inbound system where the multicast destination group includes nodes/endstations connected to the inbound subsystem.
The invention's search engine, forwarding engine, and data structures are organized in a way that supports bridging and routing functions simultaneously, where if routing criteria are not met for a received packet, then bridging functions are provided automatically.
In its present embodiment, the invention is implemented with the data link layer (Layer 2), the network layer (Layer 3) and higher layers including the transport layer (Layer 4).
Drawings
The foregoing aspects and other features of the invention will be better understood by referring to the figures, detailed description, and claims below where:
FIG. 1 is a high level view of an exemplary network application of a multi-layer distributed network element (MLDNE) of the invention.
FIG. 2 in an internal view of the MLDNE as an embodiment of the invention.
FIG. 3 illustrates an exemplary forwarding and associated memory of a subsystem in the MLDNE, including associated data for the routing of packets, according to another embodiment of the invention.
FIG. 4 is a block diagram of an embodiment of the MLDNE having only two subsystems and acting as a router between a client and a server.
FIG. 5 is a flow diagram of processing a received packet for routing purposes by the invention's network element.
FIG. 6 is a continuation of the flow diagram in FIG. 5 and includes steps performed in processing a unicast packet.
FIG. 7 shows exemplary steps and operations performed by the invention's network element for routing a multicast packet.
FIG. 8A is a simplified block diagram of a packet structure utilizied in one embodiment of the invention.
FIG. 8B is a structure for header field replacement of packets by the invention.
Detailed Description
As shown in the drawings by way of illustration, the invention defines a network element that is used to interconnect a number of nodes and endstations in a variety of different ways. In particular, an application of the multi-layer distributed network element (MLDNE) would be to route packets according to predefined routing protocols over a homogenous data link layer such as the IEEE 802.3 standard, or Ethernet. FIG. 1 illustrates the invention's use as a router in a network where the MLDNE 201 couples a client C to the Router 107 which in turn couples with the Server 105. The MLDNE 201 can interconnect a number of desktop units (endstations), while acting as an intermediate node, through its external connections 217. The MLDNE 201 is capable of providing a high performance communication path between servers and desktop units while acting as a router, where the Server 105 and the client C reside in different LANs.
The MLDNE's distributed architecture can be configured to route message traffic in accordance with a number of known routing algorithms such as RIP and OSPF. In a preferred embodiment, the MLDNE is configured to handle message traffic using the Internet suite of protocols, and more specifically the Transmission Control Protocol (TCP) and the Internet Protocol (IP) over the Ethernet LAN standard and medium access control (MAC) data link layer. The TCP is also referred to here as an exemplary Layer 4 protocol, while the IP is referred to repeatedly as a Layer 3 protocol. However, other protocols can be used to implement the concepts of the invention.
In a first embodiment of the invention's MLDNE, a network element is configured to implement packet routing functions in a distributed manner, i.e., different parts of a function are performed by identical building block subsystems in the MLDNE, while the final result of the functions remains transparent to the external nodes and endstations. As will be appreciated from the discussion below and the diagram in FIG. 2, the MLDNE has a scalable architecture which allows the designer to increase the number of external connections by adding additional subsystems.
As illustrated in block diagram form in FIG. 2, the MLDNE 201 contains a number of identical subsystems 210 that are fully meshed and interconnected using a number of internal links 241 to create a larger network element. At least one internal link couples any two subsystems. Each subsystem 210 includes a forwarding memory 213 and an associated memory 214. The forwarding memory 213 stores an address table used for matching with the headers of received packets. The associated memory stores data associated with each entry in the forwarding memory that is used to identify forwarding attributes for forwarding the packets through the MLDNE. A number of external ports (not shown) having input and output capability interface the external connections 217. Internal ports (not shown) also having input and output capability in each subsystem couple the internal links 241. In the preferred embodiment, the external and internal ports lie within a hardwired-logic switching element 211 implemented by an application specific integrated circuit (ASIC).
A received packet arrives at an inbound subsystem through one of the external connections 217, and will be forwarded to a node or endstation outside the MLDNE through another external connection in an outbound subsystem. The outbound and inbound subsystems can be either the same or different subsystems.
Referring to FIG. 2, the MLDNE 201 includes a central processing system (CPS) 260 that is coupled to the individual subsystems 210 through a communication bus 251 such as the Peripheral Components Interconnect (PCI). The CPS 260 includes a central processing unit (CPU) 261 coupled to a central memory 263. Central memory 263 includes a copy of the entries contained in the individual forwarding memories 213 of the various subsystems. The CPS has a direct control and communication interface to each subsystem 210. The CPS is also configured with a number of routing protocols that are used to identify a neighbor node as part of a route for forwarding a received packet to its ultimate destination, normally specified in the Layer 3 destination address of the packet. Other responsibilities of the CPS 260 include setting data path resources such as packet buffers between the different subsystems. Finally, the CPS 260 performs the important task of determining whether or not a type 2 entry should be added to the forwarding memory of each individual subsystem.
FIG. 3 takes a closer look at the forwarding and associated memories in each subsystem. The forwarding memory includes a number of entries of two types, type 2 entry 321 and type 1 entry 301. Each entry in the forwarding memory includes data to be compared with the headers of received packets. For the particular embodiment of TCP/IP, the data fields for each type 2 entry 321 include a class field 323, an IP source field 325, an IP destination field 327, an application source port 333, an application destination port 335, and an Inbound Port field 337. For the type 1 entry 301, a class field, a Layer 2 address field, and a VLAN identification (VID) field are shown in the exemplary embodiment. Of course, additional header information and similar definitions using alternate network and transport layer protocols can be developed and included in each entry and used for matching the headers of received packets, as will be apparent to one skilled in the art.
Associated with each type 2 entry 321 and type 1 entry 301 are associated data stored in associated memory 214. The associated data fields contain information needed to forward a matching packet received by the subsystem. The subsystem port field 347 identifies the internal or external ports of the subsystem used for forwarding the matching packet to the neighboring node in the next hop. The next hop address field 357 identifies the neighbor node's Layer 2 address which replaces the original Layer 2 destination address of a received unicast packet to be routed. A priority field 345 is used for queuing purposes by the external port which actually sends the packet outside the MLDNE. The age fields 343 and 344 help minimize the number of entries in the forwarding memory by indicating that a recently received packet has matched the corresponding type 1 or type 2 entry.
A NEW VID address field 353 allows the MLDNE to be configured to support virtual LANs (VLANs). The associated data also includes a NEW VLAN identification (VID) TAG field, used to notify the subsystem of a need to change the packet's VID, particularly when forwarding the packet across subnetworks. The inbound subsystem in response will either insert a new tag, or replace an existing tag with the value in the NEW VID field. For example, when routing between VLANs requires the forwarded packet's tag to be different from the received packet's tag, then the NEW VID field will contain the replacement tag for the subsystem to replace before forwarding the packet.
Whenever a packet is sent across an internal link, additional control information may be made available over the internal link to the outbound subsystem receiving the packet. Such information, in addition to the sa.sub.-- replace bit discussed below, includes an orig.sub.-- tag bit which indicates whether or not the received packet was originally tagged with VLAN information, a mod.sub.-- tag bit which indicates whether the tag was modified by the inbound subsystem, and a dont.sub.-- tag bit which indicates that the received packet should not be tagged by the outbound subsystem.
Finally, the associated memory can be configured to include a multicast route field 355 which activates multicast routing capability in the subsystem as further explained below.
The routing operation of the MLDNE 201 will be described for an exemplary embodiment using the flow diagram of FIGS. 5-7 in conjunction with the exemplary network application in FIG. 4. References to fields in the forwarding and associated memories are found in FIG. 3. In the example below, the journey of a packet is traced beginning with a client C in subnetwork 103 coupled to an external connection of MLDNE 201. The client C sends a packet to server 105 which is identified in the Layer 3 destination address field of the packet's header. The packet must traverse a router 107 which is assumed to have a Layer 2 address known by the MLDNE 201.
Beginning with block 503 in FIG. 5, a packet is received by the MLDNE 201 at external port E.sub.1 of the inbound subsystem 410. The packet includes a message originated from a client C having a Layer 3 address in a logically defined network subnetwork 103. Subsystem 410 is configured to recognize that external ports E.sub.1 and E.sub.2 couple the subnetwork 103.
When the packet is received by switching element 411, operation continues with decision block 507 where first header portion, including the Layer 2 destination address in the present embodiment, of the received packet is compared with a router address of the MLDNE 201. The router address may be a Layer 2 address assigned to external port E.sub.1, or a Layer 2 address assigned to the MLDNE as a whole. Normally, the MLDNE will be configured so that each external port is assigned its own router address.
If the first header portion of the received packet matches the router address, then operation proceeds to block 515 where the packet is declared to be a potential unicast route candidate. If, however, the first header portion does not match the router address, then operation proceeds to block 509 where the packet is declared as not being a unicast routable packet. As will be appreciated below, such a packet can still be a multicast packet having a multicast route available in the MLDNE.
For a unicast packet of the route class, block 517 performs a search of the forwarding memory 413 for a matching type 2 entry using "route" as the class field 323.
The search of the forwarding memory in block 517 leads to the decision block 521 where the test is whether a type 2 matching entry exists in the forwarding memory 413. If not, then operation proceeds with block 523 where relevant portions of the received packet headers are sent to the CPS via the CPS port in subsystem 410 and the CPS bus 451.
When the CPS 460 receives the portions of the headers of the "missed" packet from subsystem 410 in block 533, the CPS then examines access policies and dass of service policies that have been preconfigured in the CPS, and the CPS Layer 2 and Layer 3 topology tables. The CPS has the option of denying service to the path requested by the received packet, performing the routing function entirely in its own software, or preparing a type 2 entry in the inbound system's forwarding memory for the route.
The routing algorithms of the MLDNE 201 are implemented by the CPS. If a unicast route exists or can be readily computed for the received packet, then the CPS decides in decision block 537 to proceed with block 539 and add a route class type 2 entry 321 to the forwarding memory, and associated data to the associated memory, of the inbound subsystem 410. If the neighbor node connects to an external port of the inbound subsystem 410, as determined by the CPS consulting a Layer 2 table in the central memory, then the external port is identified in the new type 2 entry's associated subsystem port field 347. Similarly, if the neighbor node connects to the subsystem 420, then an internal port I.sub.1 or I.sub.2 is identified.
Returning to decision block 521, if the packet matches an existing route class type 2 entry in the forwarding memory 413 of the inbound subsystem 410, then the received packet is forwarded as a unicast packet as illustrated in exemplary form in FIG. 6.
Turning now to FIG. 6 and staying in the inbound subsystem, the switching element 411 evaluates whether the unicast packet's time to live has been exceeded. A time to live field is assumed to exist in the received packet's headers. If the packet has been circulating through the network too long as indicated by its time to live field, then the inbound subsystem only sends the received packet to the CPS, and then a time exceeded error message in accordance with, for example, the Internal Control Message Protocol (ICMP) or as discussed in the Request For Comments (RFC) maintained by the Internet community, is generated by the CPS as in block 609.
If, on the other hand, the packet's time to live (TTL) has not been exceeded, then operation continues with block 619 where the TTL is decremented. This modification to the packet's header will normally require compensating the packet's Layer 3 header check sum as in block 621. In block 611, the switching element 411 replaces the Layer 2 destination address of the received packet with the next hop Layer 2 address found in the associated memory corresponding to the matching type 2 entry determined in block 521 of FIG. 5.
If the MLDNE 201 is configured to support VLANs, then decision block 615 determines whether a new VLAN identification tag is required by checking the status of the NEW VID tag field 351.
Whether or not the packet is to be forwarded outside the MLDNE by another subsystem (as indicated by the subsystem port field 347 associated with the matching type 2 entry) a first control signal, such as a sa.sub.-- replace bit, is prepared in block 621. The sa.sub.-- replace bit will be handed off to the external and internal ports indicated in the subsystem port field 347, and thus may be transferred over an internal link 441, together with the packet, to the subsystem 420. The first control signal will notify the subsystem (either the inbound one or another subsystem) to replace the Layer 2 source address of the packet with the source address of the external port used for forwarding the packet.
In the example of FIG. 4, the packet together with any control information, including first control signal, are processed by internal port I.sub.2 in switching element 411, and delivered to the internal link 441 to connect with the outbound subsystem 420 in block 627. Alternatively, however, the modified packet and control information stay in the inbound subsystem and are processed by an external port, where operation continues in block 630.
In block 627, the packet is received over the internal links in outbound subsystem 420. A type 1 matching cyde then begins and decision block 629 is reached to determine whether a matching type 1 entry exists in the forwarding memory 423. If a type 1 entry exists then operation continues with block 630.
The operation from block 630 to block 637 are performed by the "outbound" subsystem where the packet leaves the MLDNE, be it the inbound subsystem 410 or a different subsystem 420. If the sa.sub.-- replace bit, as checked in decision block 630, is set, then the switching element replaces a third header portion, including at least the Layer 2 source address of the received packet, with the Layer 2 address of the external port E.sub.3 through which the packet must be forwarded. The external port E.sub.3 was identified in the associated data (in associated memory) corresponding to either the matching type 1 entry found in block 629 (the packet came across internal link) or the matching type 2 entry found in block 521 (the packet remained in inbound subsystem).
The MLDNE can be configured so that each external port is assigned a unique Layer 2 address. Alternatively, a single source address may be assigned to the MLDNE as a whole and shared by all external ports. In either case, following the replacement of the third header portion, the cyclic redundancy code (CRC) of the packet's headers is recomputed in block 635 and the packet is then forwarded to the neighbor node being the router 107 in FIG. 4.
In the above example, the packet's journey has been described originating from the client C and traveling through subsystem 410, internal link 441, and subsystem 420 in MLDNE 201. The packet is then received by router 107 and forwarded according to conventional means to server 105. The above, of course, assumed that a route for the server 105 as a destination through router 107 had been previously obtained by the MLDNE 201 using conventional techniques for determining the routes.
The above also covered the situation where although a unicast packet falls within the route class, no type 2 matching entry existed in the inbound subsystem to be used for routing the packet through the MLDNE. Thus, the decision as to whether or not a received packet will be routed is made in the inbound subsystem, in particular, in decision blocks 507 and 521 of FIG. 5. Note also that routing policies as well as class of service queuing have the granularity and flexibility of Layer 3 end-to-end addresses and protocol based classification. These routing policies and class of service queuing are identified in the associated data corresponding to each matching type 2 entry, and may be sent across the internal link to a separate outbound subsystem.
Multicast Routing
Having discussed the unicast routing aspects of the invention, the routing features of the invention for multicast packets are now presented while referring once again to the entries in the forwarding and associated memories of FIG. 3 and the flow diagram of FIG. 7. Although multicast routing in the invention's MLDNE can be supported by similar hardware structures that implement unicast routing in the MLDNE, multicast does present significantly different problems to the network element designer. For instance, the routing protocols used to derive the type 2 entries in the forwarding memory include protocols such as MOSPF and DVMRP which are well-known in the art. These multicast routing protocols produce a loop-free distribution tree for the packet's group destination network layer multicast address and a source network layer address for the sender.
The MLDNE has a local multicast forwarding rule which yields a number of external ports (and their corresponding subsystems) for forwarding the packet, as a function of a received multicast packet group destination Layer 3 address, source Layer 3 address, and the inbound subsystem port of arrival. This dependency is reflected in the type 2 entry in the forwarding memory of FIG. 3 as the fields 327, 325, and 337, respectively, to be matched with a received packet's headers. The inbound port of arrival field 337 is included to prevent forwarding duplicate packets over alternate paths.
To identify a received packet as a candidate for multicast routing, the MLDNE is configured to identify a multicast packet based on at least two criteria. First, the packet headers must match a given class. Second, the packet's headers must match an existing type 2 entry that refers to a multicast group destination address. The matching type 2 entry for the multicast case may be created as a result of executing a multicast registration protocol such as IGMP.
FIG. 7 illustrates an exemplary flow diagram for routing a received multicast packet through the MLDNE 201 of FIG. 4. When a packet is received by the subsystem 410 and the packet headers match a certain class and a type 2 entry 321 which has a multicast route field 355 indicating that the entry is for multicast routing, as in block 703, control is transferred to the decision block 705. If the packet's time to live has not been exceeded, then the routing operation continues in block 709 in the inbound subsystem 410 by decrementing the time to live field in the received packet's header. If the packet's TTL was exceeded, then in block 707 the packet may be flooded, not routed, to its VLAN. A packet's VLAN, in general, defines the Layer 2 topology used for flooding, in other words the broadcast domain.
Proceeding to block 711, the inbound subsystem 410 determines whether a new VLAN tag is required for the received packet, based on the NEW VID tag field 351 in the associated memory. If so, then the VID in the Layer 2 header of the packet is replaced with the destination VID of the next hop, as found in the associated memory, as in block 713. Note that block 713 is performed only if the Layer 3 multicast destination address of the received packet refers to endstations that lie within the same VLAN. Such a determination was made by the CPS when the type 2 entry was created.
Whether or not VLANs are supported by the MLDNE, in block 715 the inbound subsystem 410 prepares to notify the external ports that will forward the packets outside the MLDNE of a need to route the packet by setting the first control signal (sa.sub.-- replace bit) to indicate to the forwarding external ports that the Layer 2 source address of the packet to be forwarded must be replaced with the source address of the external port. Once the changes have been made to the network layer header, in particular, the portion that includes the time to live (TTL) field, the inbound subsystem compensates the packet's header check sum value in block 717. The inbound subsystem 410 then hands off copies of the packet to the external and internal ports of the inbound subsystem 410 that are identified in the subsystem ports field 347 of the associated memory as corresponding to the matching type 2 entry, as in block 719.
In the case where a copy of the packet traverses an internal link and arrives at a different subsystem 420 in block 720, operation proceeds with decision block 721 where a second control signal, here called the distributed flow (DF or distrib.sub.-- flow) bit, may be received by the outbound subsystem 420. If the DF bit is set, then a class filter determines the class of the packet, based upon the packet's headers, and a type 2 search (with the identified class) is conducted in block 722.
The distrib.sub.-- flow construct allows the CPS to define a type 2 entry in the outbound subsystem 420 corresponding to the matching multicast route entry in the inbound subsystem. This allows different priorities to be assigned by the CPS to the different external ports that will service the multicast route, to further control queuing granularity for packets traversing the MLDNE. A force.sub.-- be bit (placed by the CPS and obtained after a type 2 search in the outbound subsystem) in the associated data of the matching type 2 entry overrides the priority received over the internal link with the packet, such that the packet will be forced to the lowest priority, thus providing some granularity in queuing at the external ports.
If the distrib.sub.-- flow bit is not set, then a type 1 search is performed on the forwarding memory 423, and the packet is forwarded or flooded accordingly without the type 2 queuing granularity discussed above.
If a matching type 1 or type 2 entry is found, then the packet is handed off to the external ports identified in the associated memory corresponding to the matching entry. Thereafter, operation proceeds with block 723. Thus, a multicast route requires two type 2 entries to be created by the CPS where the inbound and outbound subsystems are different.
The operations from block 723 to block 729 are performed by the outbound subsystem, be it the subsystem 410 or subsystem 420. The outbound subsystem in decision block 723 determines whether the sa.sub.-- replace bit has been set to indicate that the Layer 2 source address of each copy of the packet should be replaced with the Layer 2 address of the corresponding external port used for forwarding the packet outside the MLDNE. If not, then the packet may be forwarded using a Layer 2 search result.
If there is an indication to replace the Layer 2 source address for routing purposes, then in block 725, the outbound subsystem, in particular an external port of the outbound subsystem, replaces the Layer 2 source address of the packet with a Layer 2 address of the external port. Operation then proceeds with block 727 where a CRC is recomputed for the modified Layer 2 header, and the packet is forwarded in block 729.
An innovative structure and method for transmitting the packet and control information across the internal link will now be described with reference to FIGS. 8A and 8B. FIG. 8A is a simplified diagram of the packet structure utilized. More particularly, as the inbound subsystem has determined certain information regarding the packet, e.g., routing, it is advantageous to simply convey this information to the outbound subsystem so that subsequent processing, such as the header field replacement, can easily be performed without reperforming the same steps performed by the inbound subsystem. Furthermore, it is desirable to maintain end-to-end error robustness. Thus, the inbound subsystem encapsulates the packet 800 with control information 805 and a cycle redundancy code (CRC) 810. The outbound system receives the encapsulated packet, determines frame validity using CRC 810, strips the CRC 810 and removes the control information 805 to determine the subsequent processing to be performed to output the packet.
The control information includes information to instruct the outbound subsystem how to update the header information, if needed, before output. In the present embodiment, the control information includes the following:
replace.sub.-- sa--when set, indicates that the source address field of the header is to be replace with the outbound subsystem's output MAC address;
orig.sub.-- tag--when set, indicates that the VLAN tag is the original tag the packet arrived with at the inbound subsystem;
mod.sub.-- tag--when set, indicates that the VLAN tag the packet arrived with has been modified;
dont.sub.-- tag--when set, indicates that the VLAN tag may not to be used regardless of the state of the orig.sub.-- tag and the mod.sub.-- tag (in the present embodiment, this is typically used when packets arrive from the CPS 260);
distributed.sub.-- flow--when set, indicates whether a Layer 3 or Layer 2 search should be conducted initially for the packet;
priority (2)--indicates the queuing priority level in the subsystem external ports for the particular packet;
reserved (9)
A simplified block diagram illustrating the process for header field replacement of packets communicated through internal links is illustrated in the diagram of FIG. 8B. For purposes of explanation, a number of functional elements not relevant to the process of performing header field replacement are not shown or described. However, it is readily apparent to one skilled in the art that the inbound subsystem includes elements to process the received packet prior to transmission to the outbound system and the outbound system includes elements that perform other function in addition to those described herein.
Referring to FIG. 8A, the inbound system 825 receives the packet and accesses the memory containing the database (not shown) to obtain information regarding the packet, e.g., if the packet is to be routed or if VLAN routing is supported. Certain control information is generated and provided to the cascading output process (COP) 835 which prepends the control information to the packet and outputs the packet with the prepended control information to the output interface 840 which generates and appends a CRC to encapsulate the packet for output to the outbound subsystem 830. Preferably the output interface is a media access controller (MAC); however, other interfaces could be used.
The outbound subsystem 830 receives the encapsulated packet at the input interface 845, which is preferably a MAC, performs frame validity checking and strips the CRC. The input interface 845 outputs to the cascading input process (CIP) 850 the packet stripped of the CRC and the CIP 850 removes the control information and forwards the packet, stripped of the encapsulating CRC and control information, to the packet memory 855. The control information is stored in the control field 857 corresponding to the packet stored in the memory 855. The output port process 860 retrieves the packet and the control information from the packet memory 855 and based upon the control information, selectively performs modifications to the packet and issues control signals to the output interface 865 (i.e., MAC).
In one embodiment, which occurs when the packet is to be routed, the OPP 860 strips the last 4 bytes of the packet corresponding to the CRC and asserts control signals to the MAC 865 to append a CRC and replace the source address with its own MAC address. For example, the OPP 860 issues a replace.sub.-- SA signal and clears a no.sub.-- CRC bit in a control word sent to the MAC 865. In another embodiment, when VLAN routing is supported, depending upon the state of the control signals, the OPP 860 removes the VLAN tag field in the packet, strips the last 4 bytes of the packet corresponding to the CRC and issues a control signal to the MAC 865 to append a CRC. More particularly, the OPP 860 decodes, orig.sub.-- tag, mod.sub.-- tag and dont.sub.-- tag and a fourth indicator, tag.sub.-- enable. Tag.sub.-- enable is an internal variable which indicates that the network segment connected to this output port does not support VLAN tagging. This variable is determined by a network management mechanism based on the underlying network topology. The result of the decoding process indicates whether the OPP 860 is to strip the tag and whether the MAC 865 is to generate a CRC. The OPP decodes according the following table:
Thus if the tag is to be stripped, the OPP 860 removes the tag, preferably as the tag is transferred to the MAC 86. If no CRC is to be generated, the OPP 860 sends a signal indicating that no CRC is to be generated (e.g., set no.sub.-- CRC) and the MAC 865 transmits the packet as it is received. If the CRC is to be generated, the last 4 bytes are removed from the packet by the OPP 860 a signal to generate the CRC is sent to the MAC 865, (e.g., clear no--CRC).
The MAC 865, based upon the control signals received from the OPP 860, replaces the source address field with its own MAC address and generates a CRC that is appended to the end of the packet as the packet is output.
The encapsulation process can potentially extend the packet by a number of bytes. This can negatively affect the capacity of the link. In order to compensate for this capacity loss and also to allow the reception of frames that may longer than standard protocols define, the protocol parameter (in the present embodiment the Ethernet protocol) are fine tuned to reduce the preamble size by 5 bytes, the interpacket gap by 5 byes and increase the maximum packet size by 10 bytes.
The embodiments of the routing apparatus and methods in the MLDNE 201 described above for exemplary purposes are, of course, subject to other variations in structure and implementation within the capabilities of one reasonably skilled in the art. Thus, the details above should be interpreted as illustrative and not in a limiting sense.
Claims
What is claimed is:
1. In a network element for receiving and forwarding packets between nodes, the network element having first and second subsystems coupled by an internal link, the subsystems having separate first and second forwarding memories, respectively, a method for relaying packets comprising the steps of: receiving a packet by the first sub-system, the packet having a first header portion; determining whether the packet should be forwarded in accordance with a routing protocol in response to parsing the first header portion of the packet; searching the first forwarding memory for a first entry that matches a second header portion of the packet; replacing part of the first header portion of the packet with a next hop address in response to the second header portion of the packet matching the first entry, the next hop address being associated with the first entry, the next hop address being an address of a neighbor node; sending the packet having the next hop address to the second subsystem through the internal link; and forwarding the packet to the neighbor node in response to matching the next hop address in the first header portion of the packet with an entry in the second forwarding memory.
2. A method as in claim 1 wherein the first header portion comprises a Layer 2 destination address, and the second header portion comprises a Layer 3 destination address.
3. A method as in claim 1 wherein the step of determining whether the packet should be forwarded includes determining whether the first header portion matches an address assigned to the network element.
4. A method as in claim 1 further comprising sending a control signal to the second subsystem over the internal link in response to the second header portion matching the first entry; and prior to forwarding the packet replacing part of a third header portion of the packet with an address of the second subsystem in response to the second subsystem receiving the control signal.
5. A method as in claim 4 wherein the third header portion comprises a Layer 2 source address.
6. A method as in claim 1, wherein the step of: sending the packet to the second subsystem includes the step of sending the packet to an internal port of the first subsystem, the internal port coupling the internal link, the internal port being identified by a value associated with the first entry.
7. A method as in claim 4 wherein the step of: replacing a third header portion of the first packet includes replacing part of the third header portion with a L2 address of a second external port in the second subsystem.
8. A method as in claim 1 further comprising the step of: updating a time to live field of the packet in response to determining by the first subsystem that the packet be forwarded in accordance with a routing protocol; and compensating a header checksum of the packet by the first subsystem.
9. A method as in claim 4 further comprising the step of: computing a cyclic redundancy code (CRC) of the packet by the second subsystem after replacing part of the third header portion of the packet.
10. A method as in claim 4 further comprising the step of: inserting a Virtual local area network IDentification (VID) of the second subsystem into the packet in response to receiving a NEW TAG notification over the internal link.
11. A network element for receiving and forwarding multicast packets between nodes, comprising: first subsystem having a first forwarding memory, the first subsystem configured to determine whether a multicast packet should be multicast routed based upon a first entry in the first forwarding memory matching a header portion of the packet, the first subsystem further including a multicast route indication associated with the first entry; second subsystem having a second forwarding memory being separate from the first forwarding memory, the second forwarding memory including a second entry; and an internal link coupling the first and second subsystems for passing the multicast packet from the first subsystem to the second subsystem, wherein the second subsystem is configured to forward a plurality of packets to a plurality of neighbor nodes in response to receiving the multicast packet over the internal link, the second entry matching the header portion of the packet, and replacing a third header portion of each of the plurality of packets with an address of the second subsystem.
12. A network element as in claim 11 wherein the first and second entries each comprises network layer addresses.
13. A network element as in claim 11 wherein the third header portion of each of the plurality of packets comprises a data link layer source address.
14. A network element as in claim 11 wherein the second subsystem receives a second control signal over the internal link from the first subsystem, in response to which a search of the second forwarding memory is conducted.
15. A network element as in claim 14 wherein the search results in the second entry matching the header portion of the packet.
16. A network element as in claim 11 wherein the internal link is further configured to pass queuing priority information from the first subsystem to the second subsystem.
17. A network element as in claim 14 wherein the search results in a third entry matching the header portion, the second subsystem further including a third control indication associated with the third entry in response to which the second subsystem overrides queuing priority indication received from the first subsystem.
18. A router comprising: first subsystem having a first forwarding memory, the first subsystem determines whether a received packet should be routed based upon a first header portion of the packet matching an address of the router; second subsystem having a second forwarding memory being separate from the first forwarding memory; and an internal link coupling the first and second subsystems, the first subsystem replaces part of the first header portion with a next hop address in response to matching a second header portion of the packet with a first entry in the first forwarding memory, the next hop address being an address of a neighbor node, the first subsystem sending the packet with the next hop address to the second subsystem over the internal link, and the second subsystem forwards the packet to the neighbor node in response to the next hop address in the first header portion matching a second entry in the second forwarding memory.
19. A router as in claim 18 wherein the address of the network element is a layer 2 address of an external port of the first subsystem that receives the packet.
20. The router of claim 18 further comprising third subsystem having a third forwarding memory and being coupled to the first and second subsystems via first and second internal links, respectively, a second packet being received by the second subsystem, the second subsystem replaces part of a third header portion of the second packet with a second next hop address of a second neighbor node in response to matching part of a fourth header portion of the second packet with a third entry in the second forwarding memory, the second subsystem sends the second packet including the second next hop address to the third subsystem over the second internal link, and the third subsystem forwards the second packet to the second neighbor node in response to the second next hop address in the third header portion matching a fourth entry in the third forwarding memory.
Patent Citations (87)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4652874 | 1987-03-01 | Loyer | |
| US4807111 | 1989-02-01 | Cohen et al. | |
| US4850042 | 1989-07-01 | Petronio et al. | |
| US4899333 | 1990-02-01 | Roediger | |
| US4922503 | 1990-05-01 | Leone | |
| US4933938 | 1990-06-01 | Sheehy | |
| US5150358 | 1992-09-01 | Punj et al. | |
| US5210746 | 1993-05-01 | Maher et al. | |
| US5220562 | 1993-06-01 | Takada et al. | |
| US5231633 | 1993-07-01 | Hluchyj et al. | |
| US5251205 | 1993-10-01 | Callon et al. | |
| US5278830 | 1994-01-01 | Kudo | |
| US5291482 | 1994-03-01 | McHarg et al. | |
| US5293379 | 1994-03-01 | Carr | |
| US5301333 | 1994-04-01 | Lee | |
| US5309437 | 1994-05-01 | Perlman et al. | |
| US5313454 | 1994-05-01 | Bustini et al. | |
| US5343471 | 1994-08-01 | Cassagnol | |
| US5386413 | 1995-01-01 | McAuley et al. | |
| US5392432 | 1995-02-01 | Engelstad et al. | |
| US5394402 | 1995-02-01 | Ross | |
| US5410540 | 1995-04-01 | Aiki et al. | |
| US5410722 | 1995-04-01 | Cornaby | |
| US5422838 | 1995-06-01 | Lin | |
| US5425028 | 1995-06-01 | Britton et al. | |
| US5426736 | 1995-06-01 | Guineau, III | |
| US5450399 | 1995-09-01 | Sugita | |
| US5455820 | 1995-10-01 | Yamada | |
| US5457681 | 1995-10-01 | Gaddis et al. | |
| US5459714 | 1995-10-01 | Lo et al. | |
| US5459717 | 1995-10-01 | Mullan et al. | |
| US5461611 | 1995-10-01 | Drake, Jr. et al. | |
| US5461624 | 1995-10-01 | Mazzola | |
| US5473607 | 1995-12-01 | Hausman | |
| US5477537 | 1995-12-01 | Dankert et al. | |
| US5481540 | 1996-01-01 | Huang | |
| US5485455 | 1996-01-01 | Dobbins et al. | |
| US5485578 | 1996-01-01 | Sweazey | |
| US5490139 | 1996-02-01 | Baker et al. | |
| US5490252 | 1996-02-01 | Macera et al. | |
| US5500860 | 1996-03-01 | Perlman et al. | |
| US5515376 | 1996-05-01 | Murthy et al. | |
| US5535202 | 1996-07-01 | Kondoh | |
| US5555405 | 1996-09-01 | Griesmaer et al. | |
| US5561666 | 1996-10-01 | Christensen et al. | |
| US5570365 | 1996-10-01 | Yodhida | |
| US5572522 | 1996-11-01 | Calamvokis et al. | |
| US5583981 | 1996-12-01 | Pleyer | |
| US5592476 | 1997-01-01 | Calamvokis et al. | |
| US5594727 | 1997-01-01 | Kolbenson et al. | |
| US5600641 | 1997-02-01 | Duault et al. | |
| US5602841 | 1997-02-01 | Lebizay et al. | |
| US5608726 | 1997-03-01 | Virgile | |
| US5610905 | 1997-03-01 | Murthy et al. | |
| US5619500 | 1997-04-01 | Hiekali | |
| US5619661 | 1997-04-01 | Crews et al. | |
| US5633865 | 1997-05-01 | Short | |
| US5636371 | 1997-06-01 | Yu | |
| US5640605 | 1997-06-01 | Johnson et al. | |
| US5649109 | 1997-07-01 | Griesmer et al. | |
| US5651002 | 1997-07-01 | Van Seters et al. | |
| US5675741 | 1997-10-01 | Aggarwal et al. | |
| US5684800 | 1997-11-01 | Dobbins et al. | |
| US5691984 | 1997-11-01 | Gardner et al. | |
| US5706472 | 1998-01-01 | Ruff et al. | |
| US5720032 | 1998-02-01 | Picazo, Jr. et al. | |
| US5724358 | 1998-03-01 | Headrick et al. | |
| US5726977 | 1998-03-01 | Lee | |
| US5734865 | 1998-03-01 | Yu | |
| US5740171 | 1998-04-01 | Mazzola et al. | |
| US5740175 | 1998-04-01 | Wakeman et al. | |
| US5740375 | 1998-04-01 | Dunne et al. | |
| US5742604 | 1998-04-01 | Edsall et al. | |
| US5742760 | 1998-04-01 | Picazo, Jr. et al. | |
| US5745048 | 1998-04-01 | Taguchi et al. | |
| US5748905 | 1998-05-01 | Hauser et al. | |
| US5751967 | 1998-05-01 | Raab et al. | |
| US5754540 | 1998-05-01 | Liu et al. | |
| US5754801 | 1998-05-01 | Lambrecht et al. | |
| US5757771 | 1998-05-01 | Li et al. | |
| US5757795 | 1998-05-01 | Schnell | |
| US5764634 | 1998-06-01 | Christensen et al. | |
| US5781549 | 1998-07-01 | Dai | |
| US5784573 | 1998-07-01 | Szczepanek et al. | |
| US5790546 | 1998-08-01 | Dobbins et al. | |
| US5802047 | 1998-09-01 | Kinoshita | |
| US5802052 | 1998-09-01 | Venkataraman |
Non-Patent Literature (19)
- International Search Report, PCT/US98/13368, 5 pages.
- International Search Report, PCT/US98/13364, 4 pages.
- International Search Report, PCT/US98/13365, 4 pages.
- International Search Report, PCT/US98/13177, 4 pages.
- International Search Report, PCT/US98/13199, 5 pages.
- International Search Report, PCT/US98/13015, 5 pages.
- Wang et al., A Novel Message Switch for Highly Parallel Systems, IEEE, pp. 150-155, 1989.
- Tobagi, Fast Packet SwitchArchitectures for Broadband Integrated Services Digital Networks, Proceedings of the IEEE, vol. 78, Issue 1, pp. 133-167, Jan. 1990.
- Fliesser et al., Design of a Multicast ATM Packet Switch, Electrical and Computer Engineering, 1993 Canadian Conference, pp. 779-783, 1993.
- Chang et al., An Overview of the Pipelined Common Buffer Architecture (PCBA) for Memory Based Packet/Cell Switching Systems, Local Computer Networks, 1994, pp. 288-297, 19th Conference, IEEE.
- Agrawal et al., A Scalable Shared Buffer ATM Switch Architecture, VLSI, 1995 5th Great Lakes Symposium, IEEE, pp. 256-261, 1994.
- Sabaa et al., Implementation of a Window-Based Scheduler in an ATM Switch, Electrical and Computer Engineering, 1995 Canadian Conference, IEEE, pp. 32-35, 1995.
- Naraghi-Pour et al., A Multiple Shared Memory Switch, System Theory, 1996 Southeastern Symposium, IEEE, pp. 50-54 1996.
- Iyengar et al., Switching Prioritized Packets, GLOBECOM '89: IEEE Global Telecommunications Conference, pp. 1181-1186, 1989.
- "Foundry Products", downloaded from Website http:/ /www.foundrynet.com/ on Jun. 19, 1997.
- Anthony J. McAuley & Paul Francis, "Fast Routing Table Lookup Using CAMs", IEEE, 1993, pp. 1382-1390.
- "Gigabit Ethernet", Network Strategy Report, The Burton Group, v2, May 8, 1997 40 pages.
- "IP On Speed", Erica Roberts, Internet-Draft, Data Communications on the Web, Mar. 1997, 12 pages.
- "Multilayer Topology", White Paper, Internet-Draft, 13 pages, downloaded from website http:/ /wwwbaynetworks.com on Apr. 18, 1997.