Distributed multi-link trunking method and apparatus

[0002] This invention relates generally to data communications networks, and more particularly, to a method and apparatus for trunking of physical ports that reside on different switches within the network.

[0003] Computer networks are used to interconnect many computing resources, including computers, workstations, servers, printers, modems, and storage devices. For example, two or more computers may be connected together through a network. Network users are able to share files, printers and other resources, send messages and run applications on remote computers. An important part of any computer network includes the physical components or network communications devices used to interconnect the computing resources.

[0004] One network communications device used to interconnect multiple computing resources is a chassis-based system designed to accommodate a number of internal cards. The computing resources such as computers are coupled to the internal cards of the chassis-based system. Once the computers are coupled to the internal cards, the computers are coupled to the network. To accommodate network growth, additional internal cards may be purchased and added to the chassis-based system. With the additional internal cards installed, additional computing resources may be added to the network. A disadvantage of these chassis-based systems is that there is a relatively large initial investment when purchasing the chassis because of the large amount of overhead associated with chassis-based systems.

[0005] An alternative to expensive chassis-based systems is the use of less expensive standalone network communications devices or units that have a fixed number of ports for connecting computing resources or stations to the network. Such standalone network communications devices include stackable switches or the like. Although additional ports can not be added to each individual standalone unit, separate standalone units can be stacked, cascaded or coupled to accommodate network growth. As a result, there is a lower startup cost when establishing a computer network with the standalone units in comparison with chassis-based systems. Furthermore, network administrators still have the flexibility to increase the size of the network with the less-expensive standalone units.

[0006] FIG. 1 illustrates multiple network communications devices 103, 105, 107, 109 and 111 coupled together to form a computer network 101. Multiple computing resources (not shown) are coupled to each network communications device 103, 105, 107, 109 and 111. In one embodiment, network communications devices 103, 105, 107, 109 and 111 are stackable switches coupled together through bus 113. Bus 113 is used to tie together the switch network fabric of computer network 101. It are noted by one of ordinary skill in the art that the utilization of bus 113 is an extension of the chassis-based designs discussed earlier. The internal cards of the chassis-based systems are commonly coupled to high speed buses within the chassis-based systems.

[0007] The use of stackable switches allows network administrators to build systems having multiple physical ports to a computing resource with a high data bandwidth requirement, such as a network file server. Multiple physical ports to a single resource is known as multi-link trunking, and allows load balancing, where the network traffic to a single computing resource can be shared across the physical ports to the resource.

[0008] Known systems allow trunking to a single computing resource only if the multiple physical ports to the resource are all connected to the same network switch. However, it is common for a single computing resource to have physical ports to more than one switch. If the physical links on different switches cannot be trunked, the available bandwidth of the extra links is often wasted. In addition, unless data frames can be sent through local trunks, bandwidth is wasted because duplicate frames typically must be sent to all switches on the network. Therefore, there is a need to enable trunking of physical ports that reside on different switches within the network. There is a future need to forward data frames through local trunks.

[0009] The present invention provides a method and apparatus for providing data communication between a source station having multiple connections to a first switch and a destination station having multiple connections to a second switch. A trunk identifier to each port on the first switch and each port on the second switch. A data frame is encoded with the trunk identifier for an ingress port on the first switch. The data frame is sent to the second switch from the first switch. A list of egress ports for the destination station is obtained from a station list contained in the second switch. An egress port is selected from the list of egress ports based upon the source address, destination address and trunk identifier. The data frame is sent to the destination station through the selected egress port.

[0010] The present invention is illustrated by way of example and may be better understood by referring to the following description in conjunction with the accompanying drawings, in which like references indicate similar elements and in which:

[0011] FIG. 1 shows a computer network coupled together with a bus architecture compatible with the present invention;

[0012] FIG. 2 shows a computer network with optimized stack memory compatible with the present invention;

[0013] FIG. 3 shows a content addressable memory (CAM) cycle compatible with the present invention;

[0014] FIG. 4 shows a packet bus and stack header compatible with the present invention;

[0015] FIG. 5 shows a preferred CAM memory organization compatible with the present invention;

[0016] FIG. 6 is a data flow diagram of ingress port processing compatible with the present invention;

[0017] FIG. 7 shows an example network system compatible with the present invention.

[0018] In the following description of an embodiment, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration a specific embodiment in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention. An embodiment of the present invention, described below, enables a remote computer system user to execute a software application on a network file server.

[0019] The present invention provides a way of directing a data frame to a remote link across a local data trunk to optimize the data bandwidth for a network switch. The forwarding process within an embodiment of the present system is distributed so that packet processing is performed at both the port level as well as by the content addressable memory (CAM) on the packet bus. The stack operation introduces a further level of distribution so that packets which are forwarded up the stack are processed by the CAM and ports in each destination unit.

[0020] Glossary

[0021] The following terms, phrases and definitions are used throughout the specification to describe an embodiment of the present invention.

[0022] CAM—Content addressable memory.

[0023] CAM CYCLE—A packet control word generated by the CAM for each packet indicating its destination.

[0024] CSMA/CD—Carrier sense multiple access collision detect.

[0025] DA—Destination address.

[0026] MAC—Media access control. The MAC address is that which matches either the DA or SA lookup. The same entry is used for both DA and SA though the remaining information in the entry is used in different ways.

[0027] Policy—Typically a 13-bit index used to access DA based policies for forwarding and filtering.

[0028] SA—Source address.

[0029] Source Port—The port number where an address was learned or migrated to. This is used for DA based forwarding.

[0030] TCP/IP—Transmission Control Protocol/Internet Protocol. A set of communication protocols developed by the U.S. Department of Defense that allows dissimilar computers to share information over a network.

[0031] Trunk Mask Group—Typically a four bit field that is accessed by the SA lookup to build an index to a trunk mask. This is used to provide SA based conditioning on a forwarding mask that has multiple trunk ports activated.

[0032] VLAN—Virtual local area network.

[0033] VLAN Tag—Virtual local area network tag. The VLAN tag originates from the packet an address was learned at. Each unique VLAN+SA creates a new CAM entry.

[0034] The following terms, phrases and definitions are used throughout the specification to describe trunking in an embodiment of the present invention.

[0035] Server Trunk—A trunk that attaches to a server that utilizes the same MAC address on each of its links. Network and Server Trunks can be either bounded or distributed.

[0036] Network Trunk—A trunk that attaches to another inter networking device, yielding a variety of addresses seen on each of the trunk members. The same source addresses can be seen on one or more of the trunk members. Network and Server Trunks can be either bounded or distributed.

[0037] Server Trunk—A trunk that attaches to a server that utilizes the same MAC address on each of its links. Network and Server Trunks can be either bounded or distributed.

[0038] Bounded Trunk—All ports that are members of the trunk group reside on the same stack module.

[0039] Bounded Server Trunk—All ports on the same stack module attach to a server.

[0040] Bounded Network Trunk—All ports on the same stack module attach to another switch.

[0041] Distributed Trunk—Port members of the trunk group can be distributed on different stack modules.

[0042] Distributed Server Trunk—Port members of the trunk group are distributed on different stack modules and attach to the server.

[0043] Distributed Network Trunk—Port members of the trunk group are distributed on different stack modules and attach to the switch.

[0044] Hardware Environment

[0045] As shown in FIG. 2, an embodiment of the present invention shows how a network 201 having network switches 203, 205, 207, 209, and 211 are coupled with a network bus 225. These switches may be stacked, allowing ports to be readily added to the network. Each switch contains a CAM 213, 221 which is used to direct data frames between ports on a switch or between local ports and remote switches, and to store address, port and other network information. A data transfer port 217, 219, allows the transfer of information and data frames between network switches. The operation of the CAM is discussed in greater detail below.

[0046] The present invention optimizes the memory of a network switch by storing only the port information associated with the switch and information about the other switches on the network. In one embodiment of the present invention, a 64-bit CAM cycle word is used to indicate the source and destination ports information for each packet transitioning a switch, as shown in the example in FIG. 3. Using a 64-bit word, it is possible to indicate, for example, the presence of 48 ports on the switch and eight switches total on the stack. The lower 56 bits of the 64-bit CAM cycle word, shown as bits 0-55, are preferably used to indicate the routing information for a data frame. The first 48 bits of the 64-bit CAM cycle word, shown as bits 0-47, are preferably used to indicate the destination ports, while the next eight bits, shown as bits 48-55, are preferably used to indicate the destination of up to eight switches. The remaining eight bits are used for other system and administrative purposes. The total number of ports available in a network using this mechanism is 48×8, or 384 ports.

[0047] The description of the present invention uses an example 64-bit CAM cycle word for illustration purposes only. It will be recognized that the number of ports recognizable on a switch and the maximum number of switches on a network may be increased or decreased by changing size of the CAM cycle word or changing the contents of the CAM cycle word without loss of compatibility with the present invention. It will also be recognized that the contents of the CAM cycle word may be reordered or rearranged without loss of compatibility with the present invention. For example, by using the first 40 bits to indicate ports and the next 16 bits to indicate switches, it is possible to have 40×16, or 640 ports on the network. In an additional example, by using a 128-bit CAM cycle word size, eight bits for administrative purposes, 104 bits for ports and 16 bits for switches, it is possible to have 104×16, or 1,664 ports on the network.

[0048] FIG. 4 shows an packet bus header and stack header which optimizes the memory of a network switch according to the present invention. In one embodiment of the present invention, a 64-bit header is used to indicate the source and destination stack only information for each set of data frames transmitted on the switch or stack. The header preferably provides six bits for the source port (SRC_PORT), shown as bits 44-49, and six bits for the destination port (DST_PORT), destination is only on the stack, this is ascertained from the cam cycle and appended to the stack header. The stack header is a combination of packet bus header and the cam cycle, shown as bits 50-55.

[0049] The description of the present invention uses an example 64-bit packet bus and stack header word for illustration purposes only. It will be recognized that the number of ports recognizable on a switch and the maximum number of switches on a network may be increased or decreased by changing size of the packet bus and stack header word or changing the contents of the packet bus and stack header word without loss of compatibility with the present invention. It will also be recognized that the contents of the packet bus and stack header word may be reordered or rearranged without loss of compatibility with the present invention.

[0050] Forwarding System Overview

[0051] In general, the type of processing that occurs in the present system can be broken down to the following sections: Port Ingress, Local CAM, Stack Egress/Ingress, Remote CAM, and Port Egress. The type of processing that occurs at each point is described in more detail below.

[0052] Port Ingress

[0053] The port ingress function is responsible for the proper reception and classification of packets. The information derived from this classification is reflected in the packet bus header that is provided by the port for each received packet. In general, the port provides the following functions:

[0054] Store and forward of packets from the physical interface to packet bus interface

[0055] Ingress port identification

[0056] Error detection and classification

[0057] Packet size indication

[0058] VLAN identification

[0059] Filtering Database Identification for shared VLANs

[0060] Multilink trunking membership identification

[0061] Spanning Tree state identification

[0062] Local CAM

[0063] The CAM function is responsible for the direction of packets towards their destination port or the filtering of packets that do not meet the forwarding criteria. The CAM accomplishes this by providing a CAM control word on the packet bus following the data phase of each packet transferred across the packet bus. This is referred to as the CAM cycle. There is significant functionality in the CAM to provide a rich set of features in the system. The CAM cycle is the end result of that processing. The CAM cycle provides the following information:

[0064] Egress port(s) identification

[0065] Flooded packet identification

[0066] The CAM function, described in greater detail below, provides the following features:

[0067] Storage of up to 32K unique 60 bit VLAN/MAC Address or address pairs indexed by a hash key

[0068] Natural learning of VLAN/MAC addresses at line rate

[0069] Known unicast address forwarding to learned port

[0070] Unknown unicast address flooding

[0071] Flooding of Broadcast and Unregistered Multicast packets

[0072] Support for forwarding registered multicast to port set

[0073] Support for port level egress VLAN filtering

[0074] Explicit Multilink Trunking support

[0075] Support of an 8K policy table that implements:

[0076] Up to 4K forwarding and filtering policies with no Multilink Trunk support

[0077] Up to 2K forwarding and filtering policies and up to 2 Multilink Trunks

[0078] Up to 1K forwarding and filtering policies with up to 6 Multilink Trunks

[0079] Ingress security for allowed ports based on SA

[0080] Stack Egress/Ingress

[0081] The stack interface subsystem is responsible for transferring packets across the stack to other switch units for further processing. The stack interface subsystem projects the packet bus information across the stack to maintain ingress and some local CAM data with the packet. The functions performed by the stack interface is:

[0082] Store and forward function onto the stack

[0083] Mapping other stack members into the local port space

[0084] Collapsing local ports onto the stack behind its stack address

[0085] Remote CAM

[0086] The remote CAM function provides all the functions of the local CAM function but is responsible for the secondary lookup that resolves a forwarding decision to one or more of its local ports. The two stage lookup that occurs in the CAM system has the local CAM resolving the packet forwarding decision to the address of a stack member. Upon reception of the packet by the remote stack member the remote CAM further resolves the forwarding decision to individual physical ports.

[0087] Port Egress

[0088] Port egress functions are the last step in the forwarding process. The egress port is responsible for accepting a packet off of the packet bus if it meets the egress criteria and forwarding it to the physical media for transmission. In particular the egress port provides the following functions:

[0089] Store and forward of packets from the packet bus to the physical interface

[0090] Tagging or not tagging packets with VLAN information based on certain criteria

[0091] Spanning tree blocked port support

[0092] Error checking on transmitted packets

[0093] CAM Description

[0094] A preferred CAM is incorporated by an application specific integrated circuit (ASIC). It is responsible for providing forwarding decisions for all packets appearing on the Packet Bus. The CAM operates by reading the packet header and DA/SA to formulate forwarding decisions.

[0095] As shown in FIG. 5, a preferred CAM utilizes the 64K×64 memory attached to the ASIC. The CAM memory is subdivided into four sections: Address entries, a policy table containing Forwarding/Filtering/Trunk Masks and two sets of VLAN filter masks.

[0096] Address entries are comprised of 96 bits and are grouped by four into a hash bucket yielding 8K hash buckets. Each hash bucket is comprised of six 64-bit words that contain the address information for the four address that are contained within. The address entries are not contiguous—four partial addresses are contained in the first two words and the remaining locations contain specific information for each of the four addresses including the remainder of the address and its VLAN tag. Specifically, the address entries contain the following information: 48 bits address, 12 bit VLAN tag, 13 bit policy, 4 bit trunk group and various other control bits as shown in FIG. 5.

[0097] The data contained in a valid entry is used along with data from the Packet Bus header to provide a forwarding decision. The forwarding decision is used to build a CAM cycle on the Packet Bus to steer the packet to its egress port(s). Table 1 below provides a list of CAM entry data.

[0098] The fields and definitions of the CAM entry data shown in Table 1 are described in greater detail below.

[0099] MAC Address—This is the MAC address that matches either the DA or SA lookup. The same entry is used for both DA and SA though the remaining information in the entry is used in different ways.

[0100] VLAN Tag—VLAN tag from the packet the address was learned at. Each unique VLAN+SA creates a new CAM entry.

[0101] Policy—A 13 bit index used to access DA based policies for forwarding and filtering.

[0102] Source Port—The port number where this address was learned or migrated to. This is used for DA based forwarding.

[0103] Trunk Mask Group—A four bit field that is accessed by the SA lookup to build an index to a trunk mask. This is used to provide SA based conditioning on a forwarding mask that has multiple trunk ports activated.

[0104] DA Based—Indicates that this address should be forwarded without the use of a trunk mask, thus always sending it to the last port learned for this address.

[0105] Filtered—Indicates that the SA is filtered for this egress port based on static filtering information. This bit is updated from the filter mask every time it migrates.

[0106] CAM VLAN Table, Hash and Lookup

[0107] The CAM preferably uses a 61 bit hash that is comprised of the 48 bit MAC address along with 12 bits of the VLAN tag. The DA and SA are hashed in the same manner. Thus, if an address is learned in one VLAN as an SA, it will use the same CAM entry when the address is seen as a DA in a packet with the same VLAN tag. The resultant hash function creates a 61 bit result, from which a 13 bit remainder is selected as a hash bucket index.

[0108] During this phase of the CAM cycle, VLAN lookups are being performed into the two VLAN mask tables to prepare for the final CAM forwarding decision. The VLAN lookup returns a 64 bit value from both tables. This contains a 56 bit port mask that contains the registered ports for that VLAN.

[0109] Once the hash function generates lookup indices for the DA and SA the CAM memory is referenced. An exact match is performed against the four addresses stored in the hash bucket for both the DA and SA referenced buckets. If either the DA or SA do not create an exact match then a secondary lookup is performed using a modification of the hash remainder. Once a valid entry is confirmed secondary lookups are initiated to reference the policy table.

[0110] CAM Forwarding Decision

[0111] The CAM forwarding decision uses information from both the DA and SA lookup. The final result that is placed on the packet bus during the CAM cycle is a 56 bit port mask plus several control bits. The 56 bit port mask that results from the CAM Forwarding decision can be logically described as:

[0112] CAM Cycle Port Mask=VLAN MASK*TRUNK MASK*(FORWARDING MASK+PORT#{circumflex over ( )}*/Filtered)

[0113] Each element of the above equation is derived from the CAM entry, indirect mask accesses and the VLAN tag, and is described in greater detail below.

[0114] The VLAN port mask is a 56-bit value that is accessed from one of the two VLAN tables. The first is used for all known multicasts, known and unknown unicasts, the second VLAN table is used when the DA is an unknown multicast. Each of the two tables consists of 4K 56-bit entries that are directly referenced by the VLAN tag.

[0115] The trunk mask is accessed from the 8K policy table that is shared between the three types of port masks. The trunk mask is used to select one of up to four parallel paths in a multi-link trunk. The trunk mask is a function of the SA as well as the ingress port. More details are given below in the trunking section. The trunk mask can also be used to disallow a SA from forwarding out the switch by providing an all zeros default for unregistered addresses.

[0116] The forwarding mask is also accessed from the policy table and is used to force traffic on specific links. The forwarding mask is used in conjunction with the trunk mask to provide multi-link trunking. The forwarding mask defines all ports that are groved into a multi-link trunk. This defines all the possible paths to a destination. Therefore, it is referenced by a destination look up. The trunk mask is referenced by the packet source information which includes the source station address and packet ingress ports. Thus the exact port forwarding decision is the logical “and” of the trunk mask with the forwarding mask. The Forwarding mask is also used to implement the static table force function.

[0117] The port number vector expansion represents the port that the address was learned on as an SA.

[0118] The filtered bit represents the logical OR of two functions: 1) whether this address has been specified as filtered for the port it was learned on. This information is derived from the specific filter mask in the policy table. The filtered bit and forwarding mask implement the static table entry function of the 802.1q specification. 2) Whether the VLAN mask indicates that ingress port is not registered for that particular VLAN. The VLAN ingress registration check can be enabled on a per port basis within the CAM.

[0119] Unknown Destination Addresses

[0120] Unknown destination addresses are flooded, or sent out to all ports, based on whether they are multicast or unicast. The flooding occurs by setting the Forwarding mask to all ones and applying the SA referenced trunk mask. Different VLAN masks are used for unknown multicasts over unknown unicasts. The forwarding and trunk masks are used to generate the “all ports” mask that is then further constrained by the VLAN mask. The resultant vector is the port mask that identifies all the ports that this packet should be flooded on for the VLAN it is in.

[0121] Learning

[0122] The basic learning operation of the CAM occurs when the hashed (SA concatenated with VLAN indexes a CAM entry that is either empty or does not contain the address—VLAN pair as one of the valid addresses. In that case a new entry is created and has the 48 bit address, VLAN and the port it was seen on written into it. The CAM defaults are written to the other fields. The default filter action bit as set or cleared (filter or not) based on the global policy. The default policy is initialized to all zeros concatenated with up to the three most significant bits of the trunk ID from the packet bus header. The default trunk group is initialized to all 1's.

[0123] Migration

[0124] Migration is detected when a source address has moved from one port to another port unless the learned and migrated to port were part of the same trunk group. The single to single port migration is detected when the source port number changes. The Trunk migration occurs when the Trunk ID changes. When a migration is detected the CAM updates the port number (non-trunked), trunk bits in the policy, filtered bit from the filter mask pointed to by the policy.

[0125] Stack Addressing and Collapsed Ports

[0126] The CAM architecture provides for stacking multiple units together to emulate a single switch. To extend the port addressing capabilities of the system, the CAM cycle provides special meaning for the upper 8 bits of the 56 bit port vector. These 8 bits represent the stack port vector—one for each member of the stack. Thus, to send a packet over the stack to another unit, the CAM sets the corresponding bit in stack port vector.

[0127] Addresses are learned from remote units with the remote unit's stack address rather than the port number that the packet came in originally. The effect here is to collapse or “hide” all the remote ports behind the stack address of the unit they are on. It is the responsibility of the local CAM on each unit to further resolve the forwarding decision of a packet received from the stack. When a CAM has learned an address on one of its local ports or has a registered address that resolves to a local port it is said to “own” that address with respect to the system.

[0128] Trunking

[0129] The trunking capabilities of the CAM architecture will support up to six trunks group with each trunk supporting up to four trunk members. There are two basic switching requirements for trunking. The first is the ability to treat multiple links as a single one for the purposes of learning and migration, and the second is to be able to select one of the member paths as the destination for a forwarding decision with no duplicate packets sent. Forwarding decisions in the CAM architecture allow the path selection in a trunk group to be based on one of the following:

[0130] A single port associated with the DA—this can be either learned or programmed.

[0131] Any port in the trunk group where that DA has been learned or programmed. The port selection is conditioned by the SA and/or the ingress port of the specific packet.

[0132] There are three basic components involved in the trunking forwarding operation. These are the Trunk Identifier, the Policy Index and the Trunk Mask Group, described in greater detail below.

[0133] Trunk Identifier

[0134] The Trunk Identifier is assigned to ports to indicate that they are members of a trunk group or to assign non-trunked ports an offset to a trunk mask index. The identifier is 6 bits and is assigned on a per port basis. This identifier appears in the packet bus header for each packet and is used by the CAM as part of the policy index into the policy table. It is defined as:

[0135] Bit 5 indicates whether the trunk identifier is part of a trunk group or a trunk mask offset. When bit 5 is zero, the Trunk Identifier is used as a trunk mask offset:

[0136] When bit is one, the Trunk Identifier is indicating trunk group membership.

[0137] The values 000000 to 011111 are assigned to identify non-trunked ports and used to offset into trunk masks. The values 101000 to 111111 are used for trunks—Either two or six trunks are allowed as shown below in Table 2:

[0138] DA Referenced Policy

[0139] The CAM architecture supports a selection of three different trunking configurations: no trunks, two trunk groups and six trunk groups. Each of these configurations has effects on the organization of the policy table. When no trunks are configured, the table can hold 4K policies, where each policy contains a forwarding mask and a filter mask. The two trunk configuration reduces the available policies to 2K, with each policy containing 3 forwarding masks and a filter mask. The six trunk configuration further reduces the policies available to 1K with each policy containing 7 forwarding masks and a filter mask.

[0140] The CAM entry contains a 13 bit policy index field per MAC address. The DA references the policy index to provide the forwarding mask that defines the port set for the trunk group that the address is known on. This policy field is stored in one of three formats. If the SA is learned or registered to a non-trunk port, or trunking is disabled the 13 bits contain a 12 bit policy index (4K policies). If the address is learned or registered to a trunk group, assuming trunking is enabled, the LSB's are modified by the trunk group identifier.

[0141] Trunk Mask Group Identifier

[0142] The Trunk Mask Group ID is the third component of the trunking mechanism. The Trunk Mask Group ID is referenced by the SA lookup of the CAM entry. The four bits are assigned to each SA to allow it to select from a set of Trunk Masks. The index into the Trunk Mask Table is comprised of the Trunk Mask Group ID, the Trunk Group ID/Unit ID and some fixed bits. The Trunk Mask Index is defined below in Table 4:

[0143] The Policy Table

[0144] The policy table is organized in three different ways based on no trunks, 2 trunks and 6 trunks. The trunk configuration selects the organizations. For no trunks the table is fully populated with forward/filter policies as defined below in Table 5:

[0145] When trunking is enabled, the trunk masks are also stored in the policy table. Trunk masks grow from the top of the table (highest addresses) and forward/filter policies grow from the bottom (lowest addresses). The policy table can hold up to 1K trunk masks. These take away from the maximum number of forward/filter that can be realized. This is shown below in Table 6 and Table 7.

[0146] Trunking Operation

[0147] The basic operation of trunking can be explained using the methods described above. The system shown in FIG. 7 represents a three unit stack with four trunks defined. This requires that the policy table be configured for six trunks and 1K policies. Unit 0 contains Trunk 0, a bounded trunk comprised of ports 1, 2, 3 and 4. Unit 2 contains two bounded trunks each with two links as trunk members. Ports 1 and 2 comprise Trunk 1 and ports 3 and 4 comprise Trunk 2. The final trunk, Trunk 3, is a distributed trunk with two trunk members, port 12 on both Unit 0 and 1. The remaining ports are singular and not members of any trunks.

[0148] Referring to the mechanisms described above—the trunk group identifier, DA referenced policy and Trunk Mask Id, the units can be configured as follows. The first assignment is the Trunk ID for each port in the system. This requires that the overall stack be initialized and that trunk assignments across the stack be consistent and synchronized such that no duplicate assignments are made. Once the ports are assigned to Trunk Ids, the policy tables on each CAM must be initialized to correspond with the desired trunking operation.

[0149] The policy tables must define the forwarding masks and trunk masks for each unit's local CAM. For the purpose of illustration and to keep the example port vectors small, the example system is defined to support only 12 local ports and 4 stack ports with a 16 bit port mask. Table 11 below corresponds to FIG. 7. The example system must also have a global policy as to whether units that are sending packets to distributed trunks utilize a single trunk member per source unit or load balance across members of the trunk group. The first option always sends packets to the trunk link on same unit for a destination trunk no matter what ingress port or source address. This also pertains to the case where the local ports always send to the local trunk member. The other option is to have different ports or source addresses on the ingress unit send to different members of a distributed trunk group.

[0150] This affects the forwarding masks for distributed trunks on the ingress box. If the first policy of using only the local trunk link member is chosen then the forwarding mask for destination trunks that are distributed always has one and only one path available—either the local port for locally originated packets or one unit that has a member of the trunk group.

[0151] For the option that allows different egress per source port/address, the forwarding masks have all possible egress paths enabled. The local lookup then applies a trunk mask to disable a local path. The implication of this is that extra bandwidth is utilized on the stack to accomplish this load balancing. In the example given Unit 2 can load balance between Unit 0 and Unit 1 for Trunk 3. The trunk masks on Unit 2 must be set up to allow both units to be valid egress paths for ingress packets—otherwise non-trunked ports on the other units could never be egress ports. But to do so requires that the CAMs in both Unit 0 and Unit 2 must be set up to filter the trunk ports based on ingress port/source address. This causes packets for distributed trunk destinations to be received by both units and one unit to filter locally while the other transmits the packet. The wasted bandwidth is the packet bus bandwidth of the unit that filters the packet.

[0152] While the invention is described in terms of preferred embodiments in a specific system environment, those of ordinary skill in the art will recognize that the invention can be practiced, with modification, in other and different hardware and software environments within the spirit and scope of the appended claims.