TCP offload device that load balances and fails-over between aggregated ports having different MAC addresses

[0001] The present application claims the benefit under 35 USC 120 from U.S. patent application Ser. No. 09/801,488, filed Mar. 7, 2001. The subject matter of U.S. application Ser. No. 09/801,488 is incorporated herein by reference.

[0002] Compact Disc Appendix, which is a part of the present disclosure, is one recordable Compact Disc (CD-R) containing information that is part of the disclosure of the present patent document. The Compact Disc contains source code for a port aggregation driver in accordance with an embodiment of the present invention. A portion of the disclosure of this patent document contains material that is subject to copyright protection. All the material on the Compact Disc is hereby expressly incorporated by reference into the present application. The copyright owner of that material has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights.

[0003] Port aggregation (also called link aggregation) is a method by which two or more network connections (sometimes called links) are grouped together at a multi-port network device to create a single logical connection (sometimes called a single logical link). Network traffic can be load balanced between multiple aggregated ports of the multi-path network device, thereby increasing bandwidth available for the single logical connection without having to increase the bandwidth of any of the associated underlying network connections. Typically associated with port aggregation is another method called port fail-over. If functionality of one physical port is lost, then another port assumes the functionality of the lost port such that the logical connection is maintained. Such a port fail-over method provides increased reliability for the single logical network connection, in comparison with the reliability of a single physical network connection.

[0004] FIG. 1 (Prior Art) is a diagram of a local device 1 that supports port aggregation, load balancing and fail-over functionality. In the illustrated example, local device 1 is both to receive network information from and to transmit network information to a remote device 2. Local device 1 includes a host portion 3 as well as a network interface device portion 4. The network interface device portion 4 may take the form of an Intelligent Network Interface Card (INIC) providing multiple media access and control (MAC) ports. Host portion 3 typically includes a central processing unit (CPU) that executes a multi-layer protocol processing stack 5. Stack 5 includes a network interface device driver 6, a port aggregation driver 7, an IP protocol processing layer 8, a TCP protocol processing layer 9, and one or more upper layers 10. In the example of FIG. 1, the port aggregation driver and the network interface device driver are together considered to be a network access layer 11 of the stack.

[0005] Incoming packets received onto local device 1 are received onto network interface device 4 and then pass through the NID device driver 6 and the port aggregation driver 7 and then up through the various layers of the stack 5 to upper layer 10. In the case of a read, for example, the upper layer 10 of the stack may be an Internet SCSI (ISCSI) layer that identifies a location in destination storage 12 where the data is to be placed. The data is transferred in the form a multi-packet message. Each packet of the message is received by local device 1, passes through network interface device 4, and is processed by each successive layer of the protocol stack 5, thereby often consuming significant processing resources of the CPU that executes the stack. This often time consuming protocol processing through successive layers of the protocol stack is called slow-path processing.

[0006] To offload the CPU of this protocol processing load, special functionality is built into network interface device 4 and a special ATCP stack portion 13 is added to stack 5. Accordingly, certain types of network communications are handled in a fast-path whereby IP and TCP protocol processing tasks are handled by the network interface device 4, thereby offloading the CPU of the host of these tasks. In the example of a read, the data payloads of the various packets of the message are placed directly into the destination storage 12 at a location identified by upper layer 10. The data payloads are placed into destination storage 12 without stack 5 doing any or substantially any IP or TCP protocol processing. For additional information on such a network interface device, on an ATCP stack, and on fast-path operation, see the discussion of the INIC and the associated ATCP stack in U.S. Pat. No. 6,247,060 (the subject matter of which is incorporated herein by reference).

[0007] To support port aggregation, load balancing and fail-over functionality, a Fast-Etherchannel switch 13 is provided such that local device 1 can send or receive information to remote device 2 via two ports (Port 1 or Port 2) and two physical connections 14 and 15, rather than just one port and one physical connection. In the case where Fast-Etherchannel switch 13 is a rack-mounted Hewlett-Packard HPJ4121A ProCurve 4000M switch, both Port 1 and Port 2 are assigned to have the same media access address (MAC address A in this example). Both of the ports Port 1 and Port 2 are grouped together into a team for port aggregation purposes. If transmission and reception for a TCP connection were passing over physical connection 14 and Port 1, and then communications over physical connection 14 were for some reason to fail, then transmission and reception for the TCP connection would be communicated via physical connection 15 and Port 2. Although this scheme works in that fast-path communications are supported in combination with port aggregation, load balancing and fail-over functionality, the need to provide the Fast-Etherchannel switch is undesirable. Fast-Etherchannel switches may be expensive. A solution is desired.

[0008] A multi-port TCP offload network interface device (NID) supports both fast-path and slow-path operations. The TCP offload device is coupled to a host. The host executes a stack of protocol processing layers.

[0009] In fast-path operation, IP and TCP protocol processing of particular TCP network connections is performed by the NID, thereby offloading the stack of the host of the task of performing IP and TCP protocol processing on network traffic over these TCP connections. In the case of a multi-packet upper layer (session layer or application layer) message being received onto the NID across a TCP connection being handled in fast-path, the NID performs all or substantially all TCP and IP protocol processing on the incoming packets, and places the data payloads of the packets free of TCP and IP headers directly into a destination on the host. The location of the destination on the host is, in one embodiment, identified by an upper layer program running on the host. In the case of the transmission of a packet over a TCP connection being handled in the fast-path, the NID stores a template header for the TCP connection. The NID obtains from the host the data to be transmitted, splits up the data into segments, adds a header to each segment of data using the template header, and then transmits the resulting packets from the NID.

[0010] In slow-path operation, on the other hand, the host stack performs substantial IP and TCP processing. The NID receives incoming packets from the network and forwards the packets to the host stack for further processing in the case of a reception. In the case of transmission, the host stack forms packets, passes these packets to the NID, and the NID transmits them onto the network.

[0011] In accordance with one embodiment of the present invention, such a multi-port TCP offload NID handles a TCP connection in fast-path over a plurality of aggregated ports (members of a team) such that each of the aggregated ports uses a different MAC address. Fail-over mechanisms are described for failing over and resuming fast-path operation after a failure is detected on a port used to receive packets of the fast-path TCP connection. Fail-over mechanisms are also described for failing over and resuming fast-path operation after a failure is detected on a port used to transmit packets of the fast-path TCP connection. In each of these fail-over situations, traffic passing over the TCP connection is initially load balanced in fast-path between multiple ports of the team, a failure then occurs on one of the ports, processing of the TCP connection then switches to slow-path, and then the function of the failed port is automatically assumed by another functioning port of the team such that fast-path processing of the TCP connection resumes. The failed port and the port that assumes its functionality use different MAC addresses. Fast-path load balancing and/or fail-over are therefore possible without the use of a Fast-Etherchannel switch that would have required the ports to use the same MAC address. A specific embodiment of port aggregation software is disclosed that manages the MAC addresses of the team members.

[0012] Other embodiments and details are also described below. This summary does not purport to define the invention. The claims, and not this summary, define the invention.

[0013] FIG. 1 (Prior Art) is a diagram of a prior art fast-path network interface device (NID) that supports port aggregation, load balancing and fail-over by the use of a Fast-Etherchannel switch.

[0014] FIG. 2 is a simplified diagram of system 100 in accordance with an embodiment of the present invention, wherein a fast-path TCP offload network interface device (NID) 103 supports port aggregation, load balancing and fail-over without the use of a Fast-Etherchannel switch.

[0015] FIG. 3 is a diagram of a list of teams (actually it is a list of PAG_ADAPT structures, where each PAG_ADAPT structure in the list identifies a team). This list is maintained by a port aggregation driver in accordance with an embodiment of the present invention.

[0016] FIG. 4 is a flowchart of a method in accordance with an embodiment of the present invention where a receive port fails.

[0017] FIG. 5 is a flowchart of a method in accordance with an embodiment of the present invention where a transmit port fails.

[0018] FIG. 6 is a flowchart of method steps that occur after the steps of FIG. 4 in accordance with one embodiment.

[0019] FIG. 7 is a flowchart of method steps that occur after the steps of FIG. 5 in accordance with one embodiment.

[0020] FIG. 2 is a diagram of system 100 in accordance with one exemplary embodiment of the present invention. System 100 includes a local device 101 comprising a host portion 102 and a TCP offload network interface device (NID) portion 103. Local device 101 receives TCP/IP network communications from and sends TCP/IP network communications to a remote device 104 via an intermediary device 105. Intermediary device 105 may, for example, be a Fast Ethernet hub or switch. In the example of FIG. 2, intermediary device 105 is a switch. In contrast to the system set forth in FIG. 1, this switch 105 is not a Fast Etherchannel switch but rather is a less complex (and hopefully less expensive) Fast Ethernet switch. Switch 105 may, for example, be a NetGear FS108 switch. Switch 105 is connected to local device 101 via two physical connections 106 and 107. Network connections 106 and 107 may, for example, be CAT5 network cables. Physical connection 106 is coupled to Port 1 of network interface device 103 whereas physical connection 107 is coupled to Port 2 of network interface device 103.

[0021] Host portion 102 includes a CPU that executes a multi-layer network protocol processing stack 108. Stack 108 includes an upper layer 109 (for example, an application layer and/or a session layer), a TCP layer 110, an IP layer 111, a port aggregation driver 112, a network interface device (NID) device driver 113, and an ATCP stack portion 114. An ISCSI layer is an example upper layer 109. The port aggregation driver 112 and the NID device driver 113 may together be considered to comprise a network access layer 115 of stack 108. Layers 109-111 are, in one embodiment, part of the stack of a Microsoft Windows operating system.

[0022] For additional details on a port aggregation driver, see U.S. patent application Ser. No. 09/801,488, filed Mar. 7, 2001, including the subject matter on its Compact Disc Appendix (the subject matter of which is incorporated herein by reference). For additional details on a stack that supports fast-path operations, see U.S. patent application Ser. No. 10/208,093, filed Jul. 29, 2002, entitled Protocol Processing Stack For Use With Intelligent Network Interface Device, by Peter Craft et al.

[0023] Network communications flowing through an ordinary network interface card (NIC) are typically processed by the various layers of a stack, one layer at a time. If in the example of FIG. 2 the CPU of host 102 were to execute the code of the various layers of the stack for all the network communications received onto and passing out of network interface device 103, then the CPU might be significantly burdened. Having to perform the network processing for all the layers of the stack would likely have the undesirable effect of reducing the amount of processing power available to carry out other desired functions.

[0024] To offload the CPU of such sequential network protocol processing tasks, TCP offload NID 103 includes special hardware and software. This special hardware and software provides a fast-path whereby all or substantially all the TCP and IP protocol processing for particular TCP network connections is performed by TCP offload NID 103 rather than by host 102.

[0025] For each such TCP connection to be handled in fast-path, an associated Transmit Control Block (TCB) is maintained on TCP offload NID 103. (The TCB used for fast-path processing is also sometimes referred to as a Communication Control Block (CCB.)) The TCB is a block of information that stores the state of the TCP connection. The TCB contains MAC source and destination addresses for the local and remote device respectively, the IP source and destination addresses of the connection, and the TCP source and destination ports of the connection. The TCB in the present example also contains a list of buffer addresses identifying a location in destination storage 116 on host 102, information on the length of a message being communicated across the TCP connection, TCP sequence numbers reflecting packets received and transmitted and acknowledged, TCP variables such as timers and receive and transmit windows for sliding window protocols, and status information regarding each of the protocol layers of the message.

[0026] TCP offload NID 103 uses the IP and TCP source and destination addresses of the TCB to identify packets. For example, when a packet is received onto NID 103 from switch 105, the NID 103 examines the headers of the packet. If the packet headers have IP source and destination addresses and TCP source and destination ports that match the IP source and destination addresses and the TCP source and destination ports in one of the TCBs (for example, TCB 117) maintained on NID 103, then the packet is identified as being associated with the TCP connection associated with TCB. If this match occurs and if the packet does not give rise to one of a set of error and/or exception conditions, then the packet is processed via the fast-path. For additional information on fast-path operation, see U.S. Pat. No. 6,247,060 (the subject matter of which is incorporated herein by reference).

[0027] In the case where a large block of data is to be transferred from remote device 104 to a location in destination storage 116 via the fast-path, the block of data is sectioned up and transferred from remote device 104 to NID 103 in the form of a plurality of packets. NID 103 receives each of packets, determines that the packet matches the associated TCB, removes the header information from the packet's data payload, and then DMA transfers the data payload directly to a section of destination storage 116 identified by upper layer 109. The data is placed into destination storage 116 by NID 103, free of headers. The data is therefore placed into the destination storage 116 without the stack 108 having to perform any or substantially any TCP and IP protocol processing. The operation of the NID 103 and ATCP stack 114 is simplified in the description above. For a more detailed description of the structure and operation of a network interface device that performs fast-path processing, and of the structure and operation of an ATCP stack, see: 1) U.S. Pat. No. 6,247,060, and 2) U.S. Published patent application 20,010,047,433, published Nov. 29, 2001 (the subject matter of these documents is incorporated herein by reference).

[0028] FIG. 4 is a flowchart illustrating a method in accordance with one embodiment of the present invention wherein a port fails that is used to receive network communications for a fast-path TCP connection. The method is carried out by the system of FIG. 2.

[0029] In an initial state (step 400), TCP/IP packets associated with TCP connection 1 are being received onto NID 103 via Port 1 and are being transmitted from NID 103 via Port 2. This operation is set forth to describe one type of load balancing that is implemented with fast-path processing. For example, if the receive from TCP connection 1 were to represent a significant amount of traffic and if the transmit from TCP connection 1 were also to represent a significant amount of traffic, then having both the receive and transmit for connection 1 occurring over the same physical connection 106 might increase latencies and delays and might overburden the available bandwidth of physical connection 106 and/or the associated ports. In the example of FIG. 2, however, the receive for connection 1 is set up to occur over physical connection 106 and the transmit for connection 1 is set up to occur over physical connection 107, thereby better distributing the traffic over the two physical connections 106 and 107.

[0030] Load balanced TCP connection 1 is being handled in fast-path. Network interface device 103 therefore has control of a TCB 117 associated with the TCP connection 1. When in fast-path, NID 103 carries out fast-path commands. Fast-path commands may, for example, include a HANDOUT1 command (the first phase of a TCB handout indicates the intention of host 102 to hand over control of a connection to NID 103,) a HANDOUT2 command (the second phase of a TCB handout that actually commits NID 103 to fast-path operation), a SENDMDL command (a transmit request), a RECEIVEMDL command, a RECEIVEINTERMEDIATEMDL command, a WINDOWUPDATE command (a transmit null), a FLUSH command (a release context), and a CLOSE command (a close request). For additional information on these commands, see U.S. patent application Ser. No. 10/208,093, filed Jul. 29, 2002, entitled Protocol Processing Stack For Use With Intelligent Network Interface Device, by Peter Craft et al. (the subject matter of which is incorporated herein by reference).

[0031] Next (step 401), a failure occurs on Port 1 (the port being used to receive). Such a failure may, for example, be due to the physical connection 106 becoming unplugged. In one embodiment, MAC hardware on NID 103 detects that the media has lost its carrier. In response to detecting this failure on Port 1, NID 103 completes execution of any active command for Port 1 with flush status (error status). A link status from NID 103 is then passed up through the NID device driver 113 and reaches the port aggregation driver 112.

[0032] Next (step 402), upon receiving this link status, port aggregation driver 112 updates a list of PAG_ADAPT structures. Each PAG_ADAPT structure in the list identifies and defines a team. Port aggregation driver 112 updates the appropriate team to indicate that the port designated as the primary port is failed and will change, and that the MAC addresses of Port 1 and Port 2 will be swapped.

[0033] FIG. 3 is a diagram of part of this list of PAG_ADAPT structures. This list is maintained by port aggregation driver 112 in the form of a linked list. Each team comprises a number (one or more) of team members. The team members are ports. In the exemplary method of FIG. 4, team 1 (see FIG. 3) represents the team associated with connection 1. Initially Port 1 is used as the receive port for TCP connection 1. Port 1 is therefore initially designated by a pointer to be the primary port. Upon the failure of Port 1, port aggregation driver 112 chooses another of the ports of the team (Port 2 in this example) to be the new primary team member. Port aggregation driver 112 therefore moves the pointer to indicate that Port 2 is the new primary team member. The moving of the pointer is indicated in FIG. 3 with the dashed arrow.

[0034] In addition to changing the primary team member, port aggregation driver 112 also swaps the MAC addresses of the two ports so that the MAC address previously associated with Port 1 is now associated with Port 2, and such that the MAC address previously associated with Port 2 is now associated with Port 1. Port aggregation driver 112 does this swapping by writing MAC A into the MAC address field for the Port 2 member, and by writing MAC C into the MAC address field for the Port 1 member.

[0035] Next (step 403), port aggregation driver 112 causes NID 103 to change the MAC address of Port 1 from MAC A to MAC C and to change the MAC address of Port 2 from MAC C to MAC A. Port aggregation driver 112 does this by calling a Microsoft operating system function, called the NDIS request function. NDIS stands for network driver interface specification. The calling of the NDIS request function is a request for the operating system to do the designated NDIS function. The NDIS request has multiple parameters including a handle, and a pointer. The handle in this case identifies a port, the MAC address of which is to be changed. The pointer in this case points to a data structure, one subfield of which identifies another data structure. The other data structure in turn identifies an OID. In the present example, the OID (OID_ADD8023_ADDRESS) is a private OID that identifies the MAC address that the port identified by the handle is to use. Accordingly, port aggregation driver 112 calls the NDIS request function a first time to change Port 1's MAC address from MAC A to MAC C, and calls the function again to change Port 2's MAC address from MAC C to MAC A.

[0036] Next (step 404), the change of the MAC address on Port 2 of NID 103 causes NID 103 to initiate a flush by executing the flush command. Execution of the flush command causes the TCB for the connection (TCB 117) to be passed back to host 102. ATCP stack 114 learns of the flush, takes control of TCP connection 1, and processing of TCP connection 1 switches from fast-path to slow-path.

[0037] Next (step 405), ATCP stack 114 in slow-path causes port aggregation driver 112 to have NID 103 send out a gratuitous Address Resolution Protocol (ARP) request to switch 105. This ARP request serves to advertise to switch 105 that the MAC address (MAC address A) that previously identified the failed Port 1 is now to be associated with Port 2. This gratuitous ARP request is sent because some intelligent switches, upon detecting the same MAC address on two physical ports, may function to isolate both ports. An isolated port is a failed port to the switch, and all communications from the isolated port would be ignored. NID 103 therefore sends the gratuitous ARP request out via Port 2 to cause such a sophisticated switch to reconfigure itself to know that MAC A is now associated with Port 2. In the case where device 105 is a less sophisticated switch or hub that does not attempt to isolate ports in this manner, receipt by device 105 of the gratuitous ARP request has no consequence. The method therefore works both in the case of a relatively sophisticated switch as well as in the case of a less complex switch or hub.

[0038] If transmit for TCP connection 1 is to remain on Port 2, and if Port 2 is now going to be associated with MAC A, then the MAC source address in TCB 119 must be changed from MAC C to MAC A. Accordingly, port aggregation driver 112 updates the TCB 119 (step 406) for TCP connection 1 by changing the MAC source address to be the MAC address of the selected member chosen for transmission. In the present example, this involves changing the MAC source address in TCB 119 from MAC C to MAC A. (The MAC destination address in TCB 119 remains MAC B).

[0039] Next (step 407), fast-path operation is resumed for TCP connection 1 such that Port 2 both receives and transmits packets for TCP connection 1. Host 102 passes control of TCP connection 1 back to NID 103 in a two-phase handout process. In the first phase (called HANDOUT1), host 102 instructs NID 103 to store and not forward to NID 103 subsequent packets for the specified TCP connection. In this case, the specified TCP connection is TCP connection 1 to be handed to the NID 103.

[0040] NID 103 receives this instruction, and responds back to host 102 with an acknowledgment. In the second phase (called HANDOUT2), host 102 passes control of the TCP connection 1 to NID 103 by moving (or copying) the TCB from memory on host 102 to memory on NID 103. The updated version of TCB 119 is illustrated in FIG. 2 as existing on NID 103 as TCB 117. Fast-path operation then automatically resumes for TCP connection 1 such that Port 2 both receives and transmits packets for TCP connection 1. NID 103 uses TCB 117 to build its transmit packets for connection 1 in accordance with teachings set forth in U.S. Pat. No. 6,247,060 (the subject matter of which is incorporated herein by reference).

[0041] Accordingly, a multi-port TCP offload NID performs initial fast-path processing of traffic for a TCP connection. A port being used to receive traffic for the TCP connection then fails. The MAC addresses of the failed port and a remaining functional port are swapped. The change of the MAC address on the new primary port causes the TCP connection to switch from fast-path processing to slow-path processing. The multi-port NID transmits an ARP request from one of its remaining functional ports to advertise that the MAC address that was on the failed port is now on another functioning port. The multi-port NID then resumes fast-path operation for the TCP connection but in this resumed fast-path operation the NID does not use the failed port. This fail-over functionality does not require a complex and sometimes unduly expensive Fast-Etherchannel switch. The MAC address of the NID seen by the switch for the TCP connection does not change, but rather moves from the failed port on the NID to another functioning port on the NID.

[0042] FIG. 5 is a flowchart illustrating a method in accordance with another embodiment of the present invention. In this embodiment, a port fails that is used to transmit network communications for a fast-path TCP connection.

[0043] In an initial state (step 500,) TCP/IP packets associated with TCP connection 1 are being received onto NID 103 via Port 1 and are being transmitted from NID 103 via Port 2 as in the prior example. As in the prior example, TCP connection 1 is being handled in fast-path.

[0044] Next (step 501), a failure occurs on Port 2 (the port being used to transmit TCP connection 1 packets from NID 103). Such a failure may, for example, be due to the physical connection 107 becoming unplugged. In one embodiment, MAC hardware on NID 103 detects that the media has host its carrier. In response to detecting this failure on Port 2, NID 103 completes execution of any active commands on Port 2 with flush status (error status). The NID device driver 113 also completes any pending commands for Port 2 with flush status. NID 103 does a flush to transfer TCB 117 back to host 102. TCB 117 is written to memory on host 102 as indicated in FIG. 2 by reference number 119. Control of TCP connection 1 passes (step 502) from NID 103 to host 102 and processing for this TCP connection switches from fast-path to slow-path. A link status from NID 103 is then passed up through the NID device driver 113 and reaches the port aggregation driver 112.

[0045] Next (step 503), upon receiving this link status, the port aggregation driver 112 updates the appropriate team to indicate that Port 2 designated as secondary port is failed. The TCB 119 for TCP connection 1 holds a template header to be used by NID 103 in fast-path for the transmission of fast-path packets. This template header includes a MAC source address field to identify the MAC address of the transmitting port. In the present example, the port used to transmit is going to change from Port 2 to Port 1. Port aggregation driver 112 therefore updates the TCB 119 for TCP connection 1 by writing the MAC address of MAC A into the MAC source address field of TCB 119. (The MAC destination address in TCB 119 remains MAC B).

[0046] Once the TCB for TCP connection 1 has been updated, fast-path operation for TCP connection 1 is resumed (step 504) such that Port 1 (designated as MAC A) both receives and transmits packets for TCP connection 1. In one embodiment, host 102 passes control of TCP connection 1 to NID 103 in the two-phase handout process described above. In the first phase, host 102 uses the HANDOUT1 command to instruct NID 103 to store incoming packets for TCP connection 1. NID 103 then responds back to host 102 with an acknowledgement. In the second phase, host 102 uses the HANDOUT2 command to send the updated TCB to NID 103 and to resume fast-path operation of TCP connection 1. The updated TCB is stored on NID 103 as TCB 117.

[0047] Accordingly, a multi-port TCP offload NID performs initial fast-path processing of traffic for a TCP connection. A port being used to transmit traffic for the TCP connection then fails. This causes processing for the TCP connection to switch from fast-path processing to slow-path processing. The multi-port NID then resumes fast-path operation for the TCP connection but in this resumed fast-path operation the NID does not use the failed port. This fail-over functionality does not require a complex and sometimes unduly expensive Fast Etherchannel switch. The MAC addresses of the NID seen by the switch for the TCP connection does not change.

[0048] FIG. 6 is a flowchart showing steps that occur, in one embodiment, after the steps, of the method of FIG. 4. At the conclusion of the method of FIG. 4, TCP connection 1 is being handled in fast-path. Port aggregation driver 112 is using the new primary team member (Port 2, MAC A) for reception and transmission after failure of the former primary member (Port 1, MAC C).

[0049] Next (step 600), the former primary member (Port 1, MAC C) recovers. Such a recovery may, for example, be due to the physical connection 106 being plugged in. In one embodiment, MAC hardware on NID 103 senses its media (detects that the media has its carrier present and determines the configuration of the link). A link status from NID 103 is then passed up through the NID device driver 113 and reaches the port aggregation driver 112. Upon receiving this link status, the port aggregation driver 112 updates the appropriate team to indicate that Port 1 (the port formerly designated as primary, but now designated as secondary) is operational.

[0050] Next (step 601), TCB 117 for TCP connection 1 is flushed back to host 102 and processing of TCP connection 1 switches from fast-path to slow-path. The flush may either be initiated by the ATCP stack 114 or be initiated by NID 103 due to a non-fatal reason. An example of a non-fatal reason is exception handling normally performed by the ATCP stack 114 in slow-path.

[0051] Next (step 602), the ATCP stack 114 handles the exception and restarts migration of TCP connection 1 from slow-path to fast-path. During the first phase of migration, the port aggregation driver 113 updates the TCB 119 for TCP connection 1 by changing the MAC source address to be the MAC address of the selected member chosen for transmission (Port 2, MAC A). The MAC destination address in TCB 119 remains MAC B.

[0052] Next (step 603), the second phase of migration of TCP connection 1 completes and control of TCP connection 1 passes from ATCP stack 114 to NID 103. Fast-path processing of TCP connection 1 resumes. The new primary member (Port 2, MAC A) is used for both reception and transmission.

[0053] Accordingly, a multi-port TCP offload NID performs initial fast-path processing of traffic for a TCP connection. A port which failed recovers and returns to operation. The TCP connection switches from fast-path processing to slow-path processing to handle an exception condition. The multi-port NID resumes fast-path operation for the TCP connection but in this resumed fast-path operation the NID does not use the recovered port for transmission.

[0054] FIG. 7 is a flowchart showing steps that occur, in one embodiment, after the steps of the method of FIG. 5. At the conclusion of the method of FIG. 5, TCP connection 1 is being handled in fast-path. Port aggregation driver 112 is using the primary team member (Port 1, MAC A) for reception and transmission after failure of the former secondary member (Port 2, MAC C).

[0055] Next (step 700), the secondary member (Port 2, MAC C) recovers. Such a recovery may, for example, be due to the physical connection 107 being plugged in. In one embodiment, MAC hardware on NID 103 senses its media (detects that the media has its carrier present and determines the configuration of the link.) A link status from NID 103 is then passed up through the NID device driver 113 and reaches the port aggregation driver 112. Upon receiving this link status, the port aggregation driver 112 updates the appropriate team to indicate that Port 2 (the port designated as secondary) is operational.

[0056] Next (step 701), TCB 117 for TCP connection 1 is flushed back to host 102 and processing of TCP connection 1 switches from fast-path to slow-path. The flush may either be initiated by the ATCP stack 114 or be initiated by NID 103 for a non-fatal reason. An example of a non-fatal reason is exception handling normally performed by the ATCP stack 114 in slow-path.

[0057] Next (step 702), the ATCP stack 114 handles the exception and restarts migration of TCP connect 1 from slow-path to fast-path. During the first phase of migration, the port aggregation driver 113 updates the TCB 119 for TCP connection 1 by changing the MAC source address to be the MAC address of the selected member chosen for transmission (Port 2, MAC C). The MAC destination address in TCB 119 remains MAC B.

[0058] Next (step 703), the second phase of migration of TCP connection 1 completes and control of TCP connection 1 passes from ATCP stack 114 to NID 103. Fast-path processing of TCP connection 1 resumes. The team primary member (Port 1, MAC A) is used for reception and the selected secondary member chosen (Port 2, MAC C) is used for transmission.

[0059] Accordingly, a multi-port TCP offload NID performs initial fast-path processing of traffic for a TCP connection. A port which failed recovers and returns to operation. The TCP connection switches from fast-path processing to slow-path processing to handle an exception condition. The multi-port NID subsequently resumes fast-path operation for the TCP connection but in this resumed fast-path operation the NID does use the recovered port for transmission.

[0060] The Compact Disc Appendix includes an implementation of a port aggregation driver in accordance with an embodiment of the present invention. The CD Appendix includes the following:

[0061] Folder i386 contains the sub-folders created as a result of compilation. Folder obj is created as a result of compilation and contains the object and symbols for a build of the port aggregation driver, either checked or free. Folder objchk is created as a result of compilation and contains the intermediary compilation files for a checked build of the port aggregation driver. Folder objfre is created as a result of compilation and contains the intermediary compilation files for a free build of the port aggregation driver. File buildchk.log is created as a result of compilation and contains a log of a checked build of the port aggregation driver. File buildfre.log is created as a result of compilation and contains a log of a free build of the port aggregation driver. File buildit.bat contains declarations and definitions for the NMAKE and BUILD facilities of the Microsoft program compilation environment. File Makefile contains declarations and definitions for the NMAKE and BUILD facilities of the Microsoft program compilation environment. File netteam.inf contains declarations and definitions for the installation of the port aggregation configurator. File nettmmp.inf contains declarations and definitions for the installation of the port aggregation NDIS miniport driver. File pag.c contains ANSI C logic implementing the main NDIS miniport driver entry point and initialization of the miniport and protocol driver functions to NDIS. File pag.h contains ANSI C logic implementing the basic data structures for the port aggregation driver. File pag.rc contains global version definitions for the ANSI C compilation facilities of the Microsoft program compilation environment. File pagmini.c contains ANSI C logic implementing the NDIS miniport driver functions. The function PSP (within file pagmini.c) modifies the MAC source address in the TCB to be the MAC address of the selected member chosen for transmission. File pagmise.c contains ANSI C logic implementing support functions for the miniport and protocol driver functions. Some support functions (within file pagmisc.c) control the swapping of the primary member upon fail-over when selecting the new primary member. File pagproto.c contains ANSI C logic implementing the NDIS protocol driver functions. File precomp.h contains declarations and definitions for the ANSI C compilation facilities of the Microsoft program compilation environment. File sources contains declarations and definitions for the NMAKE and BUILD facilities of the Microsoft program compilation environment.

[0062] Although the present invention is described in connection with certain specific embodiments for instructional purposes, the present invention is not limited thereto. The load balancing and fail-over techniques set forth above are not limited to TCP connections, but rather apply to the load balancing and fail-over of connections of other transport layer protocols. Rather than control of a TCP connection being passed from the stack to NID 103 by moving the TCB itself, the TCB in one embodiment remains in a fixed location (either on the host or on the NID) and control of the TCB is passed from host to NID or from NID to host. NID 103 may, in some embodiments, be integrated into the host. NID 103 can be part of a memory controller integrated circuit, an input/output (I/O) integrated circuit, a peripheral controller integrated circuit, or a bridge integrated circuit of a microprocessor chip-set. In some embodiments, NID 103 is part of an I/O integrated circuit chip such as, for example, the Intel 82801 integrated circuit of the Intel 820 chip set. INID 103 may be integrated into the Broadcom ServerWorks Grand Champion HE chipset, the Intel 82815 Graphics and Memory Controller Hub, the Intel 440BX chipset, or the Apollo VT8501 MVP4 North Bridge chip. The load balancing and fail over methods set forth above are not only applicable to load balancing and fail over between different ports of a single NID, but rather are also applicable to load balancing and fail over from a port on one NID to a port on another NID, or from a port on one NIC card to a port on another NIC card. The port aggregation driver structures and techniques set forth above are not only usable with and/or in a Microsoft Windows operating system, but rather are also usable with and/in other operating systems including Unix, Linux, Novel, and Apple operating systems. Accordingly, various modifications, adaptations, and combinations of various features of the described embodiments can be practiced without departing from the scope of the invention as set forth in the following claims.