Method and system for processing network and storage data

[0001] 1. Technical Field

[0002] This disclosure relates in general to networking systems, and more particularly, to processing network data and storage data in network systems.

[0003] 2. Related Art

[0004] Networking systems are commonly used to move network information (may also be referred to interchangeably, as frames, packets or commands) between computing systems (for example, servers) or between computing systems and network devices (for example, storage systems). Various hardware and software components are used to implement network communication.

[0005] Different protocols, (e.g., Ethernet, Fibre channel, iSCSI, and the like) may be used to handle network information and storage information. Each protocol has its own advantages in specific application class. Typically, a networking application may use the Ethernet protocol to handle network communication while a storage application may use the Fibre channel protocol to send and receive data from a storage area network (SAN). It is desirable to use a consolidated system, which would allow storage traffic (e.g. Fibre Channel frames) to be transmitted via a network protocol (e.g. Ethernet).

[0006] In one embodiment, a method for transmitting data using a network protocol and a storage protocol via an adapter is provided. The method includes receiving an input output control block (IOCB) from a host system for transferring data stored in a host system memory, wherein the adapter receives the IOCB and determines if an offload operation is to be performed; acquiring data from the host system memory, wherein the adapter acquires the data from the host system based on a memory address embedded in the IOCB; copying a header template in a local memory of the adapter, wherein the header template is created by a driver executed by the host system; creating a header for the network protocol and a header for the storage protocol; wherein a first module for the adapter creates the network protocol packet header and the first module uses an assist module to create the storage protocol packet header; creating a packet to transfer a portion of the acquired data, wherein a packet size is based on a payload size for the storage; and transmitting data packets until a sequence offload is complete.

[0007] In another embodiment, a system for transmitting data using a network protocol and a storage protocol is provided. The system includes a driver executed in a host system creates a header template for each sequence for transferring the data and partially executes functions related to the storage protocol; and an adapter coupled to the host system and a network link, the adapter (a) copies the data and the header template from a host system memory to a local adapter memory; (b) creates a header for the network protocol using a network processing module and a header for the storage protocol using a storage assist module, (c) creates individual packets whose size are limited by a payload size indicated by an input and output control block received from the host system; and (d) transmits the packets to a destination via a network interface using, the network link.

[0008] In yet another embodiment, an adapter or transmitting data using a network protocol and a storage protocol is provided. The adapter includes an interface for communicating with a host system that executes a driver for creating a header template for each sequence for transferring the data and the driver partially executes functions related to the storage protocol; a core module that copies the data and the header template from a host system memory to a local adapter memory; and creates a header for the network protocol; and a storage protocol assist module that creates a header for the storage protocol and interfaces with the core module; wherein the core module creates individual packets whose size are limited by a payload size indicated by an input and output control block received from the host system; and transmits the packets to a destination via a network interface using a network link.

[0009] This brief summary has been provided so that the nature of the disclosure may be understood quickly. A more complete understanding of the disclosure can be obtained by reference to the following detailed description of the embodiments thereof concerning the attached drawings.

[0021] It should be understood that the techniques of the present disclosure described below may be implemented using various technologies. For example, methods described herein may be implemented in firmware executed by a processor or state machine, or implemented in hardware using either a combination of processor or other specially designed application specific integrated circuits, programmable logic devices, or various combinations thereof.

[0022] To facilitate an understanding of the various embodiments, the general architecture and operation of a network system will first be described. The specific architecture and operation of the various embodiments will then be described with reference to the general architecture of the network system.

[0023] Network System

[0024] FIG. 1 shows a block diagram of a generic network system 100, which includes a conventional computing system 102 that can communicate with a plurality of network devices and storage devices. Computing system 102 typically includes several functional components. These components may include a central processing unit (CPU) 104, main memory 106, network interface (NIC Interface) 110, a host bus adapter (HBA) interface (HBA I/F) 112 and other devices (for example, input/output (“I/O”) devices) 114.

[0025] In computing system 102, the main memory 106 is coupled to the CPU 104 via a system bus 108 or a local memory bus (not shown). The main memory 106 is used to provide the CPU 104 access to data and/or program information that is stored in main memory 106 at execution time. Typically, the main memory 106 is composed of random access memory (RAM) circuits. A computer system with the CPU and main memory is often referred to as a host system.

[0026] Network interface 110 is coupled to a network interface card 120 via a bus/link 116 (used interchangeably through out this specification). NIC 120 handles incoming (receive) and outgoing (transmit) network traffic computing system 102 via link 126. The term incoming means network traffic that is received by NIC 120 (for example, from another host system 130 coupled to a local area network (LAN) 128). The term outgoing means network traffic that is transmitted by NIC 120 for computing system 102 to another network device (e.g. host system 130).

[0027] Various network protocols may be used by NIC 120 to handle network traffic. One common network protocol is Ethernet. The original Ethernet bus or star topology was developed for local area networks (LAN) to transfer data at 10 Mbps (mega bits per second). Newer Ethernet standards (for example, Fast Ethernet (100 Base-T) and Gigabit Ethernet) support data transfer rates between 100 Mbps and 10 gigabit (Gb). The description of the various embodiments described herein are based on using Ethernet (which includes 100 Base-T and/or Gigabit Ethernet) as the network protocol, however, the adaptive embodiments disclosed herein are not limited to any particular protocol, as long as the functional goals are met by an existing or new network protocol.

[0028] Host system 102 also uses HBA 122 to communicate with storage systems, for example, 136 coupled to a storage area network (SAN) 132. Link 124 couples HBA 122 to SAN 132. The transfer rate for link 124 continues to increase for example, from 1 Gb to 10 Gb.

[0029] In SANs (for example, 132), plural memory storage devices are made available to various host computing systems. Data in a SAN is typically moved between plural host systems and storage systems (or storage devices (e.g. server 134), used interchangeably throughout this specification) through various controllers/adapters, for example, HBA 122.

[0030] One common standard that is used to access storage systems in a SAN is Fibre Channel. Fibre channel is a set of American National Standard institute (ANSI) standards, which provide a serial transmission protocol for storage and network protocols such as HIPPI, SCSI, IP, ATM and others.

[0031] Fibre channel supports three different topologies: point-to-point, arbitrated loop and fabric. The point-to-point topology attaches two devices directly. The arbitrated loop topology attaches devices in a loop. The fabric topology attaches host systems directly (via HBAs) to a fabric, which are then connected to multiple devices. The Fibre Channel fabric topology allows several media types to be interconnected.

[0032] Fibre Channel fabric devices include a node port or “N_Port” that manages Fabric connections. The N_port establishes a connection to a Fabric element (e.g., a switch) having a fabric port or F_port.

[0033] Host systems often communicate with peripheral devices (for example, HBA 122 via bus 118) via an interface/bus such as the Peripheral Component Interconnect (“PCI”) interface, PCI-X and/or PCI-Express (incorporated herein by reference in its entirety) bus.

[0034] One challenge faced by conventional network systems is efficiently processing a storage packet (for example, Fibre Channel and a network packet (for example, an Ethernet packet. Currently network systems typically use separate NICs and HBAs (for example, 120 and 122) to send and receive network and storage traffic. These separate NICs and HBAs use complex software code, which makes the use of storage and network protocols expensive and cumbersome. The various embodiments disclosed herein overcome these shortcomings by efficiently processing FIBRE CHANNEL and Ethernet frames.

[0035] A new and upcoming standard, called Fibre Channel Over Ethernet (FCOE) is being proposed that intends handle both Ethernet (network) and Fibre Channel (storage) traffic. The current solutions for implementing the standard are not commercially desirable. For example, one proposed solution is to use a Fibre Channel host bus adapter (HBA) technique where a host system hardly does anything besides providing an I/O request and a scatter/gather list to the HBA. The FCOE HBA performs all the other I/O functions. The cost of the FCOE HBAs can prohibitive.

[0036] Another solution uses a simple network interface card (NIC) where the host implements the entire Fibre Channel stack in software. The host software performs Fibre Channel frame packetization and other I/O functionality. This solution consumes host-computing resources and tends to be slow, hence may not be commercially desirable.

[0037] The optimum solution for implementing FCOE is to use an approach that limits cost while providing optimum performance for processing both network and storage data. In one embodiment, a hybrid solution is provided, where a host performs certain functions, for example building an intelligent I/O control block (IOCB) and an ad performs packetization and other functions. This avoids the expense of a full offload to an adapter and provides better performance than a system where a host executes the entire Fibre Channel stack in software.

[0038] The adaptive aspects of the present invention, described herein are based on using the Ethernet and Fibre Channel protocols; however, any other network and storage protocol may be used to implement the various embodiments described herein.

[0039] Network/Storage Adapter

[0040] In one embodiment, an integrated network and storage adapter is provided that can handle both network and storage protocols. The adapter is referred to herein as a Fibre Channel over Ethernet (FCOE) adapter. FCOE adapter can process network (for example, Ethernet) and storage (for example, Fibre Channel) traffic efficiently. FIG. 2A shows a block diagram of an FCOE adapter 200, which is coupled to host system 102 via link 118 (which may be a PCI Express link, and adapter interface 201. FCOE adapter includes a processor 207 that executes firmware instructions out of memory 209 to control overall adapter 200 functionality.

[0041] FCOE adapter 200 interfaces with network 128 via network interface 208. Network interface 208 transmits and receives network packets via link 210.

[0042] FCOE adapter 200 interfaces with host system 102 via host interface 202. In one embodiment, host interface 202 is a PCI Express interface coupled to PCI Express link 118.

[0043] FCOE adapter 200 includes an Ethernet control module (also referred to as Ethernet core) 204, and a Fibre channel assist module (may also referred to as FC assist module) 206. Ethernet core 204, described below in detail with respect to FIG. 2B, processes network data received via link 210 and data received from a host system via host interface 202. The term data as used throughout this specification includes commands, primitives, signals and others.

[0044] FC assist module 206 may include a hardware state machine or logic (not shown) to interface with Ethernet Core 204. As described below, FC assist module 206 performs certain Fibre Channel functions related to incoming and outgoing Fibre Channel frames.

[0045] Ethernet core 204 with assistance from FC assist module 206 builds FCOE packets for sending data from host system 102. Although FC assist module 206 is shown as a separate module from Ethernet Core 204 for clarity purposes, FC assist module 206 may be integrated with Ethernet Core 204.

[0046] Referring now to FIG. 2B, Ethernet core 204 interfaces with network interface 208 for processing network data received or transmitted via link 210. Ethernet core 204 typically includes various functional components. These components may include a processor or hardware state machine (used interchangeably throughout this specification) 220, memory 218, outbound DMA (Direct Memory Access) engine 224 and inbound DMA engine 222.

[0047] Network interface 208 includes a transmit buffer 214, a receive buffer 216 and a MAC layer (medium access control layer) 212. Receive buffer 216 stores incoming network data in a receive path and transmit buffer 218 stores outgoing transmit data in a transmit path. MAC 212 processes data received from receive buffer 216 and/or transmit buffer 214. In one embodiment, MAC layer 212 may be a standard NIC layer that handles incoming and outgoing network information.

[0048] Outbound DMA engine 224 receives data from memory 106 of host system 102 and sends the data to transmit buffer 214. Inbound DMA engine 222 moves data received from the network via link 210 to host system 102. Processor 220 controls the overall functioning of the Ethernet core 204 using data and information stored in memory 218.

[0049] When only an Ethernet packet is received via network link 210, MAC 212 sends it to inbound DMA 222 (via receive buffer 216), from where it is sent to the host system 102.

[0050] When only an Ethernet packet is received from the host system 102, the outbound DMA module 224 processes the packet and sends it to MAC layer 212 via transmit buffer 214. The Ethernet packet is then forwarded to its destination via link 210.

[0051] Ethernet core 204 also assists in handling network/storage packets. For example, when FCOE adapter 200 receives a FCOE packet via link 210, Ethernet core 204 processes the FCOE packet. Ethernet core 204 may call on FC assist module 206 to perform certain Fibre Channel related functions before passing the Fibre Channel frames (in the FCOE packet) to host system 102. In one embodiment, both storage and network packets are simultaneously processed by FCOE adapter 200.

[0052] Outgoing Fibre Channel packets are also handled by Ethernet core 204 with the assistance of FC assist module 206, as described below in more detail with respect to FIG. 5.

[0053] FIG. 2C is a block diagram for host interface 202 that interfaces with host system 102. Host interface 202 includes an arbiter 240 and PCI express module 242. Arbiter 240 receives requests from various DMA modules (224, 222 and others) to access bus 118 and then arbitrates between the requests. As an example, when DMA module 224 needs to access PCI-Express bus 118 it sends a request to arbiter 240. Arbiter 240 evaluates the request and if appropriate, grants the request, which enables DMA module 224 to access the PCI-Express bus 118. Arbiter 240 may use a round robin or any other scheme to evaluate DMA requests.

[0054] FCOE Packet Format:

[0055] FIG. 3 shows an example of an FCOE packet format 300 for processing network and storage traffic. FCOE packet 300 includes an Ethernet header 302. In one embodiment, the Ethernet header 302 may be 14 bytes in length. FCOE packet also includes a FCOE header 304 that includes the Ethernet type and version information. Start of frame (SOF) 306 indicates the beginning of a frame and may be 1 byte.

[0056] FCOE packet 300 may also includes a Fibre Channel header (FC Header) 308 that may be 24 bytes long with payload 310. The Fibre Channel cyclic redundancy code (CRC) 312 may be 4 bytes and the end of frame (FOE) 42 may be 1 byte in size. EOF 314 indicates the end of a frame. Ethernet CRC 316 is inserted after EOF 314.

[0057] Overall Software Architecture:

[0058] FIG. 4 shows a top-level software architecture that may be used for handling FCOE packets. Operating system 400 may be executed at a host system (for example, 102). Application 402 initiates storage-based commands (for example, to write or read data) that are sent to the FCOE adapter 200 via a FCOE driver 404 that is executed at the host system. Firmware 406 controls overall functioning of FCOE adapter 200 and is executed by processor 207.

[0059] In one embodiment, the FCOE driver 404 performs certain Fibre Channel related functions, without, executing the entire Fibre Channel software stack. For example, the FCOE driver may perform functions related to Fibre Channel Exchange management; Fibre Channel stack (for example, FC-3 and FC-4 layer processing) and Fibre Channel Fabric management operations, for example, the PLOGI, the FLOGI, Node Discovery and other operations.

[0060] FCOE driver 404 also places data received from the network in defined user space buffers; creates a header template for each Sequence that is to be offloaded/transmitted; and creates a scatter/gather list of buffers of data to be transmitted. FCOE driver 404 further creates a Fibre Channel Sequence for data to be transmitted. FCOE driver 404 may not build individual frames for each sequence because that function is offloaded to the FCOE adapter 200.

[0061] In one embodiment, the present disclosure provides an optimum solution for handling FCOE packets. Instead a complete software solution where the host driver performs all the functions or an expensive FCOE adapter that performs most of the Ethernet and Fibre Channel related functions, a hybrid solution is provided. In the hybrid solution, the FCOE driver 404 and the FCOE adapter 200 using the FC assist module 206 perform the Fibre Channel functions. The Ethernet core 204 of the FCOE adapter 200 handles most of the Ethernet related functions and also interfaces with the FC assist module 206.

[0062] FIG. 5 shows a process flow diagram for transmitting FCOE packets via FCOE adapter 200, according to one embodiment. The process start in step S500, when host system 102 builds an input output control block (IOCB) (for example, IOCB 600, FIG. 6) and forwards the IOCB to FCOE adapter 200. FCOE adapter 200 determines if the IOCB includes a FCOE header and needs to perform any offload functions. The IOCB may include flags (or fields) that indicate to the FCOE driver 404 (and to the FCOE adapter 200) that offload operations need to be performed.

[0063] FIG. 6A shows an example of an IOCB 600 with a plurality of fields that are set by host system 102. IOCB 600 includes information for a FCOE offload sequence (602) which means that some offload (or FCOE) processing is to be performed by FCOE adapter 200.

[0064] IOCB may also include a field 604 indicating that sequence packetization is to be performed; and field 606 indicating that only Fibre Channel based CRC calculation (606) is to be performed. IOCB further includes information about the amount of data that needs to be transferred (data transfer length 608), and the Fibre Channel frame payload size 640. IOCB may also include a list of memory addresses (612) where data that is to be transmitted is stored in host memory. Field 614 is used to indicate the header type (i.e. if it is FCOE header) and field 616 provides header size.

[0065] If a FCOE header is received in step S500 for an offloaded FCOE sequence, then the process moves to step S502. In step S502, FCOE adapter 200 acquires data from host memory 106. Outbound DMA module 224 may be used to acquire the data.

[0066] In step S504, an Ethernet and Fibre Channel header template is copied to memory 218 in Ethernet core forwarded to MAC layer 212. The Ethernet core 204 uses the header template to create an FCOE packet. FIG. 6B shows an example of a header template 618 that is created by the host system 102.

[0067] Header template 618 includes an Ethernet portion 620 and a Fibre Channel portion 622. Ethernet portion 620 includes a destination MAC address 624, a source MAC address 626, optional field 628 and a field 630 indicating the frame type.

[0068] Fibre Channel portion 622 includes various Fibre Channel fields including start of frame (SOF) 632, R_CTL 634, D_ID 636, CS_CTL 638, S_ID 640, Type 642, F_CTL 644, SEQ_ID 646, DF_CTL 648, SEQ_CNT 650, OX_ID 654, RX_ID 652 and a parameter/relative offset value 656. The various Fibre Channel fields are defined by the Fibre Channel specifications. For example, R_CTL 634 is a field for routing control; D_ID 636 is a destination address of an Nx_Port; CS_CTL 638 is a field for class control; S_ID 640 is an address of a source Nx_Port of a transmitted frame; TYPE 642 indicates the frame type; F_CTL 644 is a field used for frame control; and SEQ_ID 646 identifies sequence for exchanging Fibre Channel frames. DF_CTL 648 is a field for data field control; SEQ_CNT 650 is a field for a sequence count, OX_ID 654 is an identifier assigned by an originator to identify an Exchange and RX_ID 652 is an identifier assigned by a responder identify an Exchange.

[0069] The host system sets the SOF field 632 (similar to SOF 304, FIG. 3) to indicate if the data (provided by the host system) is an entire Fibre channel sequence or just a portion of a Fibre channel sequence.

[0070] In step S506, Ethernet core 204 builds an Ethernet header (including the FCOE header) and a Fibre Channel header (with assistance from FC assist module 206). The headers (Ethernet 302, FCOE 304 and Fibre Channel 308) are used for FCOE packets, an example of which is shown in FIG. 3.

[0071] In one embodiment, the Ethernet header may be built entirely by the Ethernet core 204. Ethernet MAC header 302 fields are copied from the header template 618.

[0072] Ethernet core 204 by itself or with the assistance of FC assist module 206 also builds the Fibre Channel header 304. For example, FC assist module 206 may insert an offset value for an FC header for each FCOE packet. The offset value may be used to distinguish the Fibre Channel packets.

[0073] The offload data provided by the host system may be for an entire Fibre Channel sequence or for just a portion thereof. The offload data may be located at the start, middle or end of the Fibre Channel sequence. If the offload is at a start of a sequence, the host system indicates that by setting SOF field 306 (FIG. 3) to a code corresponding to SOFiX field for a particular class service. The host system may set the SOF field 306 to SOFnX, if the offload is not at the beginning of a sequence. For a first frame, the SOF value is copied from header template 618. For each subsequent frame, the SOFnX for a particular class of service may be inserted.

[0074] During step S506, to build the Fibre Channel header 308, the R_CTL 634, D_ID 636, CS_CTL 633, S_ID 640, OX_ID 654, RX_ID 652 and TYPE fields 642 are copied from template 618. Parameter fields are appropriately set to ensure there is no data loss.

[0075] To build the F_CTL field 644 for each packet, all bits may be copied from header template 618, except an End Sequence bit and a Sequence Initiative bit. If the host system sets an End Sequence bit in F_CTL field 644, it indicates that a last frame of the sequence is included in the offload. When the End Sequence bit is set by the host, Ethernet core 204 (or FC assist module 206) uses the bit only for last packet and clear the bit for other packets. If a Sequence Initiative bit (which is a part of F_CTL field 644) and the End Sequence bit are set in header template 618, then the Sequence Initiative bit (along with the End Sequence bit) is set only in the last frame.

[0076] For each frame header, the SEQ_ID 646 and DF_CTL 648 fields are also copied from the header template 618. The SEQ_CNT field 650 is copied from the header template 618 to the Fibre Channel header 308 (FIG. 3) of the first frame built for an offload operation. The SEQ_CNT field is then incremented and copied into the header of each subsequent frame. The FC assist module 206 may perform this function.

[0077] If the Relative Offset Present bit 656 (which indicates that the relative offset is present) in the F_CTL field 644 is clear, then the Parameter fields are copied from the header template 618 to each frame header in a sequence. If the Relative Offset Present bit was set, then the host also sets the Parameter/Relative Offset field to an initial value and this initial value is copied to the frame header.

[0078] Ethernet core 204 (and/or FC assist module 206) determines the Fibre Channel CRC 312 for each packet. Following the FC CRC 312 is the EOF field 314. Ethernet core 204 sets the EOF field 314 to EOFn for all the frames in a sequence. After the EOF field 314, Ethernet CRC 316 is inserted by network interface 208 to complete a FCOE packet

[0079] In step S508, after the headers are created, the data received from host memory is packetized based on the indicated Fibre Channel payload size in IOCB 600. If data payload 310 is less than a certain size, for example, 11 bytes, then Ethernet core 204 pads the payload 310 to meet a minimum payload size.

[0080] In step S510, an FCOE packet is sent and in step S512, a counter 700 is decremented (or decreased) by an amount of data in the transmitted packet. Counter 700 (as shown in FIG. 7) is a data-monitoring counter maintained in Ethernet core 204. Counter 700 is initially loaded with data, which is equal to the data transfer length (608) from IOCB 600 minus the size of header template 616. As each frame is created and forwarded, the counter is decremented by the amount of transmitted data. Thus when all the data is transmitted counter 700 value is zero.

[0081] In step S513, FCOE adapter 200 determines if all the data has been sent. If all the data is not sent, then the process moves back to step S504. If all the data has been sent, then the process ends in step S514.

[0082] In one embodiment, the present disclosure provides an optimum, hybrid solution to handle storage and network traffic. A host driver and an adapter share the workload in processing the network and storage packets. Instead of having a complex host driver or an expensive adapter for performing a complete offload; the host driver and the adapter efficiently split the operations.

[0083] Although the present invention has been described with reference to specific embodiments, these embodiments are illustrative only and not limiting. For example, although the foregoing examples are based on Ethernet and Fibre Channel protocols, the adaptive embodiments may be applied to any network and storage protocol. Many other applications and embodiments of the present invention will be apparent in light of this disclosure and the following claims.