Route lookup caching for a fiber channel switch

[0002] The present invention relates, in general, to the field of fibre channel switching technology. More particularly, the present invention relates to a route caching scheme for a receive port in a fibre channel switch.

[0003] Fibre Channel is a high performance, serial interconnect standard designed for bi-directional, point-to-point communications between servers, storage systems, workstations, switches, and hubs. It offers a variety of benefits over other link-level protocols, including efficiency and high performance, scalability, simplicity, ease of use and installation, and support for popular high level protocols.

[0004] Fibre channel employs a topology known as a “fabric” to establish connections between ports. A fabric is a network of switches for interconnecting a plurality of devices without restriction as to the manner in which the switch can be arranged. A fabric can include a mixture of point-to-point and arbitrated loop topologies.

[0005] In Fibre Channel, a channel is established between two nodes where the channel's primary task is to transport data from one point to another at high speed with low latency. The Fibre channel switch provides flexible circuit/packet switched topology by establishing multiple simultaneous point-to-point connections. Because these connections are managed by the switches or “fabric elements” rather than the connected end devices or “nodes”, fabric traffic management is greatly simplified from the perspective of the device.

[0006] In a fibre channel switching environment, a module within the switching element determines the appropriate route for incoming frames based upon a particular destination ID value (D_ID) located within the frame header. The D_ID identifies the exit port associated with the incoming frame. In most applications, a route lookup table provides the translation from the D_ID to the appropriate exit port.

[0007] In prior approaches, the switch dedicates a unique route lookup table to each port. Since such route lookup tables must necessarily be large to accommodate all the possible associations between the incoming frame D_ID's and the corresponding exit ports, this approach requires a significant amount of memory.

[0008] The route caching design of the present invention provides a solution to the aforementioned problem, which is vastly superior to anything currently available. It not only provides quick access to recently used D_ID and exit port combinations, but it does so in an extremely efficient manner without requiring any significant design changes and with only a relatively straightforward alteration to existing processes for networking in a fibre channel switching environment.

[0009] Particularly disclosed herein is a method for routing a data frame through a fibre channel fabric. The fibre channel fabric is comprised of a first fibre channel switch that has a plurality of fibre channel ports. The fibre channel ports are operative for transmitting and receiving a data frame. A plurality of data caches are provided such that at least one data cache is coupled to at least one of the plurality of fibre channel ports. An association is created between an exit port and a destination identification. The association is stored in at least one data cache coupled to at least one of the plurality of fibre channel ports.

[0010] In another aspect, the present invention provides a method for storing a data frame route relationship in a fibre channel fabric. A cache is provided that is operatively coupled to a port of a first fibre channel switch. Associating an exit port of the fibre channel switch with a destination identification creates a data frame route relationship. The destination identification is a field of a data frame and represents an end location for transmitting said data frame. The association is then stored in the data cache.

[0011] Still further disclosed herein is a fibre channel fabric having reduced latency and increased through put capacity. The network comprises a fibre channel switch having a plurality of fibre channel ports embodied thereon for transmitting and receiving data frames. Continuing, a route control module is coupled to one of the fibre channel ports. The route control module provides the identity of an exit port in response to a request from the fibre channel port for the exit port. The request comprises a destination identification. The network also has a data cache coupled to the route control module. The data cache stores an association between the exit port and the destination identification.

[0012] The aforementioned and other features and objects of the present invention and the manner of attaining them will become more apparent and the invention itself will be best understood by reference to the following description of a preferred embodiment taken in conjunction with the accompanying drawings, wherein:

[0013] FIG. 1 is a block diagram of a switching element, wherein the switching element has a shared memory, a central route table and a plurality of fibre channel ports;

[0014] FIG. 2 is a detailed block diagram illustrating one embodiment of an interrelationship between modules of the switching element, particularly the fibre channel ports, local control route module, shared memory, and QC module;

[0015] FIG. 2A is a block diagram illustrating a typical interface between a fibre channel port and a local route control module;

[0016] FIG. 3 is a detailed block diagram of a local route control module;

[0017] FIG. 4 is a flow chart for a data cache operation;

[0018] FIG. 5 is a block diagram of the request exit port bus and the exit port response bus.

[0019] FIG. 1 shows a generalized block diagram of a fibre channel switch for use in a fibre channel fabric implementing the method and systems of the present invention. In one embodiment, the fibre channel switching element 100 of FIG. 1 may be implemented on a single application specific integrated circuit (ASIC). However, there are many implementations of switch 100 not shown, such as frame buffer memory could be located in each GL_Port in which case shared memory is replicated with a crossbar switch.

[0020] The fibre channel fabric associated with switch 100 is the method for connecting the various N-Ports of the devices together. In this way, the fabric is capable of routing fibre channel frames using only the destination identification information in the fibre channel frame header. The destination identification information identifies which N-Port receives the frame.

[0021] Fibre channel switch 100 has a plurality of ports for receiving and transferring data through the switch. In FIG. 1, the ports are illustrated as GL-ports 130, 135 and 140. In one embodiment, switch 100 comprises 24 GL_Port modules. Each GL_Port is coupled to shared memory 120 and external optical interface 160. External optical interface couples switch 100 to the N-Port, NL-Port or E-Port of the device coupled to the fibre channel fabric. Fibre channel switch 100 is associated with a central route table 110. Central route table 110 is operatively coupled to each GL-Port associated with fibre channel switch 100 (not shown in FIG. 1). Switch 100 also has a system interface 150. System interface module 150 provides interfaces to the power supply, fans, temperature sensor, LED's, and the serial interfaces of the optical transceivers.

[0022] Continuing, fibre channel switch 100 has an embedded port 145 in addition to illustrated GL_Ports 130, 135 and 140. Within the context of the invention, embedded port 145 may be used for several functions. First, it may provide a system services processor an access point for all of the fibre channel well-known addresses for both the reception and transmission of frames. Secondly, it may handle any fibre channel frame that cannot be delivered to a destination for either busy, reject, or timeout conditions. It may also be responsible for the generation, modification and/or interpretation of all Fibre Channel-Arbitrated Loop (FC-AL) initialization frames (such as LIFA, LIPA, LIHA, and LISA frames) for the GL-Ports operating in fabric loop mode. The embedded port interface is slightly different than that of an actual GL_Port module. Since the Fibre Channel-0/1 layers are not required, embedded port 145 does not implement the low-level interface for either primitive signaling or sequences. After the system services processor has completed initialization of embedded port 145, it enters and remains in the fibre channel active state.

[0023] Embedded port 145 provides a register set accessible to the system services processor for basic initialization and low level control. Once enabled, embedded port 145 is responsible for all functions related to the transmission and reception of frames to and from shared memory 120. For data path consistency within the following description, the direction of fibre channel frame flow is referenced to shared memory 120. Thus, a transmit (TX) path actually contains paths destined to, or received by, the embedded port from shared memory 120 and a receive (RX) path contains frames generated or transmitted by embedded port 145.

[0024] Embedded port 145 creates and consumes buffers that contain complete fibre channel frames. Frames may be held in SRAM coupled to embedded port 145. SRAM will typically hold two frames, one TX frame received from shared memory 120 and one RX frame waiting to be moved into shared memory 120. All other TX frames waiting to be read by embedded port 145 and RX frames previously created by embedded port 145 are stored in shared memory buffers. In one embodiment, embedded port 145 may be allocated up to 12 shared memory buffers for storage of RX frames. Typically, TX frames utilize the shared memory buffers allocated to the GL-Ports that receive the fibre channel frames.

[0025] For example, for fast turn-around of Arbitrated Loop address initialization frames, a TX frame may be modified in place in the embedded port SRAM by software and sent via an RX path without the need to move the frame. One of the shared memory buffers allocated to an embedded port may be designated as a protected buffer. The protected buffer can be filled with an RX frame that is transmitted frequently and left intact so that the frame can be sent to an exit port without the time delay of moving the frame from the extended port SRAM to the shared memory buffer.

[0026] An embedded port receiver (RX) is used to transfer frames from a system services processor to other ports in the switch 100. The Receiver module will be identical to the GL_Port RX module described hereinafter. Similarly, an embedded port transmitter (TX) is used to transfer frames from other ports via shared memory to a system services processor. The Transmitter module will be identical to the GL_Port TX module also described hereinafter. An Embedded Port Front End is used to transfer data between the Embedded Port SRAM and the TX and RX modules.

[0027] Shared memory 120 provides buffering and switching for all fibre channel frames that flow through switch 100. Received frames are written to shared memory 120 by the receiving port then read from shared memory 120 by the transmitting port. In one embodiment, shared memory 120 has 162 total frame locations shared by the GL-Ports 130, 135 and 140 and embedded port 145. In such an example, each port may be allocated as many as 12 buffers, so long as the total of 162 buffers is not exceeded.

[0028] Central Route Control module 110 provides a common route table for all ports in switch 100. Route table provides a translation from each possible Destination ID (D_ID) value to the appropriate exit port. Additionally, the route table provides for hard zoning, which is the capability for blocking traffic from certain receive ports to certain D_IDs. Each port uses an exit port request and response bus to communicate with central route table 110.

[0029] GL_Ports 130, 135 and 140 transmit and receive fibre channel frames to and from the switch and to and from the fibre channel fabric. As shown in FIG. 1, each GL_Port is coupled to an external optical interface 160 that in turn couples the port to the fabric and ultimately to the N_Port of the destination device.

[0030] GL_Ports may function as an E_Port, an F_Port or an FL_Port, to name a few. An E_Port is an expansion port that serves as a physical interface within the fabric that is used to create multi-switch fabrics by attaching another switches E_Port through an interswitch link (ISL). An F_Port is a fabric port that operates as a physical interface within the fabric that attaches to an N_Port of a destination device through a point-to-point link connection. An FL_Port is a fabric loop port that contains arbitrated loop (AL) functions associated with the FC-AL topology. FC-AL is a fibre channel topology where ports use arbitration to establish a point-to-point circuit.

[0031] FIG. 2 illustrates one possible method of connecting GL_Ports 130 and 135, as well as possible connections between the ports and shared memory 120 and local control route module 200. GL_Port 130 has a TX module 230 and an RX module 220 for sending and receiving frames. GL_Port 135 also has a TX module 235 and RX module 225. For example, if GL_Port 130 is the receiving port for a data frame and GL_Port 135 is the exit port for the data frame, the frame would first be sent from RX module 220 to shared memory 120 and then sent from shared memory 120 to TX module 235 as illustrated.

[0032] Continuing with the illustrated example of FIG. 2, RX port module of GL_Port 130 is coupled to TX port module of GL_Port 135 through a QC module 205. QC Control Module 205 acts as the control interface between the TX module 235 and RX module 220. QC Control Module 205 routes both a request and an acknowledgement signals between GL_Ports that serve to transmit exit port information and location of the fibre channel frame in shared memory from RX module 220 as well as return a successful frame transmission message from TX module 235.

[0033] Local route control module (LR) 200 is used by the GL_Port RX module to request the exit port for a frame based on the frame's destination ID value (D_ID). As shown in FIG. 2, a send and receive connection couples RX module 220 of GL_Port 130 with LR 200. The connection allows RX module 220 to request an exit port from the local control route module 200 and also for LR 200 to transmit the identity of the exit port back to RX module 220.

[0034] FIG. 2A illustrates the communication between GL_Port 130 and local route control module 200 in greater detail. TX control module 230 and RX control module 220 are coupled through fibre channel front end module 210. Fibre channel front end (FE) 210 provides the FC-0/1 level processing requirements. FE 210 includes all of the character level state machines required to support a fibre channel link, including all of the requirements for normal data frame processing. FE 210 provides an interface to the system services processor for low level control over the fibre channel link interface.

[0035] For the processing of frame traffic, FE 210 provides independent, symmetrical RX and TX interfaces to carry frame data. These paths consist of a data bus and control signals that identify the beginning and ending frame delimiters. For the RX path, status information about the frame including CRC validation, truncated frames and other pertinent status is also included as part of the signal.

[0036] Fibre channel front end 210 continuously monitors its receive link for the detection of a start of file (SOF) delimiter in the fibre channel frame. When an SOF is detected, FE 210 then forwards the frame to the RX module 220. RX module 220 stores the frame into the next available shared memory buffer. RX module 220 uses LR 200 to make a destination port routing decision from the header information of the received frame. RX module 220 then combines the shared memory buffer number into a field, which may be referred to as a Qentry field, which is passed to a TX module of the destination port through QC Module 205. RX module 220 then waits for the TX module to return the buffer number via an AckQEntry field. When RX module 220 receives the AckQEntry field it indicates to FE 210 that the buffer is being consumed.

[0037] The time that is required for all this processing is less than 1 microsecond. For this example it is most likely that RX module 220 is still storing the received frame while the TX module is transmitting the same frame, creating a cut-through switching effect. If the TX did not immediately transmit the frame, it is possible that the entire frame has been written into the buffer memory when transmission commenced, providing for a store and forward type of switching function.

[0038] The TX logic continuously monitors the bus coupling QC Controller 205 with TX module 230 for QEntries. When a QEntry is received, it is placed in TX module 230. When FE 210 is able to transmit a new frame, the queue selects a QEntry for processing. The shared memory buffer number for the frame to transmit is extracted from the QEntry and the TX module initiates a shared memory read operation. The frame data is then passed from the shared memory 120 to the FE 210. FE 210 transmits the frame.

[0039] Continuing with FIG. 2A, RX module 220 of port 130 is coupled to LR module 200 so as to request and receive exit port information. In one possible example, RX module 220 requests the identity of an exit port for a particular frame by sending a request over ReqExitPort connection 240 to LR 200. LR module 200 performs the necessary procedure for retrieving the exit port identity based on the transmitted D_ID. LR module 200 then transmits the generated exit port information to RX module 220 over ExitPort connection 250.

[0040] FIGS. 3 and 4 illustrate the operation of local route control module 200 in greater detail. Local route control module 200 is used by GL_Port RX module 220 to request an exit port for a fibre channel frame based on the frame's destination ID value (D_ID). In the illustrated example, the request for an exit port comes in on the ReqExitPort bus 240 to local route control 200.

[0041] RX module requests an exit port by providing a D_ID from the frame header to local route control module 200 (step 400). In one embodiment, the D_ID has 24 bits, starting with the 0 bit, which is represented by a designation [23:0]. The first operation 320 of local route control module 200 is to determine if the D_ID identifies a multicast (MC), a broadcast (BC) or a well known address (WKA) (step 410). Multicast and broadcast addresses are directed to a MC/BC/WKA table 310 to identify the exit port. Well-Known Addresses and FC-AL Loop Initialization addresses always result in the Embedded Port being selected as the Exit Port. Domain controller identifier addresses are sent to the central route table for exit port lookup unless a bit is set and the frame is not a class F frame, in which case the Embedded port is selected as the exit port.

[0042] If central route table 110 has not yet been initialized, then all frames are routed to the embedded port. All other D_ID values are forwarded to central route table 110. However, prior to forwarding a request for an exit port to central route table 110, LR 200 performs an operation 330 to determine whether an association between the requested D_ID and an exit port designation is found in route cache 300 (step 430). If the D_ID to exit port association is found in route cache 300, the exit port is available immediately. Route cache 300 improves latency by caching the most recent exit port lookups. In one embodiment, the sixteen most recent lookups are stored in cache 300. Cache 300 should be cleared when either the central route table 110 or indirect exit port table 350 is modified.

[0043] If the D_ID is not located in route cache 300, then local route controller 200 performs an operation 340 to send the D_ID to a central route table 110. Central route table 110 retrieves the D_ID to exit port association and returns it to local route controller 200. If applicable, the D_ID to exit port association is stored in an indirect exit port table 350 (step 470). As shown in step 480, D_ID to exit port association is stored in route cache 300. In the illustrated example, D_ID to exit port association is sent to RX module over ExitPort bus 250 (step 490).

[0044] Route cache 300 stores information received from central route lookup table 110, more particularly the most recent exit port lookups performed by local route controller 200. In one embodiment, the associations are stored in flip-flops. However, random access memory, content addressable memory or the like may be used without departing from the intended scope of the invention. Several algorithms may be used to store and remove associations in cache 300. In one embodiment, a least recently used algorithm may be utilized. In another embodiment, a least frequently used algorithm may be utilized.

[0045] Indirect exit port table 350 provides the capability for different RX Ports to send frames destined for the same D_ID out different TX Ports. This is useful when two or more inter-element links connect two switches together. Load balancing across inter-element links can be performed without modifying central route table 110 and affecting other ports. Indirect exit port table 350 may be embodied in flip-flops or random access memory to name a few.

[0046] FIG. 5 illustrates the structure of a request exit port bus ring structure. The request exit port bus is used by the receive controller of a GL_Port to request an exit port number for a given D_ID from the central route controller. The request exit port bus is designed to operate in a ring structure in which each module that is attached to the request exit port bus pipelines and re-powers the request exit port bus signals before sending them to the next module in the ring. The local route control module is used to provide the attachment to the request exit port bus for each module. The request exit port bus signals are described in Table 1.

[0047] FIG. 5 also illustrates the exit port response bus ring. The exit port response bus is used by the central route controller to return an exit port number to an RX module of a GL_Port. The exit port response bus is designed to operate in a ring structure in which each module that is attached to the exit port response bus pipelines and re-powers the exit port response bus signals before sending them to the next module in the ring.

[0048] The central route controller may insert an exit port response on the bus in any clock cycle. A local route control module will extract the exit port response from the bus if it is addressed to that GL_Port. The local route control module is used to provide the attachment to the exit port response bus for each module. The exit port response bus signals are described in Table 2.

[0049] While there have been described above the principles of the present invention in conjunction with a specific embodiment, it is to be clearly understood that the foregoing description is made only by way of example and not as a limitation to the scope of the invention. Particularly, it is recognized that the teachings of the foregoing disclosure will suggest other modifications to those persons skilled in the relevant art. Such modifications may involve other features which are already known per se and which may be used instead of or in addition to features already described herein.

[0050] Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure herein also includes any novel feature or any novel combination of features disclosed either explicitly or implicitly or any generalization or modification thereof which would be apparent to persons skilled in the relevant art, whether or not such relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as confronted by the present invention. The applicants hereby reserve the right to formulate new claims to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.