Automatic testing of redundant switching element and automatic switchover

[0002] The present application is a divisional application of U.S. patent application Ser. No. 09/829,448, filed Apr. 9, 2001, now U.S. Pat. No. 6,556,953 which is incorporated by reference herein in its entirety.

[0003] The present invention relates, in general, to the field of fibre channel switching technology. More particularly, the present invention relates to non-disruptive, on-line testing and switchover in a high availability, fibre channel switching environment.

[0004] Fibre Channel is a high performance, serial interconnect standard designed for bi-directional, point-to-point communications between servers, storage systems, workstations, switches, and hubs. It offers a variety of benefits over other link-level protocols, including efficiency and high performance, scalability, simplicity, ease of use and installation, and support for popular high level protocols.

[0005] Fibre channel employs a topology known as a “fabric” to establish connections between ports. A fabric is a network of switches for interconnecting a plurality of devices without restriction as to the manner in which the switch can be arranged. A fabric can include a mixture of point-to-point and arbitrated loop topologies.

[0006] In Fibre Channel, a channel is established between two nodes where the channel's primary task is to transport data from one point to another at high speed with low latency. The Fibre channel switch provides flexible circuit/packet switched topology by establishing multiple simultaneous point-to-point connections. Because these connections are managed by the switches or “fabric elements” rather than the connected end devices or “nodes”, fabric traffic management is greatly simplified from the perspective of the device.

[0007] In a high availability, fibre channel switching environment, a second set of “redundant” elements are provided in the event of a failure condition. The number and make-up of the redundant elements parallel the primary elements, and operate as back-up resources if the primary elements fail. As such, in the event of such a fail condition, a switchover to the redundant elements can greatly minimize the loss of transmitted data frames.

[0008] In prior approaches, the redundant elements in a high availability environment are passive in nature. This approach provides a second set of elements that remain inactive until the occurrence of a fail condition. In such an all-or-nothing passive environment, it becomes necessary to take a system offline to perform the necessary tests on the elements to determine if they are still in working condition. Since taking a system offline is often not a viable option, the unfortunate effect is a lack of a significant method of testing the redundant elements. As a result, switching over to the redundant elements may result in a situation where an element is either partially or completely non-functional.

[0009] Another limitation of prior systems is that during switchover to the redundant elements in a failure condition, frames were lost, since there was no implementation of a completely seamless method of transition to the redundant elements. It is well understood that for a catastrophic failure event, immediate switchover is of paramount importance. However, for a non-catastrophic failure, the significance of immediacy becomes secondary to the importance of maintaining the integrity of a frame transmission.

[0010] The on-line testing and switchover design of the present invention provides a solution to the aforementioned problems which is vastly superior to anything currently available. It not only solves the critical situation where, after switchover, an element is either partially or completely non-functional, but it does so in an extremely efficient manner without requiring any significant design changes and with only a relatively straightforward alteration to existing processes for networking in a high availability, fibre channel switching environment.

[0011] The present invention advantageously provides a proactive management approach for verifying the integrity of a redundant switching element prior to actual use. More particularly, the on-line testing and switchover design of the present invention offers independent, non-disruptive online testing via an independent request/response signaling interface on a per port basis. Since the invention operates on a per port basis, it is unnecessary to shut down the other operating ports to carry out the testing operation, and therefore the ports can continue to transmit data. In addition, the invention advantageously provides the ability to switch over to a redundant switching element on a frame boundary, thereby reducing the possibility of losing a frame during the transition from a currently functioning path to a backup path.

[0012] Particularly disclosed herein is a method of performing an online, non-disruptive health check test in a fully redundant fibre channel switching network having an active switching element and a redundant switching element. The health check test is performed on the redundant switching element without the need to take the network offline. After a first and second status condition are met prior to the expiration of a timer, a working data path from a port to the redundant switching element is verified and the condition is recorded. Then, a working control path to the redundant switching element is verified and the condition is recorded.

[0013] In another aspect, the present invention provides a method of performing automatic switchover in a fully redundant fibre channel switching network having an active switching element, a redundant switching element and a port having switching element logic embodied therein. To begin, a failure condition is detected in the active switching element. The failure condition is transmitted to the port logic. In response to a possible problem with the active switching element, software notifies the port logic to perform the switchover. Notification may originate from either the active switching element or port logic via software collection. In response, the port is queried to determine if the port is transmitting a frame. If a port is in the process of transmitting a frame, then the automatic switchover is delayed until the port is not transmitting a frame.

[0014] Still further disclosed herein is a fibre channel switching network having increased bandwidth capacity. The network comprises a first switching element, a second switching element, a first reader coupled to the first switching element and a second reader coupled to the second switching element. Continuing, the switching network also has a buffer memory for storing frames, wherein the buffer memory is coupled to the first reader and the second reader. The network also has a writer coupled to the buffer memory, wherein the writer directs a buffer control to store frames in the buffer memory. The first reader or the second reader directs the buffer control to retrieve a frame from buffer memory and pass the frame to the respective reader for transmission.

[0015] The aforementioned and other features and objects of the present invention and the manner of attaining them will become more apparent and the invention itself will be best understood by reference to the following description of a preferred embodiment taken in conjunction with the accompanying drawings, wherein:

[0016] FIG. 1 is a block diagram of two switching elements, a single fibre port module having switching element control logic, an RX port and a TX port;

[0017] FIG. 2 is a block diagram showing the functionality of a fibre port module;

[0018] FIG. 3 is a flow chart for the health check test;

[0019] FIG. 4 is a flow chart for the health check test;

[0020] FIG. 5 is a prior art block drawing of a current configuration for switching elements;

[0021] FIG. 6 is a block diagram of a multi-drain configuration in which both switching elements are active thereby increasing bandwidth; and

[0022] FIG. 7 is a flow chart for the automatic switchover.

[0023] FIG. 1 shows a fully redundant, generalized fibre channel switching environment implementing the method and systems of the present invention. FIG. 1 illustrates a number of devices and connections between the devices that are indicated by connecting lines. In particular, two switching elements or SBAR's, SBAR025 and SBAR130, are shown coupled to a fibre port module 10 over receive paths or inbound paths 35, 40 and transmit or outbound paths 50, 55. Fibre port module 10 has port logic embodied thereon, however, only the SBAR control interface logic of fibre port module 10 is shown in FIG. 1. In addition, fibre port module 10 comprises receive (RX) block 20 and transmit (TX) block 15 for coupling fibre port module 10 to and from the backplane.

[0024] SBAR025 and SBAR130 each have a fully independent request/response interface mechanism for processing requests and responses to and from fibre port module cards embodied within fibre port module 10. As shown in FIG. 1, SBAR025 communicates with fibre port module 10 over request line 60 and response line 65. Similarly, SBAR1 communicates with fibre port module 10 over request line 75 and response line 70. For example, fibre port module 10 may transmit a request for connection to SBAR130 over request line 75. SBAR130 responds over response line 70 with a response, such as “destination port busy.” In one embodiment, data on both request lines 60, 75 and response lines 65, 70 is serially encoded for transmission between fibre port module 10 and SBAR025 and SBAR130.

[0025] Also shown in FIG. 1 is data control multiplexer 45, which is coupled to fibre port module 10 by connection 80. Data control multiplexer 45 is also coupled to SBAR025 via outbound path 50, and to SBAR130 via outbound path 55. Connection 80 is part of outbound paths 50, 55 respectively depending upon which SBAR is active. Using logic that is not shown, data control multiplexer 45 transmits data from the active SBAR, either SBAR025 or SBAR130, to fibre port module 10. The logic stores the identity of the active SBAR, which provides data control multiplexer 45 with sufficient information so as to pass through the data from either outbound path 50 or outbound path 55 to fibre port module 10.

[0026] Continuing with FIG. 1, one example of a frame transmission from a port to a redundant network of switching elements will now be explained in detail. In operation, a single frame appears at fibre port module 10 intended for a particular destination port. Fibre port module 10 sends a connection request to the active switching element. Logic within fibre port module 10 indicates which SBAR is active and which SBAR is redundant. In this example, SBAR130 is the active switching element and SBAR025 is the redundant switching element. For purposes of being complete, since SBAR025 is a redundant switching element, the default condition is that for each port, the redundant switching element is looped back to itself, i.e. RX 20 is looped back to TX 15 for SBAR025.

[0027] Continuing with the example, since SBAR130 is the active switching element, fibre port module 10 sends a connection request, sb_sbar1_req, to SBAR130 over request line 75 seeking to transmit the single frame to the particular destination port. SBAR130 investigates and determines if the particular destination port is available. If the particular destination port is unavailable, SBAR130 responds with a “destination port busy” response, and a connection is not established. If the particular destination port is available, an appropriate response is sent to fibre port module 10 over response line 70 indicating that the connection is established. The fibre port module then prepares to send the frame to the switching elements. In this case, the switching elements are now deemed busy and frames cannot be transmitted through the busy port until the connection is terminated.

[0028] The single frame is then transmitted from RX port 20 over inbound paths 35, 40. Since the illustrated example is a fully redundant fibre channel switching network, the frame is transmitted over both inbound path 40 to active switching element SBAR130 as well as inbound path 35 to redundant switching element

[0029] SBAR025

[0030] The return paths from SBAR025 and SBAR1 are outbound path 50 and outbound path 55 respectively through data control multiplexer 45 over connection 80 to TX port 15. Since redundant switching element 25 is looped back to itself for this port, the single frame is transmitted through SBAR025 and appears at data control multiplexer 45. As stated previously, data control multiplexer 45 contains logic to ascertain the identity of the active switching element, in this case SBAR130. Since SBAR025 is the redundant switching element, the frame from SBAR025 is discarded, and the frame transmitted from SBAR130 is passed through to TX port 15 of fibre port module 10.

[0031] FIG. 2 illustrates another aspect of fibre port module 10 for implementing the method and systems of the present invention. Fibre channel front end (FE) 200 provides an interface to a central processor (CTP). The main element of the CTP is the system services processor (SSP) that provides processing power. In the case of processing frame traffic, FE 200 provides independent, symmetrical RX and TX interfaces to carry frame data. Coupled to FE 200 is frame writer 205. In operation, FE 200 forwards a frame to frame writer 205 for storage in buffer memory 210. Frame writer 205 stores the frame in the next available buffer in buffer memory 210, modifies a free buffer list accordingly and passes the buffer location to queue manager (Q_m) 220. Frame reader 215 is responsible for reading frames stored in buffer memory 210 and transferring them to SBAR 250.

[0032] Continuing with FIG. 2, Q_m220 is the central core of fibre port module 10. Q_m220 is responsible for building queues of received frames for destination ports from the information received from frame writer 205 as well as providing buffer location information to frame reader 215. Q_m220 also interfaces with switching element (SBAR) 250 through SB control logic 225 for the establishment of connections as a source.

[0033] Transmit handler 230 provides the interface for the transmission of the frames at the port. The transmit handler logic interfaces to both SBAR's in a fully redundant switching network. However, through CTP control, only one interface is active at a time.

[0034] Health Check Test

[0035] In one embodiment, a health check test is initiated by a Central Processor (CTP) via a control register bit in the SB logic contained within fibre port module 10. The initiation of the health check test may be software driven by operational or online software, and as such, the test could be activated on a regular schedule, e.g. hourly, weekly, or monthly to name a few. By activating the control register bit in the SB logic of fibre port module 10, the CTP is in effect “taking control” of the port and the associated paths for a brief period of time to perform the health check test. During this brief period of time the port is under control, the port remains “offline” to new requests. The term “offline” simply means that the port remains unavailable for sending frames to the active switching element. However, the port still receives and buffers new frames via the front end logic until the port runs out of available buffers. The offline condition occurs only when the first status condition described later is satisfied.

[0036] FIGS. 3 and 4 illustrate, using flow charts, one embodiment of a health check test in accordance with one aspect of the present invention. For purposes of clarity, the health check test is best explained in two parts. The first part of the test verifies the integrity of a data path between a port and a redundant switching element. The second part of the test verifies the integrity of a control path between a port and a redundant switching element. In combination, the two parts of the illustrated health check test evaluate the operating condition of a redundant switching element, thereby determining if it would function properly in a switchover condition. The two parts are explained in greater detail in the sections that follow.

[0037] Data Path Test

[0038] Once the fibre port module 10 acknowledges that the CTP has set a register bit in the SB logic to begin the health check test, the logic begins the process of taking control of the port and the corresponding data and control paths to both the active and redundant switching elements. In accordance with one important aspect of the invention, certain limits are in place as to how long the port is allowed to remain in test mode since the test is designed to minimize disruption to the port. In most cases, the limits are controlled by timers that, if expire, operate to return the port to active status and terminate the health check test.

[0039] As shown in the flow chart shown in FIG. 3, the fibre port module 10 determines, through its internal port logic, whether the CTP has initiated a health check test (step 305). If a health check test has been initiated, the SB logic within fibre port module begins the process of taking control of the port and associated paths. The control logic, and more particularly a state machine within the control logic, monitors the status of the inbound and outbound paths (step 310) for the occurrence of a frame.

[0040] A first timer, or inactivity timer, starts a termination countdown (step 310) at the same time the control logic begins monitoring both the status of the inbound or transmit (TX) path and the status of the outbound or receive (RX) path. The inactivity timer places a limit on the amount of time the port will monitor the inbound and outbound path for a frame transmission before timing out and returning the port to normal operation. In one embodiment, the timer is set to expire after 100 microseconds. Advantageously, the inactivity timer prevents stalling the port while waiting for the correct transmit and receive conditions to be met on the inbound and outbound paths.

[0041] Continuing with the example of FIG. 3, the port logic of fibre port module 10 monitors the outbound path for the occurrence of a frame (step 315). It should be understood that since the logic is monitoring both paths simultaneously, the order of the steps of monitoring the inbound and outbound paths is arbitrary and is illustrated in FIG. 3 in any particular order solely for purposes of explanation. If a frame is detected on the outbound path, the logic continues to monitor the inbound path for the occurrence of a frame (step 330).

[0042] If a frame is then detected on the inbound path, the logic has detected the correct status of both the RX and TX paths. The logic then performs several operations to control the port and paths (step 340) that are explained in the paragraphs that follow. If the first timer expires at any point prior to detecting of a frame on both the inbound and outbound paths (steps 325, 345 and 350), the port immediately returns to normal operations and the health check test is terminated (step 360).

[0043] Referring back to the flow chart of FIG. 3, a first status condition, as that term is used herein, describes the occurrence of a frame on the RX path prior to the expiration of the inactivity timer (steps 320 and 330). The first status condition triggers the SB logic of the port to block any new requests from being sent through the port. If any new requests are received, the new requests will be placed in queue until the completion of the health check test. The new requests are the sent when the port resumes normal operation, which is either completion or termination of the health check test.

[0044] The occurrence of a first status condition also provides for a delay of a certain length of time to allow the completion of any frame transmission on the RX path. In one embodiment, the delay is for a period of 20 microseconds. In any case, the delay should be at least long enough to allow for the maximum length of a fibre channel frame or the length of time necessary to make certain the RX path is clear.

[0045] Referring again to FIG. 3, a second status condition, as that term is used herein, describes the appearance of a frame on the TX path prior to the expiration of the inactivity timer (steps 315 and 335). The second status condition triggers the SB logic of the port to hold off clearing the ports busy bit. The term is busy bit is used as an indication to the switching element that the port is either available or unavailable to transmit a frame. Once the health check test is either competed or terminated, the ports busy bit is cleared allowing the port to resume normal operations.

[0046] Once the first status condition and second status condition are satisfied, the port logic is temporarily in control of fibre port 10 and the associated data paths and control paths. The health check test may be engaged by the port logic (step 355). In one embodiment, prior to engaging the health check test, the active data path is changed from the active switching element to the redundant switching element. In the illustrated example of FIG. 1, a data path multiplexer 45 switches to the opposite of the programmed register value, thereby setting the inbound data to the port to come from the redundant switching element.

[0047] The port logic instructs the RX logic to transmit a test data pattern on the RX path to the redundant switching element. Concurrently, the port logic instructs the TX logic to monitor the TX path for the appearance of the test data pattern. In one embodiment, the test data pattern is programmed into a shared register in the port. Since the receive and transmit ports are looped together for a redundant switching element, the data sent on the RX side will appear at the TX side if the switching element is in operating condition.

[0048] A second timer, or an expect frame time out timer, starts at the same time the port logic instructs the RX logic to transmit the test data pattern. The expect frame time out timer places a limit on the amount of time the port logic will monitor the inbound path for the test data pattern before timing out and proceeding to the control path test. In one embodiment, the timer is set to expire after 10 microseconds.

[0049] Referring now to the flow chart of FIG. 4, the SB logic in the port monitors the receipt or non-receipt of the test data pattern. If the test data pattern is detected at the port prior to the timeout of the second timer (step 400), a data path good condition of the redundant switching element is logged at the port. However, if the second timer expires prior to the detection of the test data pattern (step 405), a data path bad condition is logged at the port. In either case, once the condition of the data path is recorded, the data path test is complete and the control path test begins.

[0050] Control Path Test

[0051] Continuing with FIG. 4, the illustrated control path test is a relatively quick method for determining the integrity of the control path between a port and a redundant switching element. Since the test is a continuation of the data path test, it should be understood that even though the data path test is complete, the port logic is still in control of fibre port 10 and the associated data paths and control paths.

[0052] As shown in FIG. 4, a connection request is sent from the port to the redundant switching element (steps 410 and 415). Since the default condition for a redundant switching element is to set all the busy bits to an on position, a connection request from the port should elicit a “destination port busy” response.

[0053] A third timer, or an expect response time out timer, starts at the same time the connection request is sent from the port to the redundant switching element. The expect response time out timer places a limit on the amount of time the port logic will monitor the control path for the “destination port busy” response before timing out and logging the results of the test. In one embodiment, the timer is set to expire after 3 microseconds.

[0054] Referring now to the flow chart of FIG. 4, the SB logic in the port monitors the receipt or non-receipt of the response to the connection request. If the “destination port busy” response is detected at the port prior to the timeout of the third timer (steps 420 or 435), a control path good condition of the redundant switching element is logged at the port. However, if the third timer expires prior to the detection of the “destination port busy” response or if some other corrupted or incorrect response is received (steps 425 and 440), a control path bad condition is logged at the port. Upon recording the condition of the control path, the control path test is complete. At this point, the health check test is over and the port resumes normal operation.

[0055] Multi-Drain Concept

[0056] FIG. 5 is a block diagram of prior art fibre channel switching system. Buffer memory 505 is coupled to a single frame writer 500 and a single frame reader 510. SBAR0515 and SBAR1520 are both coupled to frame reader 510. In operation, frames are stored in buffer memory 505 until they are switched through either SBAR0 or SBAR1. However, since only a single frame reader 510 is coupled to both SBAR0515 and SBAR1520, only a single switching element is operative at a given moment. Hence, only one read and one write operation is allowed in a particularly defined micro-cycle.

[0057] FIG. 6 is a block diagram of a fibre channel switching system according to one embodiment of the present invention. Instead of a single frame reader coupled to the switching elements, each switching element has a frame reader operatively coupled thereto. As illustrated in FIG. 5, SBAR0625 is coupled to frame reader 610 and SBAR1620 is coupled to frame reader 615. In operation, a frame can be transmitted from buffer memory 605 through frame reader 610 to SBAR0625. At the same time, a frame can be transmitted from buffer memory 605 through frame reader 615 to SBAR1620. This type of operation allows for increased bandwidth since two switching elements are operating at the same time. In addition, in a failure situation involving one of the switching elements, a second switching element remains operating resulting in decreased bandwidth but no down time.

[0058] Automatic Switchover

[0059] Referring now to FIGS. 1 and 7, one embodiment of the automatic switchover aspect of the present invention is described. The automatic switchover aspect switches from an active switching element, such as SBAR130, to a redundant switching element, such as SBAR025, in response to a failure condition. The switchover is carried out so that any frames that may be transmitting at the time of the failure are completed. For example, only one port might be bad, but the other ports are still reliably transmitting frames. If the failure is not time critical requiring immediate switchover, the present invention allows for the continued operation of the switching element until all ports are out of frame.

[0060] To begin, a failure condition is detected at a switching element (step 700). Several examples of a non-catastrophic failure condition include a bad port, one or more ports identifying a channel from the switching element as bad while other channels continue to operate or failure to clear the busy bit for one or more ports to name a few. If the failure is a catastrophic failure requiring immediate attention without regard for the status of the frame transmission (step 705), such as loss of power, all busy bits not working or component failure that keeps an active switching element from performing an operational task to name a few, the switchover is immediate. However, if it is determined that the switchover can be delayed for a period of time without harming the integrity of the switching system, the automatic switchover feature provides a method of waiting until all frames are transmitted before performing the switchover.

[0061] Continuing with FIG. 1, SB logic within fibre port module 10 receives a signal that indicates the identity of the active SBAR. In the illustrated example, SBAR130 is the active SBAR. When a changeover is to occur, the signal will indicate the redundant SBAR is the new active SBAR. In the illustrated example, SBAR025 is the new active SBAR. The SB logic queries the TX port to determine if a frame is currently being transmitted over outbound path 80. If the port is “in frame” or currently transmitting a frame, then the SB logic will wait a period of time before executing the switch until a busy bit is received from the TX port. In one embodiment, the period of time is 10 microseconds.

[0062] Once the frame transmission is complete and the busy bit is received and detected by the SB logic, the SB logic proceeds to change the inbound data select to the value indicated by the main control bit. The inbound data select identifies the active SBAR, in this case the former redundant switching element SBAR025. The switchover has taken place and the busy bit is cleared to indicate the port is available to transmit frames again.

[0063] While there have been described above the principles of the present invention in conjunction with a specific embodiment, it is to be clearly understood that the foregoing description is made only by way of example and not as a limitation to the scope of the invention. Particularly, it is recognized that the teachings of the foregoing disclosure will suggest other modifications to those persons skilled in the relevant art. Such modifications may involve other features which are already known per se and which may be used instead of or in addition to features already described herein.

[0064] Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure herein also includes any novel feature or any novel combination of features disclosed either explicitly or implicitly or any generalization or modification thereof which would be apparent to persons skilled in the relevant art, whether or not such relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as confronted by the present invention. The applicants hereby reserve the right to formulate new claims to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.