Peer link fault isolation

[0002] The present invention is directed to a method for isolating a fault in a peer-to-peer communication link between two processors.

[0003] Numerous computer systems have been assembled with two redundant processors. In particular, data storage systems are known in which a pair of storage processors are connected to an array of disk drives. For example, such a system is disclosed in U.S. Pat. No. 5,922,077 (Espy et al.). The full disclosure of said patent is hereby incorporated by reference herein. Espy et al. describes a dual data storage controller system in which the controllers are connected to one another by a peer-to-peer communication link. Each data storage controller is connected to a fibre channel loop in connection with each of the disk drives in the disk array. Fail-over switches provide each data storage controller with a means for connecting to either one of the fibre channel loops.

[0004] The peer-to-peer link between processors is typically used for configuration management. Prior systems lacking a dedicated peer-to-peer communication link may have used back end buses for transportation of configuration management information. If there is a failure in this back end link, the loss of all back end buses is implied in which case the processor will not be able to access the disk drives. As such, there is no benefit in continuing to operate the storage processor and it might as well be shut down. In such a configuration, if the back end is working and the storage processor cannot contact its peer, it is safe to assume that the peer is dead.

[0005] In the case where the peer-to-peer communication takes place over a single dedicated configuration management channel between the two processors, if a processor cannot communicate with its peer, it can't be sure if the peer is dead or if the communication channel failed. Therefore, it does not know if certain operations can be safely performed. In order to address this problem, alternative peer-to-peer communication links were developed. In particular, by the use of mailboxes on disk drives, processors may be able to communicate through the disk drives to coordinate a safe completion of operations that require coordination between the two processors. For example, when a write cache is mirrored between the two processors, such operation needs to terminate when communication over the peer-to-peer link is interrupted. Upon termination of the mirroring, a write cache needs to be dumped to complete the write operations. The alternative communication link through the mailboxes on disk drives permits coordination between the processors so that only one of them dumps the cache so that the cache dump can proceed without interference from the other processor. It is only necessary that one of the caches be dumped since they have been mirrored. While termination of the cache mirroring operation is able to be carried out through the mailboxes of the disk drives, there remains the problem of identifying the cause of the failure of the peer-to-peer communication link.

[0006] In accordance with embodiments of the invention, dual processors are programmed to perform a fault isolation method so that the processor causing the fault in the peer-to-peer communication link can be replaced while the other processor remains operational during the fault detection.

[0007] An embodiment of the method of the invention involves a number of steps beginning with detecting an inability to communicate over the peer-to-peer communication link. The method suspends a preselected one of the storage processors and suspends its operation arbitrarily indicating a fault with that storage processor. A user replaces the indicated storage processor with a new storage processor. The other storage processor detects replacement of the first storage processor. The new storage processor powers up. Before fully booting up, the storage processors test the peer-to-peer communication link. If the problem with the peer-to-peer communication is solved, operation returns to normal. If after the first storage processor was replaced and the peer-to-peer communication link still fails to provide communication between the two processors, the second storage processor recognizing itself as a survivor, instructs the first storage processor to boot up and then suspends operation.

[0008] An embodiment of the program code for powering up a storage processor includes code for attempting to communicate over the peer-to-peer communication link. Code is further included to permit communication with its peer storage processor by leaving messages in at least one of the disk drives. The code knows to suspend the storage processor in a first chassis position when there is a failure to communicate over the peer-to-peer communication link, and both storage processors are running in normal operation mode. Code is also included so that a survivor of a previous peer-to-peer communication link failure detecting replacement of the peer storage processor and an inability to communicate over the peer-to-peer communication link, instructs its peer through leaving a message in at least one of the disk drives to boot up and suspends itself. In the survivor mode, if replacement of the peer storage processor has not been detected and a peer-to-peer communication link failure is alerted, it will instruct its peer through leaving a message in at least one of the disk drives to suspend operation.

[0009] In accordance with the data storage system of an embodiment of the invention, a first and second storage processor are connected via a peer-to-peer communication link. A first communication channel connects the first storage processor to a plurality of disk drives. A second communication channel connects the second storage processor to the disk drives. A program code is maintained in each processor for enacting a survivor mode in which the processor can detect replacement of the peer storage processor. Each of the storage processors may be provided with a failure indicator which could be a light, a buzzer, or the like so that a user knows which processor to replace.

[0010] Other objects and advantages of the invention will become apparent during the following description of the presently preferred embodiments of the invention taken in conjunction with the drawings.

[0011] FIG. 1 is a schematic block diagram of a dual processor data storage system of an embodiment of the present invention.

[0012] FIG. 2 is a flow chart of a fault isolation method of an embodiment of the present invention.

[0013] FIG. 3 is a flow chart of a power-up program of an embodiment of the present invention.

[0014] An example of the dual processor data storage system is discussed with respect to FIG. 1. Storage processors 12 are each mounted on its own board. The systems are generally housed within a chassis. Insertion of a processor board into a chassis slot causes the processor to assume the identity associated with that slot. Thus, there would be a processor A slot and a processor B slot (or first processor and second processor) in the chassis for receiving the storage processors 12. Connections with other components of the system are made through a backplane or by making cable connections from one port to another. The system includes a fault indicator 14 associated with each of the storage processors 12. The fault indicator 14 may be a light or a buzzer or other indicator readily detectable by a user or technician servicing the system. The fault indicator 14 may be located on the chassis or on the processor board.

[0015] The dual processor system of an embodiment of the invention includes a single dedicated peer-to-peer communication link 20 over the backplane. The processors can use the link to coordinate their activities. In a particular preferred embodiment of the invention, each processor includes a write cache 16. Moreover, the write cache 16 is mirrored between the two processors. Mirroring of the write operations is achieved with the assistance of communications through the peer-to-peer communication link 20. A host computer 30 is connected either directly or through a network with the storage processors 12. The host uses the storage system for reading and writing data to and from a disk array. The data storage system may include any number of disk drives in communication with the storage processors. In a presently preferred embodiment, the disk drives 40 are dual ported so that redundant communication channels may be employed. The communication channel from the storage processor to the disk drives may be a fibre channel loop or other known bus for communicating data. In a presently preferred embodiment, storage processor A communicates over a first fibre channel loop with the disk drives 40 in the array and storage processor B communicates over a second fibre channel loop with the disk drives 40. Failover switches may be inserted between the fibre channel loops and the storage processors so that if one loop should be faulty, the two storage processors could share the other's fibre channel loop. Such a failover switching system is described in U.S. Pat. No. 5,922,077 (Espy et al.) which has been incorporated by reference herein.

[0016] Referring now to FIG. 2, an algorithm performed by the storage processors when the peer-to-peer communication link fails is described. The method is initiated when a peer link fault is detected 100. For example, when storage processor A sends a message over the peer-to-peer communication link 20 and it fails to receive an expected response from storage processor B in a given time period and after a given number of tries, a fault has been detected and the algorithm initiates. Operations dependent upon the peer-to-peer communication link are notified of the failure. In a preferred embodiment, the cache state machine for the write cache is alerted to move toward a frozen state 102 in anticipation of a cache dump. The operation followed will be similar to other faults which prevent caching.

[0017] It is arbitrarily predetermined which of the storage processors will be suspended so that it may be removed and replaced to determine which of the processor boards is responsible for the failure of the peer-to-peer communication link. For the purposes of the present application, we assume that it will be storage processor A which gets suspended any time a peer link failure is first detected. Thus, the following steps depend on which SP (storage processor) slot a processor is in. The SP determines 104 which SP it is. Storage processor A stops sending write complete messages to the host for any writes which fail to get mirrored 106. A diskmail manager is used to determine if diskmail is working. Storage processor A sends an “Are you there?” type message 108 to storage processor B by writing the message to the appropriate mailbox in at least one of the disk drives in the disk array. Diskmail is the process by which messages from one processor are left in mailboxes in a disk drive. In a preferred embodiment, three redundant mailboxes are used, each in a different disk drive. Messages are written to all three mailboxes. Each of the three disk drives has two mailboxes, one for each storage processor. The storage processors periodically sniff the mailboxes to pick up any diskmail messages from their peer processor. The next action depends on the status of diskmail operation 110. If diskmail faults, the cache manager will panic 112 its storage processor. A fault occurs when the diskmail manager is unable to read or unable to write to the disk drive(s). Since the storage processor can't write to disk anyway, a cache dump is moot. Panic 112 means the storage processor is halted and memory may be saved to serve diagnostic purposes.

[0018] Meanwhile, storage processor B also monitors the operation of diskmail 114. If B's diskmail manager is unable to read or unable to write to the disk drive, storage processor B will panic 116, thereby shutting down. With diskmail working, however, storage processor B sends a “suspend yourself” message to storage processor A 118. Storage processor A has been preselected as the one which will be suspended first when trying to identify which processor is the cause of a peer-to-peer communication link failure. When storage processor A receives the “suspend yourself” message, it follows the instruction. Storage processor A enters a suspended state 120 in which it monitors communications over the dedicated peer-to-peer communication link 20. In a preferred embodiment, it does nothing else. In any case, suspension of full operation requires termination of write operations to disk drives. Preferably, all substantive communications with disk drives and host computers terminate. The arbitrarily preselected storage processor A turns on its fault light 122. A user or technician addressing the problem will thus be alerted to trying to substitute a new processor board for the one in slot A.

[0019] Referring back to storage processor A looking for the diskmail message 110, if the diskmail can be read but there is no response from the processor B after waiting a reasonable time, it is presumed that the storage processor B is dead. In this case, storage processor A will follow the steps for becoming the surviving processor. On the other hand, with both processors A and B running in normal operation, the preselected processor A will be suspended and processor B will become the surviving processor. The surviving processor begins by sending a peer dead message to the cache machine to proceed on that basis 124. Thus, the cache machine proceeds with dumping the cache 126. The surviving storage processor will then continue running in survivor mode 128.

[0020] In the survivor mode 128, a storage processor will monitor operation of the peer-to-peer communication link 20. The survivor will also sniff a signal indicative of when a peer processor gets inserted into the chassis. Thus, at least once per second, the surviving processor will sniff the peer-inserted signal. This signal is toggled when a board is removed or inserted into its slot. The peer-inserted signal for each processor board is preferably carried on a backplane trace available to both processors. The survivor will also sniff the disk mailbox locations for detecting any communications from the peer processor. The survivor will also continue to communicate with the host computer.

[0021] In the survivor mode, the processor will passively monitor the peer-to-peer communication link 130. If it is working, operation can return to normal 132. If it is not working and no diskmail has been received from the other processor, survivor mode continues 134. If a diskmail is received from the other processor 134, the survivor checks to determine if the peer-inserted signal has been toggled 136. If not, that should mean that a new processor has not been inserted, so the survivor sends a suspend yourself message to the peer processor 138 and survivor mode continues. This handles the case where the suspended processor begins powering itself up for some aberrant reason. If the survivor detects that a new processor has been inserted, the survivor sends a boot up message to the peer 140. This is in preparation for now getting itself replaced, to determine if it is the problem on the peer-to-peer communication link. Upon learning through diskmail that the peer has booted up 141, the survivor suspends itself and illuminates its fault light 142. The survivor stops issuing write completes to the host and passes operation over to the peer processor 142. The peer now enters the survivor mode 144. The user or technician can now try replacing the processor with its fault light on. Most often replacement of one or the other processor will cure the peer-to-peer communication link. If not, the technician will need to investigate further.

[0022] Now we consider the case of a processor inserted into the chassis and beginning power up. The processor does not know whether or not there was any problem with the peer-to-peer communication link. Referring to FIG. 3, during power up, one of the first operations is to attempt a connection over the peer-to-peer communication link 200. If the peer-to-peer communication link works 204, then the processor can proceed with a normal boot 202. If the other processor is a survivor of a peer-to-peer communication link failure, it will also return to normal operation once it realizes the peer-to-peer communication link is working. In power up, if the peer-to-peer communication link does not work 204, the powering up processor sends a diskmail message to the peer processor 206. If the diskmail message faults 208, the processor panics 209. If there is no response after allowing an appropriate delay for the peer to power up 208, it is assumed that the peer is dead and the processor proceeds in survivor mode 210. If an answer is received through the diskmail 208, then communications with the peer are conducted so as to determine if one is in survivor mode 211. In a preferred embodiment, when a processor powers up, it sends an “I am initializing” message over diskmail to the peer processor. If both processors receive this message, then neither one is a survivor and the arbitrarily preselected routine is entered in which the storage processor A is suspended and its fault light illuminated while storage processor B boots up and becomes the survivor 212. If there is a survivor processor, the survivor sends an instruction 214 to its peer as set forth in the survivor mode algorithm of FIG. 2. If the survivor detects that this is a power up of a newly replaced processor, then it will suspend itself and turn control over to the new processor 142. If the survivor detects that the processor is the same processor that was supposed to be already suspended, it will instruct it to suspend itself again so that it can replaced by the technician 138.

[0023] By providing each of the dual processors with the fault detection algorithms described herein, the technician is easily guided by the fault indicators through the task of locating the problem. Advantageously, one of the processors remains operational during the entire replacement process in which the technician searches for the faulty board.

[0024] In alternative embodiments, the disclosed program code may be implemented as a computer program product for use with a data storage computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), pre-loaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web).

[0025] Although the embodiments hereinbefore described are preferred, many modifications and refinements which do not depart from the spirit and scope of the invention may be contemplated by those skilled in the art. It is intended that all such modifications, including but not limited to those set forth above, be covered by the following claims.