Failover and Failback System for a Direct Access Storage Device

1. Field of the Invention

The present invention relates to a storage controller for interfacing between a host system and a direct access storage device (DASD) system and, in preferred embodiments, a storage controller including a failback and failover system for handling a point of failure within the storage controller.

2. Description of the Related Art

In a large distributed computer system, a plurality of host computers and devices are typically connected to a number of direct access storage devices (DASDs) comprised of hard disk drives (HDDs). The DASDs may be organized in a redundant array of independent disks, i.e., a RAID array. A RAID array is comprised of multiple, independent disks organized into a large, high-performance logical disk. A controller stripes data across the multiple disks in the array and accesses the disks in parallel to achieve higher data transfer rates. However, utilizing multiple disks in an array increases the risk of failure. The solution in the art is to employ redundancy in the form of error-correcting codes to tolerate disk failures. The arrangement and organization of RAID arrays is described in Peter M. Chen, Edward K. Lee, Garth A. Gibson, Randy H. Katz, and David A. Patterson, "RAID: High-Performance, Reliable Secondary Storage," ACM Computing Surveys, Vol. 6, No. 2, June 1994, which is incorporated herein by reference in its entirety.

Not only is there a risk associated with the failure of a hard disk drive in a DASD system such as a RAID array, but there is also a risk of failure at a point within a storage controller which controls read and write operations between host computers and the DASDs. The conventional storage controller is typically designed to handle hardware failures. One such storage control designed to handle certain hardware failures is the storage controller utilized in the International Business Machines Corporation's (IBM) 3990 Model 3 system. This storage controller has two storage clusters, each of which provides for selective connection between a host computer and a DASD. Each cluster is on a separate power boundary. Further, a non-volatile storage unit (NVS) is associated with each cluster. The NVS is a memory array supported by a battery backup system. The NVS stores back-up copies of modified data in the event of a hardware failure within the storage controller while data is being written to a DASD. The NVS provides a storage back-up that is as secure as writing the data to a magnetic disk. Failback systems for storage controllers are described in U.S. Pat. Nos. 5,636,359, 5,437,022, 5,640,530, and 4,916,605, all of which are assigned to IBM, the assignee of the subject application, and all of which arc incorporated herein by reference in their entirety.

In U.S. Pat. Nos. 5,437,022 and 5,640,530, as with the IBM 3990, model 3, the storage controller is comprised of two clusters, wherein each cluster has a cache and a NVS. The cache buffers frequently used data. When a request is made to write data to a DASD attached to the storage controller, the storage controller may cache the data and delay writing the data to a DASD. Caching data can save time as writing operations involve time consuming mechanical operations. The cache and NVS in each cluster can intercommunicate, allowing for recovery and reconfiguration of the storage controller in the event that one of the memory elements is rendered unavailable. For instance, if one cluster and its cache fails, the NVS in the other cluster maintains a back-up of the cache in the failed cluster.

In one caching operation, referred to as DASD fast write, data buffered in the cache is backed-up in the NVS. Thus, two copies of data modifications are maintained, one in the cache and another in the NVS. DASD fast write is typically used for data that will be written to the DASD. If the data to be written to the DASD is present in the cache, instead of writing the data to the DASD, the controller copies the data into the cache and NVS without having to interrupt processing to immediately write to the DASD. If the data to be written is not present in the cache, then the data is written to the DASD and cache simultaneously and immediately. Another caching operation, referred to as cache fast write, involves writing data to the cache without using the NVS. Cache fast write is intended for data that is not needed for job completion or that can be easily reconstructed.

Notwithstanding, the storage controller disclosed in U.S. Pat. Nos. 5,437,022 and 5,640,530 comprises a complex and custom hardware structure, wherein each storage path in a cluster requires its own custom microcontroller and microcode. In fact, U.S. Pat. No. 5,640,530, at column 6, lines 50-57, makes note of the added complexity of the system which requires that each microcontroller maintains and updates information concerning the location of data for read, writes and other data transfers.

Moreover, data transfer rates in a storage controller are determined by the data-transfer rates at each connectivity point within the controller. The speed at which processed data travels between a host and DASDs, such as a RAID array, is restricted by the maximum data-transfer rate possible through each peripheral interface. In current systems, data transfer rates are significantly degraded when a cluster fails, because the remaining cluster must then handle all data transfers. In storage control systems which involve complex arrangements to handle failovers, such as the system disclosed in U.S. Pat. Nos. 5,437,022 and 5,640,530, the complexity of the system can further degrade data transfer rates when one cluster must take over data transfers previously handled by the failed clusters.

Yet further, in some prior art systems, the NVS and storage controller have single points of failure which can disable the entire cache and NVS, resulting in substantial performance degradation.

To address the shortcomings in the prior art described above, preferred embodiments of the present invention provide a system for handling failures in a storage controller interfacing between a plurality of host systems and direct access storage devices (DASDs). The storage controller directs data from the host systems through first and second data paths in the storage controller to a DASD. A first processor, first non-volatile memory unit (NVS), and a first cache are associated with the first data path and a second processor, a second NVS, and a second cache are associated with the second data path. A bridge provides communication between the first processor and the second NVS and the second processor and the first NVS.

During normal operations prior to a failure in the storage controller, data directed to the first data path is written in the first cache and communicated with the first processor to the second NVS via the bridge for storage in the second NVS. Data directed to the second data path is written in the second cache and communicated with the second processor to the first NVS via the bridge for storage in the first NVS. A point of failure within at least one of the first processor, first cache, and first NVS is handled by routing the data directed to the first data path to the second processor and writing the routed data to the second cache and the second NVS. Upon repairing the point of failure, the data directed to the first data path is rerouted to the first processor, wherein the rerouted data is written in the first cache and communicated with the first processor to the second NVS via the bridge for storage therein.

It is an object of the present invention to provide a storage controller including a first and second data paths to direct data from host systems and DASDs, wherein a processor, cache and NVS are associated with each data path.

It is a further object that the NVS in the first data path back-up data transfers through the second data path in case a component within the second data path fails.

It is yet a further object to provide improved methods for handling hardware failures within the first data path and for failing back to the first data path after the hardware failure has been repaired, such that no single point of failure will incapacitate the entire storage controller.

It is still a further object that common components be used for the processor, NVS, and cache to minimize the complexity of the storage controller.

Referring now to the drawings in which like reference numbers represent corresponding parts throughout:

FIG. 1 illustrates a block diagram of the components and architecture of a preferred embodiment of a storage controller in accordance with the present invention;

FIG. 2 illustrates a block diagram of a preferred embodiment of the arrangement of DASDs and their attachment to components of the storage controller in accordance with the present invention;

FIG. 3 illustrates a block diagram of how the storage controller embodiment of FIG. 1 handles data writes to logical subsystems within the DASDs during normal operations;

FIG. 4 is a flowchart that illustrates general logic implemented within the storage controller to handle a failure of a hardware component within the storage controller in accordance with a preferred embodiment of the present invention;

FIGS. 5a and 5b are block diagrams illustrating the storage controller embodiment of FIG. 1 handling a point of failure in accordance with the preferred logic illustrated in FIG. 4;

FIGS. 6a and 6b show a flowchart that illustrate general logic implemented within the storage controller to failback after the failed hardware component has been repaired in accordance with a preferred embodiment of the present invention;

FIGS. 7a and 7b are block diagrams illustrating the storage controller embodiment of FIG. 1 performing failback operations after the failed hardware component has been repaired in accordance with the preferred logic illustrated in FIG. 6;

FIGS. 8a and 8b are block diagrams illustrating the storage controller embodiment of FIG. 1 performing failback operations when a device adaptor has failed.

In the following description, reference is made to the accompanying drawings which form a part hereof, and which is shown, by way of illustration, several embodiments of the present invention. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

Hardware Environment and Architecture of the Storage Controller

FIG. 1 illustrates a block diagram of the components and architecture of a preferred embodiment of a storage controller 2 which interfaces between host computers or devices (not shown) and DASDs 46, 48, which include multiple RAID arrays. In preferred embodiments, the DASDs are magnetic storage units such as hard disk drives. The host computers and devices are connected to host adaptors 4, 6, 24, 26 via a bus interface (not shown), such as a SCSI bus interface. The host adaptors 4, 6, 24, 26 may be comprised of an Enterprise System Connection (ESCON) adaptor which provides access to ESCON channels and connections. Each host adaptor 4, 6, 24, 26 may be comprised of a series of host adaptors which connect to a host system.

In preferred embodiments, the storage controller 2 is divided into two clusters, cluster 0 and cluster 1. Cluster 0 consists of host adaptors 4, 6, a non-volatile storage unit (NVS) 8, a cache 10, a processor 12, a device adaptor bus 14, device adaptors 16, 18, 20, 22. Cluster 1 consists of host adaptors 24, 26, an NVS 28, a cache 30, a processor 32, a device adaptor bus 34, and device adaptors 36, 38, 40, 42. A host adaptor bridge 44 interfaces the components of cluster 0 with cluster 1. The host adaptors 4, 6, 24, 26 are connected to the host adaptor bridge 44. In preferred embodiments, the bridge 44 is a dual master bus which may be controlled by one of the processors 12, 32 or one of the host adaptors 4, 6, 24, 26. In further embodiments, the host adaptor bridge 44 may include bridge technology to allow the bus to operate at its own clock speed and provide a buffer to buffer data transferred across the bridge 44. The bridge 44 interconnects the host adaptors 4, 6, 24, 26 with the processors 12, 32. In preferred embodiments the processors 12, 32 are symmetrical multi-processors, such as the IBM RS/6000 processor. Each processor 12, 32 maintains information on the configuration of the other cluster in order to reroute data transfers directed toward the other cluster.

The caches 10, 30 may be external to the processors 12, 32 or included in the processor 12, 32 complex. A processor 12, 32 in one cluster can communicate with the other processor, NVS 8, 28 and cache 10, 30 in the other cluster via the host adaptor bridge 44. In preferred embodiments, the NVS 8, 28 consists of a random access electronic storage with a battery backup. Storage time for a fully charged battery may last a couple of days. In preferred embodiments, the NVS battery is continuously charged whenever primary power is applied during normal operations. The battery will supply power necessary to maintain contents of the NVS 8, 28 intact until power is restored. The cache 10, 30, on the other hand, is a volatile storage unit that cannot maintain data in the event of a power failure.

Device adaptor bus 14 interconnects the processor 12 with the device adaptors 16, 18, 20, 22 and device adaptor bus 34 interconnects processor 32 with device adaptors 36, 38, 40, 42. The device adaptors 16, 18, 20, 22, 36, 38, 40, 42 interface between the storage controller and the DASDs, or RAID array of hard disk drives. In preferred embodiments, the device adaptors 16, 18, 20, 22, 36, 38, 40, 42 employ the Serial Storage Architecture (SSA) developed by IBM. In such case, the DASDs may be interconnected in a loop topology including multiple RAID arrays. FIG. 2 shows a preferred embodiment of attaching a device adaptor 22, 36 from each cluster to a plurality of DASD devices using a loop topology. FIG. 2 shows two loops 54, 56, wherein each loop has eight DASD devices. Loop 54 follows the following path: port A1 of the device adaptor 22 to DASD A through DASD D, which then extends to port A1 of device adaptor 36, which then traverses from port A1 to A2 via line 58, through DASD E to DASD H, to port A2 of device adaptor 22, then to port A1 of device adaptor 22 via line 60. Loop 56 is formed in a similar manner using ports B1 and B2 of the device adaptors 22, 36. In this way, each loop 54, 56 is connected to a device adaptor 22, 36 from each of the clusters 0, 1. Connections to loop 54 are made via ports A1 and A2 of the device adaptors 22, 36 and to loop 56 via ports B1 and B2 of the device adaptors 22, 36. The DASDs within a loop 54, 56 are connected to each other, such that when a device adaptor 22, 36 accesses a disk within a loop 54, 56, the device adaptor 22, 36 communicates through the intermediary DASDs between the device adaptor 22, 36 and the DASD to be accessed. Thus, if a DASD in a loop 54, 56 or device adaptor 22, 36 fails, the DASDs may always be reached via another port or device adaptor. Moreover, the device adaptors 22, 36 may intercommunicate via the lines interconnecting the DASDs. If a device adaptor 22, 36 has a failure and cannot communicate with a DASD in the loop 54, 56, then the failed device adaptor 22, 36 could signal the functioning device adaptor 30 to access the DASD. Thus, rerouting can occur at the device adaptor level if one device adaptor connected to the loop fails.

In alternative embodiments, a loop 54, 56 of disks could include more or less DASD devices than shown in FIG. 2. Moreover, the topology of the arrangement of the DASD devices may differ, e.g., strings, complex switch strings, and cyclical paths. In yet further embodiments, the device adaptors may utilize different interface standards, such as SCSI. Still further, each loop 54, 56 could include multiple RAID arrays, depending on the total number of DASDs in the loop. In preferred embodiments, RAID 5 is used. In RAID 5, parity is striped uniformly over all of the disks. The advantage of RAID 5, along with RAID 1 and 3, is that the RAID array remains operational even if one disk within the RAID array fails. Different arrangements for connecting device adaptors to groups of disks are described in "A Practical Guide to Serial Storage Architecture for AIX" (IBM document SG24-4599-00, International Business Machines Corporation, copyright 1996).

By having one device adaptor from each cluster 0, 1 attached to each loop of DASDs, failure in one cluster and/or the device adaptors associated with the failed cluster will not prevent the functioning cluster from accessing the loop. Thus, no single point of failure in a cluster and/or in a device adaptor will prevent the other cluster from accessing a group of DASDs. Moreover, if a device adaptor, such as device adaptor 22, fails in a cluster that is otherwise functioning properly, then the re-routing to the other device adaptor 36 can occur at the device adaptor level. Alternatively, the failure of a device adaptor can be treated as a failure by the entire cluster, thereby transferring control over to the functioning cluster to access the DASD.

In the storage controller 2 embodiment of FIG. 1, each cluster 0, 1 has four device adaptors, wherein each device adaptor can be connected to two loops, each loop having numerous disks. Thus, the storage capacity of all DASDs attached to the clusters is significant. Each group, or loop, of DASDs attached to a device adaptor 16, 18, 20, 22, 36, 38, 40, 42 includes multiple logical volumes. For memory management purposes, the logical volumes or storage space available in the DASDs attached to a device adaptor can be segregated into logical subsystems (LSS). These LSSs are presented to a host. A device adaptor 16, 18, 20, 22, 36, 38, 40 or 42 can be associated with multiple LSSs, such that the associated device adaptor is responsible for accessing associated LSSs. As discussed, a group of DASDs attached to a pair of device adaptors, such as the loops 54, 56 of disks attached to device adaptors 24, 36 in FIG. 2, can include multiple RAID arrays. Each RAID arrays has multiple logical volumes. The logical volumes associated with a RAID array are mapped to a logical subsystem, which in turn is associated with a device adaptor. Thus, a logical subsystem represents a collection of logical volumes in a RAID array to which a pair of device adaptors are attached.

FIG. 3 is a block diagram illustrating how the storage controller 2 handles four logical subsystems, LSS A, B, C, D, each of which are associated with a pair of device adaptors. For illustrative purposes, device adaptors 20, 22, 36, and 38 are considered. However, the arrangement described with respect to FIG. 3 could apply to the other device adaptors 16, 18, 40, 42 and other LSS arrangements. Each LSS A, B, C, D is comprised of a group of logical volumes within the DASDs. For instance, device adaptor 20 in cluster 0 and device adaptor 36 in cluster 1 are associated with a group of DASDs having a plurality of logical volumes that for illustrative purposes are divided into two logical subsystems, LSS A and LSS B. When both clusters 0, 1 are operational, LSS B is directed to device adaptor 36 in cluster 1 and LSS A is directed to device adaptor 20 in cluster 0. Similarly, device adaptor 22 in cluster 0 and device adaptor 38 in cluster 1 are associated with a group of disks having a plurality of logical volumes that for illustrative purposes are divided into logical subsystems, LSS C and LSS D. Each host adaptor 4, 6, 24, 26 is associated with at least one LSS. Thus, a host system directed to LSS A, B interfaces with host adaptor 4, 6 in cluster 0. In normal operations, modified data stored in cache 10 is backed-up to NVS 28 and modified data stored in cache 30 is backed-up to NVS 8. Thus, the NVS 8, 28 stores data from the cache 10, 30 in the other cluster.

When a host writes data to a particular LSS, the host adaptor 4, 6, 24, 26 associated with such LSS will direct the data write to the cluster which handles requests for such LSS. For instance, if a host attached to host adaptor 4 writes data to LSS A, host adaptor 4 will direct the request to the processor 12. Processor 12 will maintain a copy of the data in the cache 10 and may back-up the write data in the NVS 28 in cluster 1. In this way, the NVS 28 provides a back-up to the cache 10 if cluster 0 fails. The processor 12 will then direct any access of LSS A to device adaptor 20, which provides access to the LSS A track.

FIGS. 4 and 6 are flowcharts illustrating the logic used to control failover and failback, i.e., recovery, operations, respectively. Such logic may be implemented in the firmware of the processors 12, 32 and other devices embedded with processors within the storage controller 2 or as hardware logic (e.g., circuits and gates) throughout the storage controller 2. Those skilled in the art will recognize that this logic is provided for illustrative purposes only and that different logic may be used to accomplish the same results. FIGS. 5a, 5b, 7a, 7b, 8a, and 8b are block diagrams illustrating how the components of the storage controller 2 interact during failover and failback operations.

FIG. 4 illustrates an embodiment of logic for handling a failure of an entire cluster, which in FIG. 4 involves a failure of cluster 0. An entire cluster fails when either the processor 12, 32 and/or the NVS 8, 28 fails. The description of the failover handling with respect to cluster 0 would equally apply to a failure in cluster 1. Block 70 represents the failure of cluster 0, which occurs when either the processor 12 and/or the NVS 8 in cluster 0 fails. When cluster 0 fails, all unmodified LSS A, C data not written to NVS 28 is lost. Control transfers to block 72 which represents the host adaptors 4, 6, 24, 26 delaying or ceasing to respond to their attached hosts until the failover is completed. Control transfers to block 74 which represents the processor 32 taking over writes to the LSS A, C tracks previously handled by processor 12 in cluster 0. In preferred embodiments, the processors 32 would signal host adaptors 4, 6, 24, 26 and the host adaptors 4, 6, 24, 26 would redirect LSS A, C track requests to processor 32.

Control then transfers to block 76 which represents the processor 32 signaling the device adaptors 36, 38 in cluster 1 to handle all accesses to LSS A, C tracks. Control transfers to block 78 which represents the host adaptors 4, 6, 24, 26 presenting busy signals to all attached host systems accessing LSS A, B, C, D. Control transfers to blocks 80 and 82 which represents the processor 32 destaging all modified LSS B, D data in the cache 30 to the DASD via the device adaptors 36, 38 handling LSS B, D. Control transfers to block 84 which represents the processor 32 creating directory entries in cache 30 for data in NVS 28 and copying modified LSS A, C data in NVS 28 to the cache 30. This is to cache a redundant copy of modified data in the NVS 28 in case of a subsequent failure of either the NVS 28 or cache 30. Control transfers to block 86 which represents the processor 32 adjusting settings so all LSS B, D backups from cache 30 go to NVS 28, whereas previously they were directed to NVS 8 in cluster 0. After destaging of LSS B, D data from the cache 30 is complete, control transfers to block 88 which represents the processor 32 signaling the host adaptors 4, 6, 24, 26 to respond to their respective hosts and direct all I/O operations for LSS A, B, C, D toward cluster 1 and processor 32. Thereafter, at block 90, all writes to LSS A, B, C, D are directed to cache 30 and backed-up in NVS 28. Any data subsequently written to tracks LSS A, C are directed through device adaptors 36, 38 (FIG. 3) in cluster 1.

FIG. 5a is a block diagram illustrating how the storage controller 2 initially responds to a failure. The processor 12, NVS 8, and device adaptors 20, 22, and cache 10 in cluster 0 are down. FIG. 5a illustrates the processor 32 copying the modified LSS A, C data from the NVS 28 to the cache 30 as described at block 84 in FIG. 4. FIG. 5a further illustrates the processor 32 destaging LSS B, D data in the cache 30 to the LSS B, D tracks in the DASD via device adaptors 36, 38.

FIG. 5b is a block diagram illustrating how the processor 32 handles I/O operations for LSS A, B, C, D in failover mode after data is destaged at blocks 70-86 in FIG. 4. All writes to LSS A, B, C, D tracks are written to cache 30 and a back-up copy of all modified LSS A, B, C, D tracks are maintained in NVS 28, which in failover mode is controlled by processor 32.

In alternative embodiments of FIG. 4, the destage process can be delayed to immediately respond to and process host requests. In such case, at block 82 instead of waiting for the device adaptors 36, 38 to destage the data to the DASDs, the destage to DASD is scheduled and the processor 32 immediately proceeds to block 88 to handle host I/O requests to LSS B, D. In this alternative embodiment, block 88 would be modified such that the host adaptors 4, 6 would not wait for destaging to complete to direct host requests to cluster 1. While host requests are directed toward cluster 1, the device adaptors 36, 38 will continue writing the LSS B, D data from the cache 30 to the DASDs. In further embodiments, the device adaptors 36, 38 would include their own NVS to store the LSS B, D to be destaged. As a further precaution, the processor 32 could mark the modified LSS B, D data in the NVS 28 in case of a failure in the cache 30. The modified LSS B, D is marked by placing the track identification for the modified LSS B, D data in the NVS 28 which identifies which tracks in LSS B, D have been modified. In case of a failure to the cache 30, the marked tracks could be used to determine which modified LSS B, D data was lost. This alternative method reduces the added security of completely destaging modified LSS B, D data to the DASDs in return for a more immediate return to handling requests from the host.

FIG. 6 is a flowchart illustrating the failback process when cluster 0 is repaired and is ready to return to handle input/output requests to LSS A, C. As with the failover process, the description of the failback handling with respect to cluster 0 would equally apply to failback of cluster 1. At block 100, the failure in cluster 0 that led to the failover is repaired, rendering cluster 0 ready for I/O operations for LSS A, C. Control transfers to block 102 which represents processor 32 formatting NVS 8 as empty and assigning NVS 8 to processor 32. Control transfers to block 104 which represents the processor 32 destaging the LSS B, D data written in NVS 28 during failover mode to the DASD via device adaptors 36, 38. Control transfers to block 106 which represents the processor 32 signaling to stop cache fast writes to LSS A, C to the cache 30. Instead, writes to LSS A, C are DASD fast written to both cache 30 and NVS 28. Control then transfers to block 108 which represents the processor 32 scanning cache 30 for any writes to LSS A, C during failover mode and destaging all cache fast writes of LSS A, C in the cache 30 and destaging DASD fast writes of LSS B, D in NVS 28 to the DASD via device adaptors 36, 38 (FIG. 3).

Control transfers to block 110 which is a decision block representing the processor 32 determining whether a host has attempted to write to LSS B, D while the destaging of LSS B, D tracks from the NVS 28 is in progress. If so, control transfers to block 112; otherwise, control transfers to block 118. Block 112 is a decision block representing the processor 32 determining whether the write is for previously modified LSS B, D data stored in the NVS 28. If so, control transfers to block 114; otherwise, control transfers to block 116. Block 114 represents the processor 32 scheduling the modified LSS B, D data that the host is attempting to update in the NVS 28 for an immediate destage to DASD. After the destage of the LSS B, D modified data from the NVS 28, the LSS B, D data involved in the write is now unmodified. If the data written is unmodified, control transfers to block 116 which represents the processor 32 storing the unmodified data in the cache 30 and journaling the cache in the NVS 8 in cluster 0. From block 116 control transfers to block 118 which is a decision block representing the processor 32 determining whether the activity scheduled for block 108 is complete. If so, control transfers to block 120; otherwise control transfers back to block 110. After the destage is complete, there is neither cache fast write data for LSS A, C existing in the cache 30 nor is there LSS B, D data in the NVS 28.

After destaging all LSS B, D data from the NVS 28, control transfers to block 120 which represents the processor 32 directing all new writes to LSS B, D to NVS 8. Control transfers to block 124 which represents the processor 32 signaling host adaptors 4, 6, 24, 26 to present busy signals to all hosts accessing LSS A, C tracks. Control transfers to block 126 which represents the processor 32 destaging all LSS A, C metadata in the cache 30 to the DASDs via device adaptors 36, 38. Control then transfers to block 128 which represents the processor 32 signaling processor 12 in cluster 0 to begin handling all I/Os for LSS A, C. Control transfers to block 130 which represents processor 12 signaling the device adaptors 20, 22 to route all LSS A, C operations to cluster 0. Control transfers to block 132 which represents the processor 12 signaling the host adaptors to drive all LSS A, C data to processor 12. Control transfers to block 134 which represents the processor 12 scanning the NVS 28 to create directory entries and copy all modified LSS A, C data in NVS 28 to cache 10 Control then transfers to block 136, which represents the processor 32 freeing all space in cache 30 of any LSS A, C data. When the space is freed, all unmodified LSS A, C data is lost and the modified LSS A, C data is maintained in cache 10 and NVS 28. Control transfers to block 138 which represents the host adaptors 4, 6, 24, 26 responding to the hosts to direct all LSS A, C writes to the processor 12 in cluster 0. At block 140, the processor 12 begins journaling all LSS A, C writes to the NVS 28 and caching all writes to LSS A, C in cache 10.

FIG. 7a is a block diagram illustrating the first part of the failback process at blocks 100-120 where all LSS A, B, C, D requests are routed to processor 32. The block diagram of FIG. 7a shows the LSS B, D destaged from NVS 28, leaving only LSS A, C tracks in NVS 28. Further, LSS A, C data written in cache 30 is destaged to DASD, leaving the LSS B, D in cache 30. New writes for LSS B, D are backed-up in NVS 8. Afterwards, there is neither cache fast write data for LSS A, C existing exclusively in cache 30, nor is there any LSS B, D in NVS 28.

FIG. 7b is a block diagram illustrating the second part of the failback process at blocks 124-140 where processor 12 in cluster 0 comes back on-line. The processor 12 copies the modified LSS A, C data from the NVS 28 into the cache 10 and reasserts control over the device adaptors 20, 22 to access LSS A, C. After cluster 0 comes back on-line, writes to LSS A, B, C, D tracks are handled as shown in FIG. 3.

The above failover and fallback processes described with respect to FIGS. 4-7 are intended to handle any failure of one of the processors 12, 32 and/or one NVS 8, 28 in a single cluster. The above process is especially suited for systems where the processor 12, 32 in the cluster with a failed NVS 8, 28 cannot share access to the NVS 8, 28 in the active cluster. Moreover, the above described failover and failback processes may apply when cross cluster communication has failed. When cross cluster communication fails, an entire cluster can be taken down and a single cluster can be used to handle all I/O operations to the DASD.

In preferred embodiments, if a device adaptor fails, then failover can be handled in the manner described with respect to FIGS. 4, 5a, and 5b as if the entire cluster including the failed device adaptor failed. FIGS. 8a and 8b are block diagrams illustrating how the failback for a failed device adaptor is handled. In the exemplar of FIGS. 8a, b, the device adaptor 30 failed and during failover (not shown), the entire cluster 0 was taken down. After failover, writes to LSS A, B, C, D are handled by the processor 32 in the second cluster 1. The failback process for a failed device adaptor 20 is the same as the failback process for a cluster as described with respect to FIG. 6 except with respect to how writes of LSS A data, initially handled by the failed device adaptor 20, are handled and routed to cluster 1. FIG. 8a illustrates the initial stage of the failback operation with the failed device adaptor 22. FIG. 8a shows the processor 32 destaging all LSS A, B, D data from NVS 28 and LSS C data from the cache 30 to DASD via device adaptors 36, 38. After destaging, all new writes to LSS A, B, D are handled by processor 32 and backed-up in NVS 8, leaving NVS 28 to handle back-ups for data writes to LSS C.

FIG. 8b is a block diagram illustrating the second part of the failback process with a failed device adaptor, and processor 12 handling writes to LSS C, and processor 32 and device adaptors 36, 38 handling writes to LSS A, B, D. Device adaptor 36 handles data writes to tracks LSS A, B as device adaptor 20 is down and can no longer write to LSS A. FIG. 8b shows the final status of how the tracks are handled when a device adaptor 20 fails. Thus, the failover process for a failed device adaptor is similar to the failback process for an entire cluster described with respect to FIG. 6, except that only the LSSs associated with the functioning device adaptor 22 failback, leaving the functioning device adaptor 36 in another cluster 1 to handle LSS A operations previously handled by the failed device adaptor 20 in cluster 0.

This concludes the description of the preferred embodiments of the invention. The following describes some alternative embodiments for accomplishing the present invention.

Preferred embodiments of the failover and failback systems are described with respect to a storage controller having a specific two cluster arrangement. However, those skilled in the art will recognize that the failover and failback procedures could apply to storage controllers having different components and a different architecture from the storage controller described with respect to FIG. 1. For instance, the storage controller may include additional clusters, a different interface arrangement between the host adaptors and the processor and between the processor and the device adaptors. Still further, a different arrangement and/or number of host adaptors, device adaptors, processors, DASDs, LSS tracks, etc., could be used.

Still further, the DASDs are described as being magnetic units. However, in alternative embodiments the DASDs could be optical memory devices, tape drives, holographic units, etc. Yet further, the DASDs could be organized into a plurality of RAID array structures. Still further, the components of the storage controller 2, including the clusters 0, 1, host adaptors 4, 6, 24, 26, host adaptor bridge 44, NVS 8, 28, processors 12, 32, cache 30, device adaptor bus 14, 34, and device adaptors 16, 18, 20, 22, 36, 38, 40, 42 and functions performed thereby may be implemented with hardware logic (e.g., gates and circuits), firmware or a combination thereof. Moreover, events may occur at times different than order presented in the flowcharts of FIGS. 4 and 6.

In summary preferred embodiments in accordance with the present invention provide a system for handling failures in a storage controller interfacing between a plurality of host systems and direct access storage devices (DASDs). The storage controller directs data from the host systems through a first and second data paths in the storage controller to a DASD. A first processor, first non-volatile memory unit (NVS), and a first cache are associated with the first data path and a second processor, a second NVS, and a second cache are associated with the second data path. A bridge provides communication between the first processor and the second NVS and the second processor and the first NVS. During normal operations prior to a failure in the storage controller, data directed to the first data path is written in the first cache and communicated with the first processor to the second NVS via the bridge for storage in the second NVS. Data directed to the second data path is written in the second cache and communicated with the second processor to the first NVS via the bridge for storage in the first NVS. A point of failure within at least one of the first processor, first cache, and first NVS is handled by routing the data directed to the first data path to the second processor and writing the routed data to the second cache and the second NVS. Upon repairing the point of failure, the data directed to the first data path is routed to the first processor, wherein the routed data is written in the first cache and communicated with the first processor to the second NVS via the bridge for storage therein.

The foregoing description of the preferred embodiments of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto. The above specification, examples and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.