Hot spare reliability for storage arrays and storage networks

[0002] The present invention relates generally to disk drive systems, and more specifically to an apparatus and method for improving the reliability of hot spare disk drives in a storage array/storage network.

[0003] RAID (Redundant Array of Independent Disks) is a set of methods and algorithms for combining multiple disk drives (i.e., a storage array) as a group in which attributes of the multiple drives are better than the individual disk drives. RAID can be used to improve data integrity (i.e., reduce the risk of losing data due to a defective or failing disk drive), cost, and/or performance.

[0004] RAID was initially developed to improve I/O performance at a time when computer CPU speed and memory size was growing exponentially. The basic idea was to combine several small inexpensive disks (with many spindles) and stripe the data (i.e., split the data across multiple drives), such that reads or writes could be done in parallel. To simplify the I/O management, a dedicated controller would be used to facilitate the striping and present these multiple drives to the host computer as one logical drive.

[0005] The problem with this approach was that the small, inexpensive PC disk drives of the time were far less reliable than the larger, more expensive drives they replaced. An artifact of striping data over multiple drives is that if one drive fails, all data on the other drives is rendered unusable. To compound this problem, by combining several drives together, the probability of one drive out of the group failing increased dramatically.

[0006] In order to overcome this pitfall, extra drives were added to the RAID group to store redundant information. In this way, if one drive failed, another drive within the group would contain the missing information, which could then be used to regenerate the lost information. Since all of the information was still available, the end user would never be impacted with down time and the rebuild could be done in the background. If users requested information that had not already been rebuilt, the data could be reconstructed on the fly and the end user would not know about it.

[0007] Today there are six base architectures (levels) of RAID, ranging from “Level 0 RAID” to “Level 5 RAID”. These levels provide alternative ways of achieving storage fault tolerance, increased I/O performance and true scalability. Three main building blocks are used in all RAID architectures: 1) Data Striping—Data from the host computer is broken up into smaller chunks and distributed to multiple drives within a RAID array. Each drive's storage space is partitioned into stripes. The stripes are interleaved such that the logical storage unit is made up of alternating stripes from each drive. Major benefits are improved I/O performance and the ability to create large logical volumes. Data striping is used in Level 0 RAID. 2) Mirroring—Data from the host computer is duplicated on a block-to-block basis across two disks. If one disk drive fails, the data remains available on the other disk. Mirroring is used in RAID levels 1 and 1+0. 3) Parity—Data from the host computer is written to multiple drives. One or more drives are assigned to store parity information. In the event of a disk failure, parity information is combined with the remaining data to regenerate the missing information. Parity is used in RAID levels 3, 4 and 5.

[0008] If a drive fails in a RAID array that includes redundancy—meaning all RAID architectures with the exception of RAID 0—it is desirable to get the drive replaced immediately so the array can be returned to normal operation. There are two reasons for this: fault tolerance and performance. If the drive is running in a degraded mode due to a drive failure, until the drive is replaced, most RAID levels will be running with no fault protection at all: a RAID 1 array is reduced to a single drive, and a RAID 3 or RAID 5 array becomes equivalent to a RAID 0 array in terms of fault tolerance. At the same time, the performance of the array will be reduced, sometimes substantially.

[0009] An extremely useful RAID feature that helps alleviate this problem is hot swapping, which when properly implemented, will let a user replace the failed drive immediately without taking down the system. Another approach is through the use of hot spares. Additional drives are attached to the controller and left in a “standby” mode. If a failure occurs, the controller can use the spare drive as a replacement for the bad drive. This simple concept is supported by most RAID implementations. Even many of the inexpensive hardware RAID cards and software RAID solutions support this approach. Typically, the only cost is another hard disk that has to be purchased but cannot be used for storing data.

[0010] The main advantage that hot sparing has over hot swapping is that with a controller that supports hot sparing, the rebuild will be automatic. The controller detects that a drive has gone bad, it disables it, and immediately rebuilds the data onto the hot spare. This is a tremendous advantage for anyone managing many arrays, or for systems that run unattended. As features, hot sparing and hot swapping are independent: you can have one, or the other, or both. They will work together, and often are used in that way. However, sparing is particularly important if hot swap (or warm swap) capability is not available, because it will enable a user to get the array back into normal operating mode quickly, delaying the time that the system has to be shut down until it is more convenient. However, when this occurs, the user loses the hot sparing capability in the meantime. When the failed drive is eventually replaced, the new drive becomes the new hot spare.

[0011] It has been discovered that hot spares can fail shortly after they are called into action. The reason for this appears to be caused by the distinctly different access patterns used for spares as compared to active drives. Thus, when a spare drive begins using areas of the disk stack not previously accessed during its idle routine, a head crash may result.

[0012] In order to address this problem, manufacturers of RAID systems have implemented a “safe mode” for priming the hot spare before it is called into service. In safe mode, when the hot spares are in idle mode, the heads are moved across the disk surfaces at some periodic interval. The problem with safe mode is that it does not emulate the track utilization present in the active drives of an array.

[0013] There is a need for a more realistic apparatus and method to prime a hot spare drive prior to use. The apparatus and method should emulate the track utilization present in the active drives of the array prior to being called into service.

[0014] The details of the present invention, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts.

[0015] A method and apparatus for improving the reliability of hot spare disk drives in disk arrays and storage networks is provided. In a preferred embodiment, the present invention intercepts commands issued to active disk drives in a disk array/storage network, analyzes the commands, and issues commands to the hot spare drives which attempt to emulate the track usage patterns of the active drives. The track usage patterns can be inferred from examining logical block addresses (LBA) of data stored to the active drives, and/or the ratio of read versus write commands issued to the active disk drives. By emulating track usage patterns of the active drives, the hot spare drives have a roughly equivalent lubricant distribution to that of the active disk drives. This provides increased reliability when the hot spare drives are later called into active service.

[0016] The present invention also provides a storage array/storage network apparatus having a plurality of active disk drives for storing data. The storage array/storage network apparatus also provides one or more hot spare disk drives for replacing any of the active disk drives if a failure occurs on any of the active disk drives during normal operation. The storage array/storage network further provides a storage array controller which provides commands to the hot spare drives during normal operation. The commands provided to the hot spare drives from the storage array controller produce track usage patterns similar to track usage patterns produced for the active disk drives during normal operation.

[0017] In one embodiment, the commands provided to the hot spare drives are derived by a statistical analysis performed on commands provided to the plurality of active drives during normal operation. The statistical analysis may include, but is not limited to, estimating track usage patterns from the logical block addresses (LBA) of commands sent to the active drives, and/or the ratio of read versus write commands sent to the active drives. In yet another embodiment, every 1 of N commands provided to the plurality of active drives are also directed to the hot spare drives.

[0018] In one embodiment of the present invention, The storage array controller begins providing commands to the hot spare drives only when a set of predictive failure analysis (PFA) measurements from at least one of the plurality of active disk drives indicate that a disk drive failure is probable in the near future. In another embodiment, the storage array controller ceases to provide command to the hot spare disk drives if a predictive failure analysis performed on a hot spare disk drive indicates that a failure is likely in the near future.

[0019] The details of the present invention, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts.

[0020] FIG. 1 is an illustration of a simple RAID storage array.

[0021] FIG. 2 is an illustration of a RAID controller.

[0022] FIG. 3 is a flowchart of a method to precondition a hot spare disk drive in a storage array/storage network.

[0023] Turning to the Drawings, wherein like numbers denote like parts throughout the several views, FIG. 1 is a high level illustration of a Redundant Array of Inexpensive Disks (RAID) storage array, shown generally at 10. RAID storage array 10 communicates with an external device 26 (e.g., a computer server) via interface 28. A plurality of disk drive modules (DDMs) 20a, 20b, 20c, 20d, and 20eare attached to a RAID controller 22 via a bus 24. RAID storage array 10 also includes one or more hot spare drives 20f. Hot spare drive 20fis connected to RAID storage array 10, but is held in reserve in case a DDM 20a, 20b, 20c, 20d, and 20efails. In a preferred embodiment, bus 24 is a SCSI bus, but several alternative bus types may be utilized (e.g., Fibre Channel) and still remain within the scope and spirit of the present invention. RAID array 10 can be part of a larger storage network.

[0024] When a read/write command is issued by external device 26 to RAID storage array 10, RAID controller 22 dispatches the command to DDMs 20a, 20b, 20c, 20d, and 20e. Even though RAID storage array 10 contains multiple DDMs, RAID storage array appears to external device 26 as a single storage unit. Depending upon the type of RAID employed within the controller, RAID controller 22 dispatches the storage operation for the single command across the plurality of DDMs 20a, 20b, 20c, 20d, and 20e. In one embodiment, data can be striped across DDMs 20a, 20b, 20c, 20d, and 20eto give more access speed. Alternatively, data can be mirrored across multiple DDMs 20a, 20b, 20c, 20d, and 20efor data security. In yet another embodiment, RAID controller 22 stripes data across DDMs 20a, 20b, 20c, 20d, and 20eand also maintains parity information on the DDMs. RAID storage array 10 also includes one or more hot spare drives 20f, which are deployed if any of the active DDMs encounter a failure.

[0025] In some prior art systems, hot spare drive 20fremains idle during the normal operation of DDMs 20a, 20b, 20c, 20d, and 20ewithin RAID storage array 10. In these systems, hot spare drive 20fonly becomes active if a failure occurs on one of the DDMs during normal operation. In other prior art systems, hot spare drive 20foperates in a “safe mode”, where the heads of the drive are moved across the surfaces of the disk at some periodic interval. In both instances, hot spare drive 20fdoes not receive the same sort of disk activity and track usage patterns encountered by DDMs 20a, 20b, 20c, 20d, and 20eduring normal operation. As a result, hot spare drive 20fdoes not have an equivalent lubricant distribution to that of DDMs 20a, 20b, 20c, 20d, and 20e. When hot spare drive 20fbegins using areas of the disk stack not previously accessed during its idle routine, the probability of a head crash is substantially increased.

[0026] The present invention improves upon prior art systems, in that it analyzes track usage patterns for DDMs 20a, 20b, 20c, 20d, and 20eduring normal operation, and attempts to send commands to hot spare drive 20fwhich attempt to emulate the track usage patterns of DDMs. In this way, hot spare drive 20fis “primed” so that it has a usage pattern and lubricant distribution similar to that of the active DDMs 20a, 20b, 20c, 20d, and 20e.

[0027] FIG. 2 illustrates RAID controller 22 in more detail. At the highest level, RAID controller 22 intercepts commands being sent to active DDMs 20a, 20b, 20c, 20d, and 20e, analyzes the commands, and calculates which commands should be sent to the hot spare drive 20f.

[0028] As shown in FIG. 2, a command processor 40 within RAID controller 22 communicates with a DDM command interception process (DCIP) 42 via SCSI interface 43. This allows DCIP 42 to send back to command processor 40 via SCSI interface 43 the commands that should be sent to the hot spare drive 20f. The commands from DCIP 42 may be sent immediately to hot spare drive 20f, or may be queued and sent at a more convenient time (i.e., depending on current workload).

[0029] DCIP 42 has several possible functions. First, DCIP 42 provides analysis of the SCSI read and write commands being sent to active DDMs 20a, 20b, 20c, 20d, and 20e. DCIP 42 also provides cognizance of the different types of DDMs 20a, 20b, 20c, 20d, and 20e(they do not all have to be the same models), and associated logical block address (LBA) to physical mapping for each of the models employed within the RAID storage array 10. Finally, DCIP 42 provides commands for hot spare drive 20f.

[0030] DCIP 42 is capable of performing several different types of analysis on the commands being sent to active DDMs 20a, 20b, 20c, 20d, and 20e. At the most general level, DCIP 42 intercepts commands destined for active DDMs 20a, 20b, 20c, 20d, and 20e, then performs a statistical analysis of the commands being sent in order to build a custom set of commands that are sent to hot spare drive 20f. The custom set of commands are constructed to “condition” hot spare drive 20fwith usage patterns similar to those generated for active DDMs 20a, 20b, 20c, 20d, and 20e.

[0031] One of the key conditioning parameters is track usage. For most disk drives, the track usage can be estimated from the write and read commands sent to a drive. This is because the data on a drive is stored in logical block addresses (LBAs) which are arranged in a more or less linear correspondence to the disk radius. For example, an LBA of 0 is near the outer disk pack diameter and Max_LBA is near the inner disk pack diameter. It is therefore possible to estimate the physical track utilization for similar disk devices on the basis of the LBAs used for writing and for reading.

[0032] Another key conditioning parameter is the ratio of write versus read commands issued to a disk drive. It is possible to estimate the fraction of writes versus reads for a disk device since the commands for writing and reading are different. This can be important when the number of writes is significantly greater than the reads. In these instances, the average head flying height is likely to be lower due to head protrusion caused by higher head temperature during writing. This will have a corresponding effect on the lubricant distribution on the drive.

[0033] DCIP 42 can consider track usage alone, the ratio of write versus read commands alone, or some combination of both conditioning parameters when analyzing the command stream sent to active DDMs 20a, 20b, 20c, 20d, and 20e. Additional conditioning parameters may be considered alone or in combination with the aforementioned conditioning parameters, and still remain within the scope and spirit of the present invention.

[0034] In one embodiment of the present invention, the issuance of commands to hot spare drive 20fis initiated only if results of predictive failure analysis (PFA) measurements performed on active DDMs 20a, 20b, 20c, 20d, and 20eindicate that one or more of the active DDMS are likely to fail in the near future. By definition, PFA monitors key device performance indicators for change over time or exceeding predefined performance limits. The device then notifies the system when an indicator surpasses the predefined performance limit. Since some amount of processing overhead is required of RAID controller to intercept active DDM commands, analyze the commands, and issue emulation commands to hot spare drive 20f, thus it may be desirable to undertake this processing overhead only when a drive failure is likely.

[0035] Similarly, in one embodiment of the present invention, the emulation of track usage patterns by hot spare drive 20fis halted if a set of PFA measurements performed on hot spare drive 20findicate that the hot spare drive is likely to fail.

[0036] The concept of conditioning a hot spare drive 20fwithin a storage array can easily be extended to storage networks, as well. In yet another embodiment of the present invention, the track utilization for a “global hot spare” drive (i.e., a hot spare drive that can be used by any number of disk storage devices in a network) is based on a statistical average of track utilization for the disk storage devices for which it serves as a spare.

[0037] FIG. 3 illustrates a flow diagram of a method for improving hot spare reliability for storage arrays and storage networks in accordance with the present invention, shown generally at 70. The method begins at block 72. At block 74, a command issued by a storage array (i.e. RAID) controller 22 to the plurality of active DDMs 20a, 20b, 20c, 20d, and 20ein the storage array is intercepted (i.e., the command is still issued to the plurality of active DDMs, but the command is also retained at storage array controller 22 for further analysis).

[0038] At block 76, the command destined for active DDMs 20a, 20b, 20c, 20d, and 20ein the storage array is analyzed. At described previously, several different types of analysis are possible. In one instance the analysis involves determining track access patterns for the active DDMs 20a, 20b, 20c, 20d, and 20eby analyzing the logical block address(es) (LBA) of the command provided. In another instance, the ratio of read-to-write commands is determined for commands issued to the active DDMs 20a, 20b, 20c, 20d, and 20e.

[0039] After the commands provided to the active DDMs 20a, 20b, 20c, 20d, and 20ehave been analyzed, a second set of commands is issued to hot spare drive(s) 20fwhich emulate the track usage patterns of the commands sent to the active DDMs 20a, 20b, 20c, 20d, and 20e, as shown at block 78. In one instance, the commands sent to the hot spare drive(s) 20fare chosen based on an LBA analysis, or an analysis of the read-to-write commands as described previously. In another instance, the commands issued to the hot spare drive(s) 20fare simply a subset of the commands issued to the active DDMs 20a, 20b, 20c, 20d, and 20e(i.e., every 1 of N commands set to the active DDMs are also sent to the hot spare drives). In one embodiment, the second set of commands is provided to the hot spare drive(s) 20fonly when a set of predictive failure analysis (PFA) measurements indicate that a drive failure is likely on one or more active DDMs 20a, 20b, 20c, 20d, and 20ein the near future. In another embodiment, the invention ceases to provide the second set of commands to the hot spare disk drive(s) 20fif a PFA performed on the hot spare drive(s) indicates that a failure is likely in the near future.

[0040] Additional modifications may be made to the illustrated embodiments without departing from the spirit or scope of the invention. Therefore, the invention lies in the claims hereinafter appended.