Method and apparatus for increasing RAID write performance by maintaining a full track write counter

[0002] 1. Field of the Invention

[0003] This invention relates in general to data storage systems, and more particularly to a method and apparatus for increasing RAID (redundant array of independent disks) write performance by maintaining a full track write counter.

[0004] 2. Description of Related Art

[0005] In recent years, there has been a growth in interest in disk arrays. Disk arrays consist of a number of disk drives connected to a host system via one or more controller elements which control the transfer of data between the host and disk drives. A disk array is designed to provide high capacity data storage, high reliability and high data transfer rates to and from the system.

[0006] RAID was conceived at the University California at Berkeley in 1987 as a means for improving storage subsystem reliability and performance. The concept initially consisted of clustering small inexpensive disk drives into an array such that the array could appear to the system as a single large expensive drive (SLED). The result of this initial testing found that the array of drives could actually deliver the same or better performance than the traditional individual hard drive. However, there was a problem with this implementation. The resulting Mean Time Before Failure (MTBF) of the array was actually reduced due to the probability of any one drive of the array failing. As a result of this finding, the Berkeley scientists proposed five levels or methods of RAID to provide a balance of performance and data protection. RAID subsystems can be optimized for performance, the highest capacity, fault tolerance or a combination of two or three of the above. Different so-called RAID levels have been defined and standardized in accordance with those general optimization parameters. There are six standardized RAID levels, called RAID 0, 1, 2, 3, 4 or 5.

[0007] A RAID controller board is the hardware element that serves as the backbone for the array of disks. It not only relays the input/output (I/O) commands to specific drives in the array, but provides the physical link to each of the independent drives so they may easily be removed or replaced. The controller also serves to monitor the integrity of each drive in the array to anticipate the need to move data should it be placed in jeopardy by faulty or failing disk drive (a feature known as “fault tolerance”). RAID utilizes some form of parity information to maintain data integrity across all drives in the subsystem. A rank is the set of logical volumes related to each other for parity protection, i.e., the number or set of drives covered by a parity scheme.

[0008] RAID Level 0 is achieved through a method known as striping and is optimized for performance at the expense of fault tolerance. Disk striping is a technique for spreading data over multiple disk drives. Disk striping can speed up operations that retrieve data from disk storage. The computer system breaks a body of data unto units and spreads these units across the available disks. Systems that implement disk striping generally allow the user to select the data unit size or stripe width.

[0009] The collection of drives in a RAID Level 0 array has data laid down in such a way that it is organized in stripes across the multiple drives. A typical array can contain any number of stripes, usually in multiples of the number of drives present in the array.

[0010] The reason RAID 0 is a performance-enhancing configuration is that striping enables the array to access data from multiple drives at the same time. In other words, since the data is spread out across a number of drives in the array, it can be accessed faster because it's not bottled up on a single drive. This is especially beneficial for retrieving very large files, since they can be spread out effectively across multiple drives and accessed as if it were the size of any of the fragments it is organized into on the data stripes.

[0011] The downside to RAID Level 0 configurations is that it sacrifices fault tolerance, raising the risk of data loss because no room is made available to store redundant data. If one of the drives in the RAID 0 fails for any reason, there is no way of retrieving the lost data as can be done in other RAID implementations described below.

[0012] The RAID Level 1 is achieved through disk mirroring, and is done to ensure data reliability or a high degree of fault tolerance. RAID 1 also enhances read performance, but the improved performance and fault tolerance are at the expense of available capacity in the drives used.

[0013] In a RAID Level 1 configuration, the RAID management software instructs the subsystem's controller to store data redundantly across a number of the drives (mirrored set) in the array. In other words, the same data is copied and stored on different disks (or “mirrored”) to ensure that, should a drive fail, the data is available somewhere else within the array. In fact, all but one of the drives in a mirrored set could fail and the data stored to the RAID 1 subsystem would remain intact. A RAID Level 1 configuration can consist of multiple mirrored sets, whereby each mirrored set can be a different capacity. Usually the drives making up a mirrored set are of the same capacity. If drives within a mirrored set are of different capacities, the capacity of a mirrored set within the RAID 1 subsystem is limited to the capacity of the smallest-capacity drive in the set.

[0014] The read performance gain can be realized if the redundant data is distributed evenly on all of the drives of a mirrored set within the subsystem. The number of read requests and total wait state times both drop significantly; inversely proportional to the number of hard drives in the RAID.

[0015] RAID Level 2 is rarely used in commercial applications, but is another means of ensuring data is protected in the event drives in the subsystem incur problems or otherwise fail. This level builds fault tolerance around Hamming error correction code (ECC), which is used as a means of maintaining data integrity. ECC tabulates the numerical values of data stored on specific blocks in the virtual drive using a special formula that yields what is known as a checksum. The checksum is then appended to the end of the data block for verification of data integrity when needed.

[0016] As data gets read back from the drive, ECC tabulations are again computed, and specific data block checksums are read and compared against the most recent tabulations. If the numbers match, the data is intact; if there is a discrepancy, the lost data can be recalculated using the first or earlier checksum as a reference point.

[0017] RAID level 3 is really an adaptation of RAID Level 0 that sacrifices some capacity, for the same number of drives, but achieves a high level of data integrity or fault tolerance. It takes advantage of RAID Level 0's data striping methods, except that data is striped across all but one of the drives in the array. This drive is used to store parity information that is used to maintain data integrity across all drives in the subsystem. The parity drive itself is divided up into stripes, and each parity drive stripe is used to store parity information for the corresponding data stripes dispersed throughout the array. This method achieves very high data transfer performance by reading from or writing to all of the drives in parallel or simultaneously but retains the means to reconstruct data if a given drive fails, maintaining data integrity for the system. RAID Level 3 is an excellent configuration for moving very large sequential files in a timely manner.

[0018] The stripes of parity information stored on the dedicated drive are calculated using an “Exclusive OR” function, which is a logical function between the two series that carries most of the same attributes as the conventional OR function. The difference occurs when the two bits in the function are both non-zero: in Exclusive OR, the result of the function is zero, wherein with conventional OR it would be one.

[0019] RAID Level 4 is similar in concept to RAID Level 3, but emphasizes performance for different applications. Another difference between the two is that RAID Level 4 has a larger stripe depth, usually of two blocks, which allows the RAID management software to operate the disks much more independently than RAID Level 3. This essentially replaces the high data throughput capability of RAID Level 3 with faster data access in read-intensive applications.

[0020] A shortcoming of RAID level 4 is rooted in an inherent bottleneck on the parity drive. As data gets written to the array, the parity encoding scheme tends to be more tedious in write activities than with other RAID topologies. This more or less relegates RAID Level 4 to read-intensive applications with little need for similar write performance. As a consequence, like its Level 3, it doesn't see much common use in commercial applications.

[0021] Finally, RAID Level 5 is the last of the most common RAID levels in use, and is probably the most frequently implemented. RAID Level 5 minimizes the write bottlenecks of RAID Level 4 by distributing parity stripes over a series of hard drives. In doing so it provides relief to the concentration of write activity on a single drive, which in turn enhances overall system performance.

[0022] The way RAID Level 5 reduces parity write bottlenecks is relatively simple. Instead of allowing any one drive in the array to assume the risk of a bottleneck, all of the drives in the array assume write activity responsibilities. The distribution frees up the concentration on a single drive, improving overall subsystem throughput.

[0023] RAID Level 5's parity encoding scheme is the same as Levels 3 and 4; it maintains the system's ability to recover any lost data should a single drive fail. This can happen as long as no parity stripe on an individual drive stores the information of a data stripe on the same drive. In other words, the parity information for any data stripe must always be located on a drive other than the one on which the data resides.

[0024] RAID strategies can be implemented using hardware or software solutions. In a hardware solution, the RAID controller interface handles the creation and regeneration of redundant information. Some vendors sell disk subsystems that implement RAID technology completely within the hardware. Some of these hardware implementations support hot-swapping of disks, which enables you to replace a failed disk while the computer is still running. A hardware implementation of RAID support can offer performance advantages over the software implementation included in Windows NT®.

[0025] Windows NT provides a software implementation for RAID using the Windows NT file system (NTFS) and the File Allocation Table (FAT) file system. Windows NT Server provides software support for two fault-tolerant disk configurations: mirror sets and stripe sets with parity. Windows NT Server and Windows NT Workstation have software support for stripe sets.

[0026] Still, computer system performance is determined by a combination of factors working in tandem to achieve the efficient results people have come to expect from even basic computer systems. In the early stages of the buying cycle, processor speeds typically get the most attention, but they are actually only one consideration in overall system performance. In addition to determining processor speed and hard drive size requirements, understanding the complexities of RAM (random access memory) is critical. RAM is the component that handles the executable tasks of any application. When an application is recalled from the hard drive, the program is put into the RAM and is ready to be used. The amount of RAM in a system has a large effect on the speed in which it will run an application. The more RAM available, the less the processor has to access the hard drive, because more instructions can be carried out by the main memory instead of temporarily holding them in the hard drive. Thus, system performance is increased. In fact, in some configurations, a system with more RAM will run faster than the same system with a faster processor but with less RAM.

[0027] However, when performing a RAID write, the RAID write may not contain a stripe width of full tracks. To perform the write, the parity must first be read, then the new parity generated, and finally the data and new parity can be written. To optimize this process, the parity read can be avoided by writing a stripe width of full tracks. During a write, an assumption that a stripe of full tracks exist must be made and then the tracks are grouped. Nevertheless, during the grouping, the controller may discover that a stripe of full tracks does not exist, yet the write will still include a parity read and the overhead doing the track grouping has been incurred.

[0028] It can be seen then that there is a need for a method and apparatus for avoiding unnecessary track grouping during writes.

[0029] It can also be seen that there is a need for a method and apparatus to indicate whether or not the track grouping will result in a stripe width of full tracks which can be written in a group to avoid the parity read.

[0030] To overcome the limitations in the prior art described above, and to overcome other limitations that will become apparent upon reading and understanding the present specification, the present invention discloses a method and apparatus for improving write performance in a disk array.

[0031] The present invention solves the above-described problems by providing a method and apparatus for avoiding unnecessary track grouping during writes by using a full track write counter. For the purpose of this invention, a write complete using the entire image in cache will herein be referred to as a full track write.

[0032] A method in accordance with the principles of the present invention includes receiving a write request, analyzing a full track write counter for tracks in a stripe of tracks associated with the write request, determining whether write request involves a full track write and subsequently executing a cache destage based on the analysis of the full track write counter for tracks in a stripe of tracks associated with the write request.

[0033] Other embodiments of a method in accordance with the principles of the invention may include alternative or optional additional aspects. One such aspect of the present invention is that the full track write counter is not incremented when the write to cache is not a full track write.

[0034] Another embodiment of the present invention is that a previous track full track count is fetched, a full track count of the tracks associated with the write request are set to be equal to the minimum of either the stripe width or the previous track's full track count plus 1 and the write to the cache is a full track write.

[0035] Another embodiment of the present invention is that the step of analyzing a full track write counter further includes receiving a de-stage request for a track and determining whether the full track write counter of the last track in the stripe is equal to a stripe width.

[0036] Another embodiment of the present invention is that each track in the stripe is processed for de-stage, new parity for the stripe is generated based on modifications to each track and new data resulting from the modifications and the new parity are written to a rank when the full track write counter of the last track in the stripe is equal to the stripe width.

[0037] Another aspect of the present invention is that the generation of new parity comprises exclusively ORing the striped data when the full track write counter of the last track in the stripe is equal to the stripe width.

[0038] Another aspect of the present invention is that only the track associated with the de-stage request is processed, an old parity and old data for the track associated with the de-stage request, new parity is generated and new data resulting from a modification to the track and the new parity are written to a rank when the full track write counter of the last track in the stripe is not equal to the stripe width.

[0039] Another aspect of the present invention is that the generation of new parity comprises exclusively ORing the old data, the new data and old parity associated with the stripe when the full track write counter of the last track in the stripe is not equal to the stripe width.

[0040] These and various other advantages and features of novelty which characterize the invention are pointed out with particularity in the claims annexed hereto and form a part hereof. However, for a better understanding of the invention, its advantages, and the objects obtained by its use, reference should be made to the drawings which form a further part hereof, and to accompanying descriptive matter, in which there are illustrated and described specific examples of an apparatus in accordance with the invention.

[0041] Referring now to the drawings in which like reference numbers represent corresponding parts throughout:

[0042] FIG. 1 illustrates a block diagram of a RAID system according to the present invention;

[0043] FIG. 2 shows the manner in which data may be stored on the disk storage devices of the array according to RAID 3;

[0044] FIG. 3 illustrates the storing of the parity records according to RAID 5;

[0045] FIG. 4 illustrates a RAID 5 de-stage without a full stripe width of tracks;

[0046] FIG. 5 illustrates a de-staging of a cache without a full stripe using a full track write counter;

[0047] FIG. 6 illustrates a de-stage with a full stripe, wherein the stripe width is four;

[0048] FIG. 7 illustrates a de-stage process with full stripe using full track write counters;

[0049] FIG. 8 illustrates the method for maintaining full track write counters; and

[0050] FIG. 9 illustrates a method of using full track write counters for write devices.

[0051] In the following description of the exemplary embodiment, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration the specific embodiment in which the invention may be practiced. It is to be understood that other embodiments may be utilized as structural changes may be made without departing from the scope of the present invention.

[0052] The present invention provides a method and apparatus for avoiding unnecessary track grouping during writes by using a full track write counter. The full track write counter indicates whether or not a track grouping will result in a stripe width of full tracks which can be written in a group. The full track write counter allows the locking of only the de-stage track and the last track in the stripe, which reduces lock contention. In addition, the full track write counter eliminates processing time for grouping tracks to determine whether a full track width exists. Finally, since each track has a control structure associated with it, processor data cache is preserved by not reading the control structure for tracks unnecessarily.

[0053] FIG. 1 illustrates a block diagram of a RAID system 100 according to the present invention. In FIG. 1, host adapters 104, 106 provide an interface between host computers or devices (not shown) and DASDs 146, 148, 150, 152, which include multiple RAID arrays. The DASDs are magnetic storage units such as hard disk drives. The host computers and devices are connected to host adapters 104, 106 via a bus interface (not shown), such as a SCSI bus interface. The host adapters 104, 106 may be comprised of an Enterprise System Connection (ESCON) adapter which provides access to ESCON channels and connections. Each host adapter 104, 106 may be comprised of a series of host adapters which connect to a host system.

[0054] In addition to host adapters 104, 106, a non-volatile storage unit (NVS) 108, a cache 110, a processor 112, a device adapter bus 114, and device adapters 116, 118, 120, 122 are provided. The host adapters 104, 106 are connected to processor 112. The processor 112 may be a symmetrical multi-processor, such as the IBM RS/6000 processor.

[0055] The cache 110 may be external to the processor 112 or included in the processor 112 complex. NVS 108 includes random access electronic storage with a battery backup. Storage time for a fully charged battery may last a couple of days. In preferred embodiments, the NVS battery is continuously charged whenever primary power is applied during normal operations. The battery will supply power necessary to maintain contents of the NVS 108 intact until power is restored. The cache 110, on the other hand, is a volatile storage unit that cannot maintain data in the event of a power failure.

[0056] Device adapter bus 114 interconnects the processor 112 with the device adapters 116, 118, 120, 122. The device adapters 116, 118, 120, 122 interface between the storage controller and the DASDs 146, 148, 150, 152, or RAID array of hard disk drives. In preferred embodiments, the device adapters 116, 118, 120, 122 employ the Serial Storage Architecture (SSA) developed by IBM. In such case, the DASDs may be interconnected in a loop topology including multiple RAID arrays.

[0057] The processor 112 implements the method for increasing write performance in a disk array by using a full track write counter according to the present invention. The program for implementing the method for increasing write performance in a disk array by using a full track write counter may be included in the processor 112, random access memory (RAM), read-only memory (ROM), and other components.

[0058] FIG. 2 shows the manner in which data may be stored 200 on the disk storage devices of the array according to RAID 3. In FIG. 2, a stripe 260 is illustrated including tracks A, B. C and P. Each track, as shown in FIG. 2, includes a predetermined number of sectors 262, e.g. three sectors of 512 bytes each. The first track T0 includes sectors A1, A2, and A3. Tracks T1, T2 and T3 are illustrated similarly. When a disk array system receives a request from the host to write new sectors 270, 271, 272 and 273 to the array, generally the rest of the data from the disk in that stripe 260 must be read, a new parity calculation performed, new sector 270, 271, 272 and 273 written, and new parity data written to the storage disk 240. For simplicity, parity blocks are not identified in FIG. 2. Protection of data on disk storage devices 234, 236, 238 is provided by exclusive ORing data records on each device, and storing the parity record resulting from the exclusive ORing on one of the storage devices 240.

[0059] FIG. 3 illustrates the storing of the parity records according to RAID 5300. In FIG. 3, each storage device 334, 336, 338, 340 is divided into blocks of data 360 and blocks of parity 370. The blocks represent physical space on the storage devices. Blocks 370, 372, 374, 376, one on each storage device, preferably having the same physical address range, are referred to as stripe 390. There are four stripes shown in FIG. 3. A protected stripe 390 has an associated parity block 370 which contains the exclusive ORing of the other blocks 372, 374, 376 in the stripe 390. That is, the first stripe 390 includes block 370 containing the parity for the remaining blocks, 372, 374, 376. A block 380 on storage device 336 contains the parity for the remaining blocks on the second stripe 392. Block 382, on storage device 338 contains the parity for the third stripe 394. Block 384, on storage device 340 contains the parity for the fourth stripe 396.

[0060] Spreading of the parity information according to the above description insures that one particular storage device is not accessed much more than the other storage devices during writing of the parity records following updates to data records on the different stripes. A change to a record stored in a block will result in a change also having to be made to the parity block for the stripe including the changed records. Since the parity blocks of the stripes are spread over more than one storage device, the parity updates will not be concentrated at one device. Thus, I/O activity is spread more evenly over all the storage devices.

[0061] However, as discussed above with reference to the prior art, writes which do not contain a stripe width of full tracks must first read the parity and the old data, generate new parity and then write the data and new parity. The parity read can be avoided by writing a stripe width of full tracks. During a write, an assumption is made that a stripe of full tracks exists and a track grouping is initiated. During the grouping, however, it may be discovered that a stripe of full tracks does not exist, yet a write will still include a parity read and the overhead for doing the track grouping has been incurred.

[0062] According to the present invention, a necessary track grouping during writes may be avoided by using a full track write counter to indicate whether or not the track grouping will result in a stripe width of full tracks. A full track write counter is maintained for each track in the control unit cache. If a write to cache is a full track write, the full track counter of a sequentially prior track is incremented and stored in the full track write counter of the track being written. When de-staging a track from cache, the ending track of the track stripe is checked for its full track write counter value. If the ending track's full track write counter is equal to the drive stripe width, all tracks in the stripe are de-staged in the same request to increase the RAID write performance. If the ending track's full track write counter is less than a stripe width, no grouping is done, thus saving the overhead of starting to group the tracks but finding there is not a stripe width of full tracks to write.

[0063] FIG. 4 illustrates a RAID 5 de-stage without a full stripe width of tracks 400. The tracks are shown at different steps in the cache 410 for de-staging during the write process. In FIG. 4, a full track write counter is not utilized according to the present invention. The content of a exclusive OR buffer 412 is shown to correspond to the steps in the write process. In step 1420, the initial state of the cache 410 is shown wherein the cache 410 includes track 0422 and track 3424. Track 3424 is indicated as being a full track that has been modified. In step one 420, the exclusive OR buffer 412 does not contain any information pertaining to track 3424.

[0064] In step two 430, the parity and the initial data in track 3 are read. In step two 430, the exclusive OR buffer 412 contains the old track 3 data 436 and the old parity 438. In step three 440, the new parity 448 is generated in the exclusive OR buffer 412. The new parity is calculated to be equal to the old parity exclusively ordered with the old track 3 data and the new track 3 data 449.

[0065] In step four 450, the modified track 3 and the new parity data are de-staged to the drive, i.e., written to the RAID rank. The contents of the cache 410 and the exclusive OR buffer 412 are unchanged. In step five 460, the write is complete. The completed write is represented by track 3464 being unmodified 469. Also in step five 460, the old track 3 data and new parity in the exclusive OR buffer 412 are discarded.

[0066] FIG. 5 illustrates a de-staging of a cache without a full stripe using a full track write counter 500. The steps illustrated in FIG. 5 are similar to the steps illustrated in FIG. 4, except that a track counter 570 is provided for each of the tracks in FIG. 5. Therefore, some overhead processing is saved by utilizing the full track write counters in each of the tracks. Since the stripe width is four, the track counter for track 3524 is examined to determine whether it is a full stripe. However, the full track write counter 572 for track 3524 is set to 1572. Thus, whether track 0522 is a full track may be determined by looking at the full track write counter 574 of track 0522. The full track write counter 574 for track 0522 is set to 0, therefore track 0522 is not a full track and track 3524 must be a full track, but not a full stripe. Thus, track 0522 is not even examined because there is not a full stripe.

[0067] FIG. 6 illustrates a de-stage with a full stripe, wherein the stripe width is four 600. The tracks in FIG. 6 do not include a full track write counter. As with FIGS. 4 and 5, the contents of the cache 610 and the exclusive OR buffer 612 are shown. The initial state of the cache 610 and exclusive OR buffer 612 are illustrated in step one 620. In step one 620, track 0621 is indicated to be a full track which has been modified. Track 1622, track 2623, and track 3624 are also full tracks which have been modified. In step 1620, the exclusive OR buffer 612 does not contain any information concerning these tracks. Thus, every track has to be examined to determine if all tracks are full.

[0068] In step two 630, the new parity 635 is generated and loaded into the exclusive OR buffer 612. The new parity 635 is equal to the exclusive ORing of the data in tracks 0-3639. In step three 640, the new parity 645 and revised data in track 0641, track 1642, track 2643, and track 3644 are de-staged to the drive from the exclusive OR buffer 612. Step four 650 illustrates that the write is complete and that track 0651, track 1652, track 2653, track 3654 are full tracks which are unmodified 659. The data in the exclusive OR buffer 612 has been discarded.

[0069] FIG. 7 illustrates a de-stage process with full stripe using full track write counters 700. The steps illustrated in FIG. 7 are similar to those illustrated in FIG. 6 except each of the tracks includes a full track write counter 770. Since track 0721 is a full track, the full track write counter 770 for track 0 is set to 1 771. The next track, track 1722, is also a full track, and therefore the counter is incremented by 1 to result in a count of 2772. Track 2723 is also a full track, therefore the full track write counter is incremented to 3 to result in a count of 3773. Finally, track 3724 is a full track and its counter is incremented by 1 to result in a count of 4774. However, rather than assemble track 0721, track 1722, track 2723, and track 3724 to determine whether a full stripe exists, the controller can look at track 3724 to determine whether the full track write counter is set to 4 (the stripe width) 774. Since the stripe width is equal to four, if the stripe is a full stripe, the full track write counter will be set to 4774. Thus, in step 3740, the controller may write the four tracks 741-744 and the new parity 745 to the drive. The data in the exclusive OR buffer 712 is discarded in step four 750.

[0070] FIG. 8 illustrates the method for maintaining full track write counters 800. In FIG. 8, a channel writes track N 810. Then a determination is made whether the track N is a full track write 820. If track N is not a full track write 822, the track N full track write counter is set to 0830. Then the write request to the cache is completed 840.

[0071] However, if track N is a full track write 824, the track N−1 full track count is fetched 850. Then, the track N's full track count is set to be equal to the minimum of either the stripe width or the track N−1 count plus 1 860. Track N's track count is set to be the minimum of the stripe width or the track N−1 count plus 1 because a number of full tracks in excess of the stripe width may be written. By limiting the full track write counter to the stripe width, the number of full tracks can still be maintained. Then the write request to the cache is completed 840.

[0072] FIG. 9 illustrates a method of using full track write counters for write devices 900. In FIG. 9, a request for de-staging track N is received 910. Then a determination of whether the last track in the stripe full track count is equal to the stripe width is made 920. If the last track in the full stripe count is equal to the stripe width 922, then each track in the stripe for de-stage is processed 930. Then the drive generates a new parity for the stripe by exclusively ORing the striped data 940. Thereafter, the drive writes the new data and the new parity 950.

[0073] If the last track in the full stripe full track count is not equal to the stripe width 924, then only track N is processed for de-stage 960. Then, the drive reads the old parity and track N old data 970. Finally, the drive generates the new parity 980 and then writes the new data and new parity 950.

[0074] The foregoing description of the exemplary embodiment of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not with this detailed description, but rather by the claims appended hereto.