US 6,351,838 B1Grant
Multidimensional parity protection system
Issue Date:2002-02-26
•10 Claims
•10 Drawing Sheets
Abstract
A data storage system for multidimensional parity protection includes a housing for containing a plurality of conventional disk drives for storing data wherein each of the conventional data disk drives is symmetrically linked to at least three parity disk drives for guarding data stored on the data disk drives. Each of the conventional data disk drives thereby have a three dimensional parity relationship with three parity data disk drives such that a three dimensional, mirrored, orthogonal parity array is established. The data storage system further includes a failure detection component coupled to the three dimensional, mirrored, orthogonal parity arrays for detecting and indicating the failure of any disk in the mirrored, orthogonal parity arrays. In addition, a data reconstruction component is coupled to the three dimensional, mirrored orthogonal parity arrays for reconstructing data from any failed disk by combining data from other parity disks in the orthogonal parity arrays.
Metadata
Assignee
- Aurora Communications, Inc
Inventor
- John R. Amelia
Application Information
Application Number:US 09/267,847
Filing Date:1999-03-12
Priority Date:1999-03-12
Art Unit:7
Classifications
IPC:
G11C 2900
Field of Search:
714755714770714800714804714799714769714784714801714 20714805714711714710714 24714 14707202707205345327725 88725 92711114711100711101711111711154711172711122711 13711136711112
Patent Drawings (10 sheets)
Description
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The present invention relates to data storage systems for multidimensional parity protection, and more particularly to data storage systems for multidimensional parity protection which may be created independently of existing systems and which may be created to be retrofittable to existing multiple drive disk systems. The invention includes a housing containing a plurality of parity disk drives for guarding data stored on conventional data disk drives with means for data reconstruction of data from conventional disk drives such that a multidimensional, mirrored, orthogonal parity array is established.
[0004] 2. Information Disclosure Statement
[0005] The following patents describe the state of the art in relevant data storage systems:
[0006] U.S. Pat. No. 5,271,012 to Blaum et al. teaches a method and means for encoding data written onto an array of M synchronous DASDs and for rebuilding onto spare DASD array capacity when up to two array DASD fail. Data is mapped into the DASD array using an (M−1) *M data array as the storage model where M is a prime number. Pairs of simple parities are recursively encoded over data in respective diagonal major and intersecting row major order array directions. The encoding traverse covers a topologically cylindrical path.
[0007] U.S. Pat. No. 5,351,246 to Blaum et al. teaches a method and means for coding an (M−1) *M data array written onto an array of M synchronous recording paths and for rebuilding and writing onto spare recording path capacity when up to a preselected number R of array DASDs fail, or one DASD becomes erroneous and up to R−2 fail. Data is mapped into the parallel paths using and (M−1)*M data and parity block array as the storage model where M is a prime number and each block extent is uniform and at least one bit in length. The (M−1)*M data and parity block array is encoded to include zero XOR sums along a traverse of slopes 0,1,2, . . . , P−1, extended cyclically over the data array.
[0008] U.S. Pat. No. 5,412,661 to Hao et al. teaches a data storage system architecture having an array of small data storage disks, organized into logical rows and columns, with each disk coupled to two disk controllers via two independent controller-disk interconnects. No two disks are coupled to the same pair of controllers. The component disks are arranged in parity groups of variable size. Within each parity group, failure of one disk sector can be recovered through data reconstruction using data from other disks in the parity group. One or more of the disks can be reserved as hot standbys for substitution on failure, automatically replacing any failed disk.
[0009] U.S. Pat. No. 5,572,659 to Iwasa et al. teaches an adapter connected between a host computer and disk storage devices providing interfaces for connecting to the host computer and the disk storage devices having the same interface design. The adapter includes control means for building a redundant disk storage system and includes means for detecting and indicating a failed disk storage device, means for replacing the failed disk storage device and means for rebuilding a redundant disk storage system after the replacement of disk storage devices.
[0010] U.S. Pat. No. 5,544,339 to Baba teaches an array of disk drives for storing information which is accessed through multiple channels by a host computer. Different channels are coupled to different sequences of disk drives. Different disk drives can be accessed simultaneously through different channels, enabling high data transfer rates. The same disk drive can be accessed through two different channels, enabling access even if one of the channels is busy or malfunctioning. In one case, the channels are divided into at least two mutually exclusive sets of channels, each set providing access to all of the disk drives.
[0011] A text by Garth A. Gibson and David A. Patterson entitled, “Designing Disk Arrays for High Data Reliability”, a text by Peter M. Chen and Garth A. Gibson entitled, “RAID-II: A High-Bandwidth Network File Server”, a text by Garth A Gibson and Jeffrey Scott Vitter and John Wilkes entitled, “Report of the Working Group on Storage I/O for Large Scale Computing”, a text by Garth A. Gibson entitled, “Redundant Disk Arrays, Reliable, Parallel Secondary Storage” and a text by Garth A. Gibson, Lisa Hellerstein, Richard M. Karp, Randy H. Katz and David A. Patterson entitled, “Coding Techniques for Handling Failures in Large Disk Arrays,” describes various types of storage protection and secondary disk arrays.
[0012] Notwithstanding the foregoing, the prior art neither teaches nor suggests a data storage system for multidimensional parity protection including for retrofit or use with a plurality of conventional disk drives for storing data wherein each of the conventional data disk drives is symmetrically linked to at least three parity disk drives for guarding data stored on the data disk drives, as taught by the present invention.
SUMMARY OF THE INVENTION
[0013] A data storage system for multidimensional parity protection includes a housing for containing a plurality parity disk drives for guarding data stored on a plurality of conventional data disk drives. Each of the conventional data disk drives are connected with the parity disk drives so as to create a multidimensional parity relationship in excess of two dimensions, with the plurality of parity data disk drives such that a multidimensional, mirrored, orthogonal parity array is established. The data storage system further includes a failure detection component coupled to the three dimensional, mirrored, orthogonal parity arrays for detecting and indicating the failure of any disk in the mirrored, orthogonal parity arrays. In addition, a data reconstruction component is coupled to the three dimensional, mirrored orthogonal parity arrays for reconstructing data from any failed disk by combining data from other parity disks in the orthogonal parity arrays.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The present invention should be more fully understood when the specification herein is taken in conjunction with the drawings appended thereto, wherein:
[0015] FIG. 1 shows a symmetrical two dimensional parity;
[0016] FIG. 2 shows symmetrical three dimensional parity;
[0017] FIG. 3 shows a system overview;
[0018] FIG. 4 shows a functional system with card interconnect;
[0019] FIG. 5 shows a functional system with card interconnect with external connectivity;
[0020] FIG. 6 shows external connectivity supporting SAN (Storage Area Network), NAT (Network Attached Storage) and conventional storage connectivity;
[0021] FIG. 7 shows a Fibre Channel variant of the Local Storage Controller card;
[0022] FIG. 8 shows a block diagram for a global storage controller system for the present invention;
[0023] FIG. 9 shows a schematic diagram of a tray enclosure for distributed architecture hierarchical control for a present invention system with single channel controllers and disk spindles and,
[0024] FIG. 10 shows a detailed schematic diagram of dynamic and static hierarchical control distributed intelligence used in the present invention system.
DETAILED DESCRIPTION OF THE PRESENT INVENTION
[0025] RAID is an acronym for Redundant Array of Independent (Inexpensive) Disks. It was conceived at the University of California at Berkeley. The initial RAID concept consisted of clustering small inexpensive disk drives into an array that appeared as a single large expensive drive. Testing of the initial concept found that the array of drives could deliver the same or better performance than an individual large expensive hard drive with a higher probability of any one drive of the array failing.
[0026] The increased probability of drive failure in the array of drives stimulated seven methods or levels of RAID to be defined. Each level defines a means that trades a balance of performance and data protection. Each method that provides protection does so at a cost of reduced storage capacity and performance. Several of these levels, from level 3 and above are classified as Parity RAID because the protection method utilizes parity to protect the data stored on the RAID disks.
[0027] The protection that different methods of RAID provide, is limited to a single disk failure per protection group with the exception of Level 6 that utilizes two parity drives. The number of drives that can participate in a protection group is defined by the architecture of the RAID system. Some are hardware based, some software based and others are combined hardware and software. The number of drives a single controller or a pair of tandem controllers can access restricts almost all architectures and implementation of RAID products. In short, they are providing RAID at the controller level.
[0028] This approach towards architecture for RAID products has several disadvantages. They include limited drive participation in protection groups as well as spare drives for automatic rebuild capabilities, limited data sharing capabilities and limited protection. As larger and larger quantities of storage are needed, more drives must be used. This increases the probability of multiple failures within a protection group.
[0029] In the present invention, the architecture does not implement RAID at the controller level and has the advantage of providing global sparing for any protection group. This distributed RAID architecture has higher degree of data sharing capable because it is not limited by a single or pair of controllers connectivity. It can also provide a higher degree of protection and deliver higher performance.
[0030] Storage and Network bandwidth are to some degree interchangeable. The invasion of caching products on the market is testament to that fact. Storage and Networking are moving closer and closer together. Terms such as NAS (Network Attached Storage), SAN (Storage Area Network) and Enterprise Storage all define methodologies that combine Networking and Storage and are now well known to the artisan.
[0031] The present invention is essentially a Reliable Rapid RAID system, “RRR” or “RCUBE”. The present invention exploits both Networking and Storage technologies. Six basic card types provide functional features in a modular system approach; together with a plurality of housing configurations, they yield a number of different systems which may be retrofitted to existing systems or a completely independent system. At the heart of the present invention in preferred embodiments is a crosspoint switch that interconnects five of the card types. The sixth type is global memory. Each crosspoint switch port moves data at 1.5 G Bytes per second. Table I below illustrates some of the features of present invention systems, including maximum and minimum drive capabilities and storage capacities. These examples are listed in Table I as Series 1000, 3000, 5000, 7000 and 9000.
[0032] The basic card types are Global Storage Controller, Local Storage Controller, Network Storage Controller, Global Cache Controller, Management and Request Queue Controller and Global Memory. All cards except Global Memory sit on dual Message and Management Buses. The Local Storage Controllers and Network Storage Controllers have one switch fabric port. The Global Storage Controllers, Global Cache Controllers and Management and Request Queue Controllers all have two switch fabric ports.
[0033] The present invention architecture and products are based on a MPSM (Massively Parallel Storage Management) system with a total fault tolerant design. This includes system components, communication channels, power, UPS and spindles/drives. The product features a revolutionary industry first multiple drive failure survivability protection scheme.
[0034] Thus, the present invention is a data storage system for multidimensional parity protection which, in many embodiments, includes a housing for containing a plurality of conventional disk drives for storing data, wherein each of the conventional data disk drives is symmetrically or non-symmetrically linked to a plurality of parity disk drives for guarding data stored on the data disk drives. Each of the conventional data disk drives thereby have a multidimensional parity relationship with a plurality of parity data disk drives such that a multidimensional, mirrored, orthogonal parity array of at least two dimensions, and in most embodiments, at least in three dimensions, is established. In addition, a data reconstruction component is coupled to the multidimensional, mirrored orthogonal parity arrays for reconstructing data from any failed disk by combining data from other parity disks in the orthogonal parity arrays.
[0035] In one embodiment of the present invention, a data storage system for multidimensional parity protection of at least two dimensions, comprising:
[0036] (a) a housing for containing a plurality of Naconventional disk drives, wherein Nais the number of disk drives and N is greater than two, and a is an integer of at least two, representing the dimensionality of parity protection;
[0037] (b) a plurality of at least Naconventional data disk drives for storing data located within said housing;
[0038] (c) a plurality of parity disk drives equal in number to 2aNc, wherein a is as defined above, and c is a−1, and wherein each of said conventional data disk drives is linked to at least a number of parity disk drives equal to a, and wherein said plurality of parity disk drives are mirrored for guarding data stored on said data disk drives in a mirrored parity arrangement, each of said conventional data disk drives thereby having a multidimensional parity relationship with at least four of said parity disk drives such that multidimensional, mirrored, orthogonal parity arrays of a dimensionality are established for each of said conventional disk drives and each of said conventional disk drives are related to at least two parity disk drives in each a dimension, thereby establishing a parity group for that conventional disk drive and other conventional disk drives sharing said same at least two parity disk drives in each a dimension; and,
[0039] (d) data reconstruction means coupled to each of said multidimensional, mirrored orthogonal parity arrays for reconstructing data from any failed conventional disk drive in a given parity group by combining data from other of said conventional disk drives and parity disk drives in its orthogonal parity arrays.
[0040] The data storage system of the present invention preferably includes a plurality of controller cards, each of said controller cards generating a plurality of channels, each channel being coupled to said data disk drives and said parity disk drives for accessing information therefrom and sending information thereto, at least one controller card being coupled to a channel for receiving access commands from at least one host computer and for selecting channels from among said plurality of channels to execute said access commands.
[0041] The data storage system of present invention preferred embodiments also includes:
[0042] (e) a plurality of controller cards for controlling parity functions of said plurality of parity disk drives;
[0043] (f) a plurality of first processors at a first level connected directly to said plurality of control cards, said first processors having control and processing intelligence;
[0044] (g) a plurality of second processors at a second level connected indirectly to said plurality of control cards, said first processors having control and processing intelligence; and,
[0045] (h) a plurality of third processors at a third level having intelligence, said third processors being connected to at least one of said plurality of first processors and said plurality of second processors;
[0046] whereby said processors have sufficient software, memory and hardware to establish both dynamic and static hierarchical control of said plurality of parity disk drives using distributing intelligence.
[0047] In another alternative embodiment, the present invention data storage system storage system includes a plurality of controller cards, with each of the controller cards generating eight channels. Each of the channels is coupled to the data disk drives and the parity disk drives for accessing information therefrom and for sending information thereto. At least one of the controller cards is coupled to a pre-selected channel adapted for receiving access commands from at least one host computer and for selecting channels from among the other plurality of channels to execute the access commands.
[0048] In one embodiment, the data storage system includes among the plurality of controller cards, a first controller card which controls and calculates linear connection between data disk drives and parity drives in a first dimension and a second controller card which controls and calculates linear connection between data disk drives and parity drives in a second dimension and a third controller card which controls and calculates linear connection between data disk drives and parity drives in a third dimension, such that the first, second and third controller cards are linked to other controller cards so as to permit reconstruction of lost data due to failure of any one or multiple data drives.
[0049] In computer systems having many disk drives, such as tens or hundreds or more, multiple drive failure survivability required methodologies not currently in use in the industry. The design goal of data survivability with multiple drive failure will require an increasing number of drives used for protection. Looking at all the available prior art RAID levels including six, none meet desirable design goals for multiple drive failure survivability. The present invention, on the other hand is a Multidimensional Parity (MDP) system which does fulfill the stringent design goals of multiple drive failure survivability. Two staging and silent target followers allow MDP to achieve higher multiple drive failure survivability than heretofore realized.
[0050] Thus, the Multidimensional Parity (MDP) of the present invention does fulfill the design goal of multiple drive failure survivability because it does not view the data drives from a linear or single dimension perspective or a two dimension perspective. Allowing multiple parity drives to have different relationships with different subsets of data drives and with mirroring within the parity disk drive arrays, are aspects of the basics of the present invention. In the present invention, a data drive will have multiple relationships with other subsets of data drives and parity drives. This multiple relationship allows multiple drive failure survivability.
[0051] MDP is based on this multiple parity relationship in excess of two and can be classified as follows:
[0052] Symmetrical Two Dimensional Parity
[0053] Non-Symmetrical Two Dimensional Parity
[0054] Symmetrical Three Dimensional Parity
[0055] Non-Symmetrical Three Dimensional Parity
[0056] Symmetrical Multidimensional Parity above three dimensions
[0057] Non-Symmetrical multidimensional Parity above three dimensions
[0058] The non-symmetrical members of this classification are included herein, but for purposes of mathematical examples, symmetrical arrangements are illustrated. It is also possible to apply this within the scope of the present invention methodology beyond the third dimension, simply by the connective relationships between, and numbers of parity disks made available. Symmetrical Multidimensional Parity (SMDP) is not very feasible with small numbers of drives, from an economic standpoint, but becomes quickly viable as the number of data drives increases. Thus, as the number of data disk drives grows, it not only becomes feasible it becomes practical and desirable.
[0059] Historically, symmetrical Two Dimensional Parity has N2data drives and 2N parity drives. Thus, every data drive has two parity drives guarding it. The worse case survivability is the lose of the two drives. Best case survivability is the lose of 2N drives. The Storage Overhead is 2N/(N2+2N). The write penalty—[read 3, write 3] is six IOs.
EXAMPLE 1
Prior Art
[0060] For 25 data disk drives, 10 parity drives are needed:
[0061] N=5, N2=25, 2N=10
[0062] 52Storage Overhead=28.6
EXAMPLE 2
Prior Art
[0063] For 900 data disk drives, 60 parity drives are needed:
[0064] N=30, N2=900, 2N=60
[0065] 302Storage Overhead=6.25%
[0066] The worse case survivability of losing two drives will occur when the two drives in question are parity drives that guard the same data drive. Even with the loss of the two parity drives, the data is still available. The probability of losing both protection drives guarding the same data drive decreases as N increases.
[0067] In the present invention, the multidimensional parity drives and resulting array are achieved by multidimensional interconnection between the data disk drives and the parity disk drives in excess of two sets of connections, i.e., in excess of two dimensions. Thus, symmetrical Three Dimensional Parity has N3data drives and 3N2parity drives. Every data drive has three parity drives guarding it. The worse case survivability is the loss of three drives. Best case survivability is the lost of 3N2drives. The Storage Overhead is 3N2/(N3+3N2). The write penalty—[read 4, write 4] is eight IOs.
EXAMPLE 3
Present Invention
[0068] When 512 data disk drives are used, 243 parity drives are needed to achieve three dimensional parity protection:
[0069] N=9, N3=512,3N2=243
[0070] 93Storage Overhead=25%
[0071] The worse case survivability of losing only three drives will occur when the three drives in question are parity drives that guard the same data drive. Even with the loss of the three parity drives, the data is still available. The probability of losing all three protection drives guarding a data drive is less than the N2model for Symmetrical Two Dimensional Parity and decreases even faster as N increases.
Symmetrical Four Dimensional Parity N
[0072] Symmetrical Four dimensional Parity has N4data drives and 4N3parity drives. Every data drive has four parity drives guarding it. The worse case survivability is the loss of four drives. Best case survivability is the lost of 4N3drives. The Storage Overhead is 4N3/(N4+4N3). The write penalty—[read 5, write 5] is ten IOs.
EXAMPLE 4
Present Invention
[0073] For four dimensional parity protection, when 4,096 data disk drives are used, 2,048 parity disk drives are needed:
[0074] N=8, N4=4,096, 4N3=2,048
[0075] 84Storage Overhead=33.3%
[0076] The overhead, complexity and voluminous number of drives needed to effect Symmetrical Four Dimensional Parity make it extremely expensive if a completely integrated system using specially designed data disk drives as in the prior art make it economically ineffective. However, in the present invention conventional data drives may be used and even retrofitted to make it practical.
[0077] The probability of losing all of the protection drives guarding a data drive is defined by the formula below.
[0078] Where N is the number of data drives in a single dimension and D is the dimension.
| TABLE I | |||||||
|---|---|---|---|---|---|---|---|
| STORAGE | INVERSE | ||||||
| N | DIMENSION | N | N | D * N | N | OVERHEAD | PROBABILITY |
[0079] The table above shows the worse case failure scenario probability of loosing a dimensional number of protection drives for different values of N and dimension. Lower probability and higher mathematics are great but in the end, Murphy's Law always rules. If you need an event not to happen then that is what will occur when you can least afford it to.
[0080] If protection drives had a higher probability of not failing than data drives this would raise the survivability MDPP (multidimensional parity protection) could afford. If protection drives could be protected by mirroring them without a performance cost than MDPP worse case survivability doubles and the probability of the event decrease depending on the dimension used and value of N. This would of course increase the storage overhead.
[0081] In the case of Symmetrical Three Dimensional Parity with N equal to eight there are 512 data drives and 192 protection drives. This case has a storage overhead of 27.27 percent. The worse case scenario is the failure of three protection drives with a probability of one in 113,095 of occurring if three drives fail. If the protection drives, were mirrored then worse case moves to losing six protection drives with a probability of one in over 1,380,000,000,000 of occurring if six drives fail. The storage overhead only increases to 42.86 percent. Best case failure moves to 6N2or 384 drives.
[0082] The probability of losing all of the protection drives guarding a data drive with the protection drives mirrored is defined as follows:
Silent Target Follower (STF)
[0083] A Silent Target Follower (STF) is in essence a mirror without the write performance cost. This can be achieved by down streaming or two staging control on the channel of drives. This will require local intelligence between the channel controller and the targets on the channel. This affords another benefit, which is it could be an upgrade or used to derive another saleable model.
Two Staging
[0084] Local intelligence between the Local Storage Controllers and the targets on the channel is what I have termed Two Staging. Two Staging will require a down stream controller from the central system embedded in the same enclosure with the drives. This is exclusive of the simple controller in the tray that provides the composite virtual drive.
[0085] The drives or targets are packaged in enclosures containing 24 trays of drives. Each tray holds two, three or four spindles to assume a virtual drive the aggregate of the spindles. The virtual drive is a single target. The enclosures have two Ultra2 SCSI buses with 12 targets on each bus. Four enclosures can be mounted into a cabinet or rack. This puts 96 targets on eight Ultra2 SCSI buses. The local controller needs to be small and provide fault tolerant redundancy.
[0086] Since the local controller is envisioned to be, a single board with redundant Ultra2 SCSI initiators fail over control will have to be upstream. The local controller cards will need to have five SCSI channels, one of which should be Fibre Channel. This provides redundancy on the eight buses with two initiators on each bus in the four enclosures and it provides four channels of Fibre channel to the upstream.
[0087] Deploying this arrangement with local controllers between the targets and upstream controllers doubles the number of drives that can be reached. The local controllers can implement STF on the buses they are attached on without any write performance to the system. A single local controller presents a channel of targets to the upstream controller.
[0088] Either an alternate means of communication between the system and the local controller is needed such as Ethernet with TCP/IP or control communication is over the SCSI channels. If control communication is over the SCSI channels this means that some vendor specific extended SCSI commands are needed. Possibly implementing the model for processor devices (see SCSI-2 specification, section 12) could server this role of control.
[0089] The present invention should be more fully understood when the specification herein is taken in conjunction with the drawings appended thereto, wherein:
[0090] FIG. 1 shows a prior art symmetrical two dimensional parity system 1 which shows a schematic arrangement wherein the outer frame 3 represents a computer system housing and peripheral operating systems hardware. Contained therein are a plurality of conventional data storage disk drives 5, 7, 9, 11, 13, 15, 17, 19 and 21. They are connected both in an X axis manner (horizontally) and in a Y axis manner (vertically) as represented by connection lines such as lines 23 and 25. Additionally, there are 6 parity disk drives 27, 29, 31, 33, 35 and 37. Theses parity disks are connected both in an X axis manner and a Y axis manner as described in conjunction with two-dimensional parity protection described above.
[0091] FIG. 2 shows symmetrical three-dimensional parity protection system schematically as diagram 51 embodied by frame 53, with three two-dimensional planes 55, 57 and 59 with each plane containing conventional data storage disk drives such as disk drive 61 and parity protection disk drives for secondary storage data such as drive 63. In this simplified embodiment, each of the flat plane arrays 55, 57 and 59 are similar to the two-dimensional system shown in FIG. 1, and are connected similarly within each plane, but additionally, each flat plane has all of the drives connected to the other planes in a Z axis manner to create a three-dimensional parity protection system such as described above.
[0092] In preferred embodiments of the present invention system for multidimensional parity protection, the probability of losing protection drives differs in mirrored and unmirrored system. The following Tables 2 and 3 illustrate these differences.
| TABLE 2 |
|---|
| UNMIRRORED |
| TABLE 3 |
|---|
| MIRRORED |
[0093] FIG. 3 shows a system overview of one preferred embodiment of the present invention system 101, which includes Interface Cards such as card 105, main switch 103, Network Storage Controller 107, Network Storage Controller 109, Global Storage Controllers, such as 111, Global Cache Controllers, such as 113, Management and Request Que Controllers, such as 115 and Memory Banks, such as 117 (Memory Bank A).
[0094] In some embodiments of the present invention, referring again to Table I, direct connectivity can be as much as 288 channels of Fibre Channel, SCSI (Ultra 2), ATM (OC12 or OC48) or Gigabit Ethernet for a series 9000 system. The 7000 series supports as many as 192 channels of the same variety and the series 5000 can support up to 96 channels. The 3000 series supports as many as 72 channels of the same variety and the series 1000 can support up to 48 channels. In addition, all series allow for in-direct connectivity that provides another level of fault tolerance and doubles the number of channels providing connectivity. The channels allow for connectivity to disk drives, host and networks.
[0095] Reference is made to FIG. 4 which shows a present invention functionable system with card interconnect arrangement in which switch 121 is connected to network storage controllers 123, local storage controllers 125, management and request queue controllers 127, global storage controllers 129 and global cache controllers 131, the latter of which is connected to global cache memory 133. This architecture for a present invention system is expanded upon in more detail in FIG. 5 illustrating a LAN/SAN arrangement, wherein identical parts shown in FIG. 4 are identically numbered, but here local storage controller 125 is externally connected to and operates in conjunction with server 135, SAN 137 and drives such as drive 141. Likewise, remote array controller 139 is connected to drives such as drive 147. Note also that both servers 135 and network storage controller 123 are connected to LAN 143.
[0096] FIG. 6 illustrates a retrofit system wherein present invention parity protection system 151 which is illustrated in more detail above and below, is connected to LAN 153, heterogenous servers such as server 155, and SAN 157, with drives 159 and storage system 161.
[0097] One of the variants of the Local Storage Controller Line Card which may be used in the present invention is depicted in FIG. 7 as a function block diagram 171. This variant is an eight port Fibre Channel card shown with channels 173, 175, 177, 179, 181, 183, 185 and 187. The overall system described above functions in conjunction with component processors 197. The SCFP 189 (Switch Channel Fabric Port) is the interface from the crosspoint switching fabric to the multichannel bus on the card 191. The I3Sp components 193 and 195 (Intelligent Interface Slave Processor) provide bridging to and from the two PCI 2.0 64 buses and the four channels of each Qcardbus 201 and 203. This allows the memory pool to be shared by the ISP's (Intelligent SCSI Processor) that drive the Fibre Channel ports and SCSI channel buses and other elements on the card. The architecture eliminates the need for each element to have its own copy of data. In designing the memory as a resource that is available to all elements on the card latency is reduced and the level of collaboration between elements is increased.
[0098] In the present invention system, several cards work with tandem members. These include the Global Cache Controllers and Management and Request Queue Controllers. The Reflective Memory Bus allows some portion of their memory space to be reflected to the tandem member.
[0099] Depicted in FIG. 8 is a function block diagram which is the Global Cache Controller system shown generally as 207. Collaboration with its tandem member is achieved through segmentation of tasks and Reflective Memory. Segmentation of tasks could include implementing a two-way interleaf for paged memory such that each Global Cache Controller is responsible for a single way in the two way interleaving approach.
[0100] There are two Global Cache Memory Buses. The Tandem Global Cache Controllers are on both buses. Each bus has a data path that is 32 bytes (256 bits) wide. Each of the two buses have 32 address lines and eight control lines. The 32 address lines allow 2TB to be addressed on each bus because the smallest addressable unit of memory is 512 bytes.
[0101] In preferred embodiments of the present invention, distributed architecture and hierarchical control and distributed cache and memory mapped storage become very important with larger systems.
[0102] The RAID write penalty occurs for RAID levels above level two. It significantly affects the performance of the system because of the extra overhead created by additional IO operations to support the parity. The extra IO operations needed include the party calculation and the write of parity. The parity calculation could require the read of the old data and the old parity. This approach has three additional IO operations for every write. This three for one write penalty is the overhead for single dimension parity.
[0103] Multidimensional parity has a greater overhead. Two additional IO operations are needed for each dimension. One to read the old parity and another to write the updated parity. A two dimensional parity would incur five additional IO operations and a three dimensional parity would incur seven additional IO operations.
Distributed CACHE and Memory Mapped Storage
[0104] A Global Memory is used for Memory Mapped Storage. Each line Card such as the various versions of Local Storage Controllers and Network Storage Controllers also have memory for buffering and caching IO operations. Most applications do not just request a write to be performed without first doing a read. Every real world application or environment has some level of sequential and deterministic aspects. How well the system finds and exploits this is a factor of the cache performance. The cache performance is delineated by read and write performance. Read performance is based on cache size; the system's ability to anticipate the next read and benefit of reusable IOs already in the cache.
[0105] Cache and Global Memory provide performance gains by allowing data to be pre read and post written. Most applications operate on a small subset of the total data during a specific time. This is known as “Locality of Reference”. There are three aspects of Locality of Reference. They are Temporal Locality, Spatial Locality and Sequential Locality. Real world applications incur all three aspects of “Locality of Reference” to some degree. The larger the pool of memory and cache is, the larger the locality that can be exploited.
[0106] The large memory and cache pools also effect the write performance penalty of RAID parity. The per-read aspect of having the old data page in memory or cache is an IO operation that is not incurred again. Post write allows the delay of dealing with the parity overhead for writes so that if one or more write requests affects the same parity page the parity overhead is spread across all the writes instead of just one.
Distributed Architecture and Hierarchical Control
[0107] Incurring overhead in dealing with parity protection is one aspect of performance. Another is where and how a system incurs the overhead of additional IOs. The distributed architecture and hierarchical control allows the parity overhead to be limited by isolating or decoupling its effects on system performance.
[0108] At the lowest level of hierarchical control, the tray enclosure has a two port controller. This is simply illustrated in FIG. 9 shown generally as diagram 207 with the legend 209 explaining each symbol. One port attaches to a bus that supports one or more disk drives and an Environmental Monitor. The other port attaches to one of the two buses in the Array Enclosure providing upstream connectivity. The tray controller also supports an option for XOR (Exclusive OR) writes. By sending the same difference record, which is the difference between the old data and the latest new data to all parity drives, for all dimensions of parity, the parity can be updated. This is accomplished by reading the old parity and XORing the difference record to obtain the new parity. This allows the system to incur the cost of the additional reads needed for parity without incurring their cost beyond the local disk drive.
[0109] The cost of mirroring every parity drive is also absorbed by the distributed architecture and hierarchical control. FIG. 10 shows a more detailed diagram 210 with switch 211 and disk array enclosures 213, 215, 217 and 219. Legend 221 describes the symbols. An optional controller referred to as the “Remote Array Controller” and shown as, for example, intelligence processor 223 with its controllers connected thereto is inserted between the Local Storage Controller and the Disk Array Enclosure. This provides a means for implementing a STF (Silent Target Follower).
[0110] The drives or targets are packaged in enclosures containing 24 trays. Each tray holds one, two, three or four spindles to assume a virtual drive the aggregate of all. The virtual drive is a single target or LUN. The enclosures have two internal SCSI buses with 12 targets on each bus. Each target has two LUNs. The first LUN, LUN zero is for accessing the Environmental Monitor. The other LUN provides access to the virtual drive. Two enclosures can be mounted into a cabinet or rack. This puts 48 targets on four Ultra2 SCSI buses.
[0111] Since the Remote Array Controller is envisioned to be a single board with redundant Ultra2 SCSI initiators, fail over control will have to be upstream. The Remote Array Controller cards will need to have five ports, one of which should be Fibre Channel. Using a buddy approach with two Remote Array Controllers provides redundancy on the four buses with two initiators on each bus in the two enclosures and it provides two channels of Fibre channel upstream to Local Storage Controllers.
[0112] Deploying this arrangement with Remote Array Controllers between the targets and upstream controllers doubles the number of drives that can be reached by the Local Storage Controller. The Remote Array Controller can implement STF on the buses they are attached on without any write performance to their upstream controllers and the system. A single Remote Array Controller presents a channel of targets to its upstream controller.
[0113] Managment and control of the Remote Array Controller should be implemented with vendor specific extended SCSI commands. This should include downloading the processor or processors on the Remote Array Controller, health handshake and assume control of channel/targets. Implementing the model for processor devices (see SCSI-2 specification, section 12) could server this role of control.
[0114] If Remote Array Controllers are not used then two ports from two different Local Storage Controller must be connected to each internal bus within the Array Enclosure. This would provide the same level of fault tolerance while reaching only 25% of the total storage that Remote Array Controllers could with a single Local Storage Controller port. To excrete the cost of mirroring the embedded Tray Enclosure Controllers implement the concept of SLAVE. The Tray Enclosure Controller acting as a slave will perform the same write commands as the master for a single XOR write request on the channel to the master. This of course limits the participation for mirrors to devices on the same enclosure bus.
Domains
[0115] A domain like a rank defines the spindles that participate within its scope. This will include spindles for data, parity and hot spares. Unlike a rank a domain is user definable within limits and not bounded by any controller. In addition, a domain does not define bounds for participation in the creation of a LUN meaning that a LUN may span one or more domains. The domain does define the protection scheme for all is of its data drives.
| Model | 9100 | 9000 | 7100 | 7000 | 5100 | 5000 | 3100 | 3000 | 1100 | 1000 |
|---|---|---|---|---|---|---|---|---|---|---|
Conclusion on Mirrored MPD
[0116] The majority of the performance cost of doing either two-dimensional mirrored parity or three-dimensional mirrored parity is mitigated by the approaches in exploiting the distributed architecture. The cost of channel contention has not been considered. Additionally, the difference between two and three dimensional mirrored parity is storage overhead and protection required; this should be a user option.
[0117] Obviously, numerous modifications and variations of the present invention are possible in light of the above teachings. It is therefore understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.
Claims
What is claimed is:
1. A data storage system for multidimensional parity protection of at least three dimensions, comprising:
(a) a housing for containing a plurality of Naconventional disk drives, wherein Nais the number of disk drives and N is greater than two, and a is an integer of at least three, representing the dimensionality of parity protection;
(b) a plurality of at least Naconventional data disk drives for storing data located within said housing;
(c) a plurality of parity disk drives sufficient in number to provide mirrored, multidimensional, orthogonal parity arrays in at least three dimensions, and equal in number to 2aNc, wherein a is as defined above, and c is a−1, and wherein each of said conventional data disk drives is linked to at least a number of parity disk drives equal to a, and wherein said plurality of parity disk drives are mirrored for guarding data stored on said data disk drives in a mirrored parity arrangement, each of said conventional data disk drives thereby having a multidimensional parity relationship with at least four of said parity disk drives such that multidimensional, mirrored, orthogonal parity arrays of a dimensionality are established for each of said conventional disk drives and each of said conventional disk drives are related to at least two parity disk drives in each a dimension, thereby establishing a parity group for that conventional disk drive and other conventional disk drives sharing said same at least two parity disk drives in each a dimension;
(d) data reconstruction means coupled to each of said multidimensional, mirrored orthogonal parity arrays for reconstructing data from any failed conventional disk drive in a given dimensional parity group by combining data from other of said conventional disk drives and parity disk drives in its orthogonal parity arrays;
(e) a plurality of first processors at a first level connected directly to said plurality of control cards, said first processors having control and processing intelligence;
(f) a plurality of second processors at a second level connected indirectly to said plurality of control cards, said first processors having control and processing intelligence; and,
(g) sufficient software, memory and hardware within said processors to establish both dynamic and static hierarchical control of said plurality of parity disk drives using distributing intelligence.
2. The data storage system of claim 1 further includes a plurality of control cards, each of said controller cards generating a plurality of channels, each channel being coupled to said data disk drives and said parity disk drives for accessing information therefrom and sending information thereto, at least one controller card being coupled to a channel for receiving access commands from at least one host computer and for selecting channels from among said plurality of channels to execute said access commands.
3. The data storage system of claim 2 wherein a collection of controllers provides access and control over a collection of disks, each of said data disk drives belonging to a parity relation in each dimension constructed.
4. The data storage system of claim 1 wherein there are a plurality of N3conventional disk drives and a plurality of 6N2parity disk drives to establish mirrored three dimensional, orthogonal parity arrays.
5. The data storage system of claim 1 wherein said controller cards are contained within a plurality of tray enclosures, each of said plurality of tray enclosures having at least one controller with at least two-ports, one port of which is connected to a bus connected to one or more disk drives and another port of which is connected to an environmental monitor contained within said tray enclosure for controlling and managing comparative data between old data and current data from the conventional disk drives to the plurality disk drives for parity updating.
6. A data storage system for multidimensional parity protection of at least three dimensions for connection to a plurality of conventional disk drives equal in number to Na, wherein Nais the number of disk drives and N is greater than two, and a is an integer of at least three, representing the dimensionality of parity protection for retrofit connection to a plurality of conventional data disk drives, which comprises:
(a) a plurality of parity disk drives sufficient in number to provide mirrored, multidimensional, orthogonal parity arrays in at least three dimensions, and equal in number to 2aNc, wherein a is as defined above, and c is a−1, and wherein each of said conventional data disk drives is linked to at least a number of parity disk drives equal to a, and wherein said plurality of parity disk drives are mirrored for guarding data stored on said data disk drives in a mirrored parity arrangement, each of said conventional data disk drives thereby having a multidimensional parity relationship with at least four of said parity disk drives such that multidimensional, mirrored, orthogonal parity arrays of a dimensionality are established for each of said conventional disk drives and each of said conventional disk drives are related to at least two parity disk drives in each a dimension, thereby establishing a parity group for that conventional disk drive and other conventional disk drives sharing said same at least two parity disk drives in each a dimension;
(b) connection means for functionally connecting said parity disk drives to said conventional data disk drives;
(c) data reconstruction means coupled to each of said multidimensional, mirrored orthogonal parity arrays for reconstructing data from any failed conventional disk drive in a given parity group by combining data from other of said conventional disk drives and parity disk drives in its orthogonal parity arrays;
(d) a plurality of controller cards for providing access and control over a collection of disks;
(e) a plurality of intelligence for parity calculation in each dimension;
(f) a plurality of first processors at a first level connected directly to said plurality of control cards, said first processors having control and processing intelligence;
(g) a plurality of second processors at a second level connected indirectly to said plurality of control cards, said first processors having control and processing intelligence;
(h) a plurality of third processors at a third level having intelligence, said third processors being connected to at least one of said plurality of first processors and said plurality of second processors; and, (i) sufficient software, memory and hardware within said processors to establish both dynamic and static hierarchical control of said plurality of parity disk drives using distributing intelligence.
7. The data storage system of claim 6 wherein each of said controller cards generating a plurality of channels, each channel being coupled to said data disk drives and said parity disk drives for accessing information therefrom and sending information thereto, at least one controller card being coupled to a channel for receiving access commands from at least one host computer and for selecting channels from among said plurality of channels to execute said access commands.
8. The data storage system of claim 7 wherein a first controller card controls and calculates linear connection in a first dimension and wherein a second controller card controls and calculates linear connection in a second dimension and wherein a third controller card controls and calculates linear connection in a third dimension, said first, second and third controllers being linked to other said controller cards so as to permit mirrored multidimensional orthogonal parity array data transmission between said data disk drives and said parity disk drives.
9. The data storage system of claim 7 wherein said controller cards are contained within a plurality of tray enclosures, each of said plurality of tray enclosures having at least one controller with at least two-ports, one port of which is connected to a bus connected to one or more disk drives and another port of which is connected to an environmental monitor contained within said tray enclosure for controlling and managing comparative data between old data and current data from the conventional disk drives to the plurality disk drives for parity updating.
10. The data storage system of claim 6 wherein there are a plurality of at least N3conventional disk drives and a plurality of at least 6N2parity disk drives.
Patent Citations (12)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US5124987(A) | 1992-06-01 | Milligan et al. | Examiner |
| US5271012(A) | 1993-12-01 | Blaum et al. | Applicant |
| US5351246(A) | 1994-09-01 | Blaum et al. | Applicant |
| US5412661(A) | 1995-05-01 | Hao et al. | Applicant |
| US5530948(A) | 1996-06-01 | Islam | Examiner |
| US5544339(A) | 1996-08-01 | Baba | Applicant |
| US5572659(A) | 1996-11-01 | Iwasa et al. | Applicant |
| US5774643(A) | 1998-06-01 | Lubbers et al. | Examiner |
| US5778426(A) | 1998-07-01 | DeKoning et al. | Examiner |
| US6154853(A) | 2000-11-01 | Kedem | Examiner |
| US6223252(B1) | 2001-04-01 | Bandera et al. | Examiner |
| US6223323(B1) | 2001-04-01 | Wescott | Examiner |
Non-Patent Literature (9)
- Menon, J et al. (Methods for improved update performance of disk arrays; IEEE, Jan. 7-10, 1992).Applicant
- Yeung, K.H. et al. (Dynamic parity logging disk arrays for engineering database systems; Computers and Digital Techniques, IEE Proceedings; Sep. 1997).Applicant
- Holland, M. et al. (Fast, on-line failure recovery in redundant disk arrays; The Twenty-Third International Symposium on Fault-Tolerant Computing, Jun. 22-24, 1993; Aug. 1993),Applicant
- Designing Disk Arrays for High Data Reliability Garth A. Gibson et al.Applicant
- RAID-II: A High-Bandwidth Network File Server Ann L. Drapeau et al.Applicant
- Report of the Working Group on Storage I/O for Large Scale Computing Garth A. Gibson et al. Nov. 1996.Applicant
- Redundant Disk Arrays Reliable Parallel Secondary Storage Garth A. Gibson 1991.Applicant
- Coding Techniques for Handling Failures in Large Disk Arrays Lisa Hellerstein et al.Applicant
- RAID: High Performance, Reliable Secondary Storage Peter M. Chen et al.Applicant