System and Method for Managing Data in a Cache System for a Disk Array

This invention relates to a cache system for use with disk arrays comprising a number of data storage disk devices and more particularly to a cache system operable to reduce the numbers of redundant accesses to the disks of such arrays.

In recent years, there has been a growth in interest in disk arrays. Disk arrays consist of a number of disk drives connected to a host system via one or more controller elements which control the transfer of data between the host and disk drives. A disk array is designed to provide high capacity data storage, high reliability and high data transfer rates to and from the using system.

A number of different array architectures have been proposed. A paper entitled `A Case for Redundant Arrays of Inexpensive disks (RAID)` (ACM SIGMOD conference proceedings, Chicago, IL., Jun. 1-3, 1988, pp.109-116) details five levels of array (RAIDS 1 to 5) which provide different levels of data management. Each of the RAID levels permits users to increase their data storage capacity by linking together a number of inexpensive disk drives. The RAID system provides protection against loss of data through the failure of a drive by either storing two copies of the data on two drives (RAID 1) or by splitting the original data into a number of subsections and striping the data across two or more drives of the array. The parity data for the striped data is calculated and stored in a dedicated parity drive (RAID-4). In the event that one of the data holding drives fails, it is possible using the parity data and the remaining data of the stripe to reconstruct the data on the failed drive (RAIDs 2 to 5). The architecture of RAIDs 2 to 5 thus allows an increase in data throughput, assuming that these data transfers are allowed a degree of concurrency. This speed-up depends upon the degree of concurrency, the overheads of partitioning the original data transfer into many, shorter, sub-transfers, and the parameter selection regarding the relationship between average transfer length and size of sub-transfer. Further details of the RAID 4 and RAID 5 configurations may be found in the above referenced conference proceedings.

The provision of parity/checksum in a Redundant Array of Inexpensive Disks (RAID-5) system confronts the system architect with many problems. These include error-recovery, data-loss protection, system performance, and implementation complexities. One penalty of employing a RAID system is the potential problem of reduced reliability. The reliability of a RAID system reduces as the number of devices increases, since any single device failure results in a complete array failure. To avoid this unacceptable degradation in system reliability, a method of providing data recovery in the event of a (single) device failure occurring has proven to be necessary. This is achieved by introducing `redundancy` into the array, in that a `second` copy of the data is stored as a parity or checksum on a device separate to the associated data devices (e.g. in the RAID 4 configuration). Thus the data is never stored in a single location, since the generation of the parity allows subsequent recovery of lost data resulting from a (single) device failure.

During normal Read operations, there is no performance impact encountered by supporting parity generation. However, during Write operations, the generation of parity becomes a concern. This is due to the fact that any alteration to a data area must have an associated update of the parity data relevant for that data area. Furthermore, the likelihood of system failure between writing the new data and writing the new parity must be minimised, or else data recovery would be impossible. This last requirement is conventionally met by generating and storing the parity on the device in conjunction with the Host storing the data on the device, as and when the data becomes available.

This method poses potentially large overheads, since the parity device is associated with a number of data devices. This poses no superfluous overheads provided all devices associated with a particular parity device are written to concurrently, the resulting parity being a composite of all data devices. This will generally not be the case, since relatively short accesses will not incur access to all devices. In this case, parity generation involves retrieving either the existing data from the unaddressed devices, or reading the existing parity for that data group together with the data prior to writing the new data.

In certain situations, the overhead of parity update due to short transfer lengths becomes unacceptable. This is also the case with longer transfers that are split into sequences of shorter transfers (the Host Filing System may do this).

The present invention seeks to overcome the drawbacks associated with the prior art and accordingly provides a cache system including a cache memory for staging data transferred between a host system and a disk array, the disk array comprising a plurality of disk storage devices in which data is stored in strides, each stride comprising a multi-sector unit of data on each of two or more devices of the array and an associated multi-sector parity block on a further array device; the cache system including means for linking the data and associated parity making up a stride within the cache as a cache data unit which is manipulated in the cache as a single entity.

Host accesses are directed to a cache memory which contains both the data and the associated parity. In the present invention, the cache is organised not as units based on the size of the smallest data transfer, but on a parity block and its associated data blocks. As an example, a cache unit may be three 4K byte-blocks and the single, 4 Kb parity block.

In a preferred cache system, a cache unit which includes data addressed by the host is moved to the head of a Most Recently Used (MRU) list of cache units. Host write data is then stored in this cache unit, possibly alongside data stored from previous accesses. As further cache units are addressed, a particular `untouched` unit will be pushed further down the cache list. Using a cache in this manner has the advantage that if the nature of accesses to data within the cache is not entirely random, but contains a degree of sequential accesses (not necessarily directly related to time), and the size of cache is sufficiently large, the likelihood of cache units residing toward the `unused` end of the cache list containing a full set of valid data will be high. If all the data in a cache unit is present, this allows a subsequent committing of the cached data to the disk array (since cache size is finite) along with the parity that can be generated with no further data accesses. The redundant accesses needed to generate the parity in an uncached architecture are thus eliminated. In the event that not all the data of a cache unit is present in the cache, then it will be necessary to access some data on disk in order to generate the updated parity data. However, the number of the redundant accesses required will be greatly reduced with a consequent improvement in system performance.

The size of cache which is necessary to reduce or eliminate the redundant accesses will be dependent on the nature of the data accesses requested by the host. If the data requests are largely sequential in nature then the amount of cache memory required to eliminate, or substantially eliminate the redundant accesses to the disk devices will be smaller than if the requests are largely random in nature.

In adopting a cache system, it is important to ensure that the data is not allowed to reside in unprotected memory. A preferred cache system employs a dual, non-volatile implementation of cache memory which provides the necessary reliability. In the event of power or hardware-failure, the data is preserved until the system is functioning again. In the event of a (single) device failure, the reconstruction of the lost data is similar to that employed by the conventional approach to parity generation.

A preferred embodiment of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:

FIG. 1 shows a data storage hierarchy in which a host system is connected via a cache and host adapter to a disk array;

FIG. 2 shows in schematic form the arrangement of a stride of data on the tracks of the data storage devices making up the disk array of FIG. 1;

FIG. 3 shows in block diagrammatical form the data request mechanism including the association of control blocks defined by the cache system according to the present invention;

FIG. 4 illustrates the hash table and side-chaining employed in the cache system according to the invention; and

FIG. 5 shows the structure of the tasks employed in the implementation of the present invention, including their relationship to the control blocks.

In the following, the cache system is implemented in software running on the host processor.

FIG. 1 shows a data processing system comprising a host system 10 connected via a host bus 15 to an array adapter 20. The adapter is connected in turn via a communication link 25 to a disk array 30 taking the form of an array controller 32 attached to four disk data storage devices 34, 36, 38, 40. Also attached to the host bus is a non-volatile cache memory 50 which is used to stage data being transferred between host memory and the disk storage devices of the disk array.

FIG. 2 shows the manner in which data is stored on the disk storage devices of the array used in conjunction with the cache system of the present invention. In FIG. 2, two strides of data are shown stored across data storage tracks 35, 37, 39, 41 in each of the devices 34, 36, 38 and 40. Each device contains a subunit of the stride known as a stripe 60 which in FIG. 2 is three sectors (3.times.512 bytes) in length. The first stride of data comprises the three A1 sectors on disk A plus the three B1 sectors on disk B plus the three C1 sectors on disk C and the associated parity data, P1, on the parity disk. Similarly, the second stride of data comprises three A2, three B2, three C2 and three P2 sectors. As described above, in prior art disk array systems, a request from the host to write new sectors 70, 71, 72 and 73 to the array will involve reading from the disks the rest of the data in the stride, the calculation of new parity data and the writing of the new sectors 70, 71, 72 and 73 plus new parity to the array.

The mechanism by which data requests by the host are serviced by the cache system will now be described with reference to FIGS. 3 to 5. FIG. 3 shows the association of the storage areas (control blocks) used by the host computer program, embodying a preferred implementation of the present invention, to hold control information. FIG. 5 shows the structure of the program (subdivided into Requester and Server tasks) including their relationship to the defined control blocks.

Incoming requests from the host 10 are placed in a queue (GPin) 100 of IO packets. The Host now waits for a Status to be returned back to it, signalling success/failure of the request. The queue is now read by the subsystem, and the resulting `Get/Put` packet (GPBLK) is `striped` into a number of jobs (JIOBs) 120, 122, 124, the completion of all these subtasks satisfying the original request. Each of these JIOBs has the property that the entire data transfer request resides in a single stripe ie. does not cross a stripe boundary. To enable progress tracking of these JIOBs, each JIOB is associated with a single parent Job Control Block (JCB) 110 which has a one-to-one correspondence with the original application request. Each of these JIOBs are now queued, ready for execution. These JIOBs are now processed by the Cache Manager (FIG. 4). If the data is available in Cache Memory, that particular JIOB may be able to complete immediately. In the case that some, or indeed all, the data is not present in the Cache, the data must be fetched from the Disk 34 (only for the case of a Read request as Writes don't require disk accesses). Cache management is composed of a doubly-linked list of Cache Control Blocks (CCBs) 150, maintained in a Most-Recently-Used (MRU) ordering scheme. To allow efficient cache searching, hashing is performed on the unique stride address. To allow for hash `collisions`, sidechaining is also used.

To enable completion of a JIOB, further subdivision is undertaken and a menu of Stripe IO requests (SIOBs) 160, 162, 164 is created. Each stripe that forms part of a stride has an associated control block called a Stripe Control Block (SCB) 130, 132, 134, 136. This maintains status of the associated memory area (Stripe Data Block--SDB), and allows queuing of JIOBs that operate upon common stripes. Each SIOB created is placed on the disk IO queue for subsequent execution.

As and when SIOBs complete, the SCB from which they were issued is updated and status is gathered. When ALL SIOBs have completed, any JIOBs that were queued because they could not immediately complete can now complete. As and when these JIOBs complete, the JCB from which they were issued is updated and status gathered. When ALL JIOBs have completed the entire request can now complete. Status is returned to the Host in the GP packet originally received into the subsystem.

The format of each of the control blocks used in this embodiment of the invention will now be described.

The following illustrates the preferred embodiment of the packet block used to receive requests from the Host Application: ##STR1##

GPBLK.sub.-- RWFLAG--specifies whether Host request is Read or Write.

GPBLK LENGTH--the length of the entire access in sectors,

GPBLK.sub.-- DISKADDR--virtual sector number (in array) of the start of the request,

GPBLK.sub.-- PARAMID--tag ID for returning completion status.

GPBLK BUFFERPTR--physical address in Host of start of data area,

GPBLK.sub.-- RETURNCODE--contains the merged statuses of each of the separate requests for return to the Host,

The merged statuses from each of the separate accesses are also returned in this packet,

The Job Control Block is a `parent` to each of the spawned JIOBs and has a one-to-one relationship with User requests (incoming GPBLKs). The completion of a JIOB causes the JCB.sub.-- counter to decrement, and status is merged with the current status.

The following illustrates the preferred embodiment of the Job Control Block: ##STR2##

JCB.sub.-- COUNTER--maintains the number of uncompleted JIOBs that need to complete before status can be returned to the Host.

JCB.sub.-- WAITALLSEM and JCB.sub.-- WAITALLHANDLE--Semaphore and Handle used to suspend the Host request until completion of all the outstanding sub-requests (JIOBs).

JCB.sub.-- RETURNCODE--Merged statuses of all the sub-requests.

The JIOB contains a portion of the Host request, but is contained within a single stripe.

The following is an illustration of the preferred embodiment of the JIOB: ##STR3##

JIOB.sub.-- QPTRS--Allows queuing of the JIOB in various queues.

JIOB.sub.-- RWFLAG--Indicates whether this IO request to the DASD is read or write.

JIOB.sub.-- DEVICE--This is the required physical device to which the request is destined.

JIOB DISKADDR--This is the physical sector address on the device of the start of transfer.

JIOB.sub.-- LENGTH--This is the length of transfer in sectors. Note that this length is such as to never cause a stripe boundary to be traversed.

JIOB BUFFERPTR--This is calculated new address in the Host of the start of data for this subtask.

JIOB.sub.-- JCBPTR--A pointer back to the parent JCB from which this subtask was issued.

JIOB.sub.-- SCBPTR--A pointer to the SCB to which this JIOB refers.

The Cache Control Block is maintained in a doubly-linked list and binds together a number (number of devices) of Stripe Control Blocks (SCBs), which allows caching on a stride basis.

The following is an illustration of the preferred embodiment of the Cache Control Block: ##STR4##

CCB.sub.-- QPTRS--Linkage pointers to maintain the CCB list and the FreeCCB queue.

CCB INUSEFLAG--Protection to avoid freeing-off CCBs that are currently in use.

CCB.sub.-- HASHPTR--Allows collisions in the Hash Table. This points to the next CCB which would have resided at this hash location, if it were available.

CCB STRIDEADDR--Holds the unique Stride Address. Needed because of the possibility of hash `collisions`.

CCB.sub.-- SCBPTR(DEVICE )--An array of pointers, initially null, referencing associated Stripe Control Blocks (SCBs). One per device.

The Stripe Control Block maintains the status of the Cache Memory also known as the Stripe Data Block (SDB). There is a one-to-one relationship between SCB and SDB, but an SDB doesn't necessarily exist if the corresponding SCB exists. The allocation of SDB/SCB memory is based on a dynamic scheme.

The following is an illustration of the preferred embodiment of the Stripe Control Block: ##STR5##

SCB.sub.-- QPTRS--Used to maintain list of `free SCBs`.

SCB.sub.-- VALIDMASK--Identifies each sector held in Stripe DataBlock (SDB) as being valid or garbage. One bit per sector.

SCB.sub.-- MODIFIEDMASK--Identifies sectors in the SDB which need to be saved to disk at some point (which DASD sectors have become invalid).

SCB.sub.-- IOSTATE--Indicates one of four SCB states--Idle,Wait Reading, Wait Writing or Wait Committing.

SCB.sub.-- COUNTER--Retains a count of uncompleted DASD requests (SIOBs) which need to be performed before the SCB can return to the Idle state.

SCB.sub.-- SIOBQPTRS--This is the head of a temporary linked list (menu) of SIOBs.

SCB.sub.-- RWAITQPTRS--This is the head of a linked list of suspended JIOBs, which all requested a Read operation on the associated SDB. Only when the SCB.sub.-- counter returns to zero can these requests be satisfied.

SCB.sub.-- WWAITQPTRS--Similar to the Read wait queue above, but these JIOBs all requested a Write operation. These can only be executed after ALL read operations have completed.

SCB.sub.-- DATAPTR--This points to the Stripe Data Block(SDB). If an SDB has not yet been allocated to this SCB, the pointer will be Null.

SCB.sub.-- RETURNCODE--Maintains a merged status from the SIOBs that were issued to satisfy this sub request.

The Stripe I/O Block is a queued request sent to the DASD, the purpose of which is to fetch uncached or store cached sectors to/from the DASD itself. Since the original JIOB is contained within a stripe, the number of SIOBs spawned from a JIOB will be at most N/2, where N is the number of sectors per stripe.

The following is an illustration of the preferred embodiment of the Stripe I/O Block: ##STR6##

SIOB.sub.-- QPTRS--Maintains the SIOB in one of a number of various queues.

SIOB.sub.-- RWFLAG--Indicates that this request is either Read or Write.

SIOB.sub.-- DEVICE--References the physical DASD device number.

SIOB.sub.-- DISKADDR--The physical LBA on the above DASD where the request is to commence.

SIOB.sub.-- LENGTH--The length in sectors of the required access. This is such that a stripe boundary will not be crossed, nor will valid & modified data in the cache be over written on a DASD Read request.

SIOB.sub.-- BUFFEROFFSET--This is the offset in sectors from the start of the SDB (SCB.sub.-- dataptr) at which data transfer is to commence. (NB. SCB.sub.-- dataptr may be Null at this time if an SDB has not yet been allocated)

SIOB.sub.-- SCBPTR--Points back to the parent SCB from which this SIOB was generated.

The management of the cache is based on a `most recently used` algorithm. An incoming data request (read or write) is initially checked to see if cache memory has been previously allocated by an earlier request. If no memory has been allocated, an attempt is made to allocate a memory area for data storage. If this fails, the command is suspended until memory becomes available.

When the requested memory area is already present, or the above allocation is successful, the host request now operates upon this cache memory. When a read requests data not currently valid in the cache, a disk IO is performed from disk into cache memory before the requested data is copied to the host application. A write request merely copies data from the application to cache memory (acts as a Fast Write Buffer).

The Cache Control Block (CCB) allows the individual stripes in a stride group to be manipulated together. These CCBs are retained in a doubly-linked list, in order of Most Recently Used (MRU) to Least Recently Used (LRU). A data access to a CCB causes that CCB to be moved to the MRU position in the CCB list. Thus unused CCBs and their corresponding Stripe Control Blocks (SCBs) and Stripe Data Blocks (SDBs) will be pushed to the LRU end of the CCB list. These CCBs will now become candidates for committing to DASD, enabling resources to be freed-up. The number of SCB's associated with a CCB will correspond to the number of disk storage devices in the array.

Rapid searching of the CCB list is performed by `hashing`. As shown diagrammatically in FIG. 4, the stride address of the current request is `hashed`, and the resulting function return-value is used as in index into a Hash Table. The table 170 is an array of pointers 172 to CCBs held in the CCB list. If the entry is `null`, the CCB is not present in the list. If the Table points to a CCB, the stride addresses must be compared. This is because many stride addresses can map onto a single Hash Table entry. These `collisions` are handled by `side-chaining`. The first CCB referenced by the Hash Table has a pointer which is the head of a potential list of CCBs with different stride addresses, but common hash values. To verify existence/nonexistence of a CCB, this side-chain must be traversed, comparing stride addresses. Generally, the length of the side chain will be small, provided the Hash Table is large and the Hash Function chosen to minimise collisions.

The computer program embodying the present invention is divided into two main tasks which are separately responsible for servicing requests from the host and for servicing disk IO requests. These two tasks are called the Requester Task and the Server Task. FIG. 5 shows (in block diagrammatical form) the structure of the cache system according to the present invention including the operational relationship between the tasks and the control blocks.

The Requester Task is responsible for servicing the User Application requests, and attempts to progress these requests as far as possible. If a resource such as disk IO or cache memory is required and is not currently available, the partially completed request is placed on one of the many queues for later processing. In FIG. 5, the Requester Task is shown subdivided into two primary functions--the Array Manager 220 and the Cache Manager 240. A listing of the sequence of instructions in the Requester Task is shown in Table 1, appended hereto.

The Server Task (250 in FIG. 5) services the waiting disk IO requests. On completion of the requested disk IO, the Requester Task is indirectly signaled of its completion by the Server updating the status of the associated control block. A listing of the sequence of instructions in the Server Task is detailed in Table 2, appended hereto.

A Background process (260 in FIG. 5) is also provided which has two primary functions--(i) committing CCBs at the bottom of the MRU list to the disk array, thereby freeing up cache resource and (ii) when cache resource is available, removing SCBs from the SCBWaitqueue and allocating SDBs thereto. As part of the process of committing a CCB to disk, the background process will perform parity generation including reading data from disk as necessary. A listing of the instructions in the Background Process is detailed in Table 3, also appended hereto.

As shown in FIG. 5, a GP packet 100 is subdivided by the Array Manager 220 into a number of JIOBs 120, 122, 124 which are held on a JIOB queue. The Cache Manager processes the JIOBs and where necessary generates a number of SIOBs 160, 162, 164 which are placed on the disk IO queue where they are subsequently processed by the Server Task 250. As the SIOBs are completed the server task updates the SCBs 130, 132, 134 & 136.

There now follows a description of an example read/write scenario using the cache system of the present invention. In order to fully understand the example scenario, reference should be made as appropriate to TABLES 1 to 3 appended hereto.

1. If the host request is READ & ALL sectors are present in the cache, complete request NOW since no further disk IO is required.

2. If the host request is READ & SCB is Idle or WaitWriting & not ALL sectors valid, set IOstate to WaitReading, queue JIOB>SCBReadWaitQ, build menu of SIOBs needed to read ALL in valid sectors into stripe (see 3. below), and IF allocation of SDB successful commence disk IO (move SIOBs to diskIOQ) ELSE put SCB>SCBWaitQ (ie. postpone access until SDB is available).

3. If the host request is READ & SCB is WaitReading & not ALL sectors valid,assume the PREVIOUS access that set IOstate to WaitReading will fulfil THIS request also, so queue JIOB>SCBReadWaitQ.

4. If the host request is READ & SCB is WaitCommitting & not ALL sectors present, can't service request yet since the one-and-only SIOB menu queue is being used for the commits, so queue JIOB>SCBReadWaitQ.

5. If the host request is WRITE & SCB is Idle & SDB available, then complete whole of request now.

6. If the host request is WRITE & SCB is waiting (read/write/commit), queue JIOB>SCBWriteWaitQ. (NB. it is not permitted to Write before ALL Reads have completed, since non-Host initiated Cache reads cannot be allowed to overwrite valid Cache data.)

7. If the host request is WRITE & SCB is idle & no SDB available, queue JIOB>SCBWriteWaitQ, queue SCB>SCBWaitQ and set state to WaitWriting (postpone entire access until SDB is available).

After the final SIOB issued from an SCB has completed (SCBcounter=0), the following occurs:

8. If SCB is WaitReading, can now complete ALL JIOBs on SCBReadWaitQ. (ALL sectors requested will now be valid) (resume 3. above )

9. If SCB is WaitCommitting restart ALL JIOBs queued on SCBReadWaitQ from the beginning. (need to build SIOBmenu)(resume 4. above)

10. If SCB is WaitReading, can now complete ALL JIOBs on SCBWriteWaitQ. (allow Writes only after ALL Reads have finished) (resume 6. above)

11. If SCB is WaitCommitting/Writing restart ALL JIOBs on SCBWriteWaitQ from the beginning.(resume 6. above)

The Background Process attempts to restart SCBs queued on the SCBWaitQ. If an SCB exists on this queue, AND an SDB has become available--

12. If SCB is WaitReading (Reads ONLY or Read/Writes) recommence disk IO (move SIOBs to diskIOQ). (resume 2. or 7. above)

13. If SCB is WaitWriting (Writes ONLY), can now complete ALL JIOBs on SCBWriteWaitQ. (complete 7. above)

Although the present invention has been described in terms of a host software implementation, it will be appreciated'that in some circumstances, this may not be the optimum solution as it will result in more Host bus traffic. However, the software design described herein is based upon an `event driven` environment which closely resembles hardware and software interrupt mechanisms. Such a design maps very closely to a dedicated second processor which could be a preferable implementation of the present invention.