Minimum read rate throughput in a disk cache system

[0002] The invention relates generally to disk storage systems having a disk cache, and in particular, to such a system in which there is a requirement for maintaining a minimum read rate in order to satisfy an application requirement.

[0003] In a typical high speed disk array system, such as that manufactured by EMC Corporation under the name Symmetrix, the disk array system is a cached disk array system which receives, from connected hosts, input/output (I/O) requests through an interconnection channel. In an open-systems environment, a typical channel is a SCSI channel and the I/O requests are composed of read operations and write operations. In the cached disk array, the write operations can all be handled by cache, thus allowing 100% of the write operations to be immediately written to cache and to be destaged later, that is, written from cache to disk. This allows for a very high rate for the write operation. On the other hand, read operations will be serviced either from cache, resulting in a high read response and designated a read “hit,” or from the disk which imposes a slower rate of response since a mechanical system is interposed into the read process. This latter operation is designated a read “miss.”

[0004] When the write rate is high, the method uses more and more of the disk cache to hold the data to be written (the pending write data) before it is destaged to disk. Accordingly, since cache memory for write operations is expended at the expense of the cache area used for read operations, less and less cache memory becomes available for the read operations as the pending write data builds up. The write operations also consume processing cycles of the control unit which also has to handle the read requests.

[0005] Typically, this may not be a problem. However in instances where the system is required to maintain a minimum rate of read operations, for example when the stored information on the disk represents digitized audio or digitized video that is being heard or seen in real time, a reduction in the read rate may interrupt the required continuous stream of audio or video information. Accordingly, such processes require at least a predetermined minimum read rate to insure uninterrupted decoding of the stored audio and/or video data.

[0006] The invention relates to a method and apparatus for maintaining a predetermined minimum data read rate from a disk drive system having a read/write cache memory. The method of the invention features the steps of sending, from a host computer to the drive system, a write cache utility factor inquiry; receiving, at the host system, the write cache utility factor information as provided by the disk drive system; determining whether a write rate to the data read/write cache should be changed based upon the utility factor information, and modifying the write rate, if necessary, in response to the determining step.

[0007] According to another aspect of the invention, the write cache utility factor information includes at least two of the following: a value representative of the total available cache memory in the disk drive system; a value representative of the maximum write data cache memory permitted to enable the minimum read rate to be maintained; and a value representative of the current write cache memory being used for pending write data.

[0008] The method further features determining the write cache utility factor information at a plurality of time instances, determining from this cache utility factor information the rate at which write data destages in the disk drive system and adjusting the write rate in response to the destage rate and the cache utility factor information. In one aspect, the write rate is maintained approximately equal to the destage rate.

[0009] In another aspect of the invention, the method features modifying the write rate in response to the unused write cache availability as determined from the cache utility factor information.

[0010] In a preferred aspect of the invention, the sending of the write cache utility factor features sending a SCSI Inquiry command to the disk drive system from the host computer and the receiving step features receiving a SCSI response to the Inquiry command that contains the write cache utility factor information.

[0011] The apparatus of the invention features a host system having a host system interface for requesting write cache utility factor information from the disk drive system and the disk drive system has a memory system interface for receiving the host factor request, and a controller connected to the memory system interface for processing the request and for generating a response to the request. The response includes the write cache utility factor information. The host further has a write rate control element for controlling the write rate in response to the drive system controller response.

[0012] This system provides advantageous operation where a minimum read rate must be maintained, and where at the same time, a mechanism is provided to service write operations consistent with the minimum read rate requirement. This system can automatically adjust to different cache sizes and operating parameters, and additional hardware units need not be added where, for example, the SCSI Inquiry command is used to acquire certain information about the cache usage, and the disk drive system has been configured to respond to that command.

[0013] Other objects, features, and advantages of the invention will be apparent from the following description taken together with the drawings in which:

[0014] FIG. 1 is a block diagram of a system in accordance with the invention;

[0015] FIG. 2 is a typical inquiry page response showing the elements of the cache utilization in accordance with a preferred embodiment of the invention;

[0016] FIG. 3 is a flow chart of the operation of the system, in accordance with one embodiment of the invention; and

[0017] FIG. 4 is a flow chart showing operation of the system in accordance with a second embodiment of the invention.

[0018] Referring to FIG. 1, a computer system 10 includes a host computer 12, a disk drive system 13 including a controller system 14 and a plurality of disk drives 16. The host computer 12 and the controller 14 communicate over a bus 18. The protocol used by the host computer 12 and controller 14 is preferably a SCSI protocol which is well defined in the field. The controller 14 operates and controls the disk drive units 16 over any of a number of buses 22 as is well known in the art.

[0019] The disk drive units are typically, in a modern system, large drives capable of storing, for example, four gigabytes each. With such large capacities, these drives will often contain audio and/or video digital information such as that used in voice messaging systems, music playback systems, and video on demand systems. Each of these systems, however, requires a continuous stream of information, at a minimum data rate, in order to decode and provide continuous, uninterrupted audio and/or video information to a user. Accordingly, the disk drive units 16 must be capable of being read at a rate sufficient to provide that continuous data for decoding.

[0020] In an operation dedicated only to reading the disks, there is more than ample capacity in a modern system to provide information at a data rate required by such continuous need systems. Typical data rates can be, for example, 4 megabytes per second. A typical disk drive system can provide information at up to, for example, 6 megabytes per second.

[0021] In a high capacity, high throughput system, however, the disk drive controller typically employs a controller element 24 and a large cache memory 30 which serves both read and write functions. Cache memories are well known in the field and enable portions of the disk drive data to be read and stored in it as a block so that read requests need not be made for each byte of data needed from the disk drives. Similarly, when the host computer writes data to the disk drive, the cache memory can provide a “hit” each time a write request is made so that the write data is stored in the cache memory until a time is available for the controller 14 to write the cached data, as a group, to the disk drive elements. Accordingly, the disk controller cache can contain a significant quantity of pending (to be destaged) write data. When this occurs, however, it occurs at the expense of cache space which would otherwise be used in connection with the disk drive read cache operation. Accordingly, if the disk drive read operation is to continue at a minimum specified rate the disk drive write cache must be controlled so that it does not exceed a certain maximum size (corresponding to a minimum read cache requirement) in order to maintain the read rate.

[0022] In accordance with the invention, the host computer application is responsible for controlling the rate of writes destined for the disk drive system in order to maintain the write cache level at or below some fixed value. While several methods can be employed according to the invention to effect this operation, two preferred embodiments are described hereinafter.

[0023] Before describing the detailed operation of the system, some description of the mechanism for communicating between host and controller is appropriate. Since many high capacity, high volume fast controllers currently exist, it is desirable to provide a host communications mechanism which does not require modification of at least the host system hardware. In this respect, therefore, for a SCSI interconnection, the host system application uses the SCSI Inquiry command to obtain the needed information and provide it to the application. Typically, the SCSI Inquiry command is used to determine the configuration of the SCSI device, that is, device type, vendor identification, model number, and other useful information. The SCSI Inquiry command will return to the host up to 256 bytes of information, and thirty-six of those bytes are defined by the SCSI standard (see FIG. 2) and compliance to the standard is mandatory. Forty other bytes are reserved for future standards definitions.

[0024] The rest of the bytes returned to the host are defined by the standard as being “Vendor Unique.” Thus a manufacturer can use them to convey additional information to a requesting host. It is a portion of these bytes which are used to return information (write cache utility factor information) to the host, describing the volume of pending write data waiting to be destaged from the cache 30 of controller 14. In a preferred embodiment of the invention, twelve bytes are allocated to the write cache utility factor information that is returned in response to the SCSI Inquiry command to determine whether to enable the host to “throttle” the write content or write rate to the disk. Specifically, the information which is returned includes the total number of cache slots available at controller 14 (each cache slot, in the illustrated embodiment, is 32K bytes in size), a write threshold cache slot count corresponding to the maximum number of write slots which can be allocated in accordance with a desired minimum read rate; and the current level of cache slots occupied by write data that is pending destaging from the cache to the disk drive(s). Referring to FIG. 2, this information is illustrated as being provided in bytes 114-125, with four bytes being allocated to each “piece” of information. Accordingly, a maximum number of bytes of cache memory in this configuration is 2⁴⁷bytes.

[0025] As illustrated in FIG. 2, substantial additional space is available for vendor specific information.

[0026] With this information available to the host, and referring now to FIG. 3, in a first embodiment of the invention the host computer first sends the SCSI Inquiry command, at 40, to the controller 14. The controller, at 42, returns a 256 byte response (illustrated in FIG. 2) which provides the data sought at bytes 114-125. The host computer then determines, at 44, in accordance with this embodiment, how many additional writes can be made by subtracting the number of writes pending from the threshold number. In this embodiment, therefore, it is not necessary to know the entire cache size.

[0027] Once that number of writes is made, no additional writes will be permitted. (Alternatively, the write rate at the host can be modified as necessary.) However, the host computer, before reaching that limit, at 46, sends another Inquiry command to the controller 14, determines again the amount of write space which is available, and accordingly can adjust a write rate so that the maximum allocated write space is not exceeded. In this manner, the host computer determines, for the time between its Inquiry commands, a write rate that prevents writing beyond the maximum size of cache memory allocated to pending writes.

[0028] Thus, expressed in a quantitative way, the host computer inquires at times T₁, T₂, T₃. . . what the cache utilization factor is. It receives in response, at each time, in this illustrated embodiment, the cache size S, the volume of cache occupied by the pending write data, P, and the maximum available write size for pending writes within the cache, W. If the write rate from the host to the disk drive is R bytes per second, then, for the time interval from T_ito T_i+1:

[0029] R<(W−P)/(T_i+1−T_i)

[0030] to ensure that the write space does not overflow. Alternatively, the total number of bytes written in the interval from T_ito T_i+1is limited to less than W−P.

[0031] In other embodiments of the invention, the write rate may be reduced from its standard value, for an interval, only if the available write cache is less than a predetermined threshold value, that is, if the available write cache is insufficient to receive all the write data which might be sent at the standard rate in the interval.

[0032] Referring now to FIG. 4, in another application, a host computer 12 will request the write cache utility factor information, and hence sample the information periodically, by sending Inquiry commands at preferably regular time intervals to the drive system. This is indicated at 50. While the time intervals are preferably of the same size, they need not be. From the responses, received at 52, the host application can determine, at 54, the rate at which write data are being destaged in the disk array (taking into account the knowledge of how much more write data are added to the pending write data in each interval of time). With the knowledge of how much write space in the cache will become available during a time interval, and further with the knowledge of how much additional space, if any, there is in the cache memory allocated to pending write operations, the rate of writes from a write application can be set, at 56, to no more than that rate which will not exceed the amount of cache memory allocated to the write data. In this manner, a higher write rate can be achieved while maintaining the minimum of required read rate.

[0033] In this example, therefore, since the destage rate is to be taken into account, one first must calculate the destage rate at 54, based on past experience. A conservative approach would be to use the lowest destage rate D_Lfor the past ten time intervals. The expected destage rate, D, for an interval is calculated as: D=(P_b+TW-P_e)Δt. This takes into account the number of pending writes, P_b, as of the beginning of the interval, the total number of writes, TW, during the interval, and the number of pending writes, P_eat the end of the interval. With this information for each past interval, the write rate, R, for the present interval (defined as the interval beginning at time T_iand ending at time T_i+1is required to be less than that which would overfill the space (W−P) available at time T_iin the write pending portion of the cache memory, and is calculated as follows:

[0034] R<(W−P)/(T_i−1−T_i)+D_L

[0035] where D_Lis the minimum value of D over the last ten time intervals. Other approaches to determining an acceptable destaging rate will be apparent to those practiced in the field and can include either a longer of shorter view of the destaging process or an averaging process. In other embodiments, the write rate can simply be limited to be less than the destage rate, D.

[0036] Other methods of operation will be apparent to those familiar with the field. In particular, other mechanisms can be employed for transmitting the write cache status from the disk controller 14 to the host computer 12, and other information, can be added to or subtracted from the write cache utilization factor information described in these preferred embodiments of the invention. For example, the total cache size need not be transmitted so long as the maximum write cache size and the write data pending to be destaged are made known to the host computer. In other embodiments, the total cache size and the number of pending writes may be the only information provided to the host computer, the host computer being provided with the capability to determine (or estimate) a maximum write cache size from that information and the information known to it regarding the minimum read rate. Indeed, the minimum read rate may vary from application to application, information about which is used to change the maximum write cache size and enable the system to dynamically adapt to varying application programs running at the host computer in order to maximize use of, and throughput for, the system. Further the disk drive system can perform calculations so that the utilization factor sent to the host is the write rate itself.

[0037] In fact, in a particular embodiment of the invention, the disk drive system itself, using the utilization factor information can effect control over the write operations to ensure a minimum data read rate required by a particular host application. Thus, the controller can, for example, effectively reject or otherwise prevent a write request from being accepted and carried out, if the information available to it regarding the utilization of the cache memory for storing pending write data indicates that the processing of the write request will prevent the disk drive system from maintaining the minimum read rate required by a specific application. In this instance, the disk drive system, and in particular the disk drive controller, will perform the steps otherwise performed at the host computer and will act to throttle write requests coming to it from the host computers, where necessary.

[0038] Additions, deletions, and other modifications of the invention will be apparent to those practiced in the field and are within the scope of the following claims.