Method and Apparatus for Limiting Data Occupancy in a Cache

The present invention relates to multilevel storage hierarchies, more particularly to the control thereof with respect to data residence in upper levels of the hierarchies.

Peripheral storage hierarchies have been used for years for providing an apparent store as suggested by Eden, et al in U.S. Pat. No. 3,569,938. Eden, et al teach that, in a demand paging or request system, caching data in a cache-type high-speed front store (buffer) can make a peripheral storage system appear to have a large capacity yet provide rapid access to data, rapid access being faster than that provided to the normal backing store. Eden, et al also teach that the backing store can be a retentive store, such as magnetic tape recorders and magnetic disk recorders, while the front store can be a volatile store such as a magnetic core store. With the advances in data storage technology, the front store typically includes semiconductive-type data storage elements. U.S. Pat. No. 3,839,704 shows another form of the storage hierarchy.

An important aspect of storage hierarchies is enabling data integrity. That is, the data received from a user, such as a central processing unit (CPU) or other data handling device, should be returned to the supplying unit either correct or with an indication that errors may exist. Also, it is typical practice in storage hierarchies to move data from a higher level to a lower level for such retentive storage, as well as to limit the data in the higher levels such that other data can be stored for rapid access. U.S. Pat. No. 4,020,466 shows copying changes from a high-level store to a backing store, while U.S. Pat. No. 4,077,059 shows forcing copy-back under predetermined conditions. Such copy-back operations can consume storage hierarchy performance time, i.e., so much data may be copied back that access to the data by a using unit may be degraded. This problem is partially solved by U.S. Pat. No. 3,588,839, which teaches that the only data that need be copied back from a high-level storage unit to a low-level storage unit is that data that is altered, i.e., where there is noncongruence between data in a backing store and data in a front store.

Storage hierarchies have taken diverse forms. For example, in accordance with the Eden, et al U.S. Pat. No. 3,569,938, a single high-speed store serviced several users. U.S. Pat. No. 3,735,360 shows that each processor can have its own high-speed store, or cache, for different performance reasons. Performance of the storage hierarchies also is affected by the algorithms and other controls used to place predetermined data into the cache, or high-speed storage portion. Accordingly, U.S. Pat. No. 3,898,624 shows that varying the time of fetching data from a backing store to a front, or caching, store can be selected by the computer operator in accordance with the programs being executed in a using CPU. In this manner, it is hoped that the data resident in the cache or upper level of the hierarchy will be that data needed by the CPU, while excess data not needed is not resident; this arrangement allows more useful data to be stored in the higher level storage portion. All of these operations become quite intricate. Accordingly, evaluation programs for storage hierarchies have been used to evaluate how best to manage a storage hierarchy. U.S. Pat. Nos. 3,964,028 and 4,068,304 show performance monitoring of storage hierarchies for achieving these goals. Even at that, much remains to be done in various types of storage hierarchies for enhancing optimum performance while ensuring data integrity. Much of the work with respect to storage hierarchies has occurred in the cache and main memory combinations connected to a using CPU. The principles and teachings from a cached main memory relate direction to caching and buffering peripheral system, as originally suggested by Eden et al, supra. Of course, main memory has been used prior to Eden et al for buffering or caching data from a magnetic tape and disk unit for a CPU, i.e., a main memory was not only used as a CPU working store but also as a buffer for peripheral devices.

The performance monitoring referred to above has indicated that it is not always in the best interest of total data processing performance and integrity to use a caching buffer interposed between a using unit and a backing store. For example, U.S. Pat. No. 4,075,686 teaches that a cache can be turned on and off by special instructions for bypassing the cache. Further, the backing store or memory was segmented into various devices with some of the devices, or segments, being bypassed, such as for serial or sequential input-output operations. This patent further teaches that, for certain commands, it is more desirable to not use cache than to use cache. U.S. Pat. No. 4,268,907 teaches that, for a command specifying the fetching of data words, an indicator flag is set to a predetermined state. Such indicator flag causes data transfer circuits to respond to subsequent predetermined commands to bypass cache storage. This bypass prevents replacement (flushing) of data instructions stored in cache during the execution of long serial data transfers. Further, U.S. Pat. No. 4,189,770 shows bypassing cache for operands, but using cache for storing instructions.

In some storage hierarchies, data integrity is ensured by transferring data to both the cache and a backing store whenever data is written to the memory. As an example see U.S. Pat. No. 4,045,781. While in many instances this may prove valuable, it does tend to have high data occupancy in the cache which may have adverse effects on total performance in certain applications. U.S. Pat. No. 4,173,781 describes a table control system for ensuring coherence between various levels in a storage hierarchy. Several storage hierarchies have involved updating a backing store after updating a cache, such as in U.S. Pat. No. 4,167,782, or controlling the updating of the cache when the memory is updated, such as shown in U.S. Pat. No. 4,157,586. Other systems always write to cache once data is resident in cache, or have space allocated in cache as taught by U.S. Pat. No. 4,084,234. For data integrity, changes to data in a cache are usually copied back to a lower level, or backing store, such as shown in U.S. Pat. No. 4,020,466. In a multilevel storage hierarchy including a direct access storage device (DASD), also termed a disk data storage device, having an electronic random-access memory cache, it is desired to always have data written to the DASD, which is a retentive memory, as soon as possible. Such writing ensures data integrity by storing data in a retentive memory as opposed to storing data only in a volatile electronic random-access memory.

It is an object of the present invention to enhance operation of a data storage hierarchy, including a DASD and a volatile caching buffer, by writing to the DASD whenever space is not allocated for data in the caching buffer, and to limit data stored in the caching buffer for encouraging writing to the DASD. Reading data is encouraged to be from the caching buffer.

In accordance with the present invention, a storage hierarchy is operated by monitoring a series of requests for access to the storage hierarchy received from a user and indicating when certain types of requests are received. In response to the indicated requests, predetermined data is removed from the buffer store such that subsequent predetermined storage access requests will relate to data stored only in the backing store. In a preferred form of the invention, the predetermined operations are: writing data to the storage hierarchy, particularly those writing operations that may relate solely to the backing store to the exclusion of the caching buffer store; or writing data when sequential data is occurring; or updating data in the caching buffer plus writing data to the backing store; or updating data in the caching buffer while sequential data is being processed; or updating plural records in a given addressable area. The data to be removed is selected in accordance with the criteria used to institute the data removal. In a peripheral system, the removal of data preferably occurs upon completion of the series of requests, which are often referred to as a chained set of commands.

A corollary aspect of the invention is to promote data to the caching buffer in response to requests for data to be read from the storage hierarchy. Accordingly, in a certain aspect of the invention, writing data to the storage hierarchy includes a preference to writing directly to the backing store to the exclusion of the caching buffer store, while reading data from the storage hierarchy gives a preference to reading data stored in the caching buffer store.

The invention also contemplates apparatus consonant with the methods set forth above.

The foregoing and other objects, features, and advantages of the invention will be apparent from the following more particular description of the preferred embodiments of the invention, as illustrated in the accompanying drawings.

FIG. 1 includes a block diagram and a machine operations chart of a storage hierarchy employing the present invention.

FIG. 2 is a block diagram of a peripheral storage system type of storage hierarchy employing the present invention as set forth in FIG. 1.

FIG. 3 is a machine operations chart detailing the operation of the FIG. 2 illustrated embodiment.

Referring now more particularly to the drawing, like numerals indicate like parts and structural features in the various diagrams. Using units 10, such as central processing units and the like, are coupled via input/output connections 11 to a storage hierarchy having a storage director 12, and preferably in the form of a peripheral data storage system. The storage hierarchy includes a backing store 14 and a cache, or caching buffer store, 15. Data is transferred between the backing store 14 and cache 15 via a set of device connections 13. Selectively, using units 10 can access cache 15 or backing store 14 in accordance with switch 17, 18 settings. For example, switch 17 set as shown couples output bus 20 of input/output connections 11 to device input connections 21 for transferring data from using units 10 directly to backing store 14. Backing store 14 supplies data directly to using units 10 via device output bus 22 of device connection 13; thence, through switch 18 and input bus 23 of input/output connections 11 to using units 10. Actuating switches 17, 18 to the alternate position connects output bus 20 to cache input bus 24, while cache output bus 25 is connected by switch 18 to using units input bus 23. Buses 24, 25 also provide for data transfer between cache 15 and backing store 14. In a preferred construction of the invention, backing store 14 includes a plurality of direct access storage devices (DASD), while cache 15 is an electronic random-access memory, preferably of the semiconductive type.

In accordance with the invention, the illustrated storage hierarchy is operated by read-hit write-miss control 26 to encourage writing directly to backing store 14 and reading data from cache 15. The operation is such that, when cache directory 58 is examined upon any storage access request received from using units 10 on a write type of operation, with no space allocated in cache 15, data is written directly to backing store 14. When DASDs are used, it is desired to write directly to the retentive DASDs as opposed to writing to the volatile cache 15. Such operation saves rewriting data from cache 15 to backing store 14. Data integrity requires storing data in a retentive store as early as possible. When data is to be supplied to using unit 10 during a read operation, it is desired to have as short an access time as possible. For reads, data integrity is ensured by retaining a copy of the data in backing store 14. Accordingly, for read operations read-hit and write-miss control 26 moves data from backing store 14 to cache 15 for ensuring that read data requests are satisfied from data stored in cache 15.

In accordance with one aspect of the invention, analyzing writing data operations and then removing data from cache 15 in accordance with such analyzation tends to encourage write misses of cache 15, hence encourages writing directly to backing store 14. This operation is important when backing store 14 consists of a plurality of disk-type data storage apparatus (DASD). Accordingly, in a series of operations such as represented as beginning at reference numeral 29 and continuing at step 30 as chained peripheral operations, all write operations of a given type are tallied at step 31. Upon completion of the chained peripheral operations, the storage hierarchy at branch 32 examines the tallied writes. If there are no writes that are of a given type, as later described, then no action is taken to limit data in cache 15. After certain write operations wherein the write operations is tallied at step 31, storage director 12 at step 34 limits the cache contents by removing addressability of predetermined data stored in cache 15. Such predetermined data is usually related to the writes tallied at step 31. To encourage read hits in cache 15, data is moved to the cache from backing store 14 at step 36.

The presently-described cache management control tends to reduce the probability of a cache write hit, usually without reducing the probability of a cache read hit. Data promotion moves to the cache that data which is likely to be referenced again soon by read accesses and where those later references are not expected to consist solely of writes. In this way, the later references can be resolved in the cache rather than by the backing store. Modeling has shown that eliminating the normal asynchronous data promotions which result from write-through to backing store processing usually does not dramatically affect the read hit ratio (and may improve it), but does reduce the write hit ratio. The elimination of updating cache when writing to the backing store also reduces the internal peripheral data storage system workload, since the impact on system performance by such data movements from the backing store 14 to cache 15 is lessened for a given workload of user accesses to the storage hierarchy. The storage hierarchy performance on writes is not diminished by increasing the probability that a given write command will be processed as a write miss rather than as a write hit. Because channel busy contributes to DASD access rotational misses for a write miss, a write miss does have a higher probability of a rotational miss than would a write hit. This fact is offset by the fact that the elimination of the dual transfer associated with write hits (first a transfer to cache followed by a transfer from cache 15 to DASD 14) reduces the control unit's contribution to a rotational miss. The net result is usually an overall reduction in rotational misses as the number of cache write hits goes down and the number of cache write misses goes up. This means an increase in performance.

The present invention also enhances data integrity over writing to cache 15, a volatile store. Writing directly to the retentive backing store results in data modifications controlled directly by the using unit. Any writing or recording errors are immediately detected. When writing to cache 15, then writing later to backing store 14, a recording error occurs such that the using unit is unaware that damage may have been done. If the using unit were made aware of the damage in some fashion, there presently exists to easy automated way for recovery from such asynchronous errors. Therefore, for data integrity, final recording status should not be given until the retentive backing store 14 has successfully recorded the data. If a pure write-through to backing store 14 were used, cache buffer 15 and backing store 14 would have divergent data, which is also avoided by controls of the present invention.

FIG. 2 illustrates a preferred embodiment of the invention as employed in a two storage director 12 arrangement. Each storage director 12 includes a plurality of so-called channel adaptors 50, also separately denominated as CAA through CAH, which connect the respective storage directors 12 to a plurality of using units 10 via a plurality of input/output connections 11. Each storage director 12 includes a programmed processor 51 which, as usual, includes a processing unit (not shown) having a control store (not shown) which contains computer programs for performing the storage director functions. The programmed processor includes programs constituting address and command evaluator ACE 52 which receives and evaluates peripheral commands. Such functions are also performed in present day storage directors for noncached DASDs as widely sold throughout the world. The programmed processor 51 also includes direct access control DAC 53, which responds to commands evaluated and decoded by ACE 52 to control data transfers between using units 10 and the DASD 14 devices addressed, as well as to provide device commands to DASD 14 for performing well known DASD access and control functions. DAC 53 also creates write tally operations at step 31 which is located in its control store (not shown). Programmed processor 51 further includes CAC 54, which is a cache access control for accessing cache 5. CD bits 60, one bit for each of the DASDs 14, are accessed by DAC 53 and CAC 54 respectively for determining whether to access cache or DASD and for setting the bits to D at a cache miss. Connections from storage director 12 to DASD 14 are via DASD circuits 55, which are constructed using known device adaptor and data flow design techniques. Cache 15 is accessed via memory circuits 56, which include those circuits for generating addresses and access requests. Cache 15 is a portion of a large random-access store 57, hereinafter referred to as a system store. A directory 58 for cache 15 is stored in store 57. The directory 58 is accessed using known hashing techniques based upon the address structure of DASDs 14. Each unit of data, track or record stored in cache 15 has an entry in directory 58, as is known in the data processing art.

Access to DASDs 14 is via a so-called string arrangement in which a plurality of DASDs 14 are connected to the storage directors 12 via controllers 65, separately denominated as DCA. Each storage director 12 connects to the controllers 65 via the daisy-chain device connection 13. A radial connection of known design may also be employed. The operation of the FIG. 2 illustrated system in accordance with the invention is best understood by referring to FIG. 3, a machine operations chart.

Referring next to FIG. 3, the chained peripheral operations 30 are performed in accordance with present-day procedures for peripheral systems that attach to computers manufactured by International Business Machines Corporation, such as the so-called series 370 computers, 4300 series and the like. During such chained peripheral operations, each storage director 12 tallies in its control store 71, a selection of write operations that occur during the chained peripheral operation. Such writing operations that are tallied include a so-called format write. Format writes include known peripheral commands associated with the above-referred-to computers, such as write home address, write record zero, write count key and data, and so forth. Updating data by writing the key and the data is written to cache or to DASD 14 in accordance with the CD bits.

A second type of write operation relates to writing in cache 15. That is, each occurrence of data updating in cache 15 during a command chain is stored such that, upon completion of the chained operations, the data which was updated in the cache during a command chain which included a format write may be demoted from cache 15 to DASD 14 and effectively erased from cache 15 by destroying addressabilities of the cache 15 data area. That is, the directory 58 entry relating to the data stored in the cache 15 is erased. Further, in any given chained operation, if sequential data is indicated by a using unit 10 supplying a so-called sequential bit (not shown), then all of the data that is updated during the chained operation is moved to DASD 14 and effectively erased from cache 15. Further, if any data area, which may include a plurality of data records, has more than one record updated, then such data residing in that cache data area is demoted to DASD 14 upon completion of the chained operation. Combinations of the above-described types of write operations as well as additional types of write operations can be used for limiting data in the cache for encouraging write misses, i.e., enabling writing directly to DASD 14 without accessing cache 15 for ensuring data integrity by writing directly to the retentive memory to the exclusion of a volatile memory. The term "data area" means either a storage area capable of storing a record track of data from DASD 14; a fixed number of data bytes, such as a 4-kilobyte virtual page; or some other addressable unit of storage as may be convenient.

Upon completion of the chained peripheral operation 30, the status bits stored at 71 are examined by storage director 12 as a branch control for executing branches 32. Branches 32 are usual branch instructions within programmed processor 51. A first branch instruction 72 examines whether sequential bit SEQ was received from a using unit 10 for chained peripheral operation 30 and a write (WR) command was received. If the commanded write is to the DASD 14 via DAC 53, then logic step 74 deletes the data contents of cache 15 corresponding to the DASD 14 altered data by erasing the directory 58 entries. That is, data was written directly to DASD 14 and cache 5 received no writing, therefore no demotion of data is required. In a similar manner, at branch 73, if a format write (FMT) was executed and cache 15 was not updated, i.e., all writing was to DASD 14, then logic step 74 is performed. If neither of the conditions in branch steps 72-73 is satisfied, then at branch 80, if a format write command was received and a write operation was made to cache 15 via CAC 54, logic path 83 is followed to step 84 for moving the altered data from cache 15 to DASD 14 and then deleting from cache the data just stored in DASD 14. In a similar manner, at 81 when a write to cache 15 occurs when sequential data is indicated, such updated tracks are removed from cache 15. Also, at branch 82 when the number of records updated in a track stored in cache is greater than one, those tracks are removed from cache 15. If none of the tallied write conditions examined in steps 72 through 82 are satisfied, then at 85 programmed processor 51 examines the sequential bit in control store 71 to determine if sequential data was being processed and none of the tallied write conditions are met. If that is the case, then at step 86, the data contents of the track to be next accessed are moved from DASD 14 to cache 15 such that the next expected request for data will have its requested data stored in cache. For nonsequential data at branch 85 or upon completion or promoting the next track to cache 15 at step 86, programmed processor 51 via logic path 87 goes to operation 36 for other read-type data promotions to cache 15 from DASD 14. That is, different data transfer characteristics may require different data to be maintained within cache 15.

In accordance with another aspect of the invention, data promotion from backing store 14 to cache 15 with respect to a given chain of commands is inhibited or reduced anytime a write operation requires access to backing store 14, i.e., cache 15 does not have storage space allocated for satisfying the write request. In other words, it is expected that references to data written to storage hierarchy will not be repeated in the immediate future; only when data is being read from the storage hierarchy is it expected to have a soon-to-be-repeated reference. When a write operation finds no data space in cache 15, such write operation is completed directly with backing store 14. Eliminating the data promotion from a write miss should not dramatically affect read hit ratios, but it does reduce write hit ratios. Further, reducing the data movement related to data promotion reduces the workload of the storage hierarchy and thereby enhances access to data stored in the storage hierarchy. Accordingly, in operation step 36, at branch 90 a branch operation is entered from logic path 87 examines tally 71 for a tallied write operation. If there was no tallied write operation in the immediately-preceding chain of commands, storage director 12 at branch 91 examines cache 15 with respect to the last track referenced (LTR); if at the completion of the chain of commands cache 15 has a complete replication of the last track referenced, an exit path 96 to logic path 75 is followed. Otherwise,, at step 94 the data contents of LTR in DASD 14 are promoted to cache 15. For a tallied write operation indicated at branch 90, a further examination for promoting data occurs at branch 92 to determine whether or not the data contents of LTR are stored in cache 15 and are an accurate replication of the contents of LTR in DASD 14 (i.e., was cache 15 or DASD 14 written to at LTR?). If both criteria are met, an exit path 97 to logic path 75 is followed; otherwise at 94 the contents of LTR in DASD 14 are promoted to cache 15. Following steps 94, other promotions of data from backing store 14 to cache 15 for purposes beyond the present description may be made at step 95. In an alternate embodiment, the exits 96 and 97 from steps 91, and 92 may instead be directed to step 95. Then logic path 75 is followed after other read promotions from step 95 are completed for completing the operation. Programmed processor 51 also follows logic path 75 to perform other operations, including subsequent chained peripheral operations which repeat the abovedescribed process, after the deletion of the tracks from the cache 15 from step 74.

Instead of waiting until a completion of the chained peripheral operations 30, the data in cache 15 can be invalidated during execution of chained operations. That is, the data to be moved out of cache by step 84 can be performed concurrently to the chained operations, particularly for those conditions recited with respect to steps 80, 81 and 82. In a similar manner step 74 can be performed concurrently to chained peripheral operations 30. Updating the DASD 14 data images and the cache 15 data images as described avoids unnecessarily invalidating data images in the cache 15 for preserving a high hit ratio for read operation; i.e., data not expected to be read by using units 10 is removed from cache 15 to allow more data expected to be read to reside in cache 15. Encouraging write misses reduces the workload of the storage hierarchy in that the double transfers associated with writing to cache 15 and then writing to DASD 14 from cache 15 are avoided. While a write miss tends to reduce performance, such a factor is minor since the number of reads in a storage hierarchy typically exceeds the number of writes. The advantage achieved by immediate data integrity is readily apparent.

Access to DASD 14 surface record areas is based upon the relationship of the reading transducer (not shown) to the particular rotational position of the disk surface. Access to such disk surface is sensitive to the relative position of the transducer to the record area being accessed when the access request is received from using unit 10. Waiting for coincidence of the record area to the transducer area for the request is called latency delay. Such latency delay delays storage access over and above access to cache 15. This latency delay is less significant than the double transfer referred to above. Once a writing operation is instituted within a cylinder (a set of storage tracks at the same radius of the storage record surfaces in a multiple record DASD), it can be followed by a series of write operations which are synchronized to DASD rotational periodicity, thereby minimizing the latency delay effect.

While the invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.