Circular and bi-directional mirroring of flexible volumes

[0001] (Not applicable)

[0002] (Not applicable)

[0003] The present invention relates to mirroring of storage devices (such as disks) in a network storage system and, more particularly, to circular or bi-directional mirroring of flexible volumes in a network storage system.

[0004] Computer workstations and application servers (collectively hereinafter referred to as “clients”) frequently access data that is stored remotely from the clients. In these cases, computer networks are used to connect the clients to storage devices (such as disks) that store the data. For example, Information Systems (IS) departments frequently maintain “disk farms,” tape backup facilities and optical and other storage devices (sometimes referred to as media) in one or more central locations and provide access to these storage devices via computer networks. This centralized storage (commonly referred to as “network storage”) enables data stored on the storage devices to be shared by many clients scattered throughout an organization. Centralized network storage also enables the IS departments to store the data on highly reliable (sometimes redundant) equipment, so the data remains available, even in case of a catastrophic failure of one or more of the storage devices. Centralized data storage also facilitates making frequent backup copies of the data and providing access to backed-up data, when necessary.

[0005] Specialized computers (variously referred to as file servers, storage servers, storage appliances, etc., collectively hereinafter referred to as “filers”) located in the central locations make the data on the storage devices available to the clients. Software in the filers and other software in the clients cooperate to make the central storage devices appear to users and application programs as though the storage devices were locally connected to the clients.

[0006] In addition, the filers can perform services that are not visible to the clients. For example, a filer can aggregate storage space in a set of storage devices and present all the space in the set of storage devices as a single “volume.” Clients treat the volume as though it were a single, large disk. The clients issue input/output (I/O) commands to read or write data from or to the volume, and the filer accepts these I/O commands. The filer then issues I/O commands to the appropriate storage device(s) of the set to fetch or store the data. The filer then returns status information (and, in the case of a read command, data) to the client. Each block of the volume maps to a particular block of one of the storage devices and vice versa. Thus, a volume represents the storage capacity of a whole number of storage devices.

[0007] Some filers implement logical volumes (sometimes called “flexible” volumes). A flexible volume does not necessarily represent the storage capacity of a whole number of storage devices. Instead, the flexible volume generally represents a portion of the total storage capacity of the set of storage devices on which the flexible volume is implemented. A filer implements the flexible volume as a container file that is stored on the set of storage devices. When a client issues I/O commands to read or write data from or to the flexible volume, the filer accepts these I/O commands. The filer then issues I/O commands to the container file to fetch or store the data. The filer then returns status information (and, in the case of a read command, data) to the client.

[0008] Storage space on the set of storage devices is not necessarily pre-allocated for the container file. Blocks on the storage devices can be allocated to the container file as needed. Furthermore, as additional space is required on the flexible volume, the container file can be extended. Thus, unlike traditional volumes, each block of a flexible volume maps to a block of the container file, but a mapping between the blocks of the flexible volume and blocks on the storage devices does not necessarily occur until these blocks of the container file are written to the storage devices. Several container files can be stored on one set of storage devices. Thus, several distinct flexible volumes can be implemented on the same set, or an overlapping set, of storage devices.

[0009] “Volume mirroring” is another service provided by some filers. A volume mirror (sometimes referred to as a “destination volume”) is an exact copy of another volume (sometimes referred to as a “source volume”). Typically, the source volume is connected to one filer (a “source filer”) and the destination volume is connected to a different filer (a “destination filer”), and the two filers are connected to each other via a network. When a client writes data to the source volume, the source filer causes a copy of the data to be written to the destination volume. Because the source and destination volumes are mirror copies of each other, volume mirrors can be used to protect against a catastrophic failure of the source or destination volume or of either filer.

[0010] For performance reasons, when file write requests are received by a filer (regardless of whether volume mirroring is involved), the filer caches the write requests and occasionally flushes the cache by writing the cache's contents to the appropriate storage device(s). The cache contains data blocks that are to be written to the storage device(s). To enable the filer to continue accepting write requests from clients while the cache is being flushed, the filer divides the cache into two halves and alternates between the halves. That is, while one half of the cache is being flushed, the other half of the cache is used to store write requests, and vice versa.

[0011] The filer also stores information about write or modify operations that are to be performed in a battery backed-up non-volatile random access memory (“NVRAM”). This memory of write/modify operations and corresponding data is arranged in an ordered log of operations called an “NVLOG.” Thus, if the filer experiences a catastrophic or power failure, upon recovery, the information in the NVLOG can be used to update the appropriate storage device(s), as though the cache had been flushed.

[0012] Volume mirroring is implemented by sending two streams of information from the source filer to the destination filer. The two streams correspond to the cache and to the NVRAM, respectively, on the source filer. The first stream is a stream of modified blocks that are to be written to the destination volume. The second stream is a log of write or modify operations that are to be performed on the destination volume. The first information stream contains much more data than the second information stream. Thus, the first information stream typically lags behind the second information stream.

[0013] The source filer directs the second stream of information to one of two log files on a storage device connected to the destination filer. The source filer alternates between the two log files, as the source filer alternates between the two halves of its cache. As with the NVRAM on the source filer, in case of a catastrophic or power failure involving the destination filer, the current log file can be used to update the destination volume, as though all the modified blocks had been written to the destination volume.

[0014] During a cache flush on the source filer, the source filer waits until the destination filer reports completion of the write operations involved in the two streams before the source filer considers its cache flush to be complete. This ensures that the destination volume is synchronized with the source volume.

[0015] However, if a volume is mirrored to a flexible volume, flushing the cache on a source filer can pose a problem. As noted, flexible volumes are implemented as container files. Therefore, write requests to flexible volumes are treated by the destination filer as ordinary file write operations to the container file, which are cached by the destination filer and reported to the source filer as being complete, before the data is actually written to the corresponding storage device(s). Because the destination filer treats both information streams from the source filer as ordinary file write requests, both streams are cached by the destination filer. The destination filer reports that the write operations involving the two information streams are complete as soon as the write operations are cached by the destination filer, i.e., before this information is actually written to the storage device(s).

[0016] Earlier, when the source filer began to flush its cache, the source filer switched to the other half of its cache and began writing to the other (second) log file on the destination filer. The next time the source filer flushes its cache, the source filer switches back to the first half of its cache and begins overwriting the first log file. The source filer assumes it is safe to reuse the first log file, because the source filer was informed that the write operations involving the two information streams were completed during the earlier cache flush. Thus, the source filer assumes the destination volume is synchronized with the source volume.

[0017] However, if the destination filer's cache is not flushed after the source filer's cache is flushed, the destination volume becomes unsynchronized with the source volume. That is, modified data blocks remain in the destination filer's cache, without having been written to the destination volume.

[0018] Furthermore, if the source filer flushes its cache a second time, the first log file begins to be overwritten. Thus, if the destination filer experiences a catastrophic or power failure, the log files do not contain enough information to update the destination volume.

[0019] To avoid this problem, a cache flush on the source filer is not deemed to be complete until the destination filer flushes its cache. However, this creates a dependency, i.e., the source filer's cache flush completion is dependent upon the destination filer's cache flush completion.

[0020] If the destination filer handles a source volume that is mirrored on the source filer, (i.e., each filer mirrors a volume that is sourced from the other filer (a situation known as “bi-directional mirroring”), completion of each filer's cache flush depends upon completion of the other filer's cache flush. This, of course, creates a deadlock, because, to flush its cache, each filer must wait until the other filer flushes its cache. Similarly, if filer A mirrors a volume on filer B, and filer B mirrors another volume on filer C, and filer C mirrors yet another volume on filer A, this “circular mirroring” creates a deadlock.

[0021] Disclosed is a method and system for mirroring flexible volumes in a circular or bi-directional configuration, without creating a deadlock. A source filer maintains a number of log files on the destination filer, where the number of log files is greater than or equal to the number of cache flush operations the source filer performs between cache flush operations on the destination filer. The source filer can create the log files in advance. Alternatively or in addition, the source filer can create the log files as needed, i.e., the source filer can create a log file when the source filer flushes its cache. Once the destination filer flushes its cache, the log files created prior to the destination filer cache flush can be deallocated or reused.

[0022] These and other features, advantages, aspects and embodiments of the present invention will become more apparent to those skilled in the art from the Detailed Description Of The Invention that follows.

[0027] Disclosed is a method and system for mirroring flexible volumes in a circular or bi-directional configuration, without creating a deadlock. A source filer maintains a number of log files on the destination filer, where the number of log files is greater than or equal to the number of cache flush operations the source filer performs between cache flush operations on the destination filer. The source filer can create the log files in advance. Alternatively or in addition, the source filer can create the log files as needed, i.e., the source filer can create a log file when the source filer flushes its cache. Once the destination filer flushes its cache, the log files created prior to the destination filer cache flush can be deallocated or reused.

[0028] As noted, workstations and application servers (collectively “clients”) are frequently connected to filers and/or storage devices via local or wide area networks (LANs or WANs) and/or storage networks. FIG. 1 is a block diagram of an exemplary network storage system in which the presently disclosed mirroring system and method can be used. Clients 100 are connected to one or more storage devices 102 and/or 104 via a packet-switched (such as in Internet Protocol (IP)) network 106 and a source filer 108. Storage device 102 is directly connected to the source filer 108, whereas storage device 104 is connected to the source filer 108 via a storage network 110. Although only one storage device is shown at each of 102 and 104, other numbers of storage devices can be used.

[0029] A destination filer 112 has one more directly connected storage devices 114, and/or as one more other storage devices 116 connected to the destination filer 112 via a storage network 118. Other numbers of clients 100, filers 108 and 112 and storage devices 102, 104, 114 and 116, can, of course, be used.

[0030] The filers 108 and 112 can communicate with each other, such as via the network 106 or via another network (not shown). Fiber Channel or other technology can be used to implement the storage networks 110 and 118, as is well known in the art. Internet SCSI (iSCSI) or other network technology can be used to communicate between the clients 100 and the filer 108, as is well known in the art.

[0031] For purposes of presenting an exemplary implementation that can support circular or bi-directional mirroring that involves a flexible volume, assume that the destination filer 112 implements a flexible volume on the one or more storage devices 114 and 116. Suitable software to support flexible volumes is available from Network Appliance, Inc. of Sunnyvale, Calif. under the trade name FlexVol.

[0032] In addition, assume that the source filer 108 implements a mirrored volume. In particular, the source filer 108 implements the mirrored volume (i.e., the source volume) on the one or more storage devices 102 and/or 104. Assume further that the source filer 108 communicates with the destination filer 112, over the network 106 and the destination filer 112 maintains a synchronous mirror copy of the source volume (i.e., the destination volume) on the flexible volume. A suitable system for maintaining a synchronous mirror copy of the source volume is described in commonly assigned U.S. patent application Ser. No. 10/692,495, filed Oct. 24, 2003, titled “Synchronous Mirroring Including Writing Image Updates To A File,” the contents of which are hereby incorporated by reference herein.

[0033] An exemplary arrangement involving mirroring on a flexible volume is depicted schematically in FIG. 2. As noted, the source filer 108 maintains a source volume 200, and the destination filer 112 maintains a destination volume. Because the destination volume is implemented in a flexible volume, the destination volume is implemented as a container file 202 on one more of the storage devices 114 and/or 116.

[0034] The source filer 108 includes a cache 204. Write or modify requests 206 (collectively hereinafter referred to as “write requests”) from a client 100 are stored in the cache 204 until the source filer 108 flushes the cache 204 by writing data blocks to the source volume 200, as indicated at 208. Until the cache 204 is flushed, the source filer 108 stores information about write or modify operations that are to be performed on the source volume 200. The filer 108 stores this information in one of two non-volatile memories 210 and 212 (referred to as NVLog0 and NVLog1). The filer 108 alternates between the two memories 210 and 212 each time the filer flushes its cache 204.

[0035] In addition to writing this information into one of the two memories 210 and 212, the source filer 108 writes (as indicated by dashed line 218) the information into a log file on a storage device maintained by the destination filer 112. Several exemplary log files 220, 222 and 224 are shown in FIG. 2. Each time the source filer 108 flushes its cache 204, the source filer 108 begins writing this information to a different one of the log files 220-224.

[0036] When the source filer 108 flushes its cache 204, in addition to writing the data blocks 208 to the source volume 200, the source filer 108 writes (as indicated by dashed line 226) the contents of the cache 204 to the destination volume (which is implemented as a container file 202). Because the destination volume is implemented by the destination filer as a container file 202, write requests to the container file 202 are stored in a cache 228, until the destination filer 212 flushes the cache 228.

[0037] Once the destination filer 112 flushes its cache 228, the log files that were created prior to flushing the cache can be deallocated or reused. Thus, the log files 220-224 can form a ring buffer of log files, in which the oldest unused log file is reused next, as indicated by arrow 229. The log files 220-224 can, however, be reused in any order. The number of required log files 220-224 depends on the number of times the cache 204 on the source filer 108 is flushed, compared to the number of times the cache 228 on the destination filer 112 is flushed.

[0038] The log files 220-224 can be pre-allocated. The number of pre-allocated log files 220-224 can be chosen based on the expected ratio of cache flushes on the source file a 108 to cache flushes on the destination filer 112. Alternatively, an arbitrary number, such as four, of log files 220-224 can be created. In either case, additional log files (not shown) can be created as needed. For example, if the source filer 108 has flushed its cache 204 as many times as there are log files 220-224, and the source filer 108 needs to flushed its cache 204 again, an additional log file can be created.

[0039] Unused log files 220-224 can be kept for reuse or deallocated. In one embodiment, the disclosed system maintains at least a predetermined number, such as four, of unused log files 220-224. If the source filer 108 flushes its cache 204 frequently enough to use the available log files 220-224 and leaves fewer than the predetermined number of log files available, the system allocates additional log files. On the other hand, if the source filer 108 flushes its cache 204 less frequently, and more than a predetermined number, such as eight, of unused log files remains unused for a predetermined amount of time, such as five minutes, the additional unused log files (beyond four unused log files) are deallocated. Thus, the system can dynamically adjust the number of log files 220-224 to match the relative rate of cache flushing by the source filer 108 and the destination filer 112.

[0040] Maintaining a number of log files 220-224 on the destination filer 112, where the number of log files 220-224 is greater than or equal to the number of cache flush operations the primary filer 108 performs between cache flush operations performed by the destination filer 112, avoids a dependency of the source filer 108 on the destination filer 112. In particular, cache flushes on the source filer 108 are not dependent upon completion of cache flushes on the destination filer 112. Thus, the destination filer 112 can maintain a source volume, such as source volume 230, and the source filer 108 can maintained a mirror copy (i.e., a destination volume 232) of the source volume 230 in a flexible volume, without incurring a deadlock. Similarly, a circular chain of filers (not shown) can maintained a set of source volumes and mirrored destination volumes on flexible volumes, without incurring a deadlock.

[0041] Operation of the disclosed method and system is also shown with reference to a flowchart shown in FIGS. 3A and 3B. At 300, a source filer receives a write request. The source filer caches the write request (such as in the cache 204 in FIG. 2) at 302. As noted, the source filer also maintains two non-volatile memories (NVLogs 210 and 212) (FIG. 2). At 304, the source filer writes information about the write request into one of the non-volatile memories. At 306, the source filer also sends the information about the write request to the destination filer for storage there in a log file. Control then returns to 300 to receive another write request.

[0042] Periodically or occasionally, the cache on the source filer is flushed. At 308, the source filer flushes its cache. Flushing the cache includes writing the contents of the cache to the source volume and sending the cache contents to the destination filer for writing to the destination volume, as discussed above. When the source filer flushes its cache, at 310, the source filer changes to the other NVLog (non-volatile memory). Thus, information about subsequently received write requests will be stored (at 304) in the other NVLog. If two NVLogs are used, as described above, the source filer alternates between the two NVLogs each time the source filer's cache is flushed.

[0043] Also each time the source filer's cache is flushed, at 312 the source filer changes to another log file on the destination filer. However, as discussed above, there can be more than two log files on the destination filer. If necessary, a new log file can be created, as discussed above. Alternatively or in addition, if one or more old log files are available, one of these old log files can be reused.

[0044] At 314, a destination filer receives a write request. The write request can contain information about a write request on the source filer, i.e., a log file entry. (Such requests are generated at 318.) Alternatively, the write request can contain cache information from the source filer that was sent as a result of the source filer flushing its cache at 308. Of course, the write request need not originate at the source filer, i.e., a client might have initiated the write request independent of the operation being described here.

[0045] At 316, the destination filer caches the write request, such as in the cache 228 in FIG. 2). At 318, the destination filer stores information about the write request in one of the destination filer's NVLogs. Control then returns to 314 to await the next write request to the destination filer.

[0046] Periodically or occasionally, the destination filer flushes its cache. The destination filer flushes its cache at 320 and, at 322, the destination filer changes to using another of its NVLogs. After flushing the cache on the destination filer, log file(s) that were created prior to flushing the cache are marked “available.” The available log files can be reused at 312. Alternatively, some or all of the log files that were created prior to flushing the cache are deleted. However, log files that were created after the cache is flushed are not marked “available,” deleted or overwritten until after the cache is subsequently flushed.

[0047] The functions describe herein can be performed by processors executing instructions stored in memories in the source and destination filers 108 and 112.

[0048] The filers 108 and 112 include host bus adapters or other network or fabric interfaces. In some embodiments, these interfaces convert (translate) messages (packets, etc.) received over the network 106 or over the Internet (not shown) according to a variety of protocols into messages according to a common message format for processing by an operating system or other control software. The interfaces also convert messages from the operating system according to the common message format into messages according to the appropriate protocol for transmission over the network 106 or the Internet. An exemplary operating system is Data ONTAP, available from Network Appliance, Inc. of Sunnyvale, Calif. However, any suitable operating system that is capable of receiving and translating data from multiple protocols is acceptable.

[0049] The interfaces and the storage devices 102 and 114 (if present) are controlled by CPUs executing the operating system, control software or other instructions stored in a memory to perform at least some of the functions described herein.

[0050] The clients 100 may be general-purpose computers configured to execute applications under the control of a variety of operating systems, including the UNIX® and Microsoft® Windows® operating systems. These computers may utilize file-based or block-based access protocols when accessing information in a storage system over a Network Attached Storage (NAS-based) or Storage Area Network (SAN-based) network, respectively. Therefore, each computer may request the services of the storage system by issuing access protocol messages (in the form of packets) to the storage system over the network. For example, a computer running the Windows operating system may communicate with the storage system using the Common Internet File System (CIFS) protocol over TCP/IP. On the other hand, a computer running the UNIX operating system may communicate with the storage system using the Network File System (NFS) protocol over TCP/IP. It will be apparent to those skilled in the art that other computers running other types of operating systems may also communicate with the storage system, also known as a multi-protocol filer, using other file access protocols. As noted, messages received by the storage system according to various protocols from various clients are translated for execution by an operating system of the storage system.

[0051] A system and method for mirroring flexible volumes has been described as comprising computer instructions that can be stored in a memory and executed by a processor. Those skilled in the art should readily appreciate that instructions or programs defining the functions of the present invention can be delivered to a processor in many forms, including, but not limited to, information permanently stored on non-writable storage media (e.g. read only memory devices within a computer such as ROM or CD-ROM disks readable by a computer I/O attachment), information alterably stored on writable storage media (e.g. floppy disks and hard drives) or information conveyed to a computer through communication media. In addition, while the invention may be embodied in software, the functions necessary to implement the invention may alternatively be embodied in part or in whole using software, firmware and/or hardware components, such as combinatorial logic, application-specific integrated circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other hardware or some combination of hardware, software and/or firmware components.

[0052] While the invention is described through the above-described exemplary embodiments, it will be understood by those of ordinary skill in the art that modifications to, and variations of, the illustrated embodiments may be made without departing from the inventive concepts disclosed herein. Moreover, while the preferred embodiments are described in connection with various illustrative data structures, one skilled in the art will recognize that the system may be embodied using a variety of data structures. Accordingly, the invention should not be viewed as limited, except by the scope and spirit of the appended claims.