Concurrent creation of persistent point-in-time images of multiple independent file systems

[0001] At least one embodiment of the present invention pertains to data storage systems, and more particularly, to a technique for concurrently creating persistent point-in-time images of multiple independent file systems.

[0002] Various forms of network-based storage systems are known today. These forms include network attached storage (NAS), storage area networks (SANs), and others. Network storage systems are commonly used for a variety of purposes, such as providing multiple users with access to shared data, backing up critical data (e.g., by data mirroring), etc.

[0003] A network-based storage system typically includes at least one storage server, which is a processing system configured to store and retrieve data on behalf of one or more client processing systems (“clients”). In the context of NAS, a storage server may be a file server, which is sometimes called a “filer”. A filer operates on behalf of one or more clients to store and manage shared files. The files may be stored in a storage subsystem that includes one or more arrays of mass storage devices, such as magnetic or optical disks or tapes, by using RAID (Redundant Array of Inexpensive Disks). Hence, the mass storage devices in each array may be organized into one or more separate RAID groups.

[0004] In a SAN context, a storage server provides clients with block-level access to stored data, rather than file-level access. Some storage servers are capable of providing clients with both file-level access and block-level access, such as certain Filers made by Network Appliance, Inc. (NetApp®) of Sunnyvale, Calif.

[0005] In conventional file servers, data is stored in logical containers called volumes and aggregates. An “aggregate” is a logical container for a pool of storage, combining one or more physical mass storage devices (e.g., disks) or parts thereof into a single logical storage object, which contains or provides storage for one or more other logical data sets at a higher level of abstraction (e.g., volumes). A “volume” is a set of stored data associated with a collection of mass storage devices, such as disks, which obtains its storage from (i.e., is contained within) an aggregate, and which is managed as an independent administrative unit, such as a complete file system. A “file system” is an independently managed, self-contained, hierarchal set of data units (e.g., files, blocks or LUNs). Although a volume or file system (as those terms are used herein) may store data in the form of files, that is not necessarily the case. That is, a volume or file system may store data in the form of other units, such as blocks or LUNs.

[0006] In older file servers, there was a fixed, one-to-one relationship between a volume and its containing aggregate, i.e., each volume was exactly coextensive with one aggregate. Consequently, there was a fixed relationship between each volume and the disks that are associated with it. This fixed relationship meant that each volume had exclusive control over the disks that are associated with the volume. Only the volume associated with the disk could read and/or write to the disk. Unused space within the disks associated with the volume could not be used by another volume. Thus, even if a volume was only using a fraction of the space on its associated disks, the unused space was reserved for the exclusive use of the volume.

[0007] To overcome these limitations and other limitations of traditional volumes, a technology called flexible volumes was developed by NetApp® and is available in NetApp® Filers as a feature of the Data ONTAP® storage operating system. A flexible volume is analogous to a traditional volume, in that it is managed as a file system; but unlike a traditional volume, a flexible volume is treated separately from the underlying physical storage that contains the associated data. A “flexible volume” is, therefore, a set of stored data associated with one or more mass storage devices, such as disks, which obtains its storage from an aggregate, and which is managed as an independent administrative unit, such as a single file system, but which is flexibly associated with the underlying physical storage.

[0008] Flexible volumes allow the boundaries between aggregates and volumes to be flexible, such that there does not have to be a one-to-one relationship between a flexible volume and an aggregate. An aggregate can contain multiple flexible volumes. Hence, flexible volumes can be very flexibly associated with the underlying physical storage block characteristics. Further, to help reduce the amount of wasted storage space, any free data block in an aggregate can be used by any flexible volume in the aggregate. A flexible volume can be grown or shrunk in size. Furthermore, blocks can be committed to flexible volumes on-the-fly from available storage.

[0009] One feature which is useful to have in a storage server is the ability to create a read-only, persistent, point-in-time image (RPPI) of a data set, such as a volume or a LUN, including its metadata. This capability allows the exact state of the data set to be restored from the RPPI in the event of, for example, a catastrophic failure of the storage system or data corruption. The ability to restore data from an RPPI provides administrators with a simple mechanism to revert the state of their data to a known previous point in time as captured by the RPPI. Typically, creation of an RPPI or restoration from an RPPI can be controlled from a client-side software tool. An example of an implementation of an RPPI is a Snapshot™ generated by SnapDrive™ or SnapManager® for Microsoft® Exchange software, both made by NetApp®. Unlike other RPPI implementations, NetApp Snapshots do not require duplication of data blocks in the active file system, because the active file system can include pointers to data blocks in a Snapshot, for any blocks that have not been modified since the Snapshot was created.

[0010] One problem with the known prior art is that there is no way to create an RPPI of multiple volumes in a single atomic operation. Consequently, if an administrator wanted to acquire an RPPI of all volumes stored by a storage server (or at east more than one volume) at a given point in time, he would have to initiate or program a separate RPPI operation for each volume. That is because each of the volumes is designed as an independent, separately managed set of data. Having to initiate or program a separate RPPI operation for each volume can be time-consuming and burdensome, since there can be many volumes maintained by a given storage server.

[0011] The present invention includes a method in which multiple file systems are maintained in a storage subsystem, and a read-only, persistent, point-in-time image of the file systems is generated in one atomic operation.

[0012] The present invention further includes a system and apparatus to perform such a method.

[0013] Other aspects of the invention will be apparent from the accompanying figures and from the detailed description which follows.

[0022] A method and apparatus for creating an RPPI of multiple independent file systems are described. References in this specification to “an embodiment”, “one embodiment”, or the like, mean that the particular feature, structure or characteristic being described is included in at least one embodiment of the present invention. Occurrences of such phrases in this specification do not necessarily all refer to the same embodiment.

[0023] FIG. 1 illustrates an aggregate 100 of storage, such as may be implemented in a storage system in connection with the technique being introduced here. The aggregate 100 is the underlying physical volume for a plurality of flexible volumes 110. Each flexible volume 110 can include named logical unit numbers (LUNs) 102, directories 104, quota trees (“qtrees”) 106, and files 108. A qtree is a special type of directory that acts as a “soft” partition, i.e., the storage used by the qtrees is not limited by space boundaries.

[0024] The aggregate 100 is layered on top of the RAID system, which is represented by at least one RAID plex 150 (depending upon whether the storage configuration is mirrored), wherein each plex 150 comprises at least one RAID group 160. Each RAID group further comprises a number of mass storage devices (e.g., disks) 130, such as one or more data (D) disks and at least one (P) parity disk.

[0025] In embodiments of the invention, the aggregate 100 is represented (implemented) as a volume within the storage system, and each flexible volume is represented (implemented) as a file (referred to as a “container file”) within that volume. That is, the aggregate 100 may include one or more files, wherein each file contains a flexible volume 110 and wherein the sum of the storage space consumed by flexible volumes associated with the aggregate 100 is physically less than or equal to the size of the overall physical volume.

[0026] The aggregate 100 utilizes a physical volume block number (PVBN) space that defines the storage space of blocks provided by the disks of the physical volume, and each flexible volume embedded within a file utilizes a “logical” or “virtual” volume block number (VVBN) space in order to organize those blocks as files. A PVBN, therefore, is an address of a physical block in the aggregate. A VVBN is an address of a block in a flexible volume (the same block as referenced by the corresponding PVBN), i.e., the offset of the block within the file that represents the flexible volume. Each VVBN space is an independent set of numbers that corresponds to locations within the file, which locations are then translated to disk block numbers (DBNs) on disk. Since a flexible volume 110 is also a logical volume, it has its own block allocation structures (e.g., active, space and summary maps) in its VVBN space.

[0027] Each flexible volume 110 is essentially a separate file system that is “mingled” onto a common set of storage in the aggregate 100 by the associated storage operating system. The RAID system of the associated storage operating system builds a RAID topology structure for the aggregate 100 that guides each file system when performing write allocation. The RAID system also presents a PVBN-to-DBN mapping to the file system.

[0028] Each file in the aggregate is represented in the form of a buffer tree. A buffer tree is a hierarchical metadata structure which used to store metadata about the file, including pointers for use in locating the data blocks for the file. A buffer tree includes one or more levels of indirect blocks (called “L1 blocks”, “L2 blocks”, etc.), each of which contains one or more pointers to lower-level indirect blocks and/or to the direct blocks (called “L0 blocks”) of the file. A “block” of data is a contiguous set of data of a known length starting at a particular offset value. In certain embodiments of the invention, each direct (Level 0) block is 4 Kbyte in length. However, a block could be of different sizes in other embodiments of the invention.

[0029] The root of a buffer tree is the “inode” of the file. An inode, as the term is used herein, is a metadata container which used to store metadata about the file, such as ownership of the file, access permissions for the file, file size, file type, and pointers to the highest level of indirect blocks for the file (see “buffer tree”). The inode is stored in a separate inode file.

[0030] FIG. 2 shows an example of the buffer tree for a file 220 within a container file, i.e., a file within a flexible volume 110. The file 220 is assigned an inode 222, which references Level 1 (L1) indirect blocks 224. In a file within a flexible volume, each indirect block stores at least one PVBN and a corresponding VVBN for each PVBN. To simplify description, only one PVBN-VVBN pair is shown in each indirect block 224 in FIG. 2; however, a real implementation could include many PVBN-VVBN pairs in each indirect block. Each PVBN references a physical block in the aggregate itself and the corresponding VVBN references the associated logical block number in the flexible volume. The inode 222 and indirect blocks 224 in FIG. 2 are shown pointing to only two lower-level blocks each in FIG. 2, to simplify description. It is to be understood, however, that an inode and any indirect block can actually include a greater number of pointers and thus may refer to a greater number of lower-level blocks.

[0031] As mentioned above, each flexible volume is represented in the storage system as a file, referred to as a “container file”, which has a buffer tree that can have a structure similar to that shown in FIG. 2. Like any other file, the container file has an inode, which is indicated as flexible volume type, and which is assigned an inode number equal to a virtual volume id (VVID). The L1 indirect blocks of the container file together may be referred to as a container map.

[0032] The container file is typically one large, sparse virtual disk, which contains all blocks owned by its flexible volume. A VVBN in a container file also represents the offset (i.e., the file block number (FBN)) of the block within the container map. A block referenced by a given VVBN, say VVBN X, in the flexible volume 110 can be found at FBN X in the container file. For example, VVBN 2000 in a flexible volume 110 can be found at FBN 2000 in its container file 200. Since each flexible volume 110 in the aggregate 100 has its own distinct VVBN space, one container file in the aggregate may have an FBN 2000 that is different from FBN 2000 in another container file in the aggregate.

[0033] As shown in FIG. 3, for each volume stored by the storage system, the inodes of each file in that volume are stored in a separate inode file 310. A separate inode file 310 is maintained for each volume in the storage system. Each inode 320 in an inode file 310 is the root of the buffer tree 330 of a corresponding file. The location of the inode file 310 for each volume is stored in a superblock (also called “fsinfo block”) 340 associated with that volume. The superblock 340 is a metadata container that contains metatdata for the volume as a whole rather than for individual files within the volume.

[0034] The aggregate 100 is also represented in the storage system as a volume. Consequently, the aggregate is assigned its own superblock, which contains metadata of the aggregate and points to the inode file for the aggregate. The inode file for the aggregate contains the inodes of all of the flexible volumes within the aggregate, or more precisely, the inodes of all of the container files within the aggregate. Hence, each volume has a structure such as shown in FIG. 3, and the aggregate itself also has a structure such as shown in FIG. 3. As such, the storage system implements a nesting of file systems, where the aggregate is one file system and each volume within the aggregate is also a file system.

[0035] As a result of this storage system structure and functionality, every actual data block (Level 0) within the aggregate is referenced by two separate buffer trees, i.e., the buffer tree of the file which contains the block and the buffer tree of the container file of the volume which contains the block. Consequently, from the superblock of the aggregate, one can locate any Level 0 data block of any file within any volume within the aggregate.

[0036] The above-described structure and functionality are advantageous for purposes of creating RPPIs. In particular, they allow an RPPI to be created of all volumes (file systems) within an aggregate in one atomic operation, by simply creating an RPPI of the aggregate. This is possible because the aggregate (including all of the volumes, files and metadata that it includes) is represented by the file system software as a self-contained administrative unit, i.e., as a volume. Therefore, an RPPI of the aggregate includes an RPPI of every volume within that aggregate.

[0037] This technique is particularly advantageous when used in conjunction with file system software that implements a write-out-of-place approach (sometimes called “write anywhere”) and an RPPI technique which does not require duplication of data blocks to create an RPPI. In a write-out-of-place file system, whenever a data block is modified, it is written to a new physical location on disk. This is in contrast with a write-in-place approach, where a data block, when modified, is written in its modified form back to the same physical location on disk. An example of file system software that implements write-out-of-place is the WAFL® file system software included in the Data ONTAP® storage operating system of NetApp®.

[0038] An example of an RPPI technique which does not require duplication of data blocks to create an RPPI is described in U.S. Pat. No. 5,819,292, which is incorporated herein by reference, and which is assigned to NetApp®. The described technique of creating an RPPI (e.g., a Snapshot™) does not require duplication of data blocks in the active file system, because the active file system can include pointers to data blocks in an RPPI, for any blocks that have not been modified since the RPPI was created. (The term “Snapshot” is used in this document without derogation of Network Appliance, Inc.'s trademark rights.) Among other advantages, this technique allows an RPPI to be created quickly, helps to reduce consumption of storage space due to RPPIs, and reduces the need to repeatedly update data block pointers as required in some prior art RPPI techniques.

[0039] FIGS. 4A through 4C further illustrate this technique. FIG. 4A shows the buffer tree of a very simple volume 41, before an RPPI of the volume is created, where levels of indirection have been removed to facilitate description. To distinguish the volume 41 from an RPPI, the volume 41 is henceforth referred to as the “active” volume or active file system (as opposed to an RPPI, which by definition is not “active”). The active volume 41 is comprised of blocks 412 and 414, which are referenced by the active file system (AFS) inode 410 of the volume 41. The inode AFS 410 is, in turn, stored in the inode file (not shown) of the volume 41, which is locatable through the superblock (fsinfo block) of the volume. The illustrated blocks 412 and 414 represent all blocks in the volume 41, including direct blocks and indirect blocks. Though only two blocks 412 and 414 are shown, each block may point to other blocks.

[0040] FIG. 4B shows the creation of an RPPI of volume 41. An RPPI 42 of the entire volume 41 is created by simply copying the AFS inode 410 of the volume 41 as the RPPI inode 422. The new RPPI inode 422 points to the highest level of indirect blocks referenced by the AFS inode 410 of the volume 41 at the time the RPPI was created. Because the inode 410 is copied, no other blocks need to be duplicated. The copied inode, i.e., RPPI inode 422, is then copied into the inode file of the volume (and identified as an RPPI inode), which dirties a block in the inode file. For an inode file comprised of one or more levels of indirection, each indirect block in the chain is in turn dirtied. This process of dirtying blocks propagates through all the levels of indirection in the inode file. A new superblock (not shown) is also created for the RPPI. The new superblock points to the inode file of the volume.

[0041] FIG. 4C shows the active volume 41 and the RPPI 42 when a change to the active volume 41 subsequently occurs after the RPPI 42 is created. As illustrated, block 414 comprising data “B” is modified after the RPPI 42 was created (in FIG. 4B). Therefore, a new block 424 containing data “B-prime” is allocated for the active volume 41. Thus, the active volume 41 now comprises blocks 412 and 424 but no longer contains block 414 containing data “B”. However, block 414 containing data “B” is not overwritten, because the file system software implements write-out-of-place and therefore does not overwrite blocks on disk. The block 414 is protected against being overwritten by a corresponding bit being set in a block map entry for block 414. Therefore, the RPPI 42 still includes unmodified block 414 as well as block 412.

[0042] It will be recognized that an RPPI of an entire aggregate, implemented as described above, can be created in essentially the same way as an RPPI of a volume.

[0043] With respect to the above-described nesting of file systems, combining this RPPI technique with the write-out-of-place file system approach is advantageous, because when a data block of a file within a volume is modified after an RPPI of the volume has been created, all of the indirect blocks in the file and in the container file of the volume will remain correct in the RPPI. The indirect blocks in the RPPI do not need to be updated when a corresponding block in the active file system is modified.

[0044] FIG. 5 illustrates an example of a network environment in which the above-described techniques can be implemented. The network environment of FIG. 5 includes a storage server 2 coupled locally to a storage subsystem 4 that includes set of mass storage devices, and to a set of clients 1 through an interconnect 3. The above-described techniques for creating an RPPI of an aggregate an its included volumes can be implemented in the storage server 2.

[0045] The storage server 2 receives various read and write requests from the clients 1 and accesses the storage subsystem 4 to service those requests. Each of the clients 1 may be, for example, a conventional personal computer (PC), workstation, or the like. The mass storage devices in storage subsystem 4 may be, for example, conventional magnetic tapes or disks, optical disks such as CD-ROM or DVD based storage, magneto-optical (MO) storage, or any other type of non-volatile storage devices suitable for storing large quantities of data, or a combination thereof. The storage subsystem 4 may be organized into one or more groups of Redundant Array of Independent Disks (RAID).

[0046] The storage server 2 may be, for example, a file server, or “filer”, such as may be used in a NAS environment. Alternatively, the storage server may provide clients with block-level access to stored data, such as in SAN environment. Or, the storage server 2 may be capable of operating in both modes. The storage server 2 can implement one or more of various different protocols, such as common Internet file system (CIFS), network file system (NFS), hypertext transport protocol (HTTP), simple network management protocol (SNMP), transfer control protocol/Internet protocol (TCP/IP), etc., and can selectively use these protocols as needed.

[0047] In a NAS implementation, the interconnect 3 may be essentially any type of computer network, such as a local area network (LAN), a wide area network (WAN), metropolitan area network (MAN) or the Internet, and may implement the Internet Protocol (IP). In a SAN implementation, the interconnect 3 may be, for example, a Fibre Channel switching fabric which implements the Fibre Channel Protocol (FCP).

[0048] FIG. 6 is a high-level block diagram of the storage server 2 of FIG. 2, according to certain embodiments of the invention. Certain standard and well-known components which are not germane to the present invention are not shown. The storage server 2 includes one or more processors 21 coupled to a bus system 23.

[0049] The bus system 23 in FIG. 6 is an abstraction that represents any one or more separate physical buses and/or point-to-point connections, connected by appropriate bridges, adapters and/or controllers. The bus system 23, therefore, may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (sometimes referred to as “Firewire”).

[0050] The processors 21 are the central processing units (CPUs) of the storage server 2 and, thus, control the overall operation of the storage server 2. In certain embodiments, the processors 21 accomplish this by executing software stored in memory 22. A processor 21 may be, or may include, one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or the like, or a combination of such devices.

[0051] The storage server 2 also includes memory 22 coupled to the bus system 43. The memory 22 represents any form of random access memory (RAM), read-only memory (ROM), flash memory, or a combination thereof. Memory 22 stores, among other things, the operating system 25 of the storage server 2, in which the techniques introduced here can be implemented.

[0052] Also connected to the processors 21 through the bus system 23 are a mass storage device 26, a storage adapter 27, and a network adapter 28. Mass storage device 26 may be or include any conventional medium for storing large quantities of data in a non-volatile manner, such as one or more disks. The storage adapter 27 allows the storage server 2 to access the storage subsystem 4 and may be, for example, a Fibre Channel adapter or a SCSI adapter. The network adapter 28 provides the storage server 2 with the ability to communicate with remote devices such as the clients 1 over a network and may be, for example, an Ethernet adapter or a Fibre Channel adapter.

[0053] Memory 22 and mass storage device 26 store software instructions and/or data, which may include instructions and/or data used to implement the techniques introduced here. These instructions and/or data may be implemented as part of the operating system 24 of the storage server 2.

[0054] A shown in FIG. 7, the operating system 24 of the storage server 2 can include several modules, or layers. These layers include a file system layer 31. The file system layer 31 is an application-level programmatic entity which imposes a structure (e.g. hierarchical) on volumes, files, directories and/or other data containers stored and/or managed by a storage server 2, and which services read/write requests from clients of the storage server. An example of a file system layer which has this functionality is the WAFL® file system software that is part of the Data ONTAP® storage operating system from NetApp®.

[0055] Logically under the file system layer 31, the operating system 24 also includes a network layer 32 and an associated network media access layer 33, to allow the storage server 2 to communicate over a network (e.g., with clients 1). The network 32 layer implements various protocols, such as NFS, CIFS, HTTP, SNMP, and TCP/IP. The network media access layer 33 includes one or more drivers which implement one or more protocols to communicate over the interconnect 3, such as Ethernet or Fibre Channel. Also logically under the file system layer 31, the operating system 24 includes a storage access layer 34 and an associated storage driver layer 35, to allow the storage server 2 to communicate with the storage subsystem 4. The storage access layer 34 implements a storage redundancy protocol, such as RAID-4 or RAID-5, while the storage driver layer 35 implements a lower-level storage device access protocol, such as Fibre Channel or SCSI. Reference numeral 37 in FIG. 7 shows the data access path through the operating system 24, associated with servicing read and write requests from clients.

[0056] The operating system 24 may also include an RPPI layer 38, which interfaces with the file system layer 31 and external RPPI client software, to allow creation of RPPIs and restoration of data from RPPIs. The operating system 24 may further include a user interface layer 36, which implements a graphical user interface (GUI) and/or a command line interface (CLI), for example, such as for purposes of administrative access to the storage server 2.

[0057] Thus, a method and apparatus for creating an RPPI of multiple independent file systems have been described.

[0058] Software to implement the technique introduced here may be stored on a machine-readable medium. A “machine-accessible medium”, as the term is used herein, includes any mechanism that provides (i.e., stores and/or transmits) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant (PDA), manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine-accessible medium includes recordable/non-recordable media (e.g., read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; etc.), etc.

[0059] The term “logic”, as used herein, can include, for example, hardwired circuitry, programmable circuitry, software, or any combination thereof.

[0060] Although the present invention has been described with reference to specific exemplary embodiments, it will be recognized that the invention is not limited to the embodiments described, but can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense.