Restartable dump that produces a consistent filesystem on tapes

[0002] 1. Field of the Invention

[0003] This invention relates to backup and recovery of data.

[0004] 2. Related Art

[0005] Computer systems typically rely on mass storage systems to store and retrieve data generated or used by the computer system. File servers (“filers”) are one such set of computer systems that offer the ability to store and retrieve relatively large amounts of data, and to make the data highly available to clients and client devices that wish to access that data. Generally, filers can use magnetic, magneto-optical, or optical mass storage, so as to provide relatively rapid access to data. These types of storage are relatively fast and reliable, particularly when used with fault-tolerant techniques, such as a RAID (redundant array of independent disks) configuration.

[0006] Filers, while relatively reliable, are occasionally subject to corruption or loss of data. Known techniques for addressing this problem include maintaining copies of the data in a separate filesystem, one that hopefully will not lose data simultaneously with the original filesystem. These copies can be maintained either (a) in a second filesystem recorded similar to the first, that is, on magnetic mass storage; or (b) on a different type of mass storage medium, such as magnetic tape. The first of these known techniques is sometimes called “mirroring”; the second these known techniques is sometimes called “dump” (or “dump to tape”). The invention described in this application primarily relates to dump operations, but those of ordinary skill in the art will recognize, after perusal of this application, that the principles of the invention can be applied to other techniques for data backup, including mirroring and related techniques.

[0007] One problem in the known art is that the amount of storage recorded at filers has increased dramatically over time. It is presently not uncommon for storage in these devices to be measured in trillions of bytes (Terabytes), and for a dump operation to take many hours, and to use dozens of tapes for storage of an entire filesystem. If there is an error (either due to a problem at the filer, or due to a problem at the tape drive) in the middle of the dump operation, the entire dump operation is aborted and restarted at the beginning. For example, the dump operation sometimes fails before completion due to errors or interruptions, such as errors in writing to tape, power failures, filesystem errors at the filer, user interruption (accidental or otherwise), and the like. If the dump operation is aborted and restarted at the beginning, it is wasteful of the effort already expended during the dump operation that was aborted (and consequently more wasteful the more of the dump operation had been completed by then). Moreover, an error during the dump operation is more and more likely when the amount of information being dumped to tape is larger and larger and the dump operation is longer and longer. Thus, the average time it takes to complete a successful dump (after restarting due to errors, and finally running to completion) increases more than linearly with the size of the filesystem.

[0008] Accordingly, it would be advantageous to provide a method for the backup of data that is not subject to the limitations of the known art. This advantage is achieved in an embodiment of the invention that allows partial results of the dump to be preserved, and the dump to be restarted at a point relatively close to where the failure occurred.

[0009] The invention provides a method and system for performing a dump operation that preserves partial results of an aborted or interrupted dump, and allows restarting the dump from near where it was stopped. Thus, tapes from the original dump, plus tapes from the restarted dump, can be combined to provide a consistent subset of a filesystem. In a preferred embodiment, the dump operation is performed on a consistent recorded snapshot of the filesystem, so that the subset of the filesystem recorded on the tapes is itself consistent. As an emergent consequence, the dump operation is freely interruptable, restartable, and provides a set of tapes that maintain a consistent subset of the filesystem that is transparent to tape-restore operations and other operations to be performed on the filesystem as it was recorded on tape.

[0010] Those of ordinary skill in the art will recognize, after perusal of this application, the many advantages provided by restartable dump. These include, but are not limited to, the following:

[0011] The invention increases the likelihood that a dump is completed in a scheduled time window. For example, if the dump operation is scheduled to begin and end within a selected time window, the ability to continue from relatively near where a failure occurred increases the likelihood that the entire dump will be completed before the time window is over.

[0012] The invention increases the effective throughput of the dump operation. If a dump stops in the middle, less time is wasted when the dump is restarted; the dump is able to restart from relatively near where it was aborted or suspended.

[0013] The invention reduces the penalty of a failed dump. As noted herein, the penalty for a dump that is aborted or suspended is limited, with restartable dump, to only a small portion of the dump that was not “committed” for restartability.

[0014] The invention provides a consistent file system recorded on tape. As noted herein, the set of tapes used prior to the dump being aborted or suspended, plus the set of tapes used when the dump is completed, can be combined to form a consistent filesystem. As an emergent consequence, knowledge of whether the dump was ever suspended and resumed, or simply ran to completion the first time, need not be considered by other elements of the filer.

[0015] In a preferred embodiment, a restartable dump allows users to manage backup operations; users can freely suspend and resume dump operations when they deem appropriate. The following are some examples of additional capabilities available to users with restartable dump:

[0016] Users can suspend and resume dump operations to avoid time windows of relatively heavy traffic for filer requests.

[0017] Users can suspend and resume dump operations to avoid contention for other relatively limited filer resources (such as tape drives).

[0018] Users can suspend and resume dump operations to avoid contention with other, higher priority, dump operations.

[0019] Designers can build other capabilities into the filer that make use of restartable dump, such making those operations that depend on dump operations also be restartable. Examples include dump copy operations and logical replication operations.

[0020] The invention has general applicability to recording of structured information (such as a filesystem) on an alternative medium (such as a tape drive) or in an alternative form (such as BSD dump format), not limited specifically to dump operations in a filer. Moreover, techniques used by a preferred embodiment of the invention for recording of structured information (such as a filesystem) can be used in contexts other than the specific applications disclosed herein.

[0021] FIG. 1 shows a block diagram of a system for performing methods shown herein.

[0022] FIG. 2 shows a data flow diagram for methods shown herein.

[0023] FIG. 3 shows a process flow diagram of methods shown herein.

[0024] In the following description, a preferred embodiment of the invention is described with regard to preferred process steps and data structures. Those skilled in the art would recognize after perusal of this application that embodiments of the invention can be implemented using one or more general purpose processors or special purpose processors or other circuits adapted to particular process steps and data structures described herein, and that implementation of the process steps and data structures described herein would not require undue experimentation or further invention.

[0025] The following terms refer or relate to aspects of the invention as described below. The descriptions of general meanings of these terms are not intended to be limiting, only illustrative.

[0026] dump—in general, copying a set of structured information (such as a subset of a filesystem) to an alternative location in an alternative format (such as a BSD tape dump format).

[0027] restore—in general, copying data, from the alternative media and the alternative format (or a copy thereof), back to the system using structured information (such as a subset of a filesystem), possibly to replace corrupted or lost data.

[0028] As noted above, these descriptions of general meanings of these terms are not intended to be limiting, only illustrative. Other and further applications of the invention, including extensions of these terms and concepts, would be clear to those of ordinary skill in the art after perusing this application. These other and further applications are part of the scope and spirit of the invention, and would be clear to those of ordinary skill in the art, without further invention or undue experimentation.

[0029] FIG. 1 shows a block diagram of a system for performing methods shown herein.

[0030] A system 100 includes a filer 110 (a file server capable of storage and retrieval of data, and capable of responding to filer requests with filer responses), mass storage 120, and secondary storage 140.

[0031] The filer 110 includes a processor, program and data memory, and software for executing instructions for performing the operations of a filer. The processor and program and data memory are known in the art, in several configurations used for filers. As described herein, the software is constructed or selected to perform the operations and steps described herein as related to embodiments of the invention.

[0032] In a preferred embodiment, the mass storage 120 includes a set of hard disk drives in a RAID configuration. In alternate embodiments, the mass storage 120 may include any combination of disk drives or other mass storage devices. Although the filer 110 and the mass storage 120 are shown in the figure as separate devices, there is no particular requirement that they must be physically separate. Rather, in preferred embodiments, the filer 110 and the mass storage 120 are coupled in a single housing.

[0033] The system 100 includes a mass storage link 121, operative to couple the filer 110 with the mass storage 120. The filer 110 uses the mass storage link 121 to control the mass storage 120, directing mass storage 120 to read from and write to selected locations thereon. In a preferred embodiment, the primary storage link 121 can include a direct wire coupling, a high-availability cluster coupling, a SAN coupling, or a more remote coupling such as reference to a secondary server using a communication link. The communication link can include any alternative form of communication device, such as for example access to an Internet, intranet, extranet, virtual private network, wireless network, or some combination thereof.

[0034] The secondary storage 140 includes a tape drive capable of recording data to one or more tapes. The system 100 also includes a secondary link 141, operative to couple the filer 110 with the secondary storage 140. In a preferred embodiment, the secondary link 141 is similar in general nature and operation to the mass storage link 121.

[0035] FIG. 2 shows a data flow diagram for methods shown herein.

[0036] A data flow diagram 200 includes representations of a tree-structured filesystem 210, a linear dump format 220, and a plurality of target tapes 230.

[0037] In a preferred embodiment, the tree-structured filesystem 210 includes at least one volume of a WAFL filesystem, as described further detail in the incorporated disclosures.

[0038] In a preferred embodiment, the linear dump format 220 includes a BSD dump format, such as for example used with respect to UNIX file systems. The BSD dump format is known in the art of dump to tape operations. As shown in the data flow diagram 200, the BSD dump format includes a plurality of map sections (indicated as “Map I” and “Map II” in the figure), a directory section (indicated as “dirs” in the figure), and a files section (indicated as “files” in the figure). In a preferred embodiment, the plurality of map sections and the directory section consume less than one full tape worth of tape storage, so it is generally not necessary to be concerned with the possibility that there will be an error in the dump operation during those sections. Rather, the files section, which represents data from the tree-structured filesystem 210 and its associated files, can be quite lengthy, and can often consume a relatively large number of tapes worth of tape storage.

[0039] In a preferred embodiment, the plurality of target tapes 230 includes a sequence of individual target tapes (not shown), successively mounted on a tape drive such as the secondary storage 140 described with regard to FIG. 1. However, in alternate embodiments, the plurality of target tapes 230 may include a sequence of individual target tapes mounted on a plurality of tape drives, so long as the individual target tapes have a definitive sequence with regard to the dump operation.

[0040] As shown in the data flow diagram 200, a processing direction 201 indicates a direction of processing for the dump operation. The dump operation proceeds for the tree-structured filesystem 210 (or subset thereof), converting that tree-structured filesystem 210 into the linear dump format 220. As the linear dump format 220 is generated, it is copied to individual target tapes within the plurality of target tapes 230.

[0041] As each individual target tape is completed, a checkpoint 202 for the processing direction 201 is set, indicating how far the dump operation has proceeded. As shown herein, the operation of setting the checkpoint 202 is “atomic”, in the sense that the checkpoint 202 is either definitively equal to an earlier value (indicating completion of the most recent tape), or definitively equal to a following value (indicating completion of the current tape). Thus, the checkpoint 202 is never set to a value that indicate anything other than the definitive end of a specific tape in the plurality of target tapes 230.

[0042] FIG. 3 shows a process flow diagram of methods shown herein.

[0043] A method 300 includes a set of flow points and process steps as described herein.

[0044] Although by the nature of textual description, the flow points and process steps are described sequentially, there is no particular requirement that the flow points or process steps must be sequential. Rather, in various embodiments of the invention, the described flow points and process steps can be performed in a parallel or pipelined manner, either by one device performing multitasking or multithreading, or by a plurality of devices operating in a cooperative manner. Parallel and pipelined operations are known in the art of computer science.

[0045] At a flow point 310, the system 100 is ready to perform an individual dump operation.

[0046] In a preferred embodiment, each individual dump operation is performed independently, even if a plurality of dump operations are to be performed substantially simultaneously. In the latter case, if the plurality of dump operations do not interfere with each other with regard to system resources (such as tape drives), they can all proceed substantially simultaneously. However, if the plurality of dump operations do interfere with each other with regard to system resources, at least one must be suspended the so that others can proceed to completion.

[0047] At a step 311, the system 100 receives an instruction, either from an operator or user, or from a timed utility (such as a utility that attempts to perform a dump operation at 3:00 AM each morning), directing the system 100 to perform a dump operation.

[0048] At a step 312, the system 100 locates a dump entry table 351. The dump entry table 351 includes at least one dump entry 352, each one of which maintains information regarding a dump operation that is currently in progress.

[0049] At a step 313, the system 100 allocates a dump entry 352 in the dump entry table 351 for the particular dump operation that has been ordered. The dump entry 352 includes a checkpoint value 353, corresponding to the checkpoint 202 described with regard to FIG. 2.

[0050] At a step 314, the system 100 proceeds with dump operation, writing data to an individual tape in the plurality of target tapes 230.

[0051] At a flow point 320, the system 100 reaches the end of the individual tape in the plurality of target tapes 230.

[0052] At a step 321, the system 100 atomically updates the checkpoint value 353 with the checkpoint 202 described with regard to FIG. 2. As noted herein, the update of the checkpoint value 353 is atomic, in the sense that is not possible for the checkpoint value to be only partially updated at any point in time. Thus, the checkpoint value 353 always indicates the end of an individual tape in the plurality of target tapes 230, for which the dump operation has actually completed.

[0053] If the dump operation proceeds properly to completion, the method 300 proceeds with the flow point 330.

[0054] If the dump operation is interrupted, either by an error or by a deliberate act (such as an instruction to suspend the dump operation), the method 300 proceeds with the flow point 340.

[0055] Dump Operation Successful

[0056] At a flow point 330, the system 100 has successfully completed a dump operation.

[0057] At a step 331, the system 100 locates the dump entry 352 (in the dump entry table 351) associated with the completed dump operation.

[0058] At a step 332, the system 100 frees the dump entry 352.

[0059] At a step 333, the method 300 continues with the flow point 310.

[0060] Dump Operation Unsuccessful

[0061] At a flow point 340, the system 100 identifies itself as having at least one dump entry 352 (in the dump entry table 351) associated with a dump operation that did not complete successfully.

[0062] At a step 341, the system 100 marks the dump entry 352 as “RESTARTABLE”, and reserves that dump entry 352 for later use. The system 100 will ultimately receive either an instruction (from an operator or user) to delete the dump operation, or an instruction to resume the dump operation.

[0063] If the system 100 receives an instruction to delete the dump operation, the system 100 proceeds with the flow point 330, just as if the dump operation had been successful (however, there will be no successful consistent plurality of target tapes 230 resulting from the dump operation).

[0064] If the system 100 receives an instruction to resume the dump operation, the system 100 proceeds with the next step 342.

[0065] At a step 342, similar to the step 331, the system 100 locates the dump entry 352 (in the dump entry table 351) associated with the suspended dump operation.

[0066] At a step 343, the system 100 examines the dump entry 352 and its corresponding checkpoint value 353, to determine which of the plurality of target tapes 230 the dump operation was last able to successfully complete.

[0067] At a step 344, the system 100 proceeds with the dump operation from the point indicated by the checkpoint value 343.

[0068] After this step, the system 100 can either successfully complete the dump operation, or might have the dump operation interrupted or suspended an additional time. If the dump operation is successfully completed, the method 300 proceeds with the flow point 330, as described above. If the dump operation is not successfully completed, the method 300 proceeds with the flow point 340, as described above.

[0069] Atomic Record of Dump Operation

[0070] In a preferred embodiment, the system 100 includes a WAFL filesystem, as described in further detail in the incorporated disclosures. Although a WAFL file system is preferred, those of ordinary skill in the art will realize, after perusal of this application, that any system 100 capable of maintaining a past consistent state of a filesystem would be workable, and is within the scope and spirit of the invention. Such a past consistent state of a filesystem is called a “snapshot” herein.

[0071] The system 100 maintains a snapshot from which the dump operation receives data indicating at least a subset of the consistent filesystem. The system 100 maintains the snapshot, not deleting any of its data or file blocks, until at least one of the following events:

[0072] The dump operation is successfully completed.

[0073] The dump operation is permanently aborted by order of an operator.

[0074] The snapshot is deliberately deleted by order of an operator (thus permanently aborting any dump operations relying on that snapshot.

[0075] If no recent snapshot is available for use by the dump operation when started, the system 100 creates a snapshot for that use. As an emergent consequence, the system 100 can continue to modify an active filesystem in response to requests by users, even while the dump operation is in process, because the snapshot is not itself altered while the active filesystem is in use. Thus, at least some reasons why a dump operation cannot proceed (due to users possibly modifying the active filesystem) do not interfere with the dump operation, and the dump operation can proceed anyway.

[0076] In a preferred embodiment, the dump table 351, and each of its dump entries 352, are maintained atomically, such as by known techniques for atomic storage of data on writable media. Alternatively, the dump table 351, and each of its dump entries 352, may be stored in a later snapshot of the filesystem, so that they can be retrieved by reference to that later snapshot.

[0077] The invention has general applicability to recording of structured information (such as a filesystem) on an alternative medium (such as a tape drive) or in an alternative form (such as BSD dump format), not limited specifically to dump operations in a filer. Moreover, techniques used by a preferred embodiment of the invention for recording of structured information (such as a filesystem) can be used in contexts other than the specific applications disclosed herein.

[0078] Other and further applications of the invention in its most general form would be clear to those skilled in the art after perusal of this application. The invention would be usable for such other and further applications without undue experimentation or further invention.

[0079] Although preferred embodiments are disclosed herein, many variations are possible which remain within the concept, scope and spirit of the invention; these variations would be clear to those skilled in the art after perusal of this application.