Apparatus and method for increasing application availability during a disaster fail-back

[0002] The invention relates generally to the field of remote storage replication. More particularly, the invention relates to a method and apparatus for increasing application availability during a disaster fail-back.

[0003] Application downtime following a server failure represents an increasing problem due to the widespread use of computerized applications as well as the ever-expanding electronic commerce-driven economy. To increase data availability and reduce application down time, customers typically build a disaster recovery site which can take over during a disaster or server failure. A significant amount of time and planning goes into insuring that following a disaster, fail-over occurs as rapidly as possible. To this end, many vendors provide methods to reduce this downtime.

[0004] Remote storage replication is a technique which is primarily used for disaster protection. The processes are optimized by vendors to expedite the fail-over process from a primary site to a secondary or disaster recovery site. A problem that is less frequently looked at, however, is the time to return the operations to the primary site once the problems causing the fail-over have been resolved.

[0005] Customer Service Level Agreements (SLAs) define the level of service provided to customers and typically state the amount of downtime acceptable when a disaster strikes. However, customer SLAs rarely provide the requirements for returning operations to the original site once the problem has been resolved. With this in mind, most companies offer solutions that minimize the fail-over process but pay less attention to the requirements of retrieving the state of the primary site (fail-back). In addition, conventional fail-back processes are rarely, if ever, tested under real, live conditions. During such customer testing, the fail-over process is likely to be well-documented. Scripts are generally written and seem to work as efficiently as possible. However, on conclusion of the fail-over process testing, operations continue on the primary site without having had the need to restore the data replicated to the secondary site.

[0006] Conventional fail-back processes are lengthy and often require the entire population of the data from the secondary site back to the primary site. These processes, if ever required, can be time consuming and involve a fair amount of application unavailability or down time. Conventional fail-back processes often require complete resynchronization of the data to get back the data to the primary server and depending on the size of the data, this can be a very long process, adding to the amount of application downtime.

[0007] The present invention provides a method and apparatus for increasing availability of an application during fail-back from a secondary site to a primary site following a failure at the primary site. The method includes copying data from active storage volumes to secondary storage volumes on the secondary site while the application runs on the secondary site and updates the active storage volumes. Once the secondary storage volumes of the secondary site are updated, the data is re-synchronized from the secondary storage volumes on the secondary site to the primary storage volumes of the primary site. The steps of copying the data and resynchronizing the data are repeated for data updated by the application, during the re-synchronization, until a time required to complete the resynchronization step for the updated data is within an acceptable downtime for the application. Once this step is complete, the application is failed-back to the primary site by bringing up the application at the primary site. Application availability is therefore increased by limiting the application downtime to an acceptable down time.

[0008] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:

[0009] FIG. 1 is a simplified view of an exemplary client-server environment in which the present invention may be implemented;

[0010] FIG. 2 depicts an exemplary high-level system architecture implementation of the present invention;

[0011] FIG. 3 is a block diagram of a client computer, a primary server computer and a secondary server computer as shown in FIG. 2; and

[0012] FIG. 4A depicts high level system architecture components in which inventive steps for replicating data may be implemented according to an embodiment of the present invention.

[0013] FIG. 4B depicts high level system architecture components in which inventive steps for generating a point-in-time image of replicated data may be implemented according to an embodiment of the present invention.

[0014] FIG. 4C depicts a flow diagram illustrating data replication and instant image generation of the replicated data according to an embodiment of the present invention.

[0015] FIG. 5A depicts high level system architecture components in which inventive steps for failing-over from a primary site to a secondary site may be implemented according to a further embodiment of the present invention.

[0016] FIG. 5B is a flow diagram illustrating steps for failing-over from a primary server computer to a secondary server computer following a failure of the primary server computer according to a further embodiment of the present invention.

[0017] FIG. 5C is a flow diagram illustrating additional method steps for failing-over from the primary server computer to the secondary server computer as described with reference to FIG. 5B.

[0018] FIG. 6 depicts high level system architecture components illustrating updating of active storage volumes by an application while running on the secondary server computers according to a further embodiment of the present invention.

[0019] FIG. 7A depicts high level system architecture components in which inventive steps for failing-back from the secondary server computer to the primary server computer according to an exemplary embodiment of the present invention.

[0020] FIG. 7B is a flow diagram illustrating inventive steps for failing-back from a secondary server computer to a primary server computer according to an exemplary embodiment of the present invention.

[0021] FIG. 7C is a flow diagram illustrating inventive steps for completing the fail-back process from the secondary server computer to the primary server computer as described in FIG. 7B according to an exemplary embodiment of the present invention.

[0022] The present invention provides a method and apparatus for increasing availability of an application during fail-back from a secondary site to a primary site following a failure at the primary site. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices not needed for an understanding of this invention are shown in block diagram form to avoid obscuring the invention.

[0023] The present invention describes a technique for increasing application availability during a disaster fail-back. According to one embodiment, the technique described by the present invention is utilized during a fail-back from a secondary site to a primary site following a failure at the primary site. The method includes copying data from active storage volumes to secondary storage volumes on the secondary site while the application runs on the secondary site and updates the active storage volumes. Once the secondary storage volumes of the secondary site are updated, the data is resynchronized from the secondary storage volumes of the secondary site to the primary storage volumes of the primary site. During this resynchronization process, the application is active and continues updating the active storage volumes. The steps of copying the data and resynchronizing the data are repeated for data updated by the application, during the re-synchronization, until a time required to complete the resynchronizing step for the updated data is within an acceptable downtime for the application. Once this step is complete, the application is failed-back to the primary site by bringing up the application at the primary site. Application availability is increased as a result of limiting downtime of the application to an acceptable downtime.

[0024] In a preferred embodiment, the steps of the present invention are embodied in machine-executable instructions. The instructions can be used to cause a general-purpose or special-purpose processor that is programmed with the instructions to perform the steps of the present invention. Alternatively, the steps of the present invention might be performed by any combination of programmed computer components and custom hardware components.

[0025] The present invention may be provided as a computer program product which may include a machine-readable medium having stored thereon instructions which may be used to program a computer (or other electronic devices) to perform a process according to the present invention. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, flash memory, or other types of media/machine-readable medium suitable for storing electronic instructions. Moreover, the present invention may also be downloaded as a computer program product, wherein the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem or network connection).

[0026] System Architecture

[0027] FIG. 1 is a simplified view of an exemplary client server environment 100 in which the teachings of the present invention may be implemented. The terms “client” and “server” are used to refer to a computer's general role as a requestor of data (“the client”) or provider of data (“the server”). Clients 102 (102A, . . . , 102N) and servers 124 (124A, . . . , 124N) communicate using a network communications protocol. The client server model is used to communicate information between clients 102 and servers 130. Servers 130 are coupled to a network 120, for example, an intranet or the Internet, and respond to data requests and/or other queries from clients. However, a failure of the server 130 will cause the client 102 to lose all access to data provided by an application while running on the server 130. In accordance with the client-server environment depicted in FIG. 1, an apparatus 110 is provided for increasing the availability of an application during a fail-back from a secondary site 170 to a primary site 130 following a failure at the primary site 130.

[0028] According to a first embodiment of the present invention as depicted in FIG. 2, the apparatus 110 includes one or more client computers 102 (102A, . . . , 102N), a primary server 130 and a secondary server 170. The primary server computer 130 and the secondary computer server 170 are connected by a communications link 120 to the one or more client computers 102. The communications link 120 generally refers to any type of wire or wireless link between computers, such as, but not limited to, a local area network, a wide area network or a combination of networks. The primary server computer 130 and a secondary server computer 170 are also connected via a server communications link 122. In a preferred embodiment of the present invention, the communications link 120 can be a network such as a corporate intranet or by a Wide Area-Network. The server communications link 122 is preferably a wire or wireless link between the primary 130 and secondary 170 server computers. A client computer 102 can be any type of computing device, such as, but not limited to desktop computers, workstations, laptops and/or mainframe computers. One or more users, not shown, can be associated with each client computer 102.

[0029] FIG. 3 illustrates the client computer 102, which includes a CPU 104, a user interface 106, a memory 108 and a communications interface 112. The communications interface 112 is used to communicate with the primary server computer 130, or the secondary server computer 170 during a failure of the primary server computer, as well as other system resources not shown. The communications interface 112 provides a network connection that is preferably a high speed or broad band data connection. The memory of the client computer 102 may be implemented as RAM (random access memory) or a combination of RAM and non-volatile memory such as a magnetic disk storage. The memory preferably contains magnetic disk storage. The memory 108 can contain the following:

[0030] an operating system 114;

[0031] Internet access procedures or network access procedures 116;

[0032] application access procedures 118;

[0033] as well as other procedures and files.

[0034] FIG. 3 also illustrates the primary server computer 130, which includes a central processing unit (“CPU”) 132, a user interface 134, a memory 136 and a communications interface 138. The primary server computer 130 can be any type of computing device, such as, but not limited to, desktop computers, workstations, laptops and/or mainframe computers. The communications interface 138 is used to communicate with the client computers 102, and the secondary server computer 170, as well as other system resources not shown. The memory 136 of the primary server computer 130 may be implemented as RAM (random access memory) or a combination of RAM and non-volatile memory such as one or more magnetic disk storage units, or redundant array of independent disk (RAID) groups or arrays of daisy chained disk-storage units. Memory 118 can contain the following:

[0035] an operating system 140;

[0036] Internet or network access procedures 142;

[0037] server procedures 144;

[0038] application procedures 146 that allow the primary server computer 130 to run the application;

[0039] a primary storage volume 148 including, for example, an application database containing data updated and processed by the application or an application file system 150 updated by the application;

[0040] fail-over procedures 152 for failing over to the secondary computer 170 in the event of a failure of the primary server computer 130;

[0041] fail-back procedures 154 for returning control to the primary server computer 130 following a fail-back to the secondary server computer 170;

[0042] data replication procedures 156 for replicating data generated by the application while running on the primary site to secondary storage volumes at the secondary server computer 170;

[0043] back-end server procedures 158;

[0044] Intranet access procedures 160; and

[0045] other procedures and data structures.

[0046] FIG. 3 also illustrates the secondary server computer 170, which includes a central processing unit (“CPU”) 172, a user interface 174, a memory 176 and a communications interface 178. The secondary server computer 170 can be any type of computing device, such as, but not limited to desktop computers, workstations, laptops and/or mainframe computers. The communications interface 178 is used to communicate with the client computers 102 and the primary server computer 130, as well as other system resources not shown. The memory 176 of the secondary server computer 170 may be implemented as RAM (random access memory) or a combination of RAM and non-volatile memory, such as one or more magnetic disk storage units, RIAD groups, or arrays of daisy chained disk storage units. Memory 176 can contain the following:

[0047] an operating system 180;

[0048] Internet access procedures 182;

[0049] server procedures 184;

[0050] application procedures 186 for running the application on the secondary server computer 170;

[0051] a secondary storage volume 188 including, for example, a secondary application database containing replicated data generated by the application while running on the primary server computer 130 or a secondary application file system 204 containing the replicated data;

[0052] a point-in-time storage volume 190, including, for example, a database or file storage system containing a point-in-time image of data stored on the secondary storage volumes 180 of the secondary server computer 170;

[0053] fail-back procedures 192 for returning control to the primary server computer 130 following a fail-over from the primary server computer 130 to the secondary server computer 170;

[0054] fail-over procedures 194 for bringing up the application on the secondary server 170 following a failure of the primary server computer 130;

[0055] data replication procedures 196 for replicating data to the secondary storage volumes 188 of the secondary server computer 170 generated by the application while running on the primary server computer 130;

[0056] instant image procedures 198 for generating a point-in-time image of data stored on the secondary storage volumes of the secondary server computer 170;

[0057] back-end server procedures 200;

[0058] Intranet access procedures 202; and

[0059] other procedures and data structures.

[0060] The storage volumes (148, 188 and 190), as described above, can include, for example, one or more disk storage units, or grouping of daisy chained disk storage units including RAID groups. Procedural method steps for implementing the application maximization techniques during disaster fail-back as described by the present invention are now provided.

[0061] Operation

[0062] Referring now to FIG. 4C, a method 300 is depicted for replicating data from the primary server computer 130 to the secondary server computer 170, for example, in the apparatus 110 as depicted in FIGS. 2, 3 and 4A. During normal operation, an application (not shown) runs on the primary server computer 130. At step 302, data generated by the application while running on the primary server computer 130 is replicated to the secondary storage volumes 188 of the secondary server computer 170. The data is replicated from the primary storage volumes 148 to the secondary storage volumes 188 using StorEdge network data replication (“SNDR”) software, such that using a fast re-synchronization process, blocks modified on the primary storage volumes 148 are immediately replicated to the secondary storage volumes 188. However, the data may be replicated using other conventional software techniques as known in the art.

[0063] Resynchronization of data from a first storage volume to a second storage volume, as described herein, refers to a process which updates data as needed on the second storage volume (fast-resynchronization). However, if all data on the second storage volume is lost, for example due to a disaster, the resynchronization process transfers all data from the first storage volume to the second storage volume (full-resynchronization).

[0064] Referring now to FIG. 4B, at step 304, a recoverable, point-in-time image (“PIT”) of the data stored on the secondary storage volumes 188 of the secondary server computer 170 is generated. The recoverable PIT copy of the data stored on the secondary storage volumes 188 is periodically generated using StorEdge instant image (“II”) software. The SNDR software and the II software are Sun StorEdge products that run on the Solaris operating system available from Sun Microsystems, Inc. Finally, at step 306, the recoverable point-in-time image is stored in a point-in-time storage volume 190 of the secondary site 170. This process is repeated until an application failure occurs or the application is terminated.

[0065] Referring now to FIG. 5B, a method 310 is depicted for performing a fail-over from the primary server 130 to the secondary server 170 following a failure of the primary server 130, for example, in the apparatus 10 as depicted in FIGS. 5A and 6. If a failure of the primary server 130 is detected, operations of the apparatus 110 and the application fail-over to the secondary server 170 as depicted in FIG. 5A. The fail-over process begins at step 312 where it is determined whether the point-in-time storage volume 190 or the secondary storage volume 180 of the secondary server 170 contains the most current or the most appropriate/consistent copy of the data on the primary storage volume 148 of the primary server 130. At step 314, when the secondary storage volume 188 contains the most current copy of the data on the primary storage volume 148, the point-in-time storage volumes 190 are updated with the most current copy of the data. For example, as depicted in FIG. 5A, the secondary storage volumes contain the most current copy of the data (timeF(+−). Finally, at step 328, the fail-over to the secondary server 170 is completed by bringing up the application on the point-in-time storage volumes 190 (step 330) of the secondary site 170. Once the fail-over process is complete, the application runs on the secondary server 170 and updates the point-in-time storage volumes 190 as depicted in FIG. 6. This process (running the application on the secondary server 170) continues until the primary server 130 failure is corrected at step 332. The secondary storage volumes (Instant Image Master Volumes) 188, are updated using the fast resynchronization process as described above. During normal operation, blocks that have been modified on the secondary storage volume are noted via a scoreboard/bit map mechanism, contrary to the replication process of step 302.

[0066] FIG. 5C depicts additional method steps 318 for performing the updating operations of step 314. At step 320, the secondary storage volumes 188 are placed into a logging mode. At step 322, an application database is brought up on the active storage volumes 190. At step 324, the application database is recovered. Finally, at step 326 archive logs from the secondary storage volumes 188 are applied to the active storage volumes 190. This process synchronizes the application database with the data contained in the primary storage volumes 148 prior to the failure. Although the method 318 for performing the updating operations of step 314 is described with reference to an application database, those skilled in the art will appreciate that additional data storage technologies, including but not limited to databases, file storage systems or other data storage means, are within the contemplation of the present invention.

[0067] Once the problem which caused the initial failure of the primary server 130 is corrected, it is necessary to schedule a time to fail-back operations from the secondary server 170 to the primary server 130. Referring now to FIG. 7B, a method 340 is depicted for failing back from the secondary server 170 to the primary server 130, for example, in the apparatus 110 as depicted in FIG. 7A. At step 342, data is copied from the active storage volumes 190 to the secondary storage volumes 188 using the fast re-synchronization process, while the application continues to run on the secondary server and update the active storage volumes 190. At step 344, data is resynchronized from the secondary storage volumes 188 to the primary storage volumes 148 of the primary server 130. When the primary server's disks associated with the primary storage volumes 148 are still intact from before the disaster, only a fast resync, as described above, is required. Otherwise a full resynchronization from the secondary storage volumes 188 to the primary storage volumes 148 is required. At step 346, steps 342 and 344 are repeated for data updated by the application while running on the secondary server 170 until a time required to complete step 344 for resynchronizing the updated data is within an acceptable downtime for the application (final resync). This final resync is performed after the application is shut down on the secondary server 170. Finally, at step 348, fail-back to the primary server 130 is completed by bringing up the application on the primary storage volumes 148 of the primary server 130 once the resynchronization of the updated data is complete. By limiting application downtime to the acceptable downtime, application unavailability is substantially increased. Acceptable down-time for the application is dependant on the specific customer's SLA (as described above).

[0068] FIG. 7C depicts additional method steps 350 for performing the step 348 of fail-back. At step 352, the application is terminated on the secondary server 170 At step 354, a final resynchronization of data is performed from the secondary storage volumes 188 of the secondary server 170 to the primary storage volumes 148 of the primary server 130. Finally, at step 356, the application is brought online on the primary server 130.

[0069] Several aspects of one implementation of the apparatus for increasing availability of an application during a disaster fail-back have been described. However, various implementations of the apparatus provide numerous features including, complementing, supplementing, and/or replacing the features described above. Features can be implemented as part of the server or as part of the user's computer system in different implementations.

[0070] It is to be understood that even though numerous characteristics and advantages of various embodiments of the present invention have been set forth in the foregoing description, together with details of the structure and function of various embodiment of the invention, this disclosure is illustrative only. Changes may be made in detail, especially matters of structure and management of parts within the principles of the present invention to the full extent indicated by the broad general meaning of the terms in which the appended claims are expressed. For example, the particular element may vary depending on the particular application for the novel mixed storage format while maintaining substantially the same functionality without departing from the scope and spirit of the present invention.

[0071] In addition, although the preferred embodiment described herein is directed to an apparatus and method for increasing availability of an application, it will be appreciated by those skilled in the art that the teaching of the present invention can be applied to other systems. In fact, systems utilizing fail-over and fail-back techniques, regardless of the type of storage volumes (database, file storage, etc.) or the type of apparatus (Internet, corporate Intranet, client-server, etc.) are within the teachings of the present invention, without departing from the scope and spirit of the present invention.

[0072] Advantages of the invention include minimizing the impact of application availability when performing a fail-back. The present invention provides a fail-back technique which can be achieved within the minimum of acceptable downtime of the application. Moreover, the technique taught by the present invention eliminates the potential possibility of data loss at a secondary site which occurs when data is merged from a primary site to a secondary site.

[0073] Having disclosed exemplary embodiments and the best mode, modifications and variations may be made to the disclosed embodiments while remaining within the scope of the invention as defined by the following claims.