System and methods for dynamically adjusting between asynchronous and synchronous data replication policies in a networked virtualization environment

[0001] This application is a continuation-in-part of U.S. application Ser. No. 14/027,576, filed on Sep. 16, 2013, entitled “SYSTEM AND METHODS FOR DYNAMICALLY ADJUSTING BETWEEN ASYNCHRONOUS AND SYNCHRONOUS DATA REPLICATION POLICIES IN A NETWORKED VIRTUALIZATION ENVIRONMENT” which is hereby incorporated by reference in its entirety.

[0002] The present application is related to U.S. Pat. No. 8,601,473, Ser. No. 13/207,345 issued on Dec. 3, 2013, entitled “ARCHITECTURE FOR MANAGING I/O AND STORAGE FOR A VIRTUALIZATION ENVIRONMENT”, and U.S. application Ser. No. 14/019,139, filed on Sep. 5, 2013, entitled “SYSTEM AND METHODS FOR PERFORMING DISTRIBUTED DATA REPLICATION IN A NETWORKED VIRTUALIZATION ENVIRONMENT”, which are all hereby incorporated by reference in their entirety.

[0003] This disclosure concerns a mechanism for performing data replication in a networked virtualization environment, and in particular to a mechanism for dynamically adjusting between asynchronous and synchronous data replication policies in a networked virtualization environment.

[0004] Data replication involves replicating data located at a source (e.g., source virtualization environment) to a destination (e.g., destination virtualization environment). This may be performed for the purpose of disaster recovery, where data replicated from the source to the destination may be later recovered at the destination when the source undergoes failure.

[0005] Two modes of data replication currently exist: asynchronous data replication and synchronous data replication. Asynchronous data replication occurs where a write operation for a piece of data at a source is committed as soon as the source acknowledges the completion of the write operation. Replication of the data at the destination may occur at a later time after the write operation at the source has been committed. Synchronous data replication occurs where a write operation for a piece of data at a source is committed only after the destination has replicated the data and acknowledged completion of the write operation. Thus, in a synchronous data replication mode, a committed write operation for data at the source is guaranteed to have a copy at the destination.

[0006] Asynchronous data replication is advantageous in certain situations because it may be performed with more efficiency due to the fact that a write operation for data at the source can be committed without having to wait for the destination to replicate the data and acknowledge completion of the write operation. However, asynchronous data replication may result in potential data loss where the source fails prior to the replication of data at the destination.

[0007] Synchronous data replication guarantees that data loss will not occur when the source fails because the write operation for data is not committed until the destination has verified that it too has a copy of the data. However, having to wait for data to be written at both the source and the destination before committing a write operation may lead to latency as well as strain on system resources (e.g., CPU usage, memory usage, network traffic, etc.).

[0008] Conventionally, data replication involves setting a fixed data replication policy (either synchronous or asynchronous). By setting a fixed data replication policy, the manner in which data replication occurs remains static regardless of the changing nature of the system (e.g. source networked virtualization environment or destination networked virtualization environment). System parameters such as the amount of data being replicated or the amount of resources being consumed by the source or destination may vary over time. Thus, fixing the data replication policy for a system fails to account for the dynamic fluctuation in system parameters and may lead to inefficiencies where the system parameters change substantially or frequently over the course of system operation.

[0009] Setting a fixed data replication policy may be efficient where the source and destination operate at a steady resource consumption rate and the amount of data to be replicated remains steady. However, where the rate of resource consumption or amount of data to be replicated exhibits volatility, the fixed data replication policy may lead to the underutilization of resources when additional resources are available or where the amount of data to be replicated significantly decreases. Similarly, inefficiency may occur where the fixed data replication policy overutilizes resource availability when fewer resources are available or where the amount of data to be replicated significantly increases.

[0010] Embodiments of the present invention provide a method for transitioning a data replication policy from an asynchronous data replication policy to a synchronous data replication policy for data replication between a source vDisk at a source networked virtualization environment and a destination vDisk at a destination networked virtualization environment, that includes taking a first snapshot of the source vDisk at a first point in time where the source vDisk has a same state as the destination vDisk prior to the data replication policy transitioning from the asynchronous data replication policy to the synchronous data replication policy; taking a second snapshot of the source vDisk at a second point in time when a replication policy for data replication between the source vDisk and the destination vDisk transitions from the asynchronous data replication policy to the synchronous data replication policy; providing metadata from the source networked virtualization environment to the destination networked virtualization environment for generating a shell destination vDisk; synchronously replicating write operations performed on the source vDisk after the second point in time at the shell destination vDisk; and lazily transmitting differences between the first snapshot and the second snapshot to the destination networked virtualization environment for populating the shell destination vDisk.

[0011] Further details of aspects, objects, and advantages of the invention are described below in the detailed description, drawings, and claims. Both the foregoing general description and the following detailed description are exemplary and explanatory, and are not intended to be limiting as to the scope of the invention.

[0027] Various embodiments are described hereinafter with reference to the figures. It should be noted that the figures are not necessarily drawn to scale. It should also be noted that the figures are only intended to facilitate the description of the embodiments, and are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an illustrated embodiment need not have all the aspects or advantages shown. An aspect or advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiments even if not so illustrated. Also, reference throughout this specification to “some embodiments” or “other embodiments” means that a particular feature, structure, material, or characteristic described in connection with the embodiments is included in at least one embodiment. Thus, the appearances of the phrase “in some embodiments” or “in other embodiments”, in various places throughout this specification are not necessarily referring to the same embodiment or embodiments.

[0028] FIG. 1 illustrates a networked virtualization environment for storage management according to some embodiments of the invention. The networked virtualization environment of FIG. 1 can be implemented for a distributed platform that contains multiple nodes (e.g., servers) 100a and 100b that manages multiple-tiers of storage. The multiple tiers of storage include storage that is accessible through a network 140, such as cloud storage 126 or networked storage 128 (e.g., a SAN or “storage area network”). Unlike the prior art, the present embodiment also permits local storage 122/124 that is within or directly attached to the node and/or appliance to be managed as part of the storage pool 160. Examples of such storage include Solid State Drives (henceforth “SSDs”) 125 or Hard Disk Drives (henceforth “HDDs” or “spindle drives”) 127. These collected storage devices, both local and networked, form a storage pool 160. Virtual disks (or “vDisks”) can be structure from the storage devices in the storage pool 160. As used herein, the term vDisk refers to the storage abstraction that is exposed by a Service VM to be used by a user VM. In some embodiments, the vDisk is exposed via iSCSI (“internet small computer system interface”) or NFS (“network file system”) and is mounted as a virtual disk on the user VM.

[0029] Each node 100a or 100b runs virtualization software, such as VMWare ESX(i), Microsoft Hyper-V, or RedHat KVM. The virtualization software includes a hypervisor 130/132 to manage the interactions between the underlying hardware and the one or more user VMs 102a, 102b, 102c and 102d that run client software.

[0030] A special VM 110a/110b is used to manage storage and I/O activities according to some embodiments of the invention, which is referred to herein as a “Service VM”. This is the “Storage Controller” in the currently described networked virtualization environment for storage management. Multiple such storage controllers coordinate within a cluster to form a single-system. The Service VMs 110a/110b are not formed as part of specific implementations of hypervisors 130/132. Instead, the Service VMs run as virtual machines above hypervisors 130/132 on the various servers 102a and 102b, and work together to form a distributed system 110 that manages all the storage resources, including the locally attached storage 122/124, the networked storage 128, and the cloud storage 126. Since the Service VMs run above the hypervisors 130/132, this means that the current approach can be used and implemented within any virtual machine architecture, since the Service VMs of embodiments of the invention can be used in conjunction with any hypervisor from any virtualization vendor.

[0031] Each Service VM 110a-b exports one or more block devices or NFS server targets that appear as disks to the client VMs 102a-d. These disks are virtual, since they are implemented by the software running inside the Service VMs 110a-b. Thus, to the user VMs 102a-d, the Service VMs 110a-b appear to be exporting a clustered storage appliance that contains some disks. All user data (including the operating system) in the client VMs 102a-d resides on these virtual disks.

[0032] Significant performance advantages can be gained by allowing the virtualization environment to access and utilize local (e.g., server-internal) storage 122. This is because I/O performance is typically much faster when performing access to local storage 122 as compared to performing access to networked storage 128 across a network 140. This faster performance for locally attached storage 122 can be increased even further by using certain types of optimized local storage devices, such as SSDs 125.

[0033] Once the virtualization environment is capable of managing and accessing locally attached storage, as is the case with the present embodiment, various optimizations can then be implemented to improve system performance even further. For example, the data to be stored in the various storage devices can be analyzed and categorized to determine which specific device should optimally be used to store the items of data. Data that needs to be accessed much faster or more frequently can be identified for storage in the locally attached storage 122. On the other hand, data that does not require fast access or which is accessed infrequently can be stored in the networked storage devices 128 or in cloud storage 126.

[0034] Another advantage provided by this approach is that administration activities can be handled on a much more efficient granular level. Recall that the prior art approaches of using a legacy storage appliance in conjunction with VMFS heavily relies on what the hypervisor can do at its own layer with individual “virtual hard disk” files, effectively making all storage array capabilities meaningless. This is because the storage array manages much coarser grained volumes while the hypervisor needs to manage finer-grained virtual disks. In contrast, the present embodiment can be used to implement administrative tasks at much smaller levels of granularity, one in which the smallest unit of administration at the hypervisor matches exactly with that of the storage tier itself.

[0035] Yet another advantage of the present embodiment of the invention is that storage-related optimizations for access and storage of data can be implemented directly within the primary storage path. For example, in some embodiments of the invention, the Service VM 110a can directly perform data deduplication tasks when storing data within the storage devices. This is far advantageous to prior art approaches that require add-on vendors/products outside of the primary storage path to provide deduplication functionality for a storage system. Other examples of optimizations that can be provided by the Service VMs include quality of service (QOS) functions, encryption and compression. The networked virtualization environment massively parallelizes storage, by placing a storage controller—in the form of a Service VM—at each hypervisor, and thus makes it possible to render enough CPU and memory resources to achieve the aforementioned optimizations.

[0036] Additional details regarding networked virtualization environments for storage management are described in U.S. Pat. No. 8,601,473, Ser. No. 13/207,345 entitled “Architecture for Managing I/O and Storage for a Virtualization Environment”, which is hereby incorporated by reference in its entirety.

[0037] Data replication involves replicating data located at a source to a destination. This may be performed for the purpose of disaster recovery, where data replicated from the source to the destination may be later recovered at the destination when the source undergoes failure. The networked virtualization environment illustrated in FIG. 1 may be representative of the source networked virtualization environment or destination networked virtualization environment for purposes of data replication. A source service VM may be utilized to perform data replication for its corresponding user VM. The source service VM does so by identifying the file(s) to be replicated for a particular user VM and coordinating with one or more destination service VMs for performing replication of the file(s) at the destination. At the destination, one or more destination service VMs are assigned to the source service VM for receiving file(s) to be replicated and storing those files. Additional details for performing such distributed data replication may be found in co-pending application Ser. No. 14/019,139, entitled “System and Methods for Performing Distributed Data Replication in a Networked Virtualization Environment”, which is incorporated by reference in its entirety.

[0038] Two modes of data replication exist: asynchronous data replication and synchronous data replication. Asynchronous data replication occurs where a write operation for a piece of data at a source is committed as soon as the source acknowledges the completion of the write operation. Replication of the data at the destination may occur at a later time after the write operation at the source has been committed. Synchronous data replication occurs where a write operation for a piece of data at a source is committed only after the destination has replicated the data and acknowledged completion of the write operation. Thus, in a synchronous data replication mode, a committed write operation for data at the source is guaranteed to have a copy at the destination.

[0039] Asynchronous data replication is advantageous in certain situations because it may be performed with more efficiency due to the fact that a write operation for data at the source can be committed without having to wait for the destination to replicate the data and acknowledge completion of the write operation. However, asynchronous data replication may result in potential data loss where the source fails prior to the replication of data at the destination.

[0040] Synchronous data replication guarantees that data loss will not occur when the source fails because the write operation for data is not committed until the destination has verified that it too has a copy of the data. However, having to wait for data to be written at both the source and the destination before committing a write operation may lead to latency as well as strain on system resources (e.g., CPU usage, memory usage, network traffic, etc.).

[0041] Conventionally, data replication involves setting a fixed data replication policy (either synchronous or asynchronous). A fixed synchronous data replication policy may be defined by various timing parameters such as the time taken for performing the data replication process or the wait time between successive data replication processes. In some situations, the fixed synchronous data replication policy may dictate the exact timing parameters for data replication such that performance of synchronous data replication under a particular policy must be made using the exact timing parameters of that particular policy. In other situations, the fixed synchronous data replication policy may provide a guideline for data replication such that performance of synchronous data replication under a particular policy attempts to meet the timing parameters of that particular policy without necessarily exactly meeting those timing parameters.

[0042] By setting a fixed data replication policy, the manner in which data replication occurs remains static regardless of the changing nature of the system (e.g., source networked virtualization environment or destination networked virtualization environment). System parameters such as the amount of data being replicated or the amount of resources being consumed by the source or destination may vary over time. Thus, fixing the data replication policy for a system fails to account for the dynamic nature of system parameters and may lead to inefficiencies where the system parameters change substantially or frequently over the course over system operation.

[0043] Setting a fixed data replication policy may be efficient where the source and destination operate at a steady resource consumption rate and the amount of data to be replicated remains steady. However, where the rate of resource consumption or amount of data to be replicated exhibits volatility, the fixed data replication policy may lead to the underutilization of resources when additional resources are available or where the amount of data to be replicated significantly decreases. Similarly, inefficiency may occur where the fixed data replication policy overutilizes resource availability when fewer resources are available or where the amount of data to be replicated significantly increases.

[0044] With the introduction of networked virtualization environments for storage management, such as the one described above in FIG. 1, various configurations between service VMs may exist for performing data replication. At any given time, any number of source service VMs may be servicing any number of user VMs. Each of the source service VMs may be assigned to any number of destination service VMs for performing data replication. Additionally, the destination service VM may assigned to numerous service VMs in addition to the source service VM for performing data replication.

[0045] FIGS. 2A-C are schematic diagrams illustrating example source/destination configurations for data replication according to some embodiments of the invention. For purposes of illustration, the examples illustrated in FIGS. 2A-C will illustrate only the configurations of service VMs at the source and destination. However one ordinarily skilled in the art will recognize that any of the service VMs illustrated in FIGS. 2A-C may be servicing any number of user VMs and may have access to various physical resource configurations (e.g., CPU, memory, network bandwidth, etc.)

[0046] In FIG. 2A, a one-to-one configuration between source service VMs 201a-d and destination service VMs 203a-d is illustrated. A source service VM 201a-d servicing any number of user VMs (not shown) may be assigned to a corresponding destination service VM 203a-d for performing data replication. In the example illustrated in FIG. 2A, each destination service VM 203a-d is assigned to only a single service VM 201a-d at and provides data replication exclusively to that service VM at the source. It is important to note, however, that each destination service VM 203a-d may additionally be assigned to any number of other service VMs (not shown) in a networked virtualization environment other than the source for providing data replication to those other networked virtualization environments.

[0047] In this configuration, parameters associated with data replication may vary over time. For example, the number of user VMs serviced by a source service VM 201a-d may change over time. When the number of user VMs being serviced by a source service VM 201a-d increases, the amount of resources available to the source service VM 201a-d for performing data replication for a given user VM may be more limited, as the same amount of resources is distributed amongst a greater number of user VMs. In this situation, providing data replication to a given user VM using a fixed data replication policy, may increase strain on the resources or decrease the availability of those resources for servicing other user VMs associated with the source service VM 210a-d. Similarly, when the number of user VMs being serviced by a source service VM 201a-d decreases, the amount of resources available to the source service VM 201a-d for performing data replication for a given user VM may be more abundant, as the same amount of resources is distributed amongst a lesser number of user VMs. In this situation, providing data replication to a given user VM using a fixed data replication policy, may result in underutilization of resources.

[0048] Likewise, when the number of user VMs serviced by the source service VM 201a-d increases, the amount of total data to be replicated for user VMs being serviced by a source service VM 201a-d may be increased. Because the total amount of data to be replicated may be increased, providing data replication to a given user VM using a fixed data replication policy may result in increased resource strain or decrease the availability of those resources available to the source service VM 201a-d for servicing other user VMs associated with the source service VM 201a-d.

[0049] Even when the number of user VMs and rate of data replication remains the same at the source, a corresponding destination service VM 203a-d may experience various parameter changes that decrease the efficiency of using a fixed data replication policy. For example, the destination service VM 203a-d assigned for performing data replication may itself be servicing an increase in user VMs. Replicating data using the same data replication policy would thereby create great strain on resources available to the destination service VM 203a-d. Similarly, the destination service VM 203a-d assigned for performing data replication may be servicing a decrease in user VMs. Replicating data using the same data replication policy would thereby underutilize resources available to the destination service VM 203a-d.

[0050] Likewise, the destination service VM 203a-d may additionally be assigned to any number of other service VMs (not shown) in a networked virtualization environment other than the source for providing data replication to user VMs in those other networked virtualization environments. The number of other service VMs assigned to the destination service VM 203a-d may also vary over time, and as such maintaining a fixed data replication policy may lead to the overutilization or underutilization of resources available to the destination service VM 203a-d.

[0051] In FIG. 2B, a one-to-many configuration between source service VMs 201a-b and destination service VMs 203a-d is illustrated. A source service VM 201a-b servicing any number of user VMs may be assigned to two or more corresponding destination service VMs 203a-d for performing data replication. In the example illustrated in FIG. 2B, each source service VM 201a-b is assigned to two destination service VMs 203a-d. The same issues that arise in the example illustrated in FIG. 2A are also present in FIG. 2B when a fixed data replication policy is used, and as such will not be described again.

[0052] In FIG. 2C, a many-to-one configuration between source service VMs 201a-d and a destination service VM 203a is illustrated. Two or more source service VMs 201a-d, each servicing any number of user VMs may be assigned to a single corresponding destination service VM 203a for performing data replication. In the example illustrated in FIG. 2C, a single destination service VM 203a is assigned to four different source service VMs 201a-d for performing data replication. The same issues that arise in the example illustrated in FIG. 2A are also present in FIG. 2B when a fixed data replication policy is used, and as such will not be described again.

[0053] As illustrated above, because of the various different configurations that may be used for performing data replication and the changing nature of parameters associated with the networked virtualization environments, various inefficiencies may arise when a fixed data replication policy is used. The dynamic nature of networked virtualization parameters such as the amount of data being replicated or the amount of resources available to and being consumed by the source service VM or destination service VM may vary over time regardless of whether an asynchronous data replication policy or synchronous data replication policy is used. As such, using a fixed data replication policy will necessarily lead to the inefficiencies described above.

[0054] What is needed, therefore, is a mechanism for dynamically adjusting between synchronous and asynchronous data replication policies. As used herein, the term dynamically adjusting between synchronous and asynchronous data replication policies may refer to the act of switching from an asynchronous data replication policy to a synchronous data replication policy or vice versa and may additionally refer to the act of transitioning between an asynchronous data replication policy with a first set of timing parameters to an asynchronous data replication policy with a different set of timing parameters.

[0055] By dynamically adjusting between different data replication policies, the fluctuations in system parameters during operation may be accounted for and the utilization of system resources may be made more optimal and efficient. For example, where the resources available to a service VM at the source or destination are heavily utilized (due to the number of user VMs being serviced or amount of data being replicated), the data replication policy may shift from a synchronous data replication policy to an asynchronous data replication policy. Alternatively, the data replication policy may shift from a synchronous data replication policy with a short replication time to a data replication policy with a longer replication time to account for the heavy resource utilization.

[0056] As another example, where the resources available to a service VM at the source or destination are underutilized (due to the number of user VMs being serviced or amount of data being replicated), the data replication policy may shift from an asynchronous data replication policy to a synchronous data replication policy. Alternatively, the data replication policy may shift from a synchronous data replication policy with a long replication time to a data replication policy with a shorter replication time to account for the low resource utilization.

[0057] FIG. 3 is a flow diagram illustrating a method for dynamically adjusting between asynchronous and synchronous data replication policies according to some embodiments of the invention. The method will be described in the context of data replication for a single user VM. However, one ordinarily skilled in the art will recognize that the method may be extended to account for replication of any number of user VMs.

[0058] The process for dynamically adjusting between data replication policies may initiate under various different circumstances. In some circumstances, the process may begin at periodic intervals, such as every two hours. Alternatively, the process may begin when a resource utilization level at either the source or the destination rises above or falls below a particular threshold. As another example, the process for dynamically adjusting between data replication policies may initiate whenever a service VM loses or gains additional user VMs.

[0059] When the process for dynamically adjusting between data replication policies initiates, a current data replication policy for a user VM is first identified as shown at 301.

[0060] An administrator or user of the user VM may establish a preferred data replication policy. In some embodiments, the data replication policy may be a synchronous data replication policy, where every write operation of data for the user VM is not committed until the data is replicated and the write operation is acknowledged at the destination. In some other embodiments, the data replication policy may be an asynchronous data replication policy, where a write operation of data for the user VM is committed once the source has acknowledged the write operation. The asynchronous data replication policy may indicate a time period for performing data replication. For example, the asynchronous data replication policy may indicate that data replication is to be performed in five minutes. Alternatively, the asynchronous data replication policy may indicate a time period between successive data replications. For example, the synchronous data replication policy may indicate that a five minute period of time passes between successive replication steps. Additionally, the asynchronous data replication policy may indicate a total time period for data replication. For example, the synchronous data replication policy may indicate that data replication is to be performed for five minutes with a five minute pause between successive replication steps.

[0061] A load level may then be determined by the source service VM as shown at 303. In some embodiments, the load level may indicate the current amount of resources being utilized by the source service VM. The load level being utilized by the source service VM may be important in determining how the data replication policy should be adjusted because it indicates the amount of additional load the source service VM can take on or the amount of load the source service VM needs to be reduced by in order to perform at an optimal level.

[0062] In other embodiments, the load level may indicate the current amount of resources being utilized by the source service VM as well as the amount of resources being utilized by the destination service VM. The load level being utilized by the destination service VM may be important in determining how the data replication policy should be adjusted because it indicates the amount of additional load the destination service VM can take on or the amount of load the destination service VM needs to be reduced by in order to perform at an optimal level.

[0063] In some embodiments, service VMs may monitor their own resources usage. In this situation, the source service VM may determine its load level by consulting its monitored resource usage and the source service VM may determine the load level at the destination service VM by communicating with the destination service VM to determine the amount of resource usage at the destination. In some other embodiments, a central controller may monitor the resource usage of both the source service VM and the destination service VM, and the source service VM may communicate with the central controller to determine the load level for both the source and the destination.

[0064] The load level at either the source service VM, destination service VM or their combination may include various resource usage parameters such as, for example, CPU usage, memory usage, and network bandwidth utilization.

[0065] The source service VM may then optionally determine the replication time history of the user VM undergoing dynamic data replication policy adjustment as shown at 305. The replication time history of the user VM illustrates the actual time needed (e.g., time for data replication or time between data replications) for previous data replications performed for the user VM. By determining the replication time history of the user VM, the source service VM may gain a better understanding of the efficiency/inefficiency of previous data replication policies. For example, where a previous data replication policy is set to 5 minutes for performing data replication and data replication actually occurs in 3 minutes, the source service VM may determine that a more aggressive asynchronous data replication policy should be used. As another example, where a previous data replication policy is set to 5 minutes for performing data replication and data replication actually occurs in 7 minutes, the source service VM may determine that a less aggressive asynchronous data replication should be used.

[0066] The source service VM may also optionally determine the data volatility for the user VM undergoing dynamic data replication policy adjustment as shown at 307. The data volatility for the user VM provides an indication of the change in data replication rate over time. When a user VM experiences high data volatility, the data replication rate may change frequently over a short period of time. This may indicate that a less aggressive data replication policy is to be used in order to account for the constant flux in data replication rate. Likewise, when the user VM experiences low data volatility, the data replication rate may remain steady over a long period of time. This may indicate that a more aggressive data replication policy is to be used in order to account for the steadiness in data replication rate.

[0067] The source service VM may then calculate a desired data replication policy for the user VM as shown at 309. The desired data replication policy may apply a mathematical algorithm or set of rules that factors the various system parameters (e.g., load level, replication time history, data volatility) to determine a desired data replication policy. The factors may be given different weights in accordance with their effect on the system's ability to perform data replication. In some embodiments, the new data replication policy may shift from an asynchronous data replication policy to a synchronous data replication policy or vice versa. In other embodiments, the new data replication policy may shift from a synchronous data replication policy with a short replication time to a data replication policy with a longer replication time or vice versa.

[0068] In some embodiments, the shift in replication time from the current data replication policy to the desired data replication policy may involve lengthening or shortening the time for performing data replication. In other embodiments, the shift in replication time from a current data replication policy to a desired data replication policy may involve lengthening or shortening the time between successive data replications.

[0069] The source service VM may maintain a learning process for updating the rules or algorithm applied to the various system parameters (e.g., load level, replication time history, data volatility) for determining the desired data replication policy. Each time the data replication policy changes, the source service VM may feedback the new data replication policy along with the parameters used to determine the new data replication policy. Information corresponding to data replication using the desired data replication policy may then be monitored to generate statistics associated with the new data replication policy for determining how effective the current rules or algorithm being used to generate desired data replication policies are. The rules or algorithm may then be updated based on these monitored statistics.

[0070] FIG. 4 is a flow diagram illustrating a method for employing a learning process for updating rules or a mathematical algorithm for dynamically adjusting between asynchronous and synchronous data replication policies according to some embodiments of the invention. Initially a desired data replication policy for a user VM is identified as shown at 401. The desired data replication policy may be identified, each time the source service VM goes through the process described above in FIG. 3. A determination is then made as to whether the desired data replication policy changed from a previous data replication policy as shown at 403. If the desired data replication policy did not change from the previous data replication policy, the current rules/algorithm for calculating data replication policies is maintained for the user VM as shown at 413.

[0071] If the desired data replication policy did change from the previous data replication policy, the desired data replication policy along with its associated system parameters are fed back to the source service VM as shown at 405. Information corresponding to data replication using the desired data replication policy may then monitored as shown at 407 and the information may be used to generate statistics associated with the desired replication policy as shown at 409. Such statistics may include such items as time taken to perform data replication using the identified data replication policy, CPU usage for data replication under the identified data replication policy, memory usage for data replication under the identified data replication policy, network traffic associated with data replication under the identified data replication policy, etc. These statistics may then be used to update the rules or algorithm to be used for performing a subsequent change in data replication policy as shown at 411.

[0072] FIG. 5 is a flow diagram illustrating a method for dynamically adjusting between asynchronous and synchronous data replication policies for a user VM according to some embodiments of the invention. Initially, a set of system parameters are received by the source service VM as shown at 501. The set of system parameters may identify certain information about the user VM undergoing data replication, as well as information about its corresponding source service VM and corresponding destination service VM, which are utilized for determining a desired data replication policy for the user VM.

[0073] As discussed above, the system parameters may include such information as a calculated load level for the source service VM or the destination service VM, a replication time history for the user VM and the data volatility associated with the user VM.

[0074] An algorithm or set of rules may be applied to the replication policy parameters to determine a new data replication policy for the user VM. In determining the new data replication policy for the user VM, the source service VM may first apply an algorithm or set of rules to the set of system parameters to determine whether the desired data replication policy should be a synchronous policy or an asynchronous policy as shown at 503.

[0075] For example, where the load level is low at both the source and the destination, the replication time history for the user VM indicates that data replication typically completes in a much shorter time period than the prescribed data replication policy, and the data volatility remains steady, a determination may be made to transition the data replication policy to a synchronous policy. Alternatively, where the load level is high at both the source and the destination, the replication time history for the user VM indicates that data replication typically completes in a longer period of time than the prescribed data replication policy, and the data is highly volatile, a determination may be made to transition the data replication policy to a synchronous data policy.

[0076] When a determination is made to transition the data replication policy to a synchronous policy, the data replication policy is set to a synchronous mode as shown at 509. The synchronous policy does not require any adjustment to replication timing parameters such as the time for performing data replication and the time between successive data replications. This is because the synchronous policy operates under a static set of timing parameters where each write operation for data at the source is not committed until the write operation is replicated and acknowledged at the destination.

[0077] When a determination is made to transition the data replication policy to an asynchronous data replication policy, timing parameters for the new asynchronous data replication policy must first be determined. Such timing parameters may be determined by applying an algorithm or set of rules to the system parameters. In determining timing parameters for the new asynchronous data replication policy, a time for performing an asynchronous mode data replication operation may first be determined as shown at 505. The time for performing the asynchronous mode data replication operation indicates how long it should take to perform data replication for the user VM. A time between successive asynchronous data replication steps may also be determined as shown at 507. The time between successive asynchronous data replication steps indicates the length of a wait time between successive asynchronous data replication steps.

[0078] FIGS. 6A to 6E illustrate an example of dynamically adjusting between asynchronous and synchronous data replication policies according to some embodiments of the invention. FIG. 6A illustrates a current data replication policy for a user VM. The illustration in the upper half of FIG. 6A provides a representation of where the current data replication policy lies between a purely synchronous data replication policy and a non-existent data replication policy where data is never replicated. The purely synchronous data replication policy is indicated by the label “Sync” and the non-existent data replication policy is indicated by the label “0”.

[0079] The current data replication policy illustrated in FIG. 6A is an asynchronous data replication policy. The timing parameters of the current asynchronous data replication policy is illustrated in the bottom half of FIG. 6A. The time for performing data replication is indicated by T₁ and the time between successive asynchronous data replication steps is indicated by T₂. Although a particular set of timing parameters in the bottom portion of FIG. 6A is shown to correspond to the current asynchronous data replication policy in the top portion of FIG. 6A illustrated, one ordinarily skilled in the art will recognize that other timing parameter configurations may also correspond to the current asynchronous data replication policy in the top portion of FIG. 6A.

[0080] FIGS. 6B-D illustrate a desired data replication policy for the user VM after the user VM undergoes a method for dynamically adjusting between asynchronous and synchronous data replication policies, such as the one described above in FIG. 3. In FIGS. 6B-D, the desired data replication policy is determined to be an asynchronous data replication policy that lies closer to the non-existent data replication policy as illustrated in the top portion of FIGS. 6B-D. In other words, the desired data replication policy is slowed down in comparison to the current data replication policy. This may be due to various factors, such as an increase in load levels of the source or destination service VMs, a replication timing history that indicates a pattern of the actual time needed for previous data replications performed for the user VM being greater than the prescribed replication time, or an increase in data volatility for the user VM.

[0081] The timing parameters for the current data replication policy are adjusted to generate the desired data replication policy. The timing parameters may be adjusted by lengthening the time for data replication T₁ and maintaining the time between successive data replication steps T₂ as illustrated in FIG. 6B. The timing parameters may alternatively be adjusted by lengthening the time between successive data replication steps T₂ while maintaining the time for data replication T₁ as shown in FIG. 6C. The timing parameters may also be alternatively adjusted by lengthening the time for data replication T₁ and lengthening the time between successive data replication steps T₂ as illustrated in FIG. 6D.

[0082] FIGS. 6E-G illustrate another example of a desired data replication policy for the user VM after the user VM undergoes a method for dynamically adjusting between asynchronous and synchronous data replication policies, such as the one described above in FIG. 3. In FIGS. 6E-G, the desired data replication policy is determined to be an asynchronous data replication policy that lies closer to the asynchronous data replication policy as illustrated in the top portion of FIGS. 6E-G. In other words, the desired data replication policy is sped up in comparison to the current data replication policy. This may be due to various factors, such as a decrease in load levels of the source or destination service VMs, a replication timing history that indicates a pattern of the actual time needed for previous data replications performed for the user VM being less than the prescribed replication time, or a decrease in data volatility for the user VM.

[0083] The timing parameters for the current data replication policy are adjusted to generate the desired data replication policy. The timing parameters may be adjusted by shortening the time for data replication T₁ and maintaining the time between successive data replication steps T₂ as illustrated in FIG. 6E. The timing parameters may alternatively be adjusted by shortening the time between successive data replication steps T₂ while maintaining the time for data replication T₁ as shown in FIG. 6F. The timing parameters may also be alternatively adjusted by shortening the time for data replication T₁ and shortening the time between successive data replication steps T₂ as illustrated in FIG. 6G.

[0084] FIG. 6H illustrates another example of a desired data replication policy for the user VM after the user VM undergoes a method for dynamically adjusting between asynchronous and synchronous data replication policies, such as the one described above in FIG. 3. In FIG. 6H, the desired data replication policy is determined to be a synchronous data replication policy as illustrated in the top portion of FIG. 6H. The synchronous replication policy continuously commits write operations for data at the source only after replication of the data and acknowledgement of the write operation at the destination, which is illustrated in the bottom portion of FIG. 6H. The transition from an asynchronous data replication policy to a synchronous data replication policy may be due to various factors, such as a significant decrease in load levels of the source or destination service VMs, a replication timing history that indicates a pattern of the actual time needed for previous data replications performed for the user VM being less than the prescribed replication time, or a significant decrease in data volatility for the user VM.

[0085] The above description illustrates various examples of dynamic adjustment between asynchronous and synchronous data replication policies. The following description will provide a mechanism for performing data replication between a source and a destination when the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy. The data replication is performed on a per vDisk basis, such that the contents of a vDisk for a user VM at the source cluster of nodes are replicated at the destination cluster of nodes. Additionally, although contents for a vDisk for a user VM at the source are replicated at the destination, the storage management parameters or storage management configuration at the source need not match the storage management parameters or storage management configuration at the destination. For example, the unit of storage at the source (e.g., block size) may differ from the unit of storage at the destination. As another example, the de-duplication policies at the source may differ from the de-duplication policies at the destination. As another example, the destination cluster of nodes may be providing data replication for different subsets of user VMs residing at different source clusters of nodes, such that the destination cluster of nodes is not identical to any one source cluster of nodes.

[0086] Transitions from an asynchronous data replication policy to a synchronous data replication policy may occur in various contexts. For example, a source and destination may be initially performing data replication in a synchronous manner. A network failure or any other condition that leads to a loss of connection between the source and the destination may cause the data replication policy to transition into an asynchronous data replication policy until the source and destination are able to re-establish connection. Such an asynchronous data replication policy may have various timing parameters. When the connection between the source and destination is re-established, the data replication policy may transition back to a synchronous data replication policy. As another example, a source and destination may be initially performing data replication in an asynchronous manner and transition into a synchronous data replication policy

[0087] Conventionally, such a transition from the asynchronous data replication policy to the synchronous data replication policy requires the destination vDisk to have the same state as the source vDisk at the time that synchronous data replication is to occur. Said differently, before a synchronous data replication policy may be utilized for replicating data between a source vDisk and a destination vDisk, any differences between the destination vDisk and the source vDisk must be resolved by applying changes to the destination vDisk to place the destination vDisk in the same state as the source vDisk.

[0088] Having to resolve all of the differences that exist between the destination vDisk and the source vDisk prior to transitioning from an asynchronous data replication policy to a synchronous data replication policy may lead to significant delay and other inefficiencies. What is needed, therefore, is a mechanism for transitioning from an asynchronous data replication policy to a synchronous data replication policy for data replication between a source vDisk and a destination vDisk without having to first resolve differences between the source vDisk and the destination vDisk.

[0089] FIG. 8 is a flow diagram illustrating a method for transitioning from an asynchronous data replication policy to a synchronous data replication policy without having to first place the destination vDisk into the same state as the source vDisk. As mentioned above, this method may be applied to any situation where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy.

[0090] Initially, a snapshot of the source vDisk is taken at a first point in time where the state of the source vDisk snapshot is the same as the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy as shown at 801. In some embodiments, the service VM facilitating data replication at the source may determine that a snapshot should be taken based on a user indicating that the data replication policy should be transitioned from an asynchronous data replication policy to a synchronous data replication policy. The snapshot is then taken at the point in time where the state of the source vDisk is equivalent to state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy.

[0091] In some other embodiments, the service VM facilitating data replication at the source may determine that a snapshot should be taken based on the service VM at the source losing connection with the service VM at the destination. The snapshot is taken at the last point in time where the source vDisk and the destination vDisk have the same state (e.g., point in time immediately preceding the loss of connection).

[0092] The snapshot taken of the source vDisk provides the state of destination vDisk at the last point in time prior to the data replication policy transitioning from an asynchronous mode to a synchronous mode.

[0093] After the first point in time, the source vDisk may continue to perform I/O operations that may change the contents of the source vDisk. Because an asynchronous data replication policy is used to replicate data between the source vDisk and destination vDisk after the first point in time, but prior to the data replication policy transitioning into a synchronous data replication policy, I/O operations performed on the source vDisk during that time are not immediately replicated at the destination vDisk. Thus, at a second point in time when the data replication policy transitions from the asynchronous data replication policy to the synchronous data replication policy, the changes made to the source vDisk after the first snapshot are not yet replicated at the destination vDisk.

[0094] At this second point in time, a second snapshot of the source vDisk is taken as shown at 803. The second snapshot provides the state of the source vDisk at the point where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy.

[0095] In order to allow for the destination vDisk to begin replicating data from the source vDisk in a synchronous manner without having to first place the destination vDisk into the same state as the source vDisk, the service VM facilitating data replication at the source must provide enough metadata to the service VM facilitating data replication at the destination to allow for a shell destination vDisk to be generated. Thus, metadata is provided from the source to the destination as shown at 805. The metadata provided from the source to the destination includes the structural parameters of the source vDisk, but does not identify the content changes that occurred between the first point in time and the second point in time. Using the metadata provided by the source, a shell destination vDisk that is structurally similar to the source vDisk at the time of the second snapshot, but does not have all of the contents of the source vDisk may be generated at the destination. The shell destination vDisk is later populated with the contents of the source vDisk through a background process, which will be described in greater detail below.

[0096] Once the shell destination vDisk has been generated at the destination, any write operations performed on the source vDisk are synchronously replicated at the shell destination vDisk as shown at 807. Although the shell destination vDisk does not yet have all the contents of the source vDisk, it is structurally similar to the source vDisk and so any write operations performed at the source vDisk after the second point in time may be replicated on the shell destination vDisk. For example, the shell destination vDisk may have an empty structure with a same number data blocks as the source vDisk, and may perform a write operation to any of those empty data blocks to replicate a write operation that occurs at the source vDisk.

[0097] By generating the shell destination vDisk, synchronous data replication between the source and the destination may begin immediately without having to first place the destination vDisk into the same state as the source vDisk. This significantly decreases the amount of delay incurred during a transition from an asynchronous data replication policy to a synchronous data replication policy.

[0098] The differences between the snapshot at the first point in time and the snapshot at the second point in time are then provided to the destination as a background process as shown at 809. Those differences are then used to populate the shell destination vDisk so that the shell destination vDisk may have the same state as the source vDisk. In some embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided in a single operation (e.g., batch process). In other embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided over several different operations.

[0099] By providing the differences between the snapshot at the first point in time and the second point in time as a background process after synchronous data replication has begun rather than providing the differences as a foreground process prior to initiating synchronous data replication, significant reduction in delay may be achieved while at the same time providing a destination vDisk that has the same state as the source vDisk.

[0100] FIG. 9 illustrates an example of transitioning from an asynchronous data replication policy to a synchronous data replication policy. FIG. 9 provides one example scenario for utilizing the method for transitioning from an asynchronous data replication policy to a synchronous data replication policy without having to first place the destination vDisk into the same state as the source vDisk. In FIG. 9, the passage of time is indicated by the arrow in the y-direction. The source vDisk and destination vDisk use an asynchronous data replication policy up until time T2, and then transition into a synchronous data replication policy at time T2.

[0101] A source vDisk and a destination vDisk initially perform data replication using an asynchronous data replication policy. As illustrated in FIG. 9, the source vDisk first performs write operations 1, 2 and 3. Because data replication is performed asynchronously, the destination vDisk does not immediately replicate write operations 1, 2 and 3. Instead, write operations 1, 2 and 3 are provided to the destination vDisk at a later time, after the write operations have already been committed at the source vDisk, as indicated by the solid arrow between the source vDisk and destination vDisk in FIG. 9. The write operations are then committed at the destination vDisk shortly after.

[0102] After the write operations 1, 2 and 3 are provided to the destination vDisk, the source vDisk continues to perform write operations 4, 5 and 6 up until time T2. The write operations 4, 5 and 6 performed by the source vDisk between the time that write operations 1, 2 and 3 are provided to the destination vDisk and time T2 are committed at the source vDisk, but are not immediately replicated at the destination vDisk. This is because of the asynchronous nature of the data replication policy between the time that write operations 1, 2 and 3 are provided to the destination vDisk and time T2.

[0103] At time T2, the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy. Thus, the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy, includes write operations 1, 2 and 3.

[0104] At a first point in time T1, the state of the source vDisk is the same as the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy (e.g., state of the destination vDisk after committing write operations 1, 2 and 3).

[0105] After time T2, write operations 7, 8 and 9 are committed at the source vDisk and also synchronously replicated at the destination vDisk. The write operations 7, 8 and 9 may be synchronously replicated at the destination vDisk even though write operations 4, 5 and 6 have not yet been committed at the destination vDisk. This will be described in detail in conjunction with FIGS. 10A-D and 11A-D.

[0106] FIGS. 10A-D are schematic diagrams illustrating the method for transitioning from an asynchronous data replication policy to a synchronous data replication policy without having to first place the destination vDisk into the same state as the source vDisk. FIGS. 10A-D will be described in conjunction with the example of transitioning from an asynchronous data replication policy to a synchronous data replication policy illustrated in FIG. 9. The example provided in FIGS. 10A-D illustrates a situation where the structure of the source vDisk does not change between the first point in time T1 where the source vDisk has the same state as the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy and the second point in time T2 where the data replication policy transitions from an asynchronous mode to a synchronous mode.

[0107] As shown in FIG. 10A, the source vDisk and destination vDisk perform data replication using an asynchronous data replication policy up until a point in time T2, where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy. At an initial point in time, both the source vDisk and the destination vDisk have the same state. The source vDisk includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C. The destination vDisk has the same state as the source vDisk and also includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C.

[0108] While operating under the asynchronous data replication policy, the source vDisk performs a first write operation that changes the contents of its third data block from data C to data D, a second write operation that changes the contents of its second data block from data B to data C, and a third write operation that changes the contents of its third data block from data D to data E. During this time, the write operations performed and committed at the source vDisk are not immediately replicated at the destination vDisk, and so prior to the point in time where the write operations at the source vDisk are transmitted to the destination vDisk, the destination vDisk continues to include 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C.

[0109] The differences (e.g., write operations not immediately replicated) are then provided to the destination vDisk as indicated by the solid arrow. The differences provided from the source vDisk to the destination vDisk indicate that the contents of the first data block have not changed, the contents of the second data block have changed from data B to data C, and the contents of the third data block have changed from data C to data E. The destination vDisk is then updated to reflect those differences.

[0110] After the differences are provided to the destination vDisk, and prior to time T2, the source vDisk performs additional write operations. The source vDisk performs a fourth write operation that changes the contents of its third data block from data E to data F, a second write operation that changes the contents of its second data block from data C to data D, and a third write operation that changes the contents of its third data block from data F to data G. During this time, the write operations performed and committed at the source vDisk are not immediately replicated at the destination vDisk, and so between the time that the differences are provided and time T2, the destination vDisk continues to include 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E.

[0111] At time T2, the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy. Thus, the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy includes 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E

[0112] At a first point in time T1, the state of the source vDisk is the same as the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy (e.g., state of the destination vDisk after committing write operations 1, 2 and 3). A first snapshot is taken of the source vDisk at time T1, where the first snapshot replicates the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy.

[0113] At time T2, a second snapshot of the source vDisk is taken. The second snapshot provides the state of the source vDisk at the point where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy.

[0114] In order to allow for the destination vDisk to begin replicating data from the source vDisk in a synchronous manner without having to first place the destination vDisk into the same state as the source vDisk, the service VM facilitating data replication at the source must provide enough metadata to the service VM facilitating data replication at the destination to allow for a shell destination vDisk to be generated. The metadata provided from the source to the destination includes the structural parameters of the source vDisk, but does not identify the content changes that occurred between the first point in time and the second point in time. Using the metadata provided by the source, a shell destination vDisk that is structurally similar to the source vDisk at the time of the second snapshot, but does not have all of the contents of the source vDisk may be generated at the destination. This shell destination vDisk is depicted in FIG. 10A by the empty structure having 3 data blocks generated after time T2.

[0115] After the shell destination vDisk is generated, write operations performed at the source vDisk may be synchronously replicated at the shell destination vDisk. A seventh write operation is performed at the source vDisk that changes the contents of its third data block from data G to data H. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its third data block to include data H as shown in FIG. 10A. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has a third data block that has the same state as the third data block of the source vDisk.

[0116] An eighth write operation is then performed at the source vDisk that changes the contents of its second data block from data D to data E as shown in FIG. 10B. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its second data block to include data E as shown in FIG. 10B. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has both a second and a third data block that has the same state as the source vDisk.

[0117] A background process may then provide the differences between the snapshot at the first point in time and the snapshot at the second point in time to the destination as shown in FIG. 10C. The differences provided from the source vDisk to the shell destination vDisk between the first point in time T1 and the second point in time T2 indicate that the contents of the first data block have not changed, the contents of the second data block have changed from data C to data D, and the contents of the third data block have changed from data E to data G. The shell destination vDisk is then updated to reflect those differences, such that the state of the shell destination vDisk is the same as the state of the source vDisk as shown in FIG. 10D.

[0118] In some embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided in a single operation (e.g., batch process) as is illustrated in FIG. 10C. In other embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided over several different operations.

[0119] By providing the differences between the snapshot at the first point in time and the second point in time as a background process after synchronous data replication has begun rather than providing the differences as a foreground process prior to initiating synchronous data replication, significant reduction in delay may be achieved while at the same time providing a destination vDisk that has the same state as the source vDisk.

[0120] FIGS. 11A-D are schematic diagrams illustrating another example of the method for transitioning from an asynchronous data replication policy to a synchronous data replication policy without having to first place the destination vDisk into the same state as the source vDisk. FIGS. 11A-D will be described in conjunction with the example of transitioning from an asynchronous data replication policy to a synchronous data replication policy illustrated in FIG. 9. The example provided in FIGS. 11A-D illustrates a situation where the structure of the source vDisk changes between the first point in time T1 where the source vDisk has the same state as the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy and the second point in time T2 where the data replication policy transitions from an asynchronous mode to a synchronous mode.

[0121] As shown in FIG. 11A, the source vDisk and destination vDisk perform data replication using an asynchronous data replication policy up until a point in time T2, where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy. At an initial point in time, both the source vDisk and the destination vDisk have the same state. The source vDisk includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C. The destination vDisk has the same state as the source vDisk and also includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C.

[0122] While operating under the asynchronous data replication policy, the source vDisk performs a first write operation that changes the contents of its third data block from data C to data D, a second write operation that changes the contents of its second data block from data B to data C, and a third write operation that changes the contents of its third data block from data D to data E. During this time, the write operations performed and committed at the source vDisk are not immediately replicated at the destination vDisk, and so prior to the point in time where the write operations at the source vDisk are transmitted to the destination vDisk, the destination vDisk continues to include 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C.

[0123] The differences (e.g., write operations not immediately replicated) are then provided to the destination vDisk as indicated by the solid arrow. The differences provided from the source vDisk to the destination vDisk between the initial point in time and the first point in time T1 indicate that the contents of the first data block have not changed, the contents of the second data block have changed from data B to data C, and the contents of the third data block have changed from data C to data E.

[0124] After the differences are provided to the destination vDisk, and prior to the time T2, the source vDisk performs another set of write operations. During this time, the source vDisk performs a fourth write operation that changes the contents of its third data block from data E to data F and adds a fourth data block having data Z, a second write operation that changes the contents of its second data block from data C to data D, and a third write operation that changes the contents of its third data block from data F to data G. During this time, the write operations performed and committed at the source vDisk are not immediately replicated at the destination vDisk, and so between time the time that the differences are provided and time T2, the destination vDisk continues to include 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E.

[0125] At time T2, the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy. Thus, the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy includes 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E.

[0126] At a first point in time T1, the state of the source vDisk is the same as the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy (e.g., state of the destination vDisk after committing write operations 1, 2 and 3). A first snapshot is taken of the source vDisk at time T1, where the first snapshot provides the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy.

[0127] At time T2, a second snapshot of the source vDisk is taken. The second snapshot provides the state of the source vDisk at the point where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy.

[0128] In order to allow for the destination vDisk to begin replicating data from the source vDisk in a synchronous manner without having to first place the destination vDisk into the same state as the source vDisk, the service VM facilitating data replication at the source must provide enough metadata to the service VM facilitating data replication at the destination to allow for a shell destination vDisk to be generated. The metadata provided from the source to the destination includes the structural parameters of the source vDisk, but does not identify the content changes that occurred between the first point in time T1 and the second point in time T2. Using the metadata provided by the source, a shell destination vDisk that is structurally similar to the source vDisk at the time of the second snapshot, but does not have all of the contents of the source vDisk may be generated at the destination. Because the structure of the source vDisk has changed from having 3 data blocks to having 4 data blocks between time T1 and time T2, the metadata provided to the destination must indicate that the shell destination vDisk should have a structure having 4 data blocks. This shell destination vDisk is depicted in FIG. 11A by the empty structure having 4 data blocks generated after time T2. Thus, even though a structural change to the source vDisk has occurred between time T1 and time T2, enough metadata is provided to the destination prior to synchronous data replication to allow for a shell destination vDisk to be generated that may be used to facilitate synchronous data replication.

[0129] After the shell destination vDisk is generated, write operations performed at the source vDisk may be synchronously replicated at the shell destination vDisk. A seventh write operation is performed at the source vDisk that changes the contents of its third data block from data G to data H. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its third data block to include data H as shown in FIG. 11A. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has a third data block that has the same state as the third data block of the source vDisk.

[0130] An eighth write operation is then performed at the source vDisk that changes the contents of its second data block from data D to data E as shown in FIG. 11B. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its second data block to include data E as shown in FIG. 11B. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has both a second and a third data block that has the same state as the source vDisk.

[0131] A background process may then provide the differences between the snapshot at the first point in time T1 and the snapshot at the second point in time T2 to the destination as shown in FIG. 11C. The differences provided from the source vDisk to the shell destination vDisk between the first point in time T1 and the second point in time T2 indicate that the contents of the first data block have not changed, the contents of the second data block have changed from data C to data D, the contents of the third data block have changed from data E to data G, and that the contents of the fourth data block include data Z. The shell destination vDisk is then updated to reflect those differences, such that the state of the shell destination vDisk is the same as the state of the source vDisk as shown in FIG. 11D.

[0132] In some embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided in a single operation (e.g., batch process) as is illustrated in FIG. 11C. In other embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided over several different operations.

[0133] By providing the differences between the snapshot at the first point in time and the second point in time as a background process after synchronous data replication has begun rather than providing the differences as a foreground process prior to initiating synchronous data replication, significant reduction in delay may be achieved while at the same time providing a destination vDisk that has the same state as the source vDisk.

[0134] FIG. 12 illustrates an example of transitioning from a synchronous data replication policy to an asynchronous data replication policy and back to a synchronous data replication policy. FIG. 12 provides one example scenario for utilizing the method for transitioning from an asynchronous data replication policy to a synchronous data replication policy without having to first place the destination vDisk into the same state as the source vDisk. In FIG. 12, the passage of time is indicated by the arrow in the y-direction. The source vDisk and destination vDisk use a synchronous data replication policy until a first point in time T1. At time T1, the data replication policy transitions to an asynchronous data replication policy up until time T2, and then transitions back into a synchronous data replication policy at time T2. The change from a synchronous data replication policy to an asynchronous data replication policy at T1 may be due to the loss of a network connection between the source and the destination either due to a network failure or a failure of the destination. Alternatively, the change from a synchronous data replication policy to an asynchronous data replication policy at T1 may be caused by a user-initiated or system-initiated change.

[0135] A source vDisk and a destination vDisk initially perform data replication using a synchronous data replication policy. As illustrated in FIG. 12, the source vDisk first performs write operations 1, 2 and 3 while data replication is occurring synchronously. Because data replication is performed synchronously, the destination vDisk immediately replicates write operations 1, 2 and 3. At time T1, the state of the source vDisk and the state of the destination vDisk are the same.

[0136] After time T1, the data replication policy transitions into an asynchronous data replication policy and the source vDisk continues to perform write operations 4, 5 and 6 up until time T2. The write operations 4, 5 and 6 performed by the source vDisk between time T1 and time T2 are committed at the source vDisk, but are not immediately replicated at the destination vDisk, because of the asynchronous nature of the data replication policy between time T1 and time T2.

[0137] At time T2, the data replication policy transitions from an asynchronous data replication policy back to a synchronous data replication policy. Thus, the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy includes write operations 1, 2 and 3.

[0138] At the first point in time T1, the state of the source vDisk is the same as the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy (e.g., state of the destination vDisk after committing write operations 1, 2 and 3).

[0139] After time T2, write operations 7, 8 and 9 are committed at the source vDisk and also synchronously replicated at the destination vDisk. The write operations 7, 8 and 9 may be synchronously replicated at the destination vDisk even though write operations 4, 5 and 6 have not yet been committed at the destination vDisk. This will be described in detail in conjunction with FIGS. 13A-D and 14A-D.

[0140] FIGS. 13A-D are schematic diagrams illustrating the method for transitioning from a synchronous data replication policy to an asynchronous data replication policy and back to a synchronous data replication policy without having to first place the destination vDisk into the same state as the source vDisk. FIGS. 13A-D will be described in conjunction with the example of transitioning from a synchronous data replication policy to an asynchronous data replication policy and back to a synchronous data replication policy illustrated in FIG. 12. The example provided in FIGS. 13A-D illustrates a situation where the structure of the source vDisk does not change between the first point in time T1 where the source vDisk has the same state as the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy and the second point in time T2 where the data replication policy transitions from an asynchronous mode to a synchronous mode.

[0141] As shown in FIG. 13A, the source vDisk and destination vDisk perform data replication using a synchronous data replication policy up until a point in time T1, where the data replication policy transitions from a synchronous data replication policy to an asynchronous data replication policy. At an initial point in time, both the source vDisk and the destination vDisk have the same state. The source vDisk includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C. The destination vDisk has the same state as the source vDisk and also includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C.

[0142] While operating under the synchronous data replication policy, the source vDisk performs a first write operation that changes the contents of its third data block from data C to data D, a second write operation that changes the contents of its second data block from data B to data C, and a third write operation that changes the contents of its third data block from data D to data E. During this time, the write operations performed and committed at the source vDisk are synchronously replicated at the destination vDisk. When the source vDisk performs the first write operation that changes the contents of its third data block from data C to data D, the change is propagated to the destination vDisk where it is committed. Likewise, when the source vDisk performs the second write operation that changes the contents of its second data block from data B to data C, the change is propagated to the destination vDisk in a synchronous manner where it is committed. Similarly, when the source vDisk performs the third write operation that changes the contents of its third data block from data D to data E, the change is propagated to the destination vDisk in a synchronous manner where it is committed. Thus, prior to the point in time T1, the destination vDisk has the same state as the source vDisk, with the first data block having data A, the second data block having data B and the third data block having data C.

[0143] At time T1, the data replication policy transitions from a synchronous data replication policy to an asynchronous data replication policy. After time T1 and prior to the time T2 where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy, the source vDisk performs another set of write operations. During this time, the source vDisk performs a fourth write operation that changes the contents of its third data block from data E to data F, a second write operation that changes the contents of its second data block from data C to data D, and a third write operation that changes the contents of its third data block from data F to data G. During this time, the write operations performed and committed at the source vDisk are not immediately replicated at the destination vDisk, and so between time T1 and T2, the destination vDisk continues to include 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E.

[0144] At time T2, the data replication policy transitions from an asynchronous data replication policy back to a synchronous data replication policy. Thus, the state of the destination vDisk prior to the data replication policy transitioning from the asynchronous data replication policy back to the synchronous data replication policy includes 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E.

[0145] At the first point in time T1, the state of the source vDisk is the same as the state of the destination vDisk prior to the data replication policy transitioning from the asynchronous data replication back to the synchronous data replication policy (e.g., state of the destination vDisk after committing write operations 1, 2 and 3). A first snapshot is taken of the source vDisk at time T1, where the first snapshot provides the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy.

[0146] At time T2, a second snapshot of the source vDisk is taken. The second snapshot provides the state of the source vDisk at the point where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy.

[0147] At this point, in order to allow for the destination vDisk to begin replicating data from the source vDisk in a synchronous manner without having to first place the destination vDisk into the same state as the source vDisk, the service VM facilitating data replication at the source must provide enough metadata to the service VM facilitating data replication at the destination to allow for a shell destination vDisk to be generated. The metadata provided from the source to the destination includes the structural parameters of the source vDisk, but does not identify the content changes that occurred between the first point in time and the second point in time. Using the metadata provided by the source, a shell destination vDisk that is structurally similar to the source vDisk at the time of the second snapshot, but does not have all of the contents of the source vDisk may be generated at the destination. This shell destination vDisk is depicted in FIG. 13A by the empty structure having 3 data blocks generated after time T2.

[0148] After the shell destination vDisk is generated, write operations performed at the source vDisk may be synchronously replicated at the shell destination vDisk. A seventh write operation is performed at the source vDisk that changes the contents of its third data block from data G to data H. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its third data block to include data H as shown in FIG. 13A. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has a third data block that has the same state as the third data block of the source vDisk.

[0149] An eighth write operation is then performed at the source vDisk that changes the contents of its second data block from data D to data E as shown in FIG. 13B. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its second data block to include data E as shown in FIG. 13B. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has both a second and a third data block that has the same state as the source vDisk.

[0150] A background process may then provide the differences between the snapshot at the first point in time and the snapshot at the second point in time to the destination as shown in FIG. 13C. The differences provided from the source vDisk to the shell destination vDisk between the first point in time T1 and the second point in time T2 indicate that the contents of the first data block have not changed, the contents of the second data block have changed from data C to data D, and the contents of the third data block have changed from data E to data G. The shell destination vDisk is then updated to reflect those differences, such that the state of the shell destination vDisk is the same as the state of the source vDisk as shown in FIG. 13D.

[0151] In some embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided in a single operation (e.g., batch process) as is illustrated in FIG. 13C. In other embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided over several different operations.

[0152] By providing the differences between the snapshot at the first point in time and the second point in time as a background process after synchronous data replication has begun rather than providing the differences as a foreground process prior to initiating synchronous data replication, significant reduction in delay may be achieved while at the same time providing a destination vDisk that has the same state as the source vDisk.

[0153] FIGS. 14A-D are schematic diagrams illustrating another example of the method for transitioning from a synchronous data replication policy to an asynchronous data replication policy and back to a synchronous data replication policy without having to first place the destination vDisk into the same state as the source vDisk. FIGS. 14A-D will be described in conjunction with the example of transitioning from a synchronous data replication policy to an asynchronous data replication policy and back to a synchronous data replication policy illustrated in FIG. 12. The example provided in FIGS. 14A-D illustrates a situation where the structure of the source vDisk changes between the first point in time T1 where the source vDisk has the same state as the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy and the second point in time T2 where the data replication policy transitions from an asynchronous mode to a synchronous mode.

[0154] As shown in FIG. 14A, the source vDisk and destination vDisk perform data replication using a synchronous data replication policy up until a point in time T1, where the data replication policy transitions from a synchronous data replication policy to an asynchronous data replication policy. At an initial point in time, both the source vDisk and the destination vDisk have the same state. The source vDisk includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C. The destination vDisk has the same state as the source vDisk and also includes 3 data blocks with the first data block having data A, the second data block having data B and the third data block having data C.

[0155] While operating under the synchronous data replication policy, the source vDisk performs a first write operation that changes the contents of its third data block from data C to data D, a second write operation that changes the contents of its second data block from data B to data C, and a third write operation that changes the contents of its third data block from data D to data E. During this time, the write operations performed and committed at the source vDisk are synchronously replicated at the destination vDisk. When the source vDisk performs the first write operation that changes the contents of its third data block from data C to data D, the change is propagated to the destination vDisk where it is committed. Likewise, when the source vDisk performs the second write operation that changes the contents of its second data block from data B to data C, the change is propagated to the destination vDisk in a synchronous manner where it is committed. Similarly, when the source vDisk performs the third write operation that changes the contents of its third data block from data D to data E, the change is propagated to the destination vDisk in a synchronous manner where it is committed. Thus, prior to the point in time T1, the destination vDisk has the same state as the source vDisk, with the first data block having data A, the second data block having data B and the third data block having data C.

[0156] At time T1, the data replication policy transitions from a synchronous data replication policy to an asynchronous data replication policy. After time T1 and prior to the time T2 where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy, the source vDisk performs another set of write operations. During this time, the source vDisk performs a fourth write operation that changes the contents of its third data block from data E to data F and adds a fourth data block having data Z, a second write operation that changes the contents of its second data block from data C to data D, and a third write operation that changes the contents of its third data block from data F to data G. During this time, the write operations performed and committed at the source vDisk are not immediately replicated at the destination vDisk, and so between time T1 and T2, the destination vDisk continues to include 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E.

[0157] At time T2, the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy. Thus, the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy includes 3 data blocks with the first data block having data A, the second data block having data C and the third data block having data E.

[0158] At the first point in time T1, the state of the source vDisk is the same as the state of the destination vDisk prior to the data replication policy transitioning from an asynchronous data replication policy to a synchronous data replication policy (e.g., state of the destination vDisk after committing write operations 1, 2 and 3). A first snapshot is taken of the source vDisk at time T1, where the first snapshot replicates the state of the destination vDisk prior to the data replication policy transitioning from the asynchronous data replication policy back to the synchronous data replication policy.

[0159] At this time T2, a second snapshot of the source vDisk is taken. The second snapshot provides the state of the source vDisk at the point where the data replication policy transitions from an asynchronous data replication policy to a synchronous data replication policy.

[0160] In order to allow for the destination vDisk to begin replicating data from the source vDisk in a synchronous manner without having to first place the destination vDisk into the same state as the source vDisk, the service VM facilitating data replication at the source must provide enough metadata to the service VM facilitating data replication at the destination to allow for a shell destination vDisk to be generated. The metadata provided from the source to the destination includes the structural parameters of the source vDisk, but does not identify the content changes that occurred between the first point in time and the second point in time. Using the metadata provided by the source, a shell destination vDisk that is structurally similar to the source vDisk at the time of the second snapshot, may be generated at the destination. While the shell destination vDisk is structurally similar to the source vDisk at the time of the second snapshot, it does not have all of the contents of the source vDisk at the time of the second snapshot. Because the structure of the source vDisk has changed from having 3 data blocks to having 4 data blocks between time T1 and time T2, the metadata provided to the destination must indicate that the shell destination vDisk should have a structure having 4 data blocks. This shell destination vDisk is depicted in FIG. 17A by the empty structure having 4 data blocks generated after time T2. Thus, even though a structural change to the source vDisk has occurred between time T1 and time T2, enough metadata is provided to the destination prior to synchronous data replication to allow for a shell destination vDisk to be generated that may be used to facilitate synchronous data replication.

[0161] After the shell destination vDisk is generated, write operations performed at the source vDisk may be synchronously replicated at the shell destination vDisk. A seventh write operation is performed at the source vDisk that changes the contents of its third data block from data G to data H. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its third data block to include data H as shown in FIG. 14A. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has a third data block that has the same state as the third data block of the source vDisk.

[0162] An eighth write operation is then performed at the source vDisk that changes the contents of its second data block from data D to data E as shown in FIG. 14B. This write operation is immediately provided to the shell destination vDisk which then updates the contents of its second data block to include data E as shown in FIG. 14B. After the write operation has been synchronously replicated at the shell destination vDisk, the shell destination vDisk has both a second and a third data block that has the same state as the source vDisk.

[0163] A background process may then provide the differences between the snapshot at the first point in time and the snapshot at the second point in time to the destination as shown in FIG. 14C. The differences provided from the source vDisk to the shell destination vDisk between the first point in time T1 and the second point in time T2 indicate that the contents of the first data block have not changed, the contents of the second data block have changed from data C to data D, the contents of the third data block have changed from data E to data G, and that the contents of the fourth data block include data Z. The shell destination vDisk is then updated to reflect those differences, such that the state of the shell destination vDisk is the same as the state of the source vDisk as shown in FIG. 14D.

[0164] In some embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided in a single operation (e.g., batch process) as is illustrated in FIG. 14C. In other embodiments, the differences between the snapshot at the first point in time and the snapshot at the second point in time may be provided over several different operations.

[0165] By providing the differences between the snapshot at the first point in time and the second point in time as a background process after synchronous data replication has begun rather than providing the differences as a foreground process prior to initiating synchronous data replication, significant reduction in delay may be achieved while at the same time providing a destination vDisk that has the same state as the source vDisk.

System Architecture

[0166] FIG. 7 is a block diagram of an illustrative computing system 1400 suitable for implementing an embodiment of the present invention. Computer system 1400 includes a bus 1406 or other communication mechanism for communicating information, which interconnects subsystems and devices, such as processor 1407, system memory 1408 (e.g., RAM), static storage device 1409 (e.g., ROM), disk drive 1410 (e.g., magnetic or optical), communication interface 1414 (e.g., modem or Ethernet card), display 1411 (e.g., CRT or LCD), input device 1412 (e.g., keyboard), and cursor control.

[0167] According to one embodiment of the invention, computer system 1400 performs specific operations by processor 1407 executing one or more sequences of one or more instructions contained in system memory 1408. Such instructions may be read into system memory 1408 from another computer readable/usable medium, such as static storage device 1409 or disk drive 1410. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and/or software. In one embodiment, the term “logic” shall mean any combination of software or hardware that is used to implement all or part of the invention.

[0168] The term “computer readable medium” or “computer usable medium” as used herein refers to any medium that participates in providing instructions to processor 1407 for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive 1410. Volatile media includes dynamic memory, such as system memory 1408.

[0169] Common forms of computer readable media includes, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0170] In an embodiment of the invention, execution of the sequences of instructions to practice the invention is performed by a single computer system 1400. According to other embodiments of the invention, two or more computer systems 1400 coupled by communication link 1415 (e.g., LAN, PTSN, or wireless network) may perform the sequence of instructions required to practice the invention in coordination with one another.

[0171] Computer system 1400 may transmit and receive messages, data, and instructions, including program, i.e., application code, through communication link 1415 and communication interface 1414. Received program code may be executed by processor 1407 as it is received, and/or stored in disk drive 1410, or other non-volatile storage for later execution

[0172] In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, the above-described process flows are described with reference to a particular ordering of process actions. However, the ordering of many of the described process actions may be changed without affecting the scope or operation of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.