Private cloud replication and recovery

[0001] Replication of data processing systems to maintain operational continuity is now required almost everywhere. The costs incurred during downtime when information technology equipment and services are not available can be significant, and sometimes even cause an enterprise to halt operations completely. With replication, aspects of data processing machines that may change rapidly over time, such as their program and data files, physical volumes, file systems, etc. can be duplicated on a scheduled or continuous basis. Replication may be used for many purposes such as assuring data availability upon equipment failure, site disaster recovery or planned maintenance operations.

[0002] Replication may be directed to either the physical or virtual processing environment and/or different abstraction levels. For example, one may undertake to is replicate each physical machine exactly as it exists at a given time. However, replication processes may also be architected along virtual data processing lines, with corresponding virtual replication processes, with the end result being to remove the physical boundaries and limitations associated with particular physical machines.

[0003] Use of a replication service as provided by a remote or hosted external service provider can have numerous advantages. Replication services can provide continuous availability and failover capabilities that are more cost effective than an approach which has the data center operator owning, operating and maintaining a complete suite of duplicate machines at its own data center. With such replication services, physical or virtual machine infrastructure is replicated at a remote and secure data center.

[0004] In the case of replication services to virtual target, a virtual disk file containing the processor type and configuration, operating system, data, and applications for each data processor in the production environment is created and retained in a dormant state. In the event of a disaster, the virtual disk file is moved to a production mode within a remote and secure data center. Applications and data can then be accessed on the remote data center, enabling the service customer to continue operating from the cloud while recovering from a disaster.

[0005] From the perspective of the service customer, the replication service provider thus offers a Recover to Cloud (R2C) service that is provided much like an on-demand utility (similar to the electricity grid) over a network (typically the Internet). This is enables a data center operator to replicate critical servers and applications in his production environment to the cloud.

[0006] Therefore, existing disaster recovery products do accommodate virtualized environments. They can also provide centralized management of recovery plans enabling non-destructive testing and automated site recovery and migration processes. These products can also be used to specify which data process resources are to be recovered. However, such products most often require provisioning of resources at the recovery site in advance of a recovery event and do not offer optimum flexibility.

[0007] In a preferred configuration, a protected private cloud infrastructure may include hosts, and virtual machines provisioned on the hosts. This environment may also include storage arrays and a management server. The host machines provide processing resources and memory to the virtual machines. The storage, typically provided by separate hardware, contains an array of disks that may preferably be formed as a storage array network that connects the host machines to the storage arrays. A management server is also responsible for orchestration of the environment as well as maintaining metadata about the components that make up the private cloud virtual infrastructure.

[0008] A preferred process operates the management server to permit a user to configure the hosts and storage arrays and to provision virtual machines on the hosts. A separate process continuously replicates the storage arrays to a recovery site. This process may be carried out as part of a continuous storage replication scheme that is operates entirely within the context of the storage array network and separate from the disaster recovery functions.

[0009] Metadata is periodically obtained from the management server. The collected metadata configuration of the hosts and virtual machines is replicated at a metadata repository accessible at or located on the recovery site. Thus, it is understood that in a preferred arrangement replicating this metadata occurs independently of replicating the storage arrays.

[0010] Upon a recovery event, such as may occur upon disaster or disaster test, replication of the storage arrays is halted. At this point, a number of target machines corresponding to the management server and the hosts are assigned. It should be noted that these hosts are not previously assigned to this task prior to the disaster event.

[0011] The assigned management server and hosts are then bare metal provisioned for example, by installing operating systems or hypervisors as specified by the metadata. The management server is then recovered from the metadata to one of the recovery target machines and the hosts are also recovered from the metadata.

[0012] It is only after the management server and all such hosts are bare metal provisioned is the next step taken of connecting the recovery target machines to the replicated storage arrays. This causes virtual machine's metadata to also be retrieved for the recovery target machines.

[0013] However, prior to accessing the metadata to determine recovery of the virtual machines, the user may access the management server to specify which virtual machines are to actually be recovered. Specification of which virtual machines are to be recovered can therefore be delayed until such time as disaster occurs and the user need not specify which virtual machines are to be recovered in advance of such an event.

[0014] This approach to private cloud replication provides distinct advantages from the perspective of a service provider. The physical machines necessary for providing the recovery service need not be specifically tied to any particular protected environment prior to a disaster event. It is only when a customer acquires a need for them, are such is recovery machines tied down to a given role. All that is needed to be persisted by the recovery service is the host and management server metadata. This can be metadata can be stored in a repository, such as a shared database, that provides secure access to different customers.

[0025] Introduction

[0026] The present disclosure describes a private cloud replication service designed to provide continuous protection for a virtual machine infrastructure. There are two parts to the replication, virtual machine storage and metadata, each describing different aspects of the infrastructure. Though it is not a requirement, storage replication is continuous—data is replicated as it is created. Metadata is replicated periodically and separate from storage replication. At time of recovery automated processes use replicated virtual machine storage and virtual infrastructure metadata to recreate a private cloud in a recovery site.

[0027] Typical Private Cloud

[0028] FIG. 1 illustrates a typical private cloud infrastructure 100 supported by this service consists of virtual machine hosts 104-A, 104-B, 104-C, storage arrays 106-1, is 106-2, and a management server 102. Hosts (physical machines) 104 run hypervisor software, which enables multiple virtual machines to run on the same physical hardware. The host machines 104 provide CPU and memory resources to the (guest) virtual machines. Storage is usually provided by separate hardware, containing an array 106 of disks. The arrays 106 can be arranged as a storage area network, or SAN, which is responsible for connecting the host machines to the storage arrays 106.

[0029] The Management Server 102 is primarily responsible for orchestration of the private cloud 100. It also maintains metadata about the components that make up the private cloud virtual infrastructure 100 in a manner to be described in more detail below.

[0030] The private cloud virtualized environment may be provided using platform such as VMWare. In such an environment, the management server may be based on a VCenter. However, other virtualization solutions may be used.

[0031] The more detailed partial view of FIG. 2 shows a representative host, Host A 104-A, and the two storage arrays 106-1, 106-2. Host A is physically connected to both Storage Arrays, represented by solid lines. In this example, Host A is providing CPU and memory to three virtual machines, VM 1 110-1, VM 2 110-2, and VM 3 110-3. Storage Array 1 160-1 is providing storage to VM 1 110-1 and VM 2 110-2, while Storage Array 2 160-2 is providing storage to VM 3 110-3. The virtual machines 110 are also logically connected to their associated virtual disks 112-1, 112-2, 112-3 residing on the storage arrays, represented by the dashed lines.

[0032] Private Cloud Replication

[0033] FIG. 3 illustrates the private cloud 100 as installed at a protected site 300. A recovery site 350 is also shown. Recovery site may typically be operated as a service to the owner/operator of the protected site 300. As such, the recovery site 350 may service many protected site 300, although only one protected site 300 is shown. Under normal operation, the storage arrays 106 are continuously replicated from the protected site 300 to the recovery site 350. This replication may occur within the content of the is SAN and separately from any recovery process described here.

[0034] Because the virtual machines 110 store their data on the storage arrays 106-1, 106-2, by way of their virtual disks 112, this continuous replication process also automatically and without further intervention by management server 102 replicates the virtual machine data. As shown in FIG. 3, the recovery site 350 would be equipped with permanent recovery storage arrays 106-R-1, 106-R-2 to accept the replicated data. An alternative configuration could have no permanently installed storage on the recovery side. Instead, the replication storage 106-R arrays could be populated at the time of need by physically bringing data to the recovery site 350. Tape based backups would be one such example of a medium for transferring data to the recovery site in a non-continuous fashion.

Metadata Capture

[0035] FIG. 4 shows how a separate, but simultaneous process handles metadata replication. The storage by itself only contains virtual disks 112, or storage for the virtual machines 110. Metadata is required to reconstruct the hosts 104 and to specify the virtual machines 110 that run inside them. Metadata replication is handled by a capture tool 410, which exports data from the management server 102 and serializes it to an metadata XML document, which can be read at a later time. This metadata replication process is periodic; it typically occurs on a similar schedule as a backup process for example, once per day, late at night. It therefore occurs preferably asynchronous to, and separate from, any changes the user might make to the VM configuration during the day. The XML document containing the metadata would then be transferred to, and persisted by, a metadata repository 450 located at or at least separately accessible by the recovery site 350.

[0036] Metadata replication is first handled by a capture tool 410, which exports data from the management server 102 and serializes it to stored form (such as an XML document) which can be read at a later time.

[0037] The capture tool 410 is pre-configured with a list of data elements (metadata), is which describe the private cloud infrastructure. This list is a subset of, but not all of, the available data concerning the private cloud infrastructure. The subset is limited to that data which must be retrieved in order to successfully reconstitute a functional copy of the original private cloud infrastructure at time of test or recovery. The capture tool 410 makes use of application programming interfaces (APIs) provided by the management server 102 to create this a subset profile of the private cloud infrastructure. For example, the capture tool can query the management server 102 for all of the host machines 104 under its control, and then present that list via the API for generating the subset. It will also look for arrangements of hosts 104, such as in clusters or hierarchical folder structures—it may be necessary to make several follow-up queries to get a complete listing of hosts 104 under the management server's 102 control.

[0038] Once a comprehensive list of hosts and host arrangements are generated, the capture tool 410 then inspects each of the hosts 104 for further data. For example, hosts 104 may contain a number of guest machines and may also contain virtual network devices, storage, or other data center resources 111 (as described in FIG. 2 and shown in FIG. 4, for example). The capture tool 410 will thus make further inquiries based on what it finds within each host 104. Each such guest and/or virtual network device will contain further relevant data, which will be captured during these later requests to the management server API. The metadata is therefore possibly further processed by metadata capture tool 410.

[0039] At the conclusion of the capture process, the capture tool 410 operates to construct a document (XML), which is used to transfer the metadata to the recovery site 350.

[0040] XML File with Virtual Infrastructure Metadata

[0041] FIG. 5 illustrates a sample of captured virtual infrastructure metadata XML file 500 contain the information about all of the virtual machines 110 and virtual machine hosts 104 and they way they are interconnected at the protected site 300. The is sample illustrates the collected metadata.

• [0042] 501 Information about the management server 102: address, datacenter name
• [0043] 502 Clusters are groupings of hosts 104 and virtual machines 110, used for fault tolerance and virtual machine movement.
• [0044] 503 Hardware information about a host machine 104.
• [0045] 504 The type of host 104. In this example, the host is VMware ESXi.
• [0046] 505 Network configuration for the host 104.
• [0047] 506 Virtual Switch (running within the host) configuration
• [0048] 507 Physical network interfaces, used by the virtual switch to route network traffic outside of the host 104.
• [0049] 508 Virtual network interface configuration—used by management network traffic.
• [0050] 509 Network configuration for the virtual network interface.
• [0051] 510 Types of traffic, which can flow over the virtual network interface.
• [0052] 511 Port Groups are groupings of virtual ports on a virtual switch.
• [0053] 512 Physical network interface configuration; referenced earlier in line 7.
• [0054] 513 Virtual Machine (or guest) definitions.
• [0055] 514 Resource Pool definitions. Resource pools are used to distribute CPU and Memory resources.

[0056] It should be understood this is a simplified example and an actual metadata file would typically include more information that is not initial to recovery.

[0057] Private Cloud Recovery

[0058] As shown in FIG. 6, time of recovery, the replication between storage devices is broken. This can be planned, in the case if a test, or can be unplanned as a result of a failure at the protected site (e.g., one or more connections in the SAN are broken).

[0059] Turning attention to FIG. 7, a preferred recovery process begins by pulling metadata from the repository 450 and matching available physical machines in the recovery site 350 to physical machines that existed on the protected site. Sample is mapping:

[0060]

Management Server	102	Recovery Target D	610-D
Host A	104-A	Recovery Target E	610-E
Host B	104-B	Recovery Target F	610-F
Host C	104-C	Recovery Target G	610-G

[0061] This mapping can be represented by another file. One example for such file 710 XML mapping the management server 102 and Host A 104-A, is shown below. It shows that role of management server 102 will be given to recovery_D 610-S and the recovery_E 610-E will take the role of Host A 104-A. This also shows the deploy flag, which is set to true by default, meaning that Host A will be recovered.

[0062]

<Site>
<VirtualCenter name=“management_01” targetid=“recovery_D” >
<DataCenter name=“Corporate_IT”>
<Folder name=“Linux Servers” type=“host”>
<Host deploy=“true” name=“host_A.example.com
targetid=“recovery_E”>

[0063] This mapping is then used by automated processes to install software and configure the target hardware 610, as needed. Recovery target servers 610 can be managed as a pool of general resources, and allocated to recovery of a specific recovery site 350 entry upon an actual disaster or disaster test.

[0064] The XML document containing the private cloud metadata may also contain information concerning physical machines which are not directly part of the virtual is infrastructure. For example, a database application may run on a non-virtualized server. This non-virtualized machine may be necessary for the proper operation of applications running within virtual machines but it is not necessary for the operation of the virtual machines themselves more generally. As such, the Host to Recovery target mapping process shown in FIG. 7 may also accommodate physical-machine to physical-machine mapping. Continuing the example, the OS for the physical database server is thus installed via the same automated process that handles the management server and the host machines, but that process is instead carried out on a recovery target 610 that is a dedicated physical machine.

[0065] Automated systems for bare-metal operating system installation 715 often make use of a technique by which the target machine boots from the network, rather than internal storage. The technique used by this system is no different. This presents a challenge when recreating a network which existed in the protected site. In most cases, the network used for deployment on the recovery site 350 will not be compatible with the network to be recreated. If this is not addressed, the import tool 600 that recovers the hosts 610-E, 610-F, 610-G (which is run following bare metal OS installation 715), would be unable to contact the recovered management server 610-D. This problem can be solved by configuring an additional network interface on the recovered management server 610-D, which was not configure on the original management server 102 running in the protected site 300. The result is that recovered management server 610-D has one network interface on the network used for automation and OS installation, and one or more other network interfaces on the network, as specified in the metadata. The recovered management server 610-D is therefore connected to two networks simultaneously, using different network interfaces.

[0066] At this point, the recovery target machines 610 have the required OS or hypervisor software installed, but no storage is connected and the machines are not aware of each other. Further the management recovery machine 610-D has no information about virtual machines or virtual network configuration. The diagram of FIG. 8 shows the next step: connecting storage. There are no lines connecting the management server 610-D to the hosts 610-E, 610-F, 610-G, because this logical is connection has not yet been made.

[0067] During the protected host to recovery host mapping process, it is also possible for an administrative user to choose which virtual machines 110 will be recovered. This is similar, but simpler, to mapping hosts. It only requires a deploy flag to be set to true or false depending upon if the host is to be recovered, no mapping is required. An example representation:

[0068]

	<Guests>
	<VMWareVM deploy=“true” memory=“2048”
	name=“USALPALTP01” vcpus=“1”>
	<description>Prod. Mail Server</description>
	<datastorepath>[us_alp_lcl_esxp01]
	USALPALTP01/USALPALTP01.vmx</datastorepath>

[0069] Finally, a metadata import tool 600 imports metadata 710 from the repository into the management server 610-D. The import tool 600 links the hosts 610-E, 610-F, 610-G with the management server 610-D, makes the management server 610-D aware of the storage, creates any required virtual networks within the hosts and registers virtual machines 110.

[0070] Metadata import 600 is a complement to the metadata capture 410 process. Data must be imported in the correct order or the import 600 will fail. For example, the protected site 300 includes other data processing resources 111 typically needed to implement a functioning Data Center, host clusters, and the like (see FIG. 4). Those other data processing resources 111, when instantiated as recovered resources 611 may need to be imported and brought on line prior to the hosts 104 being recovered. Recovered hosts 610 must also typically be imported prior to certain other resources 611 such as guests, virtual switches, and resource pools. Other storage resources 611 must also typically be configured after hosts but prior to guests. It is also important to avoid attempting to add resources which have already been imported. Virtual switch resources 611, for example, may not be imported twice.

[0071] The metadata import tool 600 thus contains and/or determines the necessary import order taking these considerations into account. For example, if the metadata import tool 600 encounters a pre-existing network item, such as a switch which exists in a default configuration, the metadata import tool 600 switches to a mode where it augments the recovery of that switch rather than attempting to add a duplicate.

[0072] Machines used as hosts 104 in the protected site 300 often contain many network interfaces. But during the early phases of the recovery process, there may be only a single network interface connected on the recovery site 350. More typically a recovered host 610-E will have one or more network interfaces configured to allow remote management of that host 610-E. If the network interface to the recovered management server 610-D, as specified in the metadata, is not physically connected at the time of metadata capture 410 the recovered management server 610-D will have lost a required management connection to that recovered host 610-E. The import tool 600 can detect this situation, prior to attempting the metadata import, and re-arrange the recovered network interfaces in a manner compatible with the original configuration, but ensuring that the recovered management interface is matched to a physically connected port on the recovered host 610-E.

[0073] Referring now to FIGS. 8 and 9, a process for recovering a private cloud virtual infrastructure can therefore proceed as follows.

[0074] At an initial time of recovery there are not yet any target machines assigned to replace the hosts or the management server. Thus the recovery process much first go through a mapping where the metadata is used to map the hosts (as specified by the metadata) to replacement machines available on the recovery site. The replacement machines may be maintained as a resource pool, and only allocated to recovery of a particular site on demand.

[0075] Once this physical machine mapping finishes, the recovery process can then perform an initial bare metal provisioning of each such recovery target. This bare metal provisioning can install software necessary for the recovery target hardware to become virtual machine hosts and/or the management server. The storage arrays at this point are not yet connected and all we have done is to prepare an environment in which to recover the virtual machines.

[0076] Once the bare metal provisioning is complete, storage for the virtual machines as available via the replicated storage portions (provided for example via replication services inherent in the storage area network (SAN) itself) are connections are connected to the respective recovery target machines. At this point the respective recovery target machines still do not have information about the virtual machines in which they are expected to host and/or any connection to the management server.

[0077] Only at this point does the process consider the metadata further via a metadata import process, and first inform the management server about the hosts that are part of its cluster. The management server can then connect to the hosts, and thus allow for recovery of the virtual machines.

[0078] Partial recovery is possible as part of this process. In particular, it may not be desirable or necessary to recovery the entire virtual infrastructure, and the user can specify at the time of recovery, which particular virtual machines to be recovered.

[0079] The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety.

[0080] It should be understood that the example embodiments described above may be implemented in many different ways. In some instances, the various “data processors” described herein may each be implemented by a physical or virtual general purpose computer having a central processor, memory, disk or other mass storage, communication interface(s), input/output (I/O) device(s), and other peripherals. The general purpose computer is transformed into the processors and executes the processes described above, for example, by loading software instructions into the processor, and then causing execution of the instructions to carry out the functions described.

[0081] As is known in the art, such a computer may contain a system bus, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. The bus or busses are essentially shared conduit(s) that connect different elements of the computer system (e.g., processor, disk storage, memory, is input/output ports, network ports, etc.) that enables the transfer of information between the elements. One or more central processor units are attached to the system bus and provide for the execution of computer instructions. Also attached to system bus are typically I/O device interfaces for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer. Network interface(s) allow the computer to connect to various other devices attached to a network. Memory provides volatile storage for computer software instructions and data used to implement an embodiment. Disk or other mass storage provides non-volatile storage for computer software instructions and data used to implement, for example, the various procedures described herein.

[0082] Embodiments may therefore typically be implemented in hardware, firmware, software, or any combination thereof.

[0083] The computers that execute the processes described above may be deployed in a cloud computing arrangement that makes available one or more physical and/or virtual data processing machines via a convenient, on-demand network access model to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Such cloud computing deployments are relevant and typically preferred as they allow multiple users to access computing resources as part of a shared marketplace. By aggregating demand from multiple users in central locations, cloud computing environments can be built in data centers that use the best and newest technology, located in the sustainable and/or centralized locations and designed to achieve the greatest per-unit efficiency possible.

[0084] In certain embodiments, the procedures, devices, and processes described herein are a computer program product, including a computer readable medium (e.g., a removable storage medium such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the system. Such a computer program product can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the is software instructions may also be downloaded over a cable, communication and/or wireless connection.

[0085] Embodiments may also be implemented as instructions stored on a non-transient machine-readable medium, which may be read and executed by one or more procedures. A non-transient machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a non-transient machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; and others.

[0086] Furthermore, firmware, software, routines, or instructions may be described herein as performing certain actions and/or functions. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

[0087] It also should be understood that the block and network diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.

[0088] Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or some combination thereof, and thus the computer systems described herein are intended for purposes of illustration only and not as a limitation of the embodiments.

[0089] While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.