System and method for migrating virtual machines on cluster systems

[0001] The present application is generally related to cluster systems and virtual machines.

[0002] A number of software products are available that “virtualize” computing resources. An example of a virtualization product for the Intel 32-bit computing architecture is the VMware ESX server product. Some virtualization software products operate by executing an operating system (the “host” operating system) on a server platform. The kernel of the host operating system typically includes a virtualization software layer. The virtualization software layer creates software constructs corresponding to hardware components. For example, virtual processors, virtual network interface cards, and virtual storage resources may be instantiated by the virtualization software layer. The number of virtual resources may exceed the physical resources available on the server platform. For example, the number of virtual processors may exceed the number of physical processors by scheduling access to the physical processors (i.e. time sharing).

[0003] The virtual resources are assigned to virtual machines. The virtual machines are used to execute “guest” operating systems as respective processes on top of the host operating system. The guest operating systems, in turn, may be used to execute one or several applications. The virtualization software layer of the host operating system may schedule execution of processes in accordance with the assignment of virtual processors to the virtual machines.

[0004] The creation and operation of virtual machines in this manner provides a number of advantages. Specifically, conventional information technology (IT) implementations involve executing each application on a discrete server platform. When such an approach is selected, the utilization of the server platforms can be relatively low and, hence, resources are essentially wasted. By executing applications in respective virtual machines instead of physical servers, the utilization rate can be much higher. Moreover, administrative costs associated with maintaining multiple server platforms can be avoided.

[0005] Known virtualization software products provide other useful functionality. For example, if two server platforms are coupled to the same storage area network (SAN), some virtualization products enable administrative personnel to transfer an application executing in a virtual machine on one platform to a virtual machine on the other platform. For example, routine maintenance may be performed for a platform and virtual machines executing on that platform can be moved to one or several other platforms. Such a migration can occur if the virtual machines to be migrated store their data using the SAN.

[0006] In one embodiment, a method comprises executing a plurality of virtual machines on a plurality of nodes of a cluster computing system, wherein at least one application is executed within each of the plurality of virtual machines, generating data that is related to performance of applications in the virtual machines, analyzing, by a management process, the data in view of parameters that encode desired performance levels of applications, and migrating, by the management process, a virtual machine on a first node to a second node of the plurality of nodes in response to the analyzing.

[0007] In another embodiment, a cluster system comprises a plurality of cluster nodes that provide hardware elements to support software operations, a plurality of virtual machines executing on the plurality of cluster nodes, wherein at least one application is executed within each of the plurality of virtual machines, and a management process located on at least one of the plurality of cluster nodes, wherein the management process analyzes performance data associated with applications executing within the plurality of virtual machines and migrates virtual machines between the plurality of cluster nodes in response to analysis of the performance data.

[0008] In another embodiment, a computer readable medium for management of applications executed within virtual machines of a cluster system comprises code for generating performance data related to execution of applications within the virtual machines of the cluster system, code for processing the performance data to determine whether applications are satisfying desired operating goals encoded within a set of parameters, and code for migrating virtual machines between cluster nodes of the cluster system in response to the code for processing.

[0012] Some representative embodiments are directed to a Single System Image (SSI) cluster architecture adapted to execute virtual machines. An SSI cluster refers to a collection of systems with hardware and software connectivity that present an “image” of a single node to system administrators and applications. By providing such an image, virtual machines may be flexibly transitioned between cluster members to optimize resource utilization of the cluster resources. Additionally, by employing a single image, the configuration of a virtual machine for a particular workload or application need only occur once for any of the nodes of the cluster. In some embodiments, workload management algorithms are employed to allocate cluster resources. Specifically, application performance data may be analyzed in view of “service level objective” parameters. The analysis may be used to reallocate resources between virtual machines. Additionally, the analysis may be used to transition a virtual machine from one cluster platform to another. By managing virtual machines in a cluster system in this manner, more efficient resource utilization may be achieved.

[0013] Referring now to the drawings, FIG. 1 depicts system 100 according to one representative embodiment. System 100 includes server platforms 110-1 and 110-2. Although only two server platforms are shown in FIG. 1, any number of server platforms 110 may be employed within a cluster architecture according to other embodiments. Each server platform 110 includes host operating system 120 that controls low-level access to hardware layer 130 of the platform. In one embodiment, host operating system 120 includes virtualization layer 121 within its kernel as an example. Virtualization layer 121 creates software constructs (logical devices) that correspond to the physical resources of hardware layer 130 of platform 110. Hardware layer 130 may include any number of physical resources such as CPU(s) 131, memory 132, network interface 133 to access network 160, input/output (I/O) interface 134 to access storage 150, and/or the like.

[0014] In one embodiment, virtual resources (e.g., one or several virtual CPUs, virtual memory, virtual network interface card, virtual I/O interface, and/or the like) are assigned to each virtual machine 141 using configuration files 155. The number of virtual CPUs may exceed the number of physical CPUs 131. Host operating system 121 may schedule the execution of the processes associated with virtual machines 141 on physical CPUs 131 in accordance with the assigned virtual CPUs.

[0015] Each virtual machine 141 is executed as a process on top of guest operating system 120 in accordance with its assigned virtual resources. CPU virtualization may occur in such a manner to cause each virtual machine 141 to appear to run on its own CPU or set of CPUs. The CPU virtualization may be implemented by providing a set of registers, translation lookaside buffers, and other control structures for each virtual CPU. Accordingly, each virtual machine 141 is isolated from other virtual machines 141. Additionally, each virtual machine 141 is used to execute a respective guest operating system 142. The virtual resources assigned to the virtual machine 141 appear to the guest operating system 142 as the hardware resources of a physical server. Guest operating system 142 may, in turn, be used to execute one or several applications 143. Each guest operating system 142 may be individually tuned and/or patched according to the characteristics of its associated applications 143.

[0016] Typical disk virtualization is implemented by creating a respective file on a network storage device for each virtual disk. The guest operating systems 142 access the virtual disks through conventional operations and the host operating system 120 translates the disk operations to access the corresponding files. In some embodiments, disk virtualization is performed using a cluster file system as will be discussed below.

[0017] Network virtualization may be implemented by creating virtual network cards having respective medium access control (MAC) addresses and Internet Protocol (IP) addresses. The virtual network cards may be mapped to a dedicated network interface 133 or virtual network interfaces from multiple virtual machines 141 may share a single network interface 133.

[0018] Host operating system 120 comprises cluster management software 123 to support cluster functionality. The cluster functionality enables multiple independent physical systems to be operated and managed as a single system. Cluster management software 123 may form a cluster, add members to a cluster, and remove members from a cluster. Cluster management software 123 may also present a comprehensive view of the resources of the entire cluster.

[0019] In some representative embodiments, cluster management software 123 implements Single System Image (SSI) functionality. An SSI cluster refers to a collection of server platforms 110 with hardware and software connectivity that present an “image” of a single node to system administrators and applications. For example, storage 150 may include direct attached devices and network attached devices. Cluster management software 123 may control access to the discrete devices of storage 150 to cause storage 150 to appear as a single file system having a single root. The shared single root enables a single copy of suitable configuration files to be used for each server platform 110. Accordingly, administrative activities may be simplified.

[0020] Referring to FIG. 2, a single system image functionality may be implemented using a number of software layers. As shown in FIG. 2, an application 143 in a virtual machine 141 may perform file operations using conventional functionality via its guest operating system 142. Guest operating system 142 performs file operations by accessing the virtual I/O interfaces and storage devices assigned to its virtual machine 141. The virtual storage resources are created from the unified cluster file system presented by cluster file system (CFS) software layer 201. Accordingly, a file located on any particular storage device 151 of storage 150 may be accessed from any server platform 110 using the same filename.

[0021] Specifically, CFS 201 may present a unified cluster file system by creation of a single, clusterwide namespace for files and directories. CFS 201 is layered upon traditional file system software modules (shown as local file system 202) such as “AdvFS” modules. When implemented as a distributed kernel component, CFSs 201 on the various platforms 110 collectively assemble the various individual file system domains into a single, unitary namespace underneath a common root. Additionally, the distributed nature of CFSs 201 may maintain cache coherency across cluster members to ensure that all cluster members have the same view of the file system. Local file system 202 performs file operations using conventional processing. Device request dispatch module 203 controls I/O to physical devices 151 through device drivers 204. Device request dispatch module 203 may enforce single-system open semantics so only one program can open a device at any one time. Additionally, device request dispatch module 203 may be implemented so that it may make physical disk and tape storage available to all cluster members regardless of where the storage is physically located.

[0022] Additional details related to SSI clusters and cluster file systems may be found in the “TruCluster Server: Cluster Technical Overview” published by Hewlett-Packard Company, September 2002, which is incorporated herein by reference.

[0023] Referring again to FIG. 1, system 100 further comprises workload management software modules. For example, within each virtual machine 141, a respective performance monitor 144 may be executed. Performance monitor 144 is a software process that monitors operations associated with application(s) 143 to generate data that is indicative of whether each application 143 is performing in an expected manner and/or whether allocation of additional resources would be beneficial. For example, performance monitor 144 may examine the length of time required to perform certain types of transactions (e.g., the length of time associated with a particular database query). Additionally or alternatively, performance monitor 144 may examine the utilization rates associated with the virtual resources assigned to its respective virtual machine 141.

[0024] System 100 further includes global workload manager (gWLM) 122. In one embodiment, gWLM 122 obtains performance data from performance monitors 144. Additionally or alternatively, gWLM 122 may obtain related performance data from host operating system 120. gWLM 122 analyzes the received performance data in view of service level objectives (SLOs). SLOs are parameters that encode the desired operating goals of applications 143. For example, an SLO may be encoded to specify that an application 143 should perform a given type of database transaction within 10 milliseconds. Additionally or alternatively, an SLO may be defined so that the utilization rate associated with a virtual machine 141 should remain below 85%.

[0025] If a particular application 143 is not achieving the respective SLO(s), gWLM 122 may allocate additional resources to the respective virtual machine 141 of the application. As previously mentioned, virtual machines 141 are processes executed on top of host operating system 120. Host operating system 120 schedules the execution of processes according to entitlement parameters associated with virtual machines 141. To allocate additional resources to a particular virtual machine 141, gWLM 122 may effect a change in the entitlement parameter associated with the virtual machine 141 through a suitable system call to virtualization layer 121. In response, host operating system 120 increases the relative scheduling of the execution of the process associated with the respective virtual machine 141. Specifically, the respective process receives a greater number of “time slices” of one or several physical CPUs 131 thereby improving the performance of the application 143.

[0026] It is possible that sufficient physical resources may not be available for allocation when gWLM 122 determines that a particular application 143 is underperforming. Sufficient virtual resources may be assigned to virtual machines 144 such that substantially all physical resources of a respective server platform 110 are consumed. If sufficient physical resources are not available, gWLM 122 may reallocate resources between virtual machines 141. Specifically, gWLM 122 may decrease the physical resources assigned to another virtual machine 141 and concurrently increase the physical resources assigned to the virtual machine 141 that has the underperforming application 143.

[0027] To perform the reassignment of physical resources, several tiers of SLOs may be defined for applications 143. Each tier of SLOs may be associated with a different amount of resources. Also, each tier may be associated with a priority level. By encoding SLOs in this manner, gWLM 141 may reassign resources to achieve the highest tier of SLOs possible given the existing workloads. Accordingly, the multiple tiers of SLOs enable allocation decisions to be made for multiple applications 143 experiencing dynamically varying workloads. Additional details regarding the use of multiple tiers of SLOs to assign resources may be found in U.S. patent application Ser. No. 10/206,594, entitled “DYNAMIC MANAGEMENT OF VIRTUAL PARTITION COMPUTER WORKLOADS THROUGH SERVICE LEVEL OPTIMIZATION,” filed Jul. 16, 2002, now U.S. Pat. No. 7,140,020, which is incorporated herein by reference.

[0028] In one embodiment, gWLM 122 may move a virtual machine 141 from one platform 110 to allocate additional resources to the virtual machine 141. For example, gWLM 122 may utilize cluster management software 123 to obtain information related to the resources available on each server platform 110, the virtual machines 141 present on each platform 110, the applications 143 executing within the virtual machines 141, and/or other suitable information. When an application 143 is underperforming and additional resources on its server platform 110 are not currently available, gWLM 122 may examine the resources available on the other platform 110. If resources are available on the other platform 110, gWLM 122 may move the respective virtual machine 141 to the other platform 110.

[0029] It is possible to move a virtual machine 141 between platforms 110, because the virtual machines 141 are executed as processes on top of host operating system 120. Specifically, the execution of a virtual machine 141 to be moved may be temporarily interrupted. The virtual processor state may be saved. Corresponding virtual resources may be created on the other platform 110 and a virtual machine process created on top of the other host operating system. The execution of the virtual machine may then be resumed using the saved processor state. The process associated with the originating platform may be terminated. Additionally, because each virtual machine 141 is allocated a virtual network interface and responds to the same network address independently of the supporting server platform 110, the transition between platforms 110 is transparent to applications 143 and client platforms (not shown).

[0030] The transition of virtual machines 141 between server platforms 110 using system 100 may occur in an advantageous manner. For example, because system 100 is an SSI system, local storage facilitates (attached storage devices) may be used by applications 143. Specifically, the cluster file system presents a consistent and unified view of storage 150 associated with the entire cluster system 100. If an application 143 uses a virtual disk associated with a local storage device 151, the cluster file system 201 enables the same virtual disk to be accessed even when the virtual machine 141 is moved between platforms 110.

[0031] FIG. 3 depicts a flowchart for managing virtual machines in a cluster system according to one representative embodiment. In block 301, performance data is generated for applications executed in a number of virtual machines. The performance data may be generated by a performance monitoring software process within the virtual machines. Additionally or alternatively, performance data may be obtained using operating system calls. The generated performance data enables the evaluation of application performance to be performed.

[0032] In block 302, the performance data is analyzed in view of service level objectives. As previously mentioned, service level objectives are parameters that encode the desired operating goals of the applications. In block 303, a logical comparison is made to determine whether the applications are meeting the SLOs. If so, the process flow returns to block 301 for continued operation.

[0033] If the applications are not meeting the SLOs, the process flow proceeds to block 304 where a logical comparison is made to determine whether sufficient resources are available on the local system to achieve the SLOs. If so, the process flow proceeds to block 305 where resources are allocated or reallocated to achieve the SLOs. Specifically, if unassigned resources are available, the unassigned resources may be allocated to the virtual machine(s) that are associated with underperforming applications. Alternatively, if a first application is “overachieving,” underutilized or idle resources may be reallocated from the virtual machine that has the overachieving application to the virtual machine that has the underperforming application. As previously mentioned, reassignment of processor resources for virtual machines may involve changing the processor scheduling associated with the virtual machines. The scheduling may occur using parameters that define the amount of “processor slices” given to each virtual machine. Accordingly, the reallocation of resources may involve changing the relative parameters of the various virtual machines. From block 305, the process flow returns to block 301.

[0034] If the logical comparison of block 304 determines that local resources are insufficient, the process flow proceeds from block 304 to block 306. In block 306, resource data of other cluster members is obtained. In block 307, a logical comparison is made to determine whether resources are available on other cluster members to enable SLOs to be achieved. If not, the process flow returns to block 301. If resources are available, the process flow proceeds to block 308. In block 308, the virtual machine containing the underperforming application is migrated to another cluster member and appropriate resources are provided to the migrated virtual machine. From block 308, the process flow returns to block 301.

[0035] When implemented in software, the elements of some representative embodiments, such as the operations of the flowchart shown in FIG. 3, are essentially the code segments to perform the necessary tasks. The program or code segments can be stored in a computer readable medium or transmitted by a computer data signal embodied in a carrier wave, or a signal modulated by a carrier, over a transmission medium. The “computer readable medium” may include any medium that can store or transfer information. Examples of the computer readable medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory, an erasable ROM (EROM), a floppy diskette, a compact disk CD-ROM, an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, etc. The computer data signal may include any signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic, RF links, etc. The code segments may be downloaded via computer networks such as the Internet, intranet, etc.

[0036] Some representative embodiments may provide a number of advantageous characteristics. For example, the allocation of resources to workloads may occur without any changes to applications that service the workloads. Specifically, virtual machines are virtualized containers that appear to application software as a separate server. Because all of the resources being consumed by a respective virtual machine are shared among multiple virtual machines, it is possible to change the resource entitlements in a completely transparent manner to the application and virtual machines.

[0037] The ability to manage virtual machines according to service level objectives enables more efficient use of resources. Specifically, when resources are shifted between virtual machines and virtual machines are migrated between cluster members, resource utilization can be maximized in response to dynamically changing workloads. Accordingly, a smaller number of resources (processors, memory, etc.) may be used to support the same number of applications than would otherwise be employed.

[0038] Additionally, the use of the SSI functionality of a cluster system enables attached storage to be used. Specifically, the cluster file system enables transparent access to both direct attached and network attached storage devices. A file implementing a virtual disk may be accessed from any cluster member without regard to the physical location of the file. Accordingly, if a virtual machine is migrated between cluster members, any storage that the virtual machine was using on the previous cluster member will continue to be available.

[0039] Administration of an SSI cluster system adapted to manage virtual machines is an efficient process, because all of the cluster members may be managed as a single entity. Typical management tasks can be performed once for all cluster members. For example, configuration file of a virtual machine to support a given workload need only occur once and the virtual machine may be placed anywhere within the cluster system.