Method and apparatus for providing at risk information in a cloud computing system having redundancy

[0001] As is known in the art, cloud computing systems contain a large number of hardware devices, components, software applications, and modules. For example, cloud computing systems can include integrated compute, network, and storage domains. In typical systems, redundancy is built into each domain to ensure services are not disrupted because of hardware failures. For example, two fabrics and two switches provide multiple paths for each server blade to get to network or storage resources. In the storage domain, RAID technology with hot spare disks prevents data loss due to disk failures.

[0002] Traditionally, the redundancy on a group of resources has been measured using a fractional threshold, e.g., 2 out of 4 links are down. One example of using this method is the analytic model in the EMC ITOI SMARTS domain manager. In other known systems, redundancy is classified into categories, such as redundancy lost, redundancy degraded, etc., based on the number of available resources. A classification of this type appears in some VMware vCenter events such as storage path redundancy events. Some conventional systems use analytics that measure risk by predicting the headroom in terms of capacity. This technology is being used in the VMware Capacity IQ, for example. However, this concerns a different aspect of resources, namely, capacity, but not redundancy or availability.

[0003] In one aspect of the invention, a method comprises: in a pre-integrated cloud computing system having a compute domain, a network domain, a storage domain, a virtual layer and a management layer, to provide services, the cloud computing system providing redundancy, determining an at risk probability in the compute domain, determining an at risk probability for storage volumes in the storage domain, determining an at risk probability for storage paths to the storage domain from the compute domain, and determining an overall at risk probability for a first one of the services using system resources.

[0004] The method can further include one or more of the following features: providing an overall risk index for the first one of the services from the overall risk probability, using a logarithm to generate the overall risk index, the at risk probability for the compute domain is derived from a number of blades used by the first one of the services, a failover rate relating to a number of servers that can fail without putting a compute cluster service into an overcommitted state, an annual failure rate for the blades used by the first one of the services, and an amount of time to replace one of the blades used by the first one of the services, the at risk probability for the storage volumes is derived from a number of storage volumes used by the first one of the services, a number of disks in the storage volumes, a number of disks that can fail in the storage volumes without data loss, an annual disk failure rate, and an amount of time needed to convert a hot spare disk to a contributing disk of the storage volume, the at risk probability for the storage paths is derived from a number of storage volumes used by the first one of the services, a number of blades used by the first one of the services, a number of cables used by the storage volumes, an annual failure rate for the cables, and an amount of time to replace a failed one of the cables, determining a revised overall risk index after one or more component failures, and/or determining a first risk index from the at risk probability in the compute domain, determining a second risk index from the at risk probability for storage volumes in the storage domain, determining a third risk index from the at risk probability for storage paths to the storage domain from the compute domain, and generating a revised first, second, third risk index for one or more component failures to assist a user in selecting a first one of the one or more components to fix first.

[0005] In another aspect of the invention, an article comprises: a computer readable medium containing non-transitory stored instructions that enable a machine to perform, in a pre-integrated cloud computing system having a compute domain, a network domain, a storage domain, a virtual layer and a management layer, to provide services, the cloud computing system providing redundancy, determining an at risk probability in the compute domain, determining an at risk probability for storage volumes in the storage domain, determining an at risk probability for storage paths to the storage domain from the compute domain, and determining an overall at risk probability for a first one of the services using system resources.

[0006] The article can further include one or more of the following features: instructions for providing an overall risk index for the first one of the services from the overall risk probability, instructions for using a logarithm to generate the overall risk index, the at risk probability for the compute domain is derived from a number of blades used by the first one of the services, a failover rate relating to a number of servers that can fail without putting a compute cluster service into an overcommitted state, an annual failure rate for the blades used by the first one of the services, and an amount of time to replace one of the blades used by the first one of the services, the at risk probability for the storage volumes is derived from a number of storage volumes used by the first one of the services, a number of disks in the storage volumes, a number of disks that can fail in the storage volumes without data loss, an annual disk failure rate, and an amount of time needed to convert a hot spare disk to a contributing disk of the storage volume, the at risk probability for the storage paths is derived from a number of storage volumes used by the first one of the services, a number of blades used by the first one of the services, a number of cables used by the storage volumes, an annual failure rate for the cables, and an amount of time to replace a failed one of the cables, including instructions for determining a revised overall risk index after one or more component failures, and/or instructions for determining a first risk index from the at risk probability in the compute domain, determining a second risk index from the at risk probability for storage volumes in the storage domain, determining a third risk index from the at risk probability for storage paths to the storage domain from the compute domain, and generating a revised first, second, third risk index for one or more component failures to assist a user in selecting a first one of the one or more components to fix first.

[0007] In a further aspect of the invention, a pre-integrated cloud computing system having a compute domain, a network domain, a storage domain, a virtual layer and a management layer, to provide services, the cloud computing system providing redundancy, comprises: a processor, and a memory containing instructions to enable the processor to: determine an at risk probability in the compute domain, determine an at risk probability for storage volumes in the storage domain, determine an at risk probability for storage paths to the storage domain from the compute domain, and determine an overall at risk probability for a first one of the services using system resources.

[0018] FIG. 1 shows an exemplary cloud computing environment 100 having a redundancy risk index in accordance with exemplary embodiments of the invention. The environment includes an infrastructure layer 112, a virtual layer 114 and a management layer 108. The infrastructure layer is further divided into a compute domain 102, a network domain 104, and a storage domain 106. The management layer 108 contains a risk index module 110, which is described more fully below. The environment may be referred to as a platform. It is understood that any practical number of platforms can be combined into a cloud storage environment. It is further understood that the risk index module 110 can be provided in any practical location within the system.

[0019] The compute domain 102 comprises components, such as blade servers, chassis and fabric interconnects that provide the computing power for the platform. The storage domain 106 comprises the storage components for the platform. The network domain 104 comprises the components that provide switching and routing between the compute and storage domains 102, 106 within and between platforms, and to the client or customer network.

[0020] FIG. 2 shows further detail for the environment 100 of FIG. 1. The storage domain 106 can include storage components 150, such as CLARIION storage components from EMC Corporation of Hopkinton Mass. The network domain 104 can include a pair of switches 152, such as MDS 9000 Series Multilayer SAN Switches from Cisco of San Jose, Calif., coupled to the storage components and to a LAN. The compute domain 102 can include a pair of fabric interconnects 154, such as CISCO 6100 series devices. The compute domain can further include a number of blade servers 156, such as CISCO 5100 blade chassis.

[0021] FIG. 3 shows further detail of an exemplary cloud environment having a compute domain 302, a network domain 304 and a storage domain 306. The network domain 302 is coupled to a customer network 308 in a manner known in the art. The network domain 302 includes switches 310 coupled to the customer network 308. The network domain 302 also includes multilayer fabric switches 312 coupled to fabric interconnects 314 in the compute domain 302 and to storage processors 316 in the storage domain 306. The fabric interconnects 314 are coupled to blade server chassis 318 containing blades. Data movers 320 in the storage domain 306 are coupled between the storage processors 316 and the switches 310 in the network domain. Disk array enclosures 322 are coupled to the storage processors 316. FIG. 4 shows interconnections for a system similar to that shown in FIG. 3 with physical disks 326. In the illustrated embodiment, the storage layer includes 8 to 16 front end fibre channel ports and 2-4 GB iSCSI front end ports.

[0022] It is understood that a variety of other configurations having different interconnections and storage configuration can be provided to meet the needs of a particular application.

[0023] The management layer can include a number of applications to perform various functions for overall control, configuration, etc of the various platform components. A management application can be provided as the Unified Computing System (UCS) Manager by Cisco. It is understood that the blade chassis and fabric interconnection can be considered part of the UCS. Another management application can include a management interface, such as EMC UNISPHERE, to provide a flexible, integrated experience for managing existing storage systems, such as CLARIION and CELERRA storage devices from EMC. A further management application includes a platform element manager, such as Unified Infrastructure Manager (UIM) by EMC, for managing the configuration, provisioning, and compliance of the platform.

[0024] FIG. 5 shows an exemplary unified infrastructure manager 500 having a risk redundancy module 501 in accordance with exemplary embodiments of the invention. In one embodiment, the unified infrastructure manager 500 includes a configuration center module 502, a provisioning center module 504, and an operations center module 506. Below these modules, is a platform infrastructure service catalog 506 and a cross domain context and visibility module 508.

[0025] The unified infrastructure manager 500 further includes a change and configuration management module 510, a policy-based compliance and analysis module 512, a unified infrastructure provisioning module 514, a consolidation topology and event service module 516, and an operational awareness module 518. The various modules interact with platform elements, such as devices in compute, network and storage domains, and other management applications.

[0026] The unified infrastructure manager 500 performs platform deployment by abstracting the overall provisioning aspect of the platform(s) and offering granular access to platform components for trouble shooting and fault management.

[0027] It is understood that various vendor specific terminology, product name, jargon, etc., may be used herein. It is further understood that such vendor specific information is used to facilitate an understanding of embodiments of the invention and should not limit the invention in any way. Any specific vendor information should be construed mean a generic product, function, or module.

[0028] Some exemplary items are set forth below. It is understood that one of ordinary skill in the art is familiar with the generic architecture and functionality of a vendor specific terms.

[0029] UIM/Provisioning or UIM/P: EMC Unified Infrastructure Management/Provisioning that provides simplified management for VCE VBLOCK by managing the components of VBLOCK platforms as a single entity and easily define and create infrastructure service profiles to match business requirements.

Cisco UCS: Cisco Unified Computing System.

VMWARE VSPHERE: A virtualization platform for building cloud infrastructures

ESX/ESXi: An enterprise-level computer virtualization product offered by VMware.

VM: Virtual Machine

VBLOCK: A pre-architected and pre-qualified environment for virtualization at scale: storage, fabric, compute, hypervisor, management and security.

[0030] In one aspect of the invention, a cloud computing system includes risk quantification by evaluating the failure probability of a service or application in a Vblock, for example. The failure probability is converted into a risk index of scale 0 to 10, for example, to enable a user to easily assess the risk level. In one particular embodiment, a value of 5.5 indicates a failure probability between 10^-5 (a five-9s quality) and 10^-6 (a six-9s quality).

[0031] With this arrangement, a risk index renders the redundancy in each domain measurable and comparable, thereby helping balance resources across domains. In addition, a normalized risk index can be used as a health indication for the Vblock, service or application. Also, risk information can be used to provide an effective way to evaluate the severity of a failure, which can be measured as the change to the risk index associated with a failure.

[0032] FIG. 6 shows an exemplary system 100, such as a Vblock, having redundancy risk evaluation in accordance with exemplary embodiments of the invention. The system 100 includes physical and virtual redundancy in each domain. As can be seen, the UCS service VMs are running on server blades 604, each of which is connected to first and second fabrics 606a,b. If one of the fabrics 606a,b fails, or a fabric port fails, the availability of the services running on the blades 604 is not affected as long as the other fabric is functioning.

[0033] There are multiple technologies used to protect a storage volume from being unavailable due to disk failures. For example, Redundant Array Inexpensive Disk (RAID) technology replicates data using an n+1 coding such that the data can be computed given any of the n instances. This ensures the volume survives with a single disk failure.

[0034] In the case where a disk fails and the volumes lose their n+1 protection, the failed disk will be replaced by a hot spare if one is available. The process of building the hot spare from the remaining n disks can take a significant amount of time. However, only if another disk fails during this period of time are the volumes affected.

[0035] FIG. 7 shows a simplified block diagram of cloud system 700 having a storage layer 702 coupled to a UCS blade 704 via fabric interconnects 706a,b and SAN switches 708a,b. A storage path 710 is for storage traffic between an ESX 712 and a storage volume 714. There are multiple storage paths between the ESX 712 and the storage volume 714. In the illustrative embodiment, there are four storage paths between the ESX 712 and the storage volume 714. Each storage path goes through UCSBlade 704 to UCSFabric 706 to MDS 708 to StorageController 716, via three stages of cables. As can be seen each cable stage provides redundancy between the components.

[0036] In an exemplary embodiment, a cluster of ESX servers is formed to deliver compute services. A cluster is equipped with High Availability (HA) capability to mitigate the effects of ESX failures. If an ESX server fails, VMs running at that ESX will be restarted in a different ESX server to continue service. It is possible that, after some ESX server fails, the overall computing power of a cluster can no longer meet the configured needs. In this case, an overcommitted warning is raised.

[0037] In a UIM service model, infrastructure resources of multiple domains are treated as a single service. A UIM service is a collection of resources in each of the compute, network and storage domains. Logically, a UIM service includes an exclusive set of ESXs, an exclusive set of storage volumes, and a set of network resources shared with other services in the same Vblock.

[0038] A service is considered to be at risk if: (storage) disks are down causing any storage volume to fail and lose data; (network) storage paths are down causing any storage volume to become inaccessible; or a (compute) ESX is down causing the HA-enabled cluster to become overcommitted. Thus, the failure probability of a service is the sum of the failure probability in each domain.

[0039] FIG. 8 shows an exemplary at risk evaluation in accordance with exemplary embodiments of the invention. In step 800, the at risk probability is computed for the compute domain. In step 802, the at risk probability is computed for the storage volumes in the storage domain. In step 804, the at risk probability is computed for storage paths.

[0040] In step 806, the at risk probabilities for the compute domain 800, the storage volumes 802, and the storage paths 804 are summed to determine an overall at risk probability for the UIM service in step 808. In step 810, the logarithm of the overall at risk probability is taken to generate a risk index for the UIM service in step 812. Exemplary at risk probability computations for the compute, storage volumes, and storage paths are set forth below.

[0041] In an exemplary embodiment, the probability that a service is at risk in the storage domain is given by:

[0042] $\sum _{i=1}^v\left(\begin{array}{c}{n}_i\\ k_i+1\end{array}\right)·r·{\left(\frac{\mathrm{mr}}{365⨯24⨯60}\right)}^{k_i}$
where v is the number of storage volumes used by this service, n, is the number of disks in each of the storage volumes, k_i is the number of disks that could fail in the storage volume without data loss, 0≦k_i≦n_i, r is the annual failure rate of a disk, and m is the number of minutes it takes to convert a hot spare to a contributing disk of a storage volume.

[0043] There is considerable information relating to the annual failure rate for disk drives, which serves as the basis to estimate the failure rate r for storage volumes. Depending on which RAID technology is used, the volume failure rate without hot spare can vary. For example, the failure rate is n(n−1)r² for RAID5-based volumes that are tolerant of a single disk failure.

[0044] Assume that a hot spare takes m minutes to build. A storage volume will fail only if other k disks fail during this period of time. The chance that a disk fails in an m-minute interval is:

[0045] $\frac{m·r}{365*24*60}$

[0046] For any combination of (k+1) disks in a storage volume, the probability that they all fail within an m-minute interval in a given year is then given by:

[0047] $r·{\left(\frac{m·r}{365*24*60}\right)}^k$

[0048] In a storage volume of n disks, there are

[0049] $\left(\begin{array}{c}{n}_i\\ k+1\end{array}\right)\hspace{1em}$
combinations of (k+1) disks. Therefore, we have the annual failure rate of a storage volume without hot spare as:

[0050] $\left(\begin{array}{c}{n}_i\\ k+1\end{array}\right)·r·{\left(\frac{m·r}{365*24*60}\right)}^k$

[0051] If n=5, k=1, r=0.1, m=5, the annual failure probability of a storage volume will be 1.42e-6, or 1.4 in one million. For a service with four storage volumes, the overall at risk probability with storage volumes is 5.68e-6.

[0052] In one embodiment, the probability that a service is at risk in the storage path domain is given by:

[0053] $b·v·\sum _{i=1}^s{p}_i·{\left(\frac{p_i·h}{365⨯24}\right)}^{c_i-1}$
where v is the number of storage volumes used by this service, b is the number of ESX servers (blades) used by this service, c_i is the number of cables in each of the storage stages, p_i is the annual failure rate of a cable in stage i, and h is the number of hours it takes to replace a defected cable.

[0054] An ESX will lose its connectivity to a storage volume if the cables in one of the stages are all down. Suppose that a storage path includes s stages in tandem. In stage i, there are a group of c_i cables. Denote, the annual failure rate p_i of a cable, and suppose it on average takes h hours to physically replace a cable. The probability that all cables in stage i are down within the same h hours in a given year is:

[0055] $p_i·{\left(\frac{p_i·h}{365⨯24}\right)}^{c_i-1}$

[0056] Then, the overall annual probability that an ESX will lose connectivity to a storage volume is given by:

[0057] $\sum _{i=1}^s{p}_i·{\left(\frac{p_i·h}{365⨯24}\right)}^{c_i-1}$

[0058] In an exemplary Vblock environment, s=3, c₁=2, c₂=2, c₃=4. If all p_i is 0.005 and h is 3 hours, then the annual rate that an ESX will lose connectivity with a storage volume is 2.56e-8. For a service with eight blade servers and four storage volumes, there are thirty-two ESX-Volume combinations. The overall risk probability related to all the storage paths is approximately 2.56e-8×32=8.22e-7.

[0059] In one embodiment, the probability that a service is at risk in the compute domain, i.e. the compute cluster becomes overcommitted, is given by:

[0060] $\left(\begin{array}{c}b\\ a+1\end{array}\right)·q·{\left(\frac{q·d}{365}\right)}^a$
where b is the number of ESX servers (blades) used by this service, a is the failover tolerate, i.e. the number of servers that could fail without putting the compute cluster service into overcommitted state, q is the annual failure rate of a blade/ESX server, and d is the number of days it takes to replace a blade. For an 8-ESX cluster with failover tolerate set to 1, a=1 and b=8. If q=0.2% and d=2, then the probability the service is at risk in the compute domain is about 6.12e-7.

[0061] In an exemplary embodiment, the raw risk data is manipulated to enable a user to more easily understand the risk information. In one embodiment, the risk index is obtained by taking the logarithm of the at risk probability. The larger the index, the better assurance the service is given. Table 1 below summarizes previous calculations of the at risk probability in each domain and the overall risk index for a UIM service.

[0062]

TABLE 1
			Network
			(Storage
Domain	Compute	Storage	Path)
Resources &	8-ESX cluster;	4 volumes from a 4 + 1	4 storage
Configuration	Host failover tolerate	raid-5 group	paths
	set to 1		between
			an ESX
			and a
			volume
Annual	Blade/ESX - 0.002	Disk - 0.1	Cable -
Failure Rate			0.005
Recovery time	2 days	5 minutes	3 hours
At-risk	6.12e − 7	5.68e − 6	8.22e − 7
probability
Risk index	Overall failure probability:
	6.12e − 7 + 5.68e − 6 + 8.22e − 7 = 7.11e − 6
	Risk index = 5.15

[0063] By comparing the at risk probability at each domain, it can be seen in this example that storage is the most at risk domain. Given this information, it may be worth considering ways to improve and/or balance the risk.

[0064] In a system where there are multiple failures present, one would want to know the severity of each failure, based on which the priority of remediation can be defined. For example, assume there are three failures in the system: a blade fails, a disk fails, and a UCS-MDS cable is down. In an exemplary embodiment, the risk indexes can be used to determine which failure should be addressed first.

[0065] Table 2 below illustrates a situation in which three failures are present. It recalculates the at risk probability in each domain after the failure, and evaluates the difference in terms of the risk index in that domain. The bigger the difference in risk index, the more severe the failure is. In this example, the disk failure introduces the biggest risk.

[0066]

	TABLE 2
				Network
				(Storage
	Domain	Compute	Storage	Path)
	Original At-	6.12e − 7	5.68e − 6	8.22e − 7
	risk probability
	Equivalent	6.21	5.24	6.09
	Risk Index
	Example	A blade fails	A disk fails	A cable
	failure scenario			between
				UCS and
				MDS is
				down,
				affecting
				4 ESX-
				volume
				combinations
	New at-risk	0.014	0.344	0.02
	probability
	New Risk	1.85	0.46	1.70
	Index
	Change in Risk	6.21 − 1.85 =	5.24 − 0.46 =	6.09 − 1.70 =
	Index (=	4.36	4.78	4.39
	Impact)

[0067] FIG. 9 shows an exemplary computer that can perform at least a part of the processing described herein. A computer includes a processor 902, a volatile memory 904, an output device 905, a non-volatile memory 906 (e.g., hard disk), and a graphical user interface (GUI) 908 (e.g., a mouse, a keyboard, a display, for example). The non-volatile memory 906 stores computer instructions 912, an operating system 916 and data 918, for example.

[0068] In one example, the computer instructions 912 are executed by the processor 902 out of volatile memory 904 to perform all or part of the processing described above. An article 919 can comprise a machine-readable medium that stores executable instructions causing a machine to perform any portion of the processing described herein.

[0069] Processing is not limited to use with the hardware and software described herein and may find applicability in any computing or processing environment and with any type of machine or set of machines that is capable of running a computer program. Processing may be implemented in hardware, software, or a combination of the two. Processing may be implemented in computer programs executed on programmable computers/machines that each includes a processor, a storage medium or other article of manufacture that is readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. Programs may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs may be implemented in assembly or machine language. The language may be a compiled or an interpreted language and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. A computer program may be stored on a storage medium or device (e.g., CD-ROM, hard disk, or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform processing.

[0070] One skilled in the art will appreciate further features and advantages of the invention based on the above-described embodiments. Accordingly, the invention is not to be limited by what has been particularly shown and described, except as indicated by the appended claims. All publications and references cited herein are expressly incorporated herein by reference in their entirety.