US 10,205,677 B2Grant
Cloud resource placement optimization and migration execution in federated clouds
Issue Date:2019-02-12
•20 Claims
•8 Drawing Sheets
Abstract
The present disclosure describes a method for cloud resource placement optimization. A resources monitor monitors state information associated with cloud resources and physical hosts in the federated cloud having a plurality of clouds managed by a plurality of cloud providers. A rebalance trigger triggers a rebalancing request to initiate cloud resource placement optimization based on one or more conditions. A cloud resource placement optimizer determines an optimized placement of cloud resources on physical hosts across the plurality of clouds in the federated cloud based on (1) costs including migration costs, (2) the state information, and (3) constraints, wherein each physical host is identified in the constraints-driven optimization solver by an identifier of a respective cloud provider and an identifier of the physical host. A migrations enforcer determines an ordered migration plan and transmits requests to place or migrate cloud resources according to the ordered migration plan.
Metadata
Assignee
- CISCO TECHNOLOGY, INC.
Inventors
- Yathiraj B. Udupi
- Xinyuan Huang
- Aparupa Das Gupta
- Marc Solanas Tarre
- Rishabh Jain
Application Information
Application Number:US 14/951,110
Filing Date:2015-11-24
Priority Date:2015-11-24
Art Unit:2449
Classifications
IPC:
H04L29/08H04L12/911
Patent Drawings (8 sheets)
Description
Technical Field
[0001] This disclosure relates in general to the field of computing and, more particularly, to cloud resource placement optimization and migration execution in federated clouds.
Background
[0002] Cloud computing allows cloud providers to pool a collection of physical computing devices and enables the pool of computing devices to be shared and used on demand by a plurality of tenants. Clouds can provide Infrastructure as a Service (IaaS), where virtual resources are provided in the cloud and made accessible to tenants as a service. Tenants, such as users and enterprises, can access virtual machines, virtual storage volumes, and even run applications on demand in the cloud. Cloud providers incur significant capital expenses and operating expenses when implementing and maintaining the pool of resources. Tenants and cloud providers often have high demands for performance. Accordingly, cloud providers are always looking for ways to improve the efficiency and performance of the cloud so that resources are used efficiently and in a cost-effective manner.
Brief Description of the Drawings
[0003] To provide a more complete understanding of the present disclosure and features and advantages thereof, reference is made to the following description, taken in conjunction with the accompanying figures, wherein like reference numerals represent like parts, in which:
[0004] FIG. 1 shows an exemplary cloud resource placement and rebalancing system for a federated cloud and an illustrative federated cloud, according to some embodiments of the disclosure;
[0005] FIG. 2 is a flow diagram illustrating a method for cloud resource placement and migration optimization for a federated cloud, according to some embodiments of the disclosure;
[0006] FIG. 3 is an illustration of federated cloud resource placement and rebalancing data flow, according to some embodiments of the disclosure;
[0007] FIG. 4 shows initial virtual machine placements, according to some embodiments of the disclosure;
[0008] FIG. 5 shows desired virtual machine placements, according to some embodiments of the disclosure;
[0009] FIG. 6 shows resource requirements associated with the virtual machines, according to some embodiments of the disclosure;
[0010] FIG. 7 shows host capacities associated with physical hosts, according to some embodiments of the disclosure;
[0011] FIGS. 8-10 illustrate an example implementing a backtracking-based method for determining a sequence of migrations, according to some embodiments of the disclosure; and
[0012] FIG. 11 shows an aggregated physical host in a federated cloud, according to some embodiments of the disclosure.
Detailed Description of Example Embodiments
Overview
[0013] The present disclosure describes, among other things, a method for cloud resource placement and migration optimization for a federated cloud. A resources monitor monitors state information associated with cloud resources and physical hosts in the federated cloud having a plurality of clouds managed by a plurality of cloud providers. A rebalance trigger triggers, based on one or more conditions, a rebalancing request to initiate cloud resource placement optimization. A constraints-driven optimization cloud resource placement solver (or a cloud resource placement optimizer) determines an optimized placement of cloud resources on physical hosts across the plurality of clouds in the federated cloud based on (1) costs including migration costs, (2) the state information, and (3) constraints, wherein each physical host is identified in the constraints-driven optimization solver by an identifier of a respective cloud provider and an identifier of the physical host. A migrations enforcer determines an ordered migration plan for the optimized placement of cloud resources based on the optimized placement of cloud resources and the state information, and transmits one or more requests to place or migrate cloud resources in the plurality of clouds in the federated cloud according to the ordered migration plan. The identification mechanism for each physical host allows a global optimization solution to be determined and executed across different cloud providers.
[0014] According to one aspect, monitoring the state information in the federated cloud comprises normalizing different types of state information associated with different cloud providers to a common schema.
[0015] According to one aspect, monitoring the state information in the federated clouds comprises receiving state information from respective agents deployed in respective clouds in the federated cloud.
[0016] According to one aspect, state information comprises one or more of the following: current placements of cloud resources to physical hosts; number of cloud resources; resource requirements of cloud resources; number of workloads; number of physical hosts; capacities of physical hosts; one or more events occurred in the federated cloud; one or more alarms in the federated cloud; one or more metrics associated with application(s) running on the federated cloud; one or more metrics associated with physical hosts in the federated cloud; and one or more metrics associated with network resources in the federated cloud.
[0017] According to one aspect, state information associated with resources in the federated cloud comprises time-series information.
[0018] According to one aspect, the resources monitor can apply a predictive model on the time-series information to determine predicted state information. The one or more conditions can include the predicted state information meeting a predetermined criteria.
[0019] According to one aspect, the one or more conditions include a scheduled time condition.
[0020] According to one aspect, the one or more conditions include the state information meeting a predetermined criteria.
[0021] According to one aspect, determining the optimized placement of cloud resources comprises: (1) defining M×N number of assignment variables indicating whether a particular cloud resource is to be placed on a particular physical host, wherein M is the number of cloud resources and N is the number of available physical hosts in the federated cloud, (2) defining N×M number of cost variables indicating cost of migrating a particular cloud resource from a current physical host to another physical host in the federated cloud and/or cost of placing the particular cloud resource on a particular physical host, (3) defining constraints, in terms of the assignment variables and/or cost variables, based on one or more of the following: the state information, one or more policies, one or more business rules, one or more affinity rules, and one or more anti-affinity rules, and (4) solving for the optimized placement of cloud resources based on the constraints and an objective function in terms of the assignment variables and cost variables.
[0022] According to one aspect, the resources monitor and/or the cloud resource placement optimizer can aggregate physical hosts managed by a first cloud provider as an aggregated physical host. The cloud resource placement optimizer can defining the assignment variables and the cost variables by defining assignment variables and cost variables based on the aggregated physical host and other physical hosts managed by cloud providers different from the first cloud provider.
[0023] According to one aspect, determining the ordered migration plan comprises determining a sequence of migrations of cloud resources from source physical hosts to destination physical hosts which does not violate capacity requirements of the destination physical hosts during the sequence of migrations.
[0024] According to one aspect, determining the ordered migration plan includes: (A) determining a series of migrations from the optimized placement of cloud resources, (B) determining capacities of physical hosts and requirements of cloud resources associated cloud resources and physical hosts of the series of migrations based on the state information, and (C) executing a backtracking-based method. The backtracking-based method can include (1) determining if a current capacity of a particular destination physical host for a particular migration accommodates requirement of a particular cloud resource for the particular migration, (2) if the current capacity accommodates the requirement, adding the particular migration from the series of migrations to the ordered migration plan and updating the current capacity of the particular destination physical host and a particular source physical host for the particular migration based on the requirement, and (3) if the current capacity does not accommodate the requirement, repeat steps (1) and (2) for other migration(s) in the series of migrations which have not been added to the ordered migration plan based on current capacities of physical hosts until all migrations in the series of migrations are added to the ordered migration plan.
[0025] According to one aspect, determining the ordered migration plan comprises determining host capacities based on the state information, wherein determining the host capacities comprises aggregating state information associated with physical hosts managed by a first cloud provider as an aggregated physical host, and determining a sequence of migrations of cloud resources from source physical hosts including the aggregated physical host to destination physical hosts including the aggregated physical host which does not violate capacity requirements of the destination physical hosts including the aggregated physical host during the sequence of migrations.
Understanding Federated Clouds
[0026] Traditionally, a cloud provider manages one or more clouds, where a cloud includes a pool of physical computing devices, or referred herein as pools of physical hosts on which virtual resources can be placed. Virtual resources (referred herein as “cloud resources”) can include virtual machines, virtual storage volumes, virtual platforms, virtual networks, software applications, services, workloads, and so on. Physical hosts can include physical computing devices such as servers, computer processors, computer-readable storage media, networks, network links, etc. As used herein, physical hosts can encompass one or more of the following: compute, storage, and network resources. Tenants can make requests to use the cloud, e.g., to place or access cloud resources on the physical hosts. Examples of requests from the tenants include: requesting a workload to be carried out by virtual machines in the cloud, requesting a number of virtual storage volumes to be allocated in the cloud, requesting a service on demand. The cloud provider is responsible for providing and maintaining the physical computing resources in the cloud(s) and placing virtual cloud resources on the physical computing resources as demanded by the tenants.
[0027] Generally speaking, running applications in a virtualized cloud environment such as OpenStack involves creating virtual machines (VMs) in a cluster of physical host servers (referred to as “physical hosts”). All the VMs running in the cloud share the common physical resources—compute, storage, and network. Depending on the application workload or the VM density on a physical host, sometimes it could lead to poor performing VMs because of overloading. As a way to mitigate this effect, cloud infrastructures such as OpenStack provide software support for migrating VMs from one host to another host. However in the OpenStack cloud, currently the only way to achieve migration is by letting the cloud operator manually decide which VM to migrate and to which physical host. There is no automatic support, or any guidance on the migrations. Similarly for managing the virtual storage migrations, in terms of logical volumes, or for virtual networks distribution. Managing overloading in a cloud remains a difficult task.
[0028] Recent developments in cloud computing has motivated a new type of cloud called the federated cloud (or called “cloud federation”), where clouds managed by different cloud providers are collected together to provide a common service. For instance, tenants may have cloud storage resources spread across multiple clouds (e.g., internal cloud(s) and external cloud(s), multiple internal clouds, multiple external clouds, etc.), which may be managed by different cloud providers, or even running on different platforms. Cloud providers do not always use the same platform, and application programming interfaces are generally different from one cloud provider to another. As a result, clouds managed by different cloud providers do not share resources or information with each other, since the platforms are very different and cloud providers are generally competitors against each other. Nevertheless, federation is useful for tenants because the different clouds can provide a diversity of provision and delivery of services, but cooperation or sharing of resources between clouds in the federated cloud is minimal. The different platforms can make it very difficult for cloud providers to manage the federated cloud and improve the federated cloud's efficiency. Federated clouds are not to be confused with hybrid clouds where public and private clouds managed by the same cloud operator are collected together.
[0029] For the federated cloud scenario, one or more technical goals remain to be achieved by state of the art systems:
-
- [0030] determining the best possible placements or migrations for cloud resources running in the cluster of physical hosts across federated cloud providers, such that the cloud cluster is balanced, without impacting the cloud resource placement requirements, as well as keeping the cost of migrations to a minimum;
- [0031] addressing issues of platform differences;
- [0032] addressing issues related to uncooperative cloud providers;
- [0033] providing suitable conditions for reactively responding and rebalancing the federated cloud;
- [0034] providing a proactive way to predict when a cloud rebalance may avoid catastrophic VM failures;
- [0035] providing support for triggering a periodic cloud rebalance operations; and
- [0036] determining a proper migration plan to ensure rebalancing can be executed.
[0037] In one example, OpenStack live migration expects an administrator user to manually trigger the migration operation on a single VM basis, and expects the final VM destination to be specified. It fails if the destination has not enough resources to handle the migration, and expects the administrator user to manually determine if this migration is possible.
[0038] In another example, VMWare vSphere Dynamic Resource Scheduler (DRS) and Dynamic Power Management (DPM) functionality offers dynamic functionality but fails to address issues related to a federated cloud scenario. The DRS functionality handles resource demand in a cluster of hosts, by guaranteeing resources to virtual machines, and allowing for automatic migration of virtual machines during maintenance without service disruption. Depending on the automation levels provided, it provides recommendations or automatically migrates to keep the cluster load balanced and to maximize performance. It provides constraint correction abilities with support for user defined constraints such as affinity or anti-affinity requirements. The DPM functionality manages energy consumption by keeping hosts in standby mode when demand for resources is low, and powering on additional hosts when needed. While these dynamic functionalities are useful, the dynamic functionalities operate under limited conditions.
[0039] Cloud Resource Placement and Rebalancing in a Federated Cloud Environment
[0040] The solution to achieve the one or more aforementioned technical goals involves the following series of technical tasks: unified resources monitoring, comprehensive triggering for rebalancing of the federated cloud, smart placement optimization across the federated cloud, and systematic migration for carrying out the rebalancing of the federated cloud. The present disclosure describes various aspects of the system architecture and methods for implementing and carrying out these technical tasks. These tasks are designed specifically to handle the federated cloud scenario and to provide a more intelligent rebalancing triggering mechanism.
[0041] FIG. 1 shows an exemplary cloud resource placement and rebalancing system for a federated cloud 102 and an illustrative federated cloud 104, according to some embodiments of the disclosure. In this example, the federated cloud 104 comprises three clouds or three clusters of physical hosts: a first cloud 122, a second cloud 124, and a third cloud 126. The three clouds are managed by different cloud providers. For purposes of illustration, the first cloud 122 is managed by cloud provider A (“CP_A”). The second cloud 124 is managed by cloud provider B (“CP_B”). The third cloud 126 is managed by cloud provider C (“CP_C”). Each cloud may have different number of physical hosts, and a number of cloud resources may already be placed (or allocated) in the physical hosts. The cloud resource placement and rebalancing system for a federated cloud 102 includes one or more memory elements 110 (e.g., non-transitory computer readable storage medium), one or more processors 108 coupled to the one or more memory elements 110. The system 102 can further include resources monitor 112, a rebalance trigger 114, cloud resource placement optimizer 116, and migrations enforcer 118. Instructions stored on one or more memory elements 110 can be executed by one or more processors 108 to carry out or provide functionalities associated with parts including one or more of: resources monitor 112, a rebalance trigger 114, cloud resource placement optimizer 116, and migrations enforcer 118. These parts that when executed by the at least one processor are configured to carry out or provide functionalities described herein. The functionalities and data flow of these parts are described in greater detail in later sections of the present disclosure.
[0042] To obtain information about the state of the three clouds, resources monitor 112 can be communicably connected to respective agents deployed in respective clouds in the federated cloud 104. A first agent 132 is deployed in the first cloud 122. A second agent 134 is deployed in the second cloud 124. A third agent 136 is deployed in the third cloud 126. Each agent for a respective cloud provides an interface which can respond to requests from the resources monitor 112 for state information associated with the respective cloud and/or push state information associated with the respective cloud to the resources monitor 112.
[0043] The resources monitor 112 can communicate with rebalance trigger 114, cloud resource placement optimizer 116, and migrations enforcer 118, e.g., to provide state information. Rebalance trigger 114 can communicate with cloud resource placement optimizer 116 to signal when to rebalance the federated cloud.
[0044] To execute migrations according to an optimized placement of cloud resources determined by cloud resource placement optimizer 116, migrations enforcer 118 can be communicably connected to respective services deployed in respective clouds in the federated cloud 104. A first service 142 is deployed in the first cloud 122. A second service 144 is deployed in the second cloud 124. A third service 146 is deployed in the third cloud 126. Typically, each service for a respective cloud is provided by the cloud provider to allow requests to be made by tenants to use the cloud, e.g., to allocate X number of cloud resources in the cloud, to move a cloud resource from one host to another host. These services can receive requests to place or migrate cloud resources from the migrations enforcer 118 to implement migrations for rebalancing the federated cloud.
[0045] The cloud resource placement and rebalancing system for a federated cloud 102 provides an intelligent system for rebalancing the clouds even if they are managed by different cloud providers. Resources monitor 112 can keep track of the state of the federated cloud including, e.g., information relating to existing cloud resources and physical hosts, consumption and/or utilization information, behavior of workloads, etc. Through a comprehensive framework defined by the system 102, rebalance trigger 114 can determine when rebalancing should occur. To rebalance the federated cloud, cloud resource placement optimizer 116 can determine an optimized placement of cloud resources across the clouds. Migrations enforcer 118 can determine an ideal order in which migrations are to be executed.
[0046] Exemplary Method for Cloud Resource Placement and Migration Optimization for a Federated Cloud Involving a Global Namespace to Treat the Federated Cloud as a Single Cloud
[0047] Many state of the art migrations schemes do not address how to rebalance a federated cloud to actively prevent overloading of resources in the federated cloud. Treating the clouds managed by different cloud providers as a single cloud, a global optimization can be applied to determine effectively determine the optimized or improved placements or migrations for cloud resources across the clouds, such that the federated cloud as a whole is balanced. The global optimization can ensure cloud resource placement requirements are met, as well as keep the cost of migrations to a minimum by defining a proper objective function.
[0048] FIG. 2 is a flow diagram illustrating a method for cloud resource placement and migration optimization for a federated cloud, according to some embodiments of the disclosure. The method is being described in conjunction with the parts illustrated in FIG. 1 . Details related to various tasks of the method are described in later sections of the present disclosure.
[0049] Resources monitor 112 can monitor state information associated with cloud resources and physical hosts in the federated cloud having a plurality of clouds managed by a plurality of cloud providers (task 202). Resources monitor can optionally normalize the state information from different cloud providers (task 204). Rebalance trigger 114 can trigger, based on one or more conditions, a rebalancing request to initiate cloud resource placement optimization (task 206). A constraints-driven optimization solver of cloud resource placement optimizer 116 can determine an optimized placement of cloud resources on physical hosts (task 208) across the plurality of clouds in the federated cloud based on (1) costs (in some cases including migration costs), (2) the state information, and (3) constraints, wherein each physical host is identified in the constraints-driven optimization solver by an identifier of a respective cloud provider and an identifier of the physical host. Migrations enforcer 118 can determine an ordered migration plan for the optimized placement of cloud resources based on the optimized placement of cloud resources and the state information (task 210). Migrations enforcer 118 can transmit one or more requests to place or migrate cloud resources in the plurality of clouds in the federated cloud according to the ordered migration plan (task 212).
[0050] For task 208, a distinguishing feature involves identifying each physical host in the constraints-driven optimization solver by an identifier of a respective cloud provider and an identifier of the physical host. For cloud provider A's Host_1's identifier used by the optimization solver can be prefixed with an identifier to indicate the location or corresponding cloud provider by specifying the host as “CP_A:Host_1”. As a result, each physical host across the federated cloud can be treated as part of a single cloud and a global optimization can be applied across the federated cloud. The namespace ensures the cloud resource placement and rebalancing system for a federated cloud can identify to which cloud provider a host belongs. Furthermore, the namespace abstraction allows the global optimization to be performed without requiring the cloud providers to change their respective clouds, interfaces, or application programming interfaces. An optimization solver can optimize placement and migrations for multiple clouds across the federated cloud as if it is performing the optimization for a single cloud. The namespace abstraction allows the migrations enforcer to transmit requests to the correct cloud provider(s) to execute migrations.
[0051] The namespace can be used across different parts of the cloud resource placement and rebalancing system 102. Resources monitor 112 can associate state information to different hosts using the namespace. Variables and optimized placements in the cloud resource placement optimizer 116 can use the namespace. Migrations enforcer can resolve the namespace to generate requests to the appropriate cloud provider when executing migrations.
[0052] FIG. 3 is an illustration of federated cloud resource placement and rebalancing data flow, according to some embodiments of the disclosure. The data flow between different parts of the cloud resource placement and rebalancing system further clarifies the functionalities of the parts illustrated in FIG. 1 and tasks illustrated by FIG. 2 . Each part of the cloud resource placement and rebalancing system and their respective functionalities are described in greater detail in the following sections.
[0053] Unified Resources Monitoring in the Federated Cloud Scenario
[0054] Resources monitor 112 can monitor state information associated with cloud resources and physical hosts in the federated cloud having a plurality of clouds managed by a plurality of cloud providers (task 202 of FIG. 2 ). As illustrated in FIG. 3 , resources monitor 112 can receive state information, which may include one or more of the following: current placements of cloud resources to physical hosts, number of cloud resources, requirements of cloud resources; number of physical hosts, capacities of physical hosts; number of workloads, one or more events occurred in the federated cloud, one or more alarms in the federated cloud, one or more metrics associated with application(s) running on the federated cloud, one or more metrics associated with physical hosts in the federated cloud, and one or more metrics associated with network resources in the federated cloud.
[0055] In terms of applications running on the federated cloud, some metric examples include: application latency/response times, transactions rate, etc. Physical host metric examples include: CPU usage, memory (RAM) usage, disk usage, network utilization, network resource metrics such as packets read/write rates, etc. Some of these host metrics can be captured even for the cloud resources such as virtual machines, virtual storage volumes, etc.
[0056] Because the clouds are managed by different cloud providers, there is generally no centralized entity which has state information for all the clouds, accordingly, resources monitor 112 is provided to receiving state information from respective agents deployed in respective clouds in the federated cloud. Resources monitor 112 can collect and monitor measurable state information, including metrics from the underlying cloud infrastructure, e.g., the first cloud 122, the second cloud 124, and the third cloud 126, by communicating with the cloud infrastructures components and services, e.g., the first agent 132, the second agent 134, and the third agent 136.
[0057] For example, in OpenStack clouds, there is a service called Ceilometer, which publishes periodic metrics such as Computer Processing Unit (CPU) Utilization, Memory Utilization, and other usage metrics associated with the cloud. Resources monitor 112 can talk to these kind of services (referred here in as “agents”), and also some (native) monitoring agents running on the hosts, to gather and consolidate all the data. Advantageously, resources monitor 112 in a federated cloud scenario can receive and maintain monitorable data from the clouds managed by all the different cloud providers, and ensures it can appropriately process the information to make decisions for each of the clouds. As used herein, agents can include any one or more of the above entities or other suitable entities which can report state information to the resources monitor 112; agents are computer-implemented services or devices associated with the clouds. The agents can be implemented in the physical hosts. The agents can be implemented in the cloud resources (e.g., as a service running on a virtual machine). The agents can be implemented in applications running in the cloud (e.g., as a part of the application).
[0058] When clouds are managed by different cloud providers, certain types of state information may not always be available. In some cases, even when the type of state information is available, the format and/or terminology of the state information may not be the same across the clouds managed by different service providers. To address this issue, resources monitor 112, include a normalizer 302 to normalize the state information from different cloud providers (task 204 of FIG. 2 ). The normalizer 302 can normalizing different types of state information associated with different cloud providers to a common schema. Once the state information is normalized, the (normalized) state information as illustrated in FIG. 3 can then be provided to and used by other parts of the cloud resource placement and rebalancing system 102 (e.g., rebalance trigger 114, cloud resource placement optimizer 116, and migration enforcer 118).
[0059] Comprehensive Triggering for Rebalancing of the Federated Cloud
[0060] Rebalance trigger 114 can trigger, based on one or more conditions, a rebalancing request to initiate cloud resource placement optimization (task 206 of FIG. 2 ). As illustrated in FIG. 3 , rebalance trigger 114 can receive (normalized) state information from resources monitor 112 to check one or more conditions and decide whether rebalancing is needed. To avoid overloading issues in a federated cloud, resources monitor 112 monitors state information, e.g., metrics for workloads, and the rebalance trigger 114 can check the state information against one or more conditions to assess whether rebalancing is needed for the federated cloud. For instance, when workloads are running, rebalance trigger 114 can take into account time-series data on CPU, memory, disk utilization. The time-series data can indicate whether a host is overloaded, or even whether a virtual machine itself is overloaded. In another example, rebalance trigger 114 can take into account of network load for transfers to a host's interface, which can also indicate whether a host is overloaded and whether a virtual machine needs to be migrated elsewhere. In yet another example, rebalance trigger 114 can take into account an application-level metric (for an application running on a VM), such as a web server running on a virtual machine. For instance, rebalance trigger 114 can take into account whether the application has a slow response time, to determine whether rebalancing is needed.
[0061] Rebalance trigger 114 can determine a rebalancing of the federated cloud is desirable or needed using one or more of these modes: reactive triggering, proactive triggering, and scheduled triggering. Any of these modes includes one or more conditions for triggering a cloud resource rebalance request to be transmitted to the cloud resource placement optimizer 116. The modes provide a comprehensive scheme for intelligently triggering a rebalancing of the federated cloud for a variety of situations (i.e., based one on or more conditions, e.g., as specified by an administrator of the federated cloud).
[0062] For reactive triggering, rebalance trigger 114 can respond to events or alarms notifications from the clouds in the federated cloud. For instance, rebalance trigger 114 can implement rules and/or threshold checkers that monitor certain types of state information. In some embodiments, the rebalance trigger 114 can assess whether the state information meets a predetermined criteria (e.g., rules, threshold checks). In some embodiments, the rebalance trigger 114 can assess whether a certain alarm went off and/or event has occurred. The types of state information, the rules, and/or threshold checkers can be configured by an administrator. Example events or alarm notifications from the clouds may include one or more of the following: host failing events (typically indicating a cloud resource on the host needs to be migrated), some other alarms that are configured in the infrastructure service directly, etc. In some embodiments, Ceilometer service in OpenStack supports triggering alarms based on values of some metrics, and such alarms can be used for the rebalance trigger 114 to reactively trigger rebalancing of the federated cloud. In some cases, rebalance trigger 114 can assess state information at the cloud resource level, e.g., based on virtual machine specific performance metrics. With the help of monitoring agents running in the virtual machines directly, rebalance trigger 114 can measure the application performance and if the application performance is deteriorating, e.g., when a virtual machine migration might benefit.
[0063] For proactive triggering, rebalance trigger 114 predicts can predict the performance of the cloud, and use predicted state information to determine if one or more criteria has been met for triggering a rebalancing of the federated cloud. The performance of the cloud can include performance information at one or more levels including: at the network level, at the physical host level, at the cloud resource level, and/or at the application level. Typically, rebalance trigger 114 monitors state information to determine performance of the cloud (i.e., performance of the resources in the cloud). The state information associated with cloud resources and physical hosts in the federated cloud can include time-series information. Rebalance trigger 114 can apply a predictive model on the time-series information to determine predicted state information, and determine whether the predicted state information meets one or more of the predetermined criteria (e.g., based on rules and/or threshold checks). Predictive models can include advanced techniques such as Holt-Winters model, autoregressive integrated moving average (ARIMA) model, and any suitable time-series analysis and prediction model. Effectively, rebalance trigger 114 can determine when it is appropriate or needed, e.g., to rebalance the federated cloud to prevent overloading, to perform a migration for the current workload, etc. Rebalance trigger 114 leverages a unified resources monitor 112, which continuously monitors state information associated with the cloud resources and physical hosts in the federated cloud. Based on the state information, rebalance trigger 114 can apply special data analysis techniques such as forecasting to study demand, loads, usage behavior, etc. to determine predicted state information. The predicted state information can then be used as a basis for triggering a cloud resource rebalance request to be transmitted to cloud resource placement optimizer 116. For instance, based on N number of time-series data points of a particular metric, it is possible to predict the value of the metric using a suitable predictive model. If the predicted value of the particular metric exceeds a threshold, rebalance trigger 114 can trigger a cloud resource rebalance request. Advantageously, rebalance trigger can proactively rebalance the federated cloud to avoid any downtimes and performance degradations. For instance, rebalance trigger 114 can execute predictive machine learning algorithms on state information (e.g., metrics in time-series format, and usage behaviors based on the historical data collected from the cloud infrastructure), and take decisions to rebalance the federated cloud proactively. In some cases, anomalies detected using an anomaly detection algorithms can also be used as a basis for determining whether to trigger a rebalance request to avoid failures.
[0064] For scheduled triggering, rebalance trigger 114 may include one or more timers. Based on a scheduled time condition, rebalance trigger 114 can trigger a cloud resource rebalance request to be transmitted to cloud resource placement optimizer. Examples of scheduled time conditions may include: periodic maintenance window support, scheduled maintenance, periodic scheduled rebalancing, etc.
[0065] Smart Placement Optimization Across the Federated Cloud
[0066] A constraints-driven optimization cloud resource placement solver of cloud resource placement optimizer 116 can determine an optimized placement of cloud resources on physical hosts (task 208 of FIG. 2 ) across the plurality of clouds in the federated cloud. As illustrated in FIG. 3 , cloud resource placement optimizer 116 includes constraints-driven optimization resource placement solver 310. For the federated cloud scenario, the solver effectively combines the resources from all the clouds managed by different cloud providers and forms a single cloud, i.e., a common resource pool. For instance, suppose the federated cloud has two clouds: a local cloud and an external cloud. The resource of external cloud could be seen as some additional capacity of local cloud. The external cloud can be modeled as one or more additional hosts to the local cloud with available capacity and associated costs. By aggregating the resources from different cloud providers, the constraints-driven optimization cloud resource placement can work across physical hosts from different clouds managed by different cloud providers. Each physical host is identified in the constraints-driven optimization solver by an identifier of a respective cloud provider and an identifier of the physical host. As previously explained, the namespace ensures the cloud resource placement and rebalancing system for a federated cloud can identify to which cloud provider a host belongs, and the namespace abstraction allows the global optimization to be performed without requiring the cloud providers to change their respective clouds, interfaces, or application programming interfaces.
[0067] The constraints-driven optimization cloud resource placement solver 310 determines a balanced and optimized set of new placements of cloud resources (e.g., virtual machines, storage, network) while ensuring the set of new placements satisfy requirement(s) of the various cloud providers. In other words, the optimized placement of cloud resources on physical hosts across the plurality of clouds aim to keep the clouds balanced, increase performance, and ensure user requested placement constraints are satisfied.
[0068] Some examples herein are related to virtual machine placement and virtual machine migrations. These examples are serve to illustrate how the constraints-driven optimization cloud resource placement solver 310 operates; these examples are not meant to be limiting. The present disclosure encompasses other types of placements involving other types of cloud resources, such as storage resources and network resources. To rebalance the federated cloud, the solver 310 tries to find an optimized placement of virtual machines by solving optimal placements as if the existing virtual machines are new requests for virtual machines to be placed in the cloud. In one example, the solver 310 determines an optimal placement for each virtual machine (i.e., which physical host is best for hosting the virtual machine), by taking the following into consideration:
-
- [0069] consider all the existing virtual machines as a new request of virtual machines with their specific demand of resources;
- [0070] consider the physical hosts as available (as if it was a fresh start);
- [0071] consider all the existing virtual machines constraints, e.g., special requirements in terms of policies, business rules, affinity/anti-affinity requirements, or any other special constraints virtual machines had when they were initial placed;
- [0072] minimize cost, including cost of migration, e.g., the number of migrations;
- [0073] maximize the cloud rebalancing and performance impact, in terms of the final values of a set of cloud metrics after the resulting new distribution.
[0074] As illustrated in FIG. 3 , the constraints-driven optimization cloud resource placement solver determines an optimized placement of cloud resources based on (1) costs 304, (2) state information 306, and (3) constraints 308. When optimizing placement of cloud resources in a federated cloud, special considerations are given to (1) costs 304, (2) state information 306, and (3) constraints 308 (when compared with optimizations performed for a single cloud), since migrating cloud resources across clouds managed by different cloud providers is different from migrating cloud resources within a single cloud.
[0075] Special set of costs are considered by the solver 310 in the federated cloud scenario. Examples of costs 304 can include: the cost for a particular physical host hosting a particular virtual resource, the cost of running in a federated cloud (e.g., comparing cloud A vs cloud B using a cost matrix with effective cost of running in each of the clouds available for federation), cost of migration, etc. In particular, examples of cost of migration can include: a number of physical hosts running/utilized after all migrations, number of migrations (and associated costs), time required for executing migrations, amount of data transfer required for executing migrations, the cost of migrating a cloud resource from a source cloud provider to a destination cloud provider.
[0076] As explained previously, state information 306 for the solver 310 utilizes the namespace to identify which cloud provider a physical host is located, and the state information across the clouds are unified and optionally normalized by resource monitor 112.
[0077] Special set of constraints can be taken into account in the federated cloud scenario. Examples of constraints 308 can include: geo-location-based policies, tenant-specific requirements in terms of what clouds are allowed or not allowed, resource-based constraints in terms of capabilities in the clouds managed by different cloud providers, etc.
[0078] In some embodiments, the solver 310 can determine the optimized placement of cloud resources by setting up a constraints-driven optimization problem. The solver 310 can define M×N number of assignment variables indicating whether a particular cloud resource is to be placed on a particular physical host, wherein M is the number of (existing) cloud resources and N is the number of available physical hosts in the federated cloud. An assignment variable can be a binary variable, whose value indicates whether a particular virtual resource is to be placed on a physical host. The solver 310 can define N×M number of cost variables indicating cost of migrating a particular cloud resource from a current physical host to another physical host in the federated cloud and/or cost of placing the particular cloud resource on a particular physical host. The solver 310 can define constraints, in terms of the assignment variables and/or cost variables, based on one or more of the following: the state information, one or more policies, one or more business rules, one or more affinity rules, and one or more anti-affinity rules. Constraints can be mathematically defined in terms of the assignment variables and/or cost variables. For instance, one constraint can specify that the sum of all assignment variables must equal to the number of (existing) cloud resources. In another instance, one constraint can specify that the required/requested amount of resources for particular cloud resource(s) to be placed on a particular physical host must not exceed the available capacity of a particular physical host. The solver 310 can solve for the optimized placement of cloud resources based on the constraints and an objective function in terms of the assignment variables and cost variables. Specifically, the solver 310 searches through possible placements to find an optimal placement of cloud resources which can optimize the objective function. The objective function can be a cost function which takes into account of the total cost of placing the cloud resources and the total cost of needed migrations. The solver 310 would find an optimal placement which minimizes the cost function while ensuring constraints are satisfied. The objective function can be a utility function which rewards performance and/or placements which involve less migrations. The solver 310 would find an optimal placement which maximizes the utility function while ensuring constraints are satisfied. Any suitable constraints-driven optimization solver can be used. The result generated by the solver 310, i.e., the optimized placement of cloud resources, includes mapping of each cloud resource to a particular physical host. The optimized placement of cloud resources is transmitted to migrations enforcer 118.
[0079] Systematic Migration for Carrying Out the Rebalancing of the Federated Cloud
[0080] Based on the optimized placement from cloud resource placement optimizer 116, migrations enforcer 118 determines an (unordered) series of migrations. Depending on the optimized placement of cloud resources, one or more existing cloud resources may be assigned to its current physical host (i.e., the physical host on which the cloud resource is currently placed) and one or more existing cloud resources may be assigned to a physical host different from its current physical host. When a cloud resource is assigned to a physical host different from its current physical host, a migration is needed. The series of migration(s) includes one or more such migrations. In an example dealing with migrating virtual machines, the series of migrations includes a set of all required migrations involving virtual machines migrating from their source physical hosts to a new destination physical hosts. A migration can be represented as a tuple, e.g., <VM, Cloud_Provider:Host_source, Cloud_Provider:Host_destination>, indicating that the virtual machine VM is to be migrated from source host Cloud_Provider:Host_source to the destination host Cloud_Provider:Host_destination. For instance, if currently VM_1 was in Cloud_Provider_A:Host_1, and the optimized placement requiring VM_1 to be in Cloud_Provider_B:Host_4, a migration is required for VM_1 from Cloud_Provider_A:Host_1 to Cloud_Provider_B:Host_4. This migration can be represented step as a tuple <VM_1, Cloud_Provider_A:Host_1, Cloud_Provider_B:Host_4>. A series of migrations can include a set of such tuples {<VM_1, Cloud_Provider_A:Host_1, Cloud_Provider_B:Host_4>, <VM_4, Cloud_Provider_B:Host_2, Cloud_Provider_A:Host_2>, <VM_7, Cloud_Provider_A:Host_3, Cloud_Provider_A:Host_4> . . . }
[0081] With the series of migrations, the migrations enforcer 118 can communicate with services in respective clouds to enable the migrations. For instance, for virtual machine migrations in OpenStack Clouds, the migrations enforcer 118 can communicate migration request to compute service Nova, and use the Nova virtual machine live migration application programming interfaces, to trigger the migrations. In some cases, after some cloud resource migrations are executed, the resources of a physical host may get used up completely, even if there were additional migrations that would free up the physical host. This situation happens because the cloud resource placement optimizer 116 determines a “global” solution, and doesn't determine a migration plan. To address this issue, migrations enforcer 118 can determine an ordered migration plan for the optimized placement of cloud resources based on the optimized placement of cloud resources the state information (task 210 of FIG. 2 ). Determining an ordered migration plan (from the series of migrations) includes determining a sequence of migrations of cloud resources from source physical hosts to destination physical hosts which does not violate capacity requirements of the destination physical hosts during the sequence of migrations. Migration enforcer 118 can implement a method which ensures there is capacity for each physical host during the sequence of migrations as each cloud resource is migrated one by one. Specifically, migration enforcer 118 solves for a valid ordered migration plan to migrate cloud resources one by one to achieve the final optimized placement of cloud resources from the cloud resource placement optimizer 116. After the migration plan is determined, migrations enforcer 118 can transmit one or more requests to place or migrate cloud resources in the plurality of clouds in the federated cloud according to the ordered migration plan (task 212 of FIG. 2 ).
[0082] To illustrate, FIGS. 4-10 outlines executing the method for determining a sequence of migrations, i.e., the migration plan, in a simplified example. The simplified example involves three virtual machines (VM-1, VM-2, and VM-3) and three physical hosts (HOST-1, HOST-2, and HOST-3). FIG. 4 shows initial virtual machine placements, according to some embodiments of the disclosure. VM-1 is placed on HOST-1. VM-2 is placed on HOST-2. VM-3 is placed on HOST-3. Suppose the constraints-driven optimization cloud resource placement solver determines an optimized placement, referred to as the “desired virtual machine placements”. FIG. 5 shows desired virtual machine placements, according to some embodiments of the disclosure. VM-1 is to be placed on HOST-3. VM-2 is to be placed on HOST-1. VM-3 is to be placed on HOST-2. In this example, all virtual machines are being migrated. The series of migrations can be represented as {<VM-1, Host-1, Host-3>, <VM-2, Host-2, Host-1>, <VM-3, Host-3, Host-2>}. The series of migrations is not ordered. FIG. 6 shows resource requirements associated with the virtual machines, according to some embodiments of the disclosure. VM-1 requires 200 units of resources. VM-2 requires 100 units of resources, VM-3 requires 300 units of resources. FIG. 7 shows host capacities associated with physical hosts, according to some embodiments of the disclosure. The physical host capacities are initial (free) capacities of the physical hosts (amount of resources available when the virtual machines are still in their initial virtual machine placements). VM-1 has 100 units of resources. VM-2 requires 200 units of resources, VM-3 requires 100 units of resources.
[0083] Determining the ordered migration plan can include determining a series of migrations from the optimized placement of cloud resources. Furthermore, determining the ordered migration plan includes determining capacities of physical hosts (e.g., FIG. 7 ) and requirements of cloud resources (e.g., FIG. 6 ) associated cloud resources and physical hosts of the series of migrations based on the state information. Migration enforcer 118 can execute a backtracking-based method to determine the ordered migration plan based on the series of migrations and associated state information.
[0084] In some embodiments, the backtracking-based method includes: (1) determining if a current capacity of a particular destination physical host for a particular migration accommodates requirement of a particular cloud resource for the particular migration, (2) if the current capacity accommodates the requirement, adding the particular migration from the series of migrations to the ordered migration plan and updating the current capacity of the particular destination physical host and a particular source physical host for the particular migration based on the requirement, and (3) if the current capacity does not accommodate the requirement, repeat steps (1) and (2) for other migration(s) in the series of migrations which have not been added to the ordered migration plan based on current capacities of physical hosts until all migrations in the series of migrations are added to the ordered migration plan. The backtracking-based method can return an ordered migration plan, i.e., an ordered sequence of migrations, if a valid sequence exists. The backtracking-based method avoids random migration and determines a concrete and valid series of migrations to be executed. After an ordered migration plan is determined, migration enforcer 118 can communicate to respective services of the respective clouds (e.g., cloud Infrastructure application programming interfaces such as the OpenStack Nova migration application programming interface to trigger the migrations one by one in the order as specified by the ordered migration plan. The following is pseudocode for the backtracking-based method which recursively backtracks ordering of migrations if a capacity requirement is violated.
[0085]
| Variable Description | |
| 1. | current_hosts_capacity : Existing Host Capacity |
| 2. | current_vm_host_mapping : Current VM to its Host mapping |
| 3. | proposed_migration_plan : The new set of proposed migration |
| mappings of the form { <VM-i, Host_source, Host_destination>,.... } | |
| as a result of the cloud rebalancing | |
| 4. | vm_requirement : Resource requirements of the VM |
| Order = [ ] | |
| bool FindOrder(current_hosts_capacity, current_vm_host_ | |
| mapping, proposed_migration_plan, vm_requirement): | |
| If all new hosts mapping have been ordered: | |
| return success | |
| For each mapping i in proposed_migration_plan: | |
| v = VM in mapping i | |
| new_host = Host to which v is mapped | |
| curren_host = current_vm_host_mapping[v] | |
| If mapping [i] not in Order: | |
| if Host new_host has enough capacity for VM v: | |
| // Update old and new host | |
| Consume vm_requirement[v] capacity from new_host | |
| Add capacity to current_host | |
| // Update current_mapping | |
| Change current_host to new_host | |
| // Append to order | |
| Append mapping to Order | |
| //Make recursive call | |
| result = FindOrder(current_hosts_capacity, | |
| current_vm_host_mapping, proposed_migration_plan, | |
| vm_requirement) | |
| if result is True: | |
| return True | |
| else: | |
| // Revert the changes made before recursive call | |
| Revert current_mapping[v] to its old mapping | |
| Revert capacity changes | |
| Remove this mapping from order | |
| // backtrack to earlier level of recursion since none of new mapping could | |
| lead a | |
| // successful ordering | |
| return False | |
[0086] FIGS. 8-10 illustrate an example implementing a backtracking-based method for determining a sequence of migrations, according to some embodiments of the disclosure. Using the same example as before, the series of migrations is represented by {<VM-1, HOST-1, HOST-3>, <VM-2, HOST-2, HOST-1>, <VM-3, HOST-3, HOST-2>}. In FIG. 8 , the method tries to migrate VM-1 from HOST-1 to HOST-3, but HOST-3 having a current capacity of 100 units of resources cannot accommodate VM-1, which requires 200 units of resources. The method backtracks to try a different virtual machine migration in the series. In FIG. 9 , the method tries to migrate VM-2 from HOST-2 to HOST-1 and succeeds because HOST-1 having a current capacity of 100 units of resources can accommodate VM-2, which requires 100 units of resources. The current capacities of HOST-1 (destination physical host) and HOST-2 (source physical host) are updated based on the requirement of VM-2: {HOST-1: 0, HOST-2: 300, HOST-3: 100}. In FIG. 10 , based on the updated current capacity, the method tries to migrate VM-1 from HOST-1 to HOST-3 but fails again because the migration would violate capacity requirements. The method backtracks and tries to migrate VM-3 to HOST-2 and succeeds because HOST-2 having current capacity of 300 units of resources can accommodate VM-3, which requires 300 units of resources. The current capacities of HOST-2 (destination physical host) and HOST-3 (source physical host) are updated based on the requirement of VM-3: {HOST-1: 0, HOST-2: 0, HOST-3: 400}. Based on the updated current capacity, the method tries to migrate VM-1 from HOST-1 to HOST-3 and succeeds (finally) because HOST-3 having a current capacity of 400 units of resources can accommodate VM-1, which requires 200 units of resources. If desired, the current capacities of HOST-2 (destination physical host) and HOST-1 (source physical host) are updated based on the requirement of VM-1: {HOST-1: 200, HOST-2: 0, HOST-3: 200}. The method ends because all of the migrations are now ordered in a migration plan, which can be represented by {<VM-2, HOST-2, HOST-1>, <VM-3, HOST-3, HOST-2>, <VM-1, HOST-1, HOST-3>}. As explained in a previous section, the namespace identifying the cloud provider can be used in the host identifier to identify the host and the cloud provider to which the host belongs.
[0087] Recursive Method for Treating a Cloud as an Aggregated Physical Host
[0088] When operating in a federated cloud scenario, there are additional unique challenges to optimizing placement and rebalancing the cloud resources across the clouds. For instance, some cloud providers may not expose or provide access to execute migrations within a cloud (e.g., some cloud providers may not have an application programming interface for executing migrations within the cloud). FIG. 11 shows an aggregated physical host in a federated cloud, according to some embodiments of the disclosure. In this example, the first cloud 122 may not allow migrations to be executed by external requests. In some cases, the first cloud 122 may not provide host-by-host state information to resources monitor 112 of FIGS. 1 and 3 . In another instance, there can be numerous physical hosts when many clouds are in a federated cloud. The constraints-driven optimization cloud resource placement solver increases in complexity significantly when the number of physical hosts (and the number of cloud resources even) increases.
[0089] To address these issues, the disclosed mechanisms for cloud resource placement and rebalancing can be implemented as a recursive placement decision scheme. When determining the optimized placement of cloud resources, cloud resource placement optimizer 116 and/or resources monitor 112 can aggregate physical hosts managed by a first cloud provider as an aggregated physical host. Defining assignment variables and cost variables can include defining assignment variables and cost variables based on the aggregated physical host and other physical hosts managed by cloud providers different from the first cloud provider.
[0090] In some embodiments, the decision scheme can pool capacities of physical hosts of a cloud managed by a particular cloud provider and logically present the collective capacity available as a single (aggregate) physical host. The optimization can determine that some cloud resources is to be migrated to or from this logical aggregate physical host. The particular cloud provider can then internally execute those migrations.
[0091] In some embodiments, the recursive placement decision scheme can be viewed as hierarchical scheduling, where a high level scheduling is first performed according to aggregated information from each cloud in the federated cloud. First decision involves optimizing placement of a particular cloud resource at the cloud level (e.g., deciding which cloud should the cloud resource be placed). Second decision involves optimizing placement of the particular cloud resource at the physical host level within the optimal cloud (e.g., deciding which physical host should the cloud resource be placed in the optimal cloud). Both of these decisions can be performed by cloud resource placement optimizer 116 of FIGS. 1 and 3 .
[0092] In the federated cloud scenario, some cloud providers do not allow tenants to request or demand a specific order for migrating cloud resources. To address this issue, the same recursive scheme can also be applied in the migrations enforcer 118. The backtracking-based method can treat physical hosts of a particular cloud as a single physical host. In some embodiments, determining the ordered migration plan comprises determining host capacities based on the state information, wherein determining the host capacities comprises aggregating state information associated with physical hosts managed by a first cloud provider as an aggregated physical host, and determining a sequence of migrations of cloud resources from source physical hosts (including the aggregated physical host) to destination physical hosts (including the aggregated physical host) which does not violate capacity requirements of the destination physical hosts (including the aggregated physical host) during the sequence of migrations. The same backtracking-based method is applied, but the method treats physical hosts of a particular cloud as a single aggregated physical host. The migrations enforcer 118 can still transmit requests to migrate cloud resources to or from the particular cloud, but the specific ordering of migrations from one physical host to another physical host within the particular cloud is decided by the cloud provider of the particular cloud.
[0093] Advantages
[0094] The present disclosure describes systems and methods for enabling a smart and optimized monitoring data driven cloud rebalancing via cloud resource migration designed specifically for the scenario of dealing with resources from federated cloud providers. When compared to conventional solutions, the systems and methods described herein greatly improves upon the migration process. In one aspect, a resources monitor collects information across the federated cloud. In another aspect, a rebalance trigger triggers the cloud rebalancing with comprehensive support for reactive, proactive, and periodic triggering. For instance, the rebalance trigger can respond to events or alarms, or when some monitoring data crosses the thresholds to trigger a rebalance of the cloud (reactive triggering). The rebalance trigger can also study and analyze the historical data, and also monitor the user loads, to predict the rebalances (proactive trigger). The rebalance trigger can also allow for periodic scheduling rebalances (scheduled triggering). In yet another aspect, an optimizer determines optimized placement of cloud resources across the clouds. The optimization can ensure all the existing placement constraints or policies imposed by tenants and/or cloud providers are satisfied, while optimizing all the costs. The optimization advantageously enables the required cloud resource migrations to be determined, and optimizes these migrations such that the physical hosts are optimally used by the cloud resources, and also such that the cost and impact of migration is minimized (e.g., keeping the number of migrations to a minimum). In yet another aspect, a migration enforcer implements a migration ordering mechanism to calculate a feasible ordering of the migrations. Advantageously, migrations can be executed without service disruption. Smarter rebalancing can help federated clouds guarantee quality of service and other service level agreements.
[0095] Variations and Implementations
[0096] Some examples herein are related to virtual machine placement and virtual machine migrations. These examples are serve to illustrate how the constraints-driven optimization cloud resource placement solver 310 operates; these examples are not meant to be limiting. The present disclosure encompasses other types of placements involving other types of cloud resources, such as storage resources and network resources. The teachings herein easily extends to the other cloud resource migrations, in terms of the Storage (logical storage volumes placement) or the network (logical networks distribution). Storage and Network resources are other resources that can also be balanced in the cloud, based on their utilizations. For instance, the teachings extend to migrations of logical virtual storage volumes from one host disks to another host disk. This can involve detaching the storage volumes and then migrating the storage volumes. Storage cloud metrics can be monitored, and the cloud metrics can include such as Storage IOPS (IO operations per second), storage capacity, etc., and optimized migrations can be determined in a similar manner.
[0097] Within the context of the disclosure, a cloud of physical hosts generally includes physical hosts communicably connected over a network. A network used herein represents a series of points, nodes, or network elements of interconnected communication paths for receiving and transmitting packets of information that propagate through a communication system. A network offers communicative interface between sources and/or hosts, and may be any local area network (LAN), wireless local area network (WLAN), metropolitan area network (MAN), Intranet, Extranet, Internet, WAN, virtual private network (VPN), or any other appropriate architecture or system that facilitates communications in a network environment depending on the network topology. A network can comprise any number of hardware or software elements coupled to (and in communication with) each other through a communications medium.
[0098] As used herein in this Specification, the term ‘network element’ or parts shown in FIGS. 1, 3, and 11 , are meant to encompass suitable elements such as servers (physical or virtually implemented on physical hardware), machines (physical or virtually implemented on physical hardware), end user devices, routers, switches, cable boxes, gateways, bridges, loadbalancers, firewalls, inline service nodes, proxies, processors, modules, or any other suitable device, component, element, proprietary appliance, or object operable to exchange, receive, and transmit information in a network environment. These network elements or parts shown in FIGS. 1 and 4-9 may include any suitable hardware, software, components, modules, interfaces, or objects that facilitate the virtual machine placement operations thereof. This may be inclusive of appropriate algorithms and communication protocols (which can be carried out by one or more processors) that allow for the effective exchange of data or information.
[0099] In some implementations, elements including the cloud resource placement and rebalancing system for a federated cloud and any parts therein may include software to achieve (or to foster) the functions discussed herein where the software is executed on one or more processors to carry out the functions. This could include the implementation of instances of resources monitor, rebalance trigger, cloud resource placement optimizer, migrations enforcer, and/or any other suitable part that would foster the activities discussed herein. Additionally, each of these elements can have an internal structure (e.g., one or more suitable processors, one or more memory elements, processor 108 of FIG. 1 , memory 110 of FIG. 1 , etc.) to facilitate some of the operations described herein. In other embodiments, these functions for cloud resource placement and rebalancing may be executed externally to these elements, or included in some other network element to achieve the intended functionality. Alternatively, the cloud resource placement and rebalancing system for a federated cloud and any parts therein may include software (or reciprocating software) that can coordinate with other network elements in order to achieve the functions described herein. In still other embodiments, one or several devices may include any suitable algorithms, hardware, software, components, modules, interfaces, or objects that facilitate the operations thereof.
[0100] In certain example implementations, the cloud resource placement and rebalancing functionalities outlined herein may be implemented by logic encoded in one or more non-transitory, tangible media (e.g., embedded logic provided in an application specific integrated circuit [ASIC], digital signal processor [DSP] instructions, software [potentially inclusive of object code and source code] to be executed by one or more processors, or other similar machine, etc.). In some of these instances, one or more memory elements can store data used for the operations described herein. This includes the memory element being able to store instructions (e.g., software, code, etc.) that are executed to carry out the activities described in this Specification. The memory element is further configured to store databases/data structures related to costs, constraints, rules, host states, etc. The processor can execute any type of instructions associated with the data to achieve the operations detailed herein in this Specification. In one example, the processor could transform an element or an article (e.g., data) from one state or thing to another state or thing. In another example, the activities outlined herein may be implemented with fixed logic or programmable logic (e.g., software/computer instructions executed by the processor) and the elements identified herein could be some type of a programmable processor, programmable digital logic (e.g., a field programmable gate array [FPGA], an erasable programmable read only memory (EPROM), an electrically erasable programmable ROM (EEPROM)) or an ASIC that includes digital logic, software, code, electronic instructions, or any suitable combination thereof.
[0101] Any of these elements (e.g., the cloud resource placement and rebalancing system for a federated cloud and any parts therein, etc.) can include memory elements for storing information to be used in cloud resource placement and rebalancing, as outlined herein. Additionally, each of these devices may include a processor that can execute software or an algorithm to perform the cloud resource placement and rebalancing activities as discussed in this Specification. These devices may further keep information in any suitable memory element [random access memory (RAM), ROM, EPROM, EEPROM, ASIC, etc.], software, hardware, or in any other suitable component, device, element, or object where appropriate and based on particular needs. Any of the memory items discussed herein should be construed as being encompassed within the broad term ‘memory element.’ Similarly, any of the potential processing elements, modules, and machines described in this Specification should be construed as being encompassed within the broad term ‘processor.’ Each of the network elements can also include suitable interfaces for receiving, transmitting, and/or otherwise communicating data or information in a network environment.
[0102] Additionally, it should be noted that with the examples provided above, interaction may be described in terms of two, three, or four network elements or parts shown in FIGS. 1, 3, and 11 . However, this has been done for purposes of clarity and example only. In certain cases, it may be easier to describe one or more of the functionalities of a given set of flows by only referencing a limited number of network elements. It should be appreciated that the systems described herein are readily scalable and, further, can accommodate a large number of components, as well as more complicated/sophisticated arrangements and configurations. Accordingly, the examples provided should not limit the scope or inhibit the broad techniques of cloud resource placement and rebalancing, as potentially applied to a myriad of other architectures.
[0103] It is also important to note that the steps/tasks illustrated by the FIGS. 2 and 8-10 (or any steps/tasks described herein) illustrate only some of the possible scenarios that may be executed by, or within, the cloud resource placement and rebalancing system described herein. Some of these steps may be deleted or removed where appropriate, or these steps may be modified or changed considerably without departing from the scope of the present disclosure. In addition, a number of these operations have been described as being executed concurrently with, or in parallel to, one or more additional operations. However, the timing of these operations may be altered considerably. The preceding operational flows have been offered for purposes of example and discussion. Substantial flexibility is provided by the cloud resource placement and rebalancing system (and any parts therein) in that any suitable arrangements, chronologies, configurations, and timing mechanisms may be provided without departing from the teachings of the present disclosure.
[0104] Numerous other changes, substitutions, variations, alterations, and modifications may be ascertained to one skilled in the art and it is intended that the present disclosure encompass all such changes, substitutions, variations, alterations, and modifications as falling within the scope of the appended claims. In order to assist the United States Patent and Trademark Office (USPTO) and, additionally, any readers of any patent issued on this application in interpreting the claims appended hereto, Applicant wishes to note that the Applicant: (a) does not intend any of the appended claims to invoke paragraph six (6) of 35 U.S.C. section 112 as it exists on the date of the filing hereof unless the words “means for” or “step for” are specifically used in the particular claims; and (b) does not intend, by any statement in the specification, to limit this disclosure in any way that is not otherwise reflected in the appended claims.
Claims
What is claimed is:
1. A method for cloud resource placement and migration optimization for a federated cloud, the method comprising:
monitoring state information associated with cloud resources and physical hosts in the federated cloud having a plurality of clouds managed by a plurality of cloud providers;
triggering, based on one or more conditions, a rebalancing request to initiate cloud resource placement optimization;
determining, by a constraints-driven optimization cloud resource placement solver, an optimized placement of cloud resources on physical hosts across the plurality of clouds in the federated cloud based on (1) costs including migration costs, (2) the state information, and (3) constraints, wherein each physical host is identified in the constraints-driven optimization cloud resource placement solver by an identifier of a respective cloud provider and an identifier of the physical host, comprising:
defining M×N number of assignment variables indicating whether a particular cloud resource is to be placed on a particular physical host, wherein M is the number of cloud resources and N is the number of available physical hosts in the federated cloud;
defining N×M number of cost variables indicating cost of migrating a particular cloud resource from a current physical host to another physical host in the federated cloud and/or cost of placing the particular cloud resource on a particular physical host; and
solving for the optimized placement of cloud resources based on the assignment variables and the cost variables;
determining an ordered migration plan for the optimized placement of cloud resources based on the optimized placement of cloud resources and the state information; and
transmitting one or more requests to place or migrate cloud resources in the plurality of clouds in the federated cloud according to the ordered migration plan.
2. The method of claim 1, wherein monitoring the state information in the federated cloud comprises:
normalizing different types of state information associated with different cloud providers to a common schema.
3. The method of claim 1, wherein monitoring the state information in the federated clouds comprises:
receiving state information from respective agents deployed in respective clouds in the federated cloud.
4. The method of claim 1, wherein state information comprises one or more of the following:
current placements of cloud resources to physical hosts;
number of cloud resources;
resource requirements of cloud resources;
number of workloads;
number of physical hosts; and
capacities of physical hosts.
5. The method of claim 1, wherein state information comprises one or more of the following:
one or more events that occurred in the federated cloud;
one or more alarms in the federated cloud;
one or more metrics associated with application(s) running on the federated cloud;
one or more metrics associated with physical hosts in the federated cloud; and
one or more metrics associated with network resources in the federated cloud.
6. The method of claim 1, wherein state information associated with resources in the federated cloud comprises time-series information.
7. The method of claim 6, further comprising:
applying a predictive model on the time-series information to determine predicted state information;
wherein the one or more conditions include the predicted state information meeting a predetermined criteria.
8. The method of claim 1, wherein the one or more conditions include a scheduled time condition.
9. The method of claim 1, wherein the one or more conditions include the state information meeting a predetermined criteria.
10. The method of claim 1, wherein determining the optimized placement of cloud resources comprises:
defining constraints, in terms of the assignment variables and/or the cost variables, based on one or more of the following: the state information, one or more policies, one or more business rules, one or more affinity rules, and one or more anti-affinity rules; and
wherein the solving comprises solving for the optimized placement of cloud resources based on the constraints and an objective function in terms of the assignment variables and cost variables.
11. The method of claim 10, wherein determining the optimized placement of cloud resources comprises:
aggregating physical hosts managed by a first cloud provider as an aggregated physical host; and
wherein defining the assignment variables and the cost variables comprises defining assignment variables and cost variables based on the aggregated physical host and other physical hosts managed by cloud providers different from the first cloud provider.
12. The method of claim 1, wherein determining the ordered migration plan comprises:
determining a sequence of migrations of cloud resources from source physical hosts to destination physical hosts which does not violate capacity requirements of the destination physical hosts during the sequence of migrations.
13. The method of claim 1, wherein determining the ordered migration plan includes:
determining a series of migrations from the optimized placement of cloud resources;
determining capacities of physical hosts and requirements of cloud resources associated cloud resources and physical hosts of the series of migrations based on the state information; and
executing a backtracking-based method comprising:
(1) determining if a current capacity of a particular destination physical host for a particular migration accommodates requirement of a particular cloud resource for the particular migration;
(2) if the current capacity accommodates the requirement, adding the particular migration from the series of migrations to the ordered migration plan and updating the current capacity of the particular destination physical host and a particular source physical host for the particular migration based on the requirement; and
(3) if the current capacity does not accommodate the requirement, repeat steps (1) and (2) for other migration(s) in the series of migrations which have not been added to the ordered migration plan based on current capacities of physical hosts until all migrations in the series of migrations are added to the ordered migration plan.
14. The method of claim 1, wherein determining the ordered migration plan comprises:
determining host capacities based on the state information, wherein determining the host capacities comprises aggregating state information associated with physical hosts managed by a first cloud provider as an aggregated physical host; and
determining a sequence of migrations of cloud resources from source physical hosts including the aggregated physical host to destination physical hosts including the aggregated physical host which does not violate capacity requirements of the destination physical hosts including the aggregated physical host during the sequence of migrations.
15. A cloud resource placement and rebalancing system for a federated cloud, the system comprising:
one or more memory elements;
one or more processors coupled to one or more memory elements; and
a resources monitor that when executed by the at least one processor is configured to monitor state information associated with cloud resources and physical hosts in the federated cloud having a plurality of clouds managed by a plurality of cloud providers;
a rebalance trigger that when executed by the at least one processor is configured to trigger, based on one or more conditions, a rebalancing request to initiate cloud resource placement optimization;
a cloud resource placement optimizer that when executed by the at least one processor is configured to determine an optimized placement of cloud resources on physical hosts across the plurality of clouds in the federated cloud based on (1) costs including migration costs, (2) the state information, and (3) constraints, wherein each physical host is identified in the constraints-driven optimization cloud resource placement solver by an identifier of a respective cloud provider and an identifier of the physical host, comprising:
defining M×N number of assignment variables indicating whether a particular cloud resource is to be placed on a particular physical host, wherein M is the number of cloud resources and N is the number of available physical hosts in the federated cloud;
defining N×M number of cost variables indicating cost of migrating a particular cloud resource from a current physical host to another physical host in the federated cloud and/or cost of placing the particular cloud resource on a particular physical host; and
solving for the optimized placement of cloud resources based on the assignment variables and the cost variables; and
a migrations enforcer that when executed by the at least one processor is configured to:
determine an ordered migration plan for the optimized placement of cloud resources based on the optimized placement of cloud resources and the state information; and
transmit one or more requests to place or migrate cloud resources in the plurality of clouds in the federated cloud according to the ordered migration plan.
16. The cloud resource placement and rebalancing system of claim 15, wherein monitoring the state information in the federated cloud comprises:
normalizing different types of state information associated with different cloud providers to a common schema.
17. The cloud resource placement and rebalancing system of claim 15, wherein:
state information associated with resources in the federated cloud comprises time-series information;
the rebalance trigger is further configured to apply a predictive model on the time-series information to determine predicted state information; and
the one or more conditions include the predicted state information meeting a predetermined criteria.
18. A computer-readable non-transitory medium comprising one or more instructions, for cloud resource placement and migration optimization for a federated cloud, that when executed on a processor configure the processor to perform one or more operations comprising:
monitoring state information associated with cloud resources and physical hosts in the federated cloud having a plurality of clouds managed by a plurality of cloud providers;
triggering, based on one or more conditions, a rebalancing request to initiate cloud resource placement optimization;
determining, by a constraints-driven optimization cloud resource placement solver, an optimized placement of cloud resources on physical hosts across the plurality of clouds in the federated cloud based on (1) costs including migration costs, (2) the state information, and (3) constraints, wherein each physical host is identified in the constraints-driven optimization cloud resource placement solver by an identifier of a respective cloud provider and an identifier of the physical host, comprising:
defining M×N number of assignment variables indicating whether a particular cloud resource is to be placed on a particular physical host, wherein M is the number of cloud resources and N is the number of available physical hosts in the federated cloud;
defining N×M number of cost variables indicating cost of migrating a particular cloud resource from a current physical host to another physical host in the federated cloud and/or cost of placing the particular cloud resource on a particular physical host; and
solving for the optimized placement of cloud resources based on the assignment variables and the cost variables;
determining an ordered migration plan for the optimized placement of cloud resources based on the optimized placement of cloud resources and the state information; and
transmitting one or more requests to place or migrate cloud resources in the plurality of clouds in the federated cloud according to the ordered migration plan.
19. The computer-readable non-transitory medium of claim 18, wherein: determining the optimized placement of cloud resources comprises:
aggregating physical hosts managed by a first cloud provider as an aggregated physical host; and
wherein defining assignment variables and cost variables comprises defining the assignment variables and the cost variables based on the aggregated physical host and other physical hosts managed by cloud providers different from the first cloud provider;
defining constraints, in terms of the assignment variables and/or cost variables, based on one or more of the following: the state information, one or more policies, one or more business rules, one or more affinity rules, and one or more anti-affinity rules; and
wherein the solving comprises solving for the optimized placement of cloud resources based on the constraints and an objective function in terms of the assignment variables and cost variables.
20. The computer-readable non-transitory medium of claim 18, wherein determining the ordered migration plan comprises:
determining a sequence of migrations of cloud resources from source physical hosts to destination physical hosts which does not violate capacity requirements of the destination physical hosts during the sequence of migrations.
Patent Citations (440)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US5812773(A) | 1998-09-01 | Norin | Applicant |
| US5889896(A) | 1999-03-01 | Meshinsky et al. | Applicant |
| US6108782(A) | 2000-08-01 | Fletcher et al. | Applicant |
| US6178453(B1) | 2001-01-01 | Mattaway et al. | Applicant |
| US6298153(B1) | 2001-10-01 | Oishi | Applicant |
| US6343290(B1) | 2002-01-01 | Cossins et al. | Applicant |
| US6643260(B1) | 2003-11-01 | Kloth et al. | Applicant |
| US6683873(B1) | 2004-01-01 | Kwok et al. | Applicant |
| US6721804(B1) | 2004-04-01 | Rubin et al. | Applicant |
| US6733449(B1) | 2004-05-01 | Krishnamurthy et al. | Applicant |
| US6735631(B1) | 2004-05-01 | Oehrke et al. | Applicant |
| US6996615(B1) | 2006-02-01 | McGuire | Applicant |
| US7054930(B1) | 2006-05-01 | Cheriton | Applicant |
| US7058706(B1) | 2006-06-01 | Lyer et al. | Applicant |
| US7062571(B1) | 2006-06-01 | Dale et al. | Applicant |
| US7111177(B1) | 2006-09-01 | Chauvel et al. | Applicant |
| US7212490(B1) | 2007-05-01 | Kao et al. | Applicant |
| US7277948(B2) | 2007-10-01 | Igarashi et al. | Applicant |
| US7313667(B1) | 2007-12-01 | Pullela et al. | Applicant |
| US7379846(B1) | 2008-05-01 | Williams et al. | Applicant |
| US7480672(B2) | 2009-01-01 | Hahn et al. | Applicant |
| US7496043(B1) | 2009-02-01 | Leong et al. | Applicant |
| US7536476(B1) | 2009-05-01 | Alleyne | Applicant |
| US7567504(B2) | 2009-07-01 | Darling et al. | Applicant |
| US7583665(B1) | 2009-09-01 | Duncan et al. | Applicant |
| US7606147(B2) | 2009-10-01 | Luft et al. | Applicant |
| US7644437(B2) | 2010-01-01 | Volpano | Applicant |
| US7647594(B2) | 2010-01-01 | Togawa | Applicant |
| US7773510(B2) | 2010-08-01 | Back et al. | Applicant |
| US7808897(B1) | 2010-10-01 | Mehta et al. | Applicant |
| US7881957(B1) | 2011-02-01 | Cohen et al. | Applicant |
| US7917647(B2) | 2011-03-01 | Cooper et al. | Applicant |
| US8010598(B2) | 2011-08-01 | Tanimoto | Applicant |
| US8028071(B1) | 2011-09-01 | Mahalingam et al. | Applicant |
| US8041714(B2) | 2011-10-01 | Aymeloglu et al. | Applicant |
| US8121117(B1) | 2012-02-01 | Amdahl et al. | Applicant |
| US8171415(B2) | 2012-05-01 | Appleyard et al. | Applicant |
| US8234377(B2) | 2012-07-01 | Cohn | Applicant |
| US8244559(B2) | 2012-08-01 | Horvitz et al. | Applicant |
| US8250215(B2) | 2012-08-01 | Stienhans et al. | Applicant |
| US8280880(B1) | 2012-10-01 | Aymeloglu et al. | Applicant |
| US8284664(B1) | 2012-10-01 | Aybay et al. | Applicant |
| US8301746(B2) | 2012-10-01 | Head et al. | Applicant |
| US8345692(B2) | 2013-01-01 | Smith | Applicant |
| US8406141(B1) | 2013-03-01 | Couturier et al. | Applicant |
| US8407413(B1) | 2013-03-01 | Yucel et al. | Applicant |
| US8448171(B2) | 2013-05-01 | Donnellan et al. | Applicant |
| US8477610(B2) | 2013-07-01 | Zuo et al. | Applicant |
| US8495356(B2) | 2013-07-01 | Ashok et al. | Applicant |
| US8495725(B2) | 2013-07-01 | Ahn | Applicant |
| US8510469(B2) | 2013-08-01 | Portolani | Applicant |
| US8514868(B2) | 2013-08-01 | Hill | Applicant |
| US8532108(B2) | 2013-09-01 | Li et al. | Applicant |
| US8533687(B1) | 2013-09-01 | Greifeneder et al. | Applicant |
| US8547974(B1) | 2013-10-01 | Guruswamy et al. | Applicant |
| US8560639(B2) | 2013-10-01 | Murphy et al. | Applicant |
| US8560663(B2) | 2013-10-01 | Baucke et al. | Applicant |
| US8589543(B2) | 2013-11-01 | Dutta et al. | Applicant |
| US8590050(B2) | 2013-11-01 | Nagpal et al. | Applicant |
| US8611356(B2) | 2013-12-01 | Yu et al. | Applicant |
| US8612625(B2) | 2013-12-01 | Andries et al. | Applicant |
| US8630291(B2) | 2014-01-01 | Shaffer et al. | Applicant |
| US8639787(B2) | 2014-01-01 | Lagergren et al. | Applicant |
| US8656024(B2) | 2014-02-01 | Krishnan et al. | Applicant |
| US8660129(B1) | 2014-02-01 | Brendel et al. | Applicant |
| US8719804(B2) | 2014-05-01 | Jain | Applicant |
| US8775576(B2) | 2014-07-01 | Hebert et al. | Applicant |
| US8797867(B1) | 2014-08-01 | Chen et al. | Applicant |
| US8805951(B1) | 2014-08-01 | Faibish et al. | Applicant |
| US8850002(B1) | 2014-09-01 | Dickinson et al. | Applicant |
| US8850182(B1) | 2014-09-01 | Fritz et al. | Applicant |
| US8856339(B2) | 2014-10-01 | Mestery et al. | Applicant |
| US8909928(B2) | 2014-12-01 | Ahmad et al. | Applicant |
| US8918510(B2) | 2014-12-01 | Gmach et al. | Applicant |
| US8924720(B2) | 2014-12-01 | Raghuram et al. | Applicant |
| US8930747(B2) | 2015-01-01 | Levijarvi et al. | Applicant |
| US8938775(B1) | 2015-01-01 | Roth et al. | Applicant |
| US8959526(B2) | 2015-02-01 | Kansal et al. | Applicant |
| US8977754(B2) | 2015-03-01 | Curry, Jr. et al. | Applicant |
| US9009697(B2) | 2015-04-01 | Breiter et al. | Applicant |
| US9015324(B2) | 2015-04-01 | Jackson | Applicant |
| US9043439(B2) | 2015-05-01 | Bicket et al. | Applicant |
| US9049115(B2) | 2015-06-01 | Rajendran et al. | Applicant |
| US9063789(B2) | 2015-06-01 | Beaty et al. | Applicant |
| US9065727(B1) | 2015-06-01 | Liu et al. | Applicant |
| US9075649(B1) | 2015-07-01 | Bushman et al. | Applicant |
| US9130846(B1) | 2015-09-01 | Szabo et al. | Applicant |
| US9164795(B1) | 2015-10-01 | Vincent | Applicant |
| US9167050(B2) | 2015-10-01 | Durazzo et al. | Applicant |
| US9201701(B2) | 2015-12-01 | Boldyrev et al. | Applicant |
| US9201704(B2) | 2015-12-01 | Chang et al. | Applicant |
| US9203784(B2) | 2015-12-01 | Chang et al. | Applicant |
| US9223634(B2) | 2015-12-01 | Chang et al. | Applicant |
| US9244776(B2) | 2016-01-01 | Koza et al. | Applicant |
| US9251114(B1) | 2016-02-01 | Ancin et al. | Applicant |
| US9264478(B2) | 2016-02-01 | Hon et al. | Applicant |
| US9294408(B1) | 2016-03-01 | Dickinson et al. | Applicant |
| US9313048(B2) | 2016-04-01 | Chang et al. | Applicant |
| US9361192(B2) | 2016-06-01 | Smith et al. | Applicant |
| US9379982(B1) | 2016-06-01 | Krishna et al. | Applicant |
| US9380075(B2) | 2016-06-01 | He et al. | Applicant |
| US9432245(B1) | 2016-08-01 | Sorenson, III et al. | Applicant |
| US9432294(B1) | 2016-08-01 | Sharma et al. | Applicant |
| US9444744(B1) | 2016-09-01 | Sharma et al. | Applicant |
| US9473365(B2) | 2016-10-01 | Melander et al. | Applicant |
| US9503530(B1) | 2016-11-01 | Niedzielski | Applicant |
| US9558078(B2) | 2017-01-01 | Farlee et al. | Applicant |
| US9571570(B1) | 2017-02-01 | Mutnuru | Applicant |
| US9613078(B2) | 2017-04-01 | Vermeulen et al. | Applicant |
| US9628471(B1) | 2017-04-01 | Sundaram et al. | Applicant |
| US9658876(B2) | 2017-05-01 | Chang et al. | Applicant |
| US9692802(B2) | 2017-06-01 | Bicket et al. | Applicant |
| US9755858(B2) | 2017-09-01 | Bagepalli et al. | Applicant |
| US2001/0055303(A1) | 2001-12-01 | Horton et al. | Applicant |
| US2002/0073337(A1) | 2002-06-01 | Ioele et al. | Applicant |
| US2002/0143928(A1) | 2002-10-01 | Maltz et al. | Applicant |
| US2002/0166117(A1) | 2002-11-01 | Abrams et al. | Applicant |
| US2002/0174216(A1) | 2002-11-01 | Shorey et al. | Applicant |
| US2003/0018591(A1) | 2003-01-01 | Komisky | Applicant |
| US2003/0056001(A1) | 2003-03-01 | Mate et al. | Applicant |
| US2003/0212598(A1) | 2003-11-01 | Raman | Examiner |
| US2003/0228585(A1) | 2003-12-01 | Inoko et al. | Applicant |
| US2004/0004941(A1) | 2004-01-01 | Malan et al. | Applicant |
| US2004/0034702(A1) | 2004-02-01 | He | Applicant |
| US2004/0088542(A1) | 2004-05-01 | Daude et al. | Applicant |
| US2004/0095237(A1) | 2004-05-01 | Chen et al. | Applicant |
| US2004/0131059(A1) | 2004-07-01 | Ayyakad et al. | Applicant |
| US2004/0197079(A1) | 2004-10-01 | Latvala et al. | Applicant |
| US2004/0264481(A1) | 2004-12-01 | Darling et al. | Applicant |
| US2005/0060418(A1) | 2005-03-01 | Sorokopud | Applicant |
| US2005/0125424(A1) | 2005-06-01 | Herriott et al. | Applicant |
| US2006/0062187(A1) | 2006-03-01 | Rune | Applicant |
| US2006/0104286(A1) | 2006-05-01 | Cheriton | Applicant |
| US2006/0126665(A1) | 2006-06-01 | Ward et al. | Applicant |
| US2006/0146825(A1) | 2006-07-01 | Hofstaedter et al. | Applicant |
| US2006/0155875(A1) | 2006-07-01 | Cheriton | Applicant |
| US2006/0168338(A1) | 2006-07-01 | Bruegl et al. | Applicant |
| US2006/0233106(A1) | 2006-10-01 | Achlioptas et al. | Applicant |
| US2007/0174663(A1) | 2007-07-01 | Crawford et al. | Applicant |
| US2007/0223487(A1) | 2007-09-01 | Kajekar et al. | Applicant |
| US2007/0242830(A1) | 2007-10-01 | Conrado et al. | Applicant |
| US2008/0005293(A1) | 2008-01-01 | Bhargava et al. | Applicant |
| US2008/0080524(A1) | 2008-04-01 | Tsushima et al. | Applicant |
| US2008/0084880(A1) | 2008-04-01 | Dharwadkar | Applicant |
| US2008/0165778(A1) | 2008-07-01 | Ertemalp | Applicant |
| US2008/0198752(A1) | 2008-08-01 | Fan et al. | Applicant |
| US2008/0198858(A1) | 2008-08-01 | Townsley et al. | Applicant |
| US2008/0201711(A1) | 2008-08-01 | Amir Husain | Applicant |
| US2008/0235755(A1) | 2008-09-01 | Blaisdell et al. | Applicant |
| US2009/0006527(A1) | 2009-01-01 | Gingell, Jr. et al. | Applicant |
| US2009/0019367(A1) | 2009-01-01 | Cavagnari et al. | Applicant |
| US2009/0031312(A1) | 2009-01-01 | Mausolf et al. | Applicant |
| US2009/0083183(A1) | 2009-03-01 | Rao et al. | Applicant |
| US2009/0138763(A1) | 2009-05-01 | Arnold | Applicant |
| US2009/0177775(A1) | 2009-07-01 | Radia et al. | Applicant |
| US2009/0178058(A1) | 2009-07-01 | Stillwell, III et al. | Applicant |
| US2009/0182874(A1) | 2009-07-01 | Morford et al. | Applicant |
| US2009/0265468(A1) | 2009-10-01 | Annambhotla et al. | Applicant |
| US2009/0265753(A1) | 2009-10-01 | Anderson et al. | Applicant |
| US2009/0293056(A1) | 2009-11-01 | Ferris | Applicant |
| US2009/0300608(A1) | 2009-12-01 | Ferris et al. | Applicant |
| US2009/0313562(A1) | 2009-12-01 | Appleyard et al. | Applicant |
| US2009/0323706(A1) | 2009-12-01 | Germain et al. | Applicant |
| US2009/0328031(A1) | 2009-12-01 | Pouyadou et al. | Applicant |
| US2010/0036903(A1) | 2010-02-01 | Ahmad et al. | Applicant |
| US2010/0042720(A1) | 2010-02-01 | Stienhans et al. | Applicant |
| US2010/0050172(A1) | 2010-02-01 | Ferris | Examiner |
| US2010/0061250(A1) | 2010-03-01 | Nugent | Applicant |
| US2010/0115341(A1) | 2010-05-01 | Baker et al. | Applicant |
| US2010/0131765(A1) | 2010-05-01 | Bromley et al. | Applicant |
| US2010/0149966(A1) | 2010-06-01 | Achlioptas et al. | Applicant |
| US2010/0191783(A1) | 2010-07-01 | Mason et al. | Applicant |
| US2010/0192157(A1) | 2010-07-01 | Jackson et al. | Applicant |
| US2010/0205601(A1) | 2010-08-01 | Abbas et al. | Applicant |
| US2010/0211782(A1) | 2010-08-01 | Auradkar et al. | Applicant |
| US2010/0293270(A1) | 2010-11-01 | Augenstein et al. | Applicant |
| US2010/0318609(A1) | 2010-12-01 | Lahiri et al. | Applicant |
| US2010/0325199(A1) | 2010-12-01 | Park et al. | Applicant |
| US2010/0325441(A1) | 2010-12-01 | Laurie et al. | Applicant |
| US2010/0333116(A1) | 2010-12-01 | Prahlad et al. | Applicant |
| US2011/0016214(A1) | 2011-01-01 | Jackson | Applicant |
| US2011/0035754(A1) | 2011-02-01 | Srinivasan | Applicant |
| US2011/0055396(A1) | 2011-03-01 | Dehaan | Applicant |
| US2011/0055398(A1) | 2011-03-01 | Dehaan et al. | Applicant |
| US2011/0055470(A1) | 2011-03-01 | Portolani | Applicant |
| US2011/0072489(A1) | 2011-03-01 | Parann-Nissany | Applicant |
| US2011/0075667(A1) | 2011-03-01 | Li et al. | Applicant |
| US2011/0110382(A1) | 2011-05-01 | Jabr et al. | Applicant |
| US2011/0116443(A1) | 2011-05-01 | Yu et al. | Applicant |
| US2011/0126099(A1) | 2011-05-01 | Anderson et al. | Applicant |
| US2011/0138055(A1) | 2011-06-01 | Daly et al. | Applicant |
| US2011/0145413(A1) | 2011-06-01 | Dawson et al. | Applicant |
| US2011/0145657(A1) | 2011-06-01 | Bishop et al. | Applicant |
| US2011/0173303(A1) | 2011-07-01 | Rider | Applicant |
| US2011/0185063(A1) | 2011-07-01 | Head et al. | Applicant |
| US2011/0185065(A1) | 2011-07-01 | Stanisic et al. | Applicant |
| US2011/0206052(A1) | 2011-08-01 | Tan et al. | Applicant |
| US2011/0213966(A1) | 2011-09-01 | Fu et al. | Applicant |
| US2011/0219434(A1) | 2011-09-01 | Betz et al. | Applicant |
| US2011/0231715(A1) | 2011-09-01 | Kunii et al. | Applicant |
| US2011/0231899(A1) | 2011-09-01 | Pulier et al. | Applicant |
| US2011/0239039(A1) | 2011-09-01 | Dieffenbach et al. | Applicant |
| US2011/0252327(A1) | 2011-10-01 | Awasthi et al. | Applicant |
| US2011/0261811(A1) | 2011-10-01 | Battestilli et al. | Applicant |
| US2011/0261828(A1) | 2011-10-01 | Smith | Applicant |
| US2011/0276675(A1) | 2011-11-01 | Singh et al. | Applicant |
| US2011/0276951(A1) | 2011-11-01 | Jain | Applicant |
| US2011/0283013(A1) | 2011-11-01 | Grosser et al. | Applicant |
| US2011/0295998(A1) | 2011-12-01 | Ferris et al. | Applicant |
| US2011/0305149(A1) | 2011-12-01 | Scott et al. | Applicant |
| US2011/0307531(A1) | 2011-12-01 | Gaponenko et al. | Applicant |
| US2011/0320870(A1) | 2011-12-01 | Kenigsberg et al. | Applicant |
| US2012/0005724(A1) | 2012-01-01 | Lee | Applicant |
| US2012/0036234(A1) | 2012-02-01 | Staats et al. | Applicant |
| US2012/0054367(A1) | 2012-03-01 | Ramakrishnan et al. | Applicant |
| US2012/0072318(A1) | 2012-03-01 | Akiyama et al. | Applicant |
| US2012/0072578(A1) | 2012-03-01 | Alam | Applicant |
| US2012/0072581(A1) | 2012-03-01 | Tung et al. | Applicant |
| US2012/0072985(A1) | 2012-03-01 | Davne et al. | Applicant |
| US2012/0072992(A1) | 2012-03-01 | Arasaratnam et al. | Applicant |
| US2012/0084445(A1) | 2012-04-01 | Brock et al. | Applicant |
| US2012/0084782(A1) | 2012-04-01 | Chou et al. | Applicant |
| US2012/0089726(A1) | 2012-04-01 | Doddavula | Examiner |
| US2012/0096134(A1) | 2012-04-01 | Suit | Applicant |
| US2012/0102193(A1) | 2012-04-01 | Rathore et al. | Applicant |
| US2012/0102199(A1) | 2012-04-01 | Hopmann et al. | Applicant |
| US2012/0131174(A1) | 2012-05-01 | Ferris et al. | Applicant |
| US2012/0137215(A1) | 2012-05-01 | Kawara | Applicant |
| US2012/0158967(A1) | 2012-06-01 | Sedayao et al. | Applicant |
| US2012/0159097(A1) | 2012-06-01 | Jennas, II et al. | Applicant |
| US2012/0167094(A1) | 2012-06-01 | Suit | Applicant |
| US2012/0173710(A1) | 2012-07-01 | Rodriguez | Applicant |
| US2012/0179909(A1) | 2012-07-01 | Sagi et al. | Applicant |
| US2012/0180044(A1) | 2012-07-01 | Donnellan et al. | Applicant |
| US2012/0182891(A1) | 2012-07-01 | Lee et al. | Applicant |
| US2012/0185913(A1) | 2012-07-01 | Martinez et al. | Applicant |
| US2012/0192016(A1) | 2012-07-01 | Gotesdyner et al. | Applicant |
| US2012/0192075(A1) | 2012-07-01 | Ebtekar et al. | Applicant |
| US2012/0201135(A1) | 2012-08-01 | Ding et al. | Applicant |
| US2012/0214506(A1) | 2012-08-01 | Skaaksrud et al. | Applicant |
| US2012/0222106(A1) | 2012-08-01 | Kuehl | Applicant |
| US2012/0236716(A1) | 2012-09-01 | Anbazhagan et al. | Applicant |
| US2012/0240113(A1) | 2012-09-01 | Hur | Applicant |
| US2012/0265976(A1) | 2012-10-01 | Spiers et al. | Applicant |
| US2012/0272025(A1) | 2012-10-01 | Park et al. | Applicant |
| US2012/0281706(A1) | 2012-11-01 | Agarwal et al. | Applicant |
| US2012/0281708(A1) | 2012-11-01 | Chauhan et al. | Applicant |
| US2012/0290348(A1) | 2012-11-01 | Hackett | Examiner |
| US2012/0290647(A1) | 2012-11-01 | Ellison et al. | Applicant |
| US2012/0297238(A1) | 2012-11-01 | Watson et al. | Applicant |
| US2012/0311106(A1) | 2012-12-01 | Morgan | Applicant |
| US2012/0311568(A1) | 2012-12-01 | Jansen | Applicant |
| US2012/0324092(A1) | 2012-12-01 | Brown et al. | Applicant |
| US2012/0324114(A1) | 2012-12-01 | Dutta et al. | Applicant |
| US2013/0003567(A1) | 2013-01-01 | Gallant et al. | Applicant |
| US2013/0013248(A1) | 2013-01-01 | Brugler et al. | Applicant |
| US2013/0036213(A1) | 2013-02-01 | Hasan et al. | Applicant |
| US2013/0044636(A1) | 2013-02-01 | Koponen et al. | Applicant |
| US2013/0066940(A1) | 2013-03-01 | Shao | Applicant |
| US2013/0080509(A1) | 2013-03-01 | Wang | Applicant |
| US2013/0080624(A1) | 2013-03-01 | Nagai et al. | Applicant |
| US2013/0091557(A1) | 2013-04-01 | Gurrapu | Applicant |
| US2013/0097601(A1) | 2013-04-01 | Podvratnik et al. | Applicant |
| US2013/0104140(A1) | 2013-04-01 | Meng et al. | Applicant |
| US2013/0111540(A1) | 2013-05-01 | Sabin | Applicant |
| US2013/0117337(A1) | 2013-05-01 | Dunham | Applicant |
| US2013/0124712(A1) | 2013-05-01 | Parker | Applicant |
| US2013/0125124(A1) | 2013-05-01 | Kempf et al. | Applicant |
| US2013/0138816(A1) | 2013-05-01 | Kuo et al. | Applicant |
| US2013/0144978(A1) | 2013-06-01 | Jain et al. | Applicant |
| US2013/0152076(A1) | 2013-06-01 | Patel | Applicant |
| US2013/0152175(A1) | 2013-06-01 | Hromoko et al. | Applicant |
| US2013/0159097(A1) | 2013-06-01 | Schory et al. | Applicant |
| US2013/0159496(A1) | 2013-06-01 | Hamilton et al. | Applicant |
| US2013/0160008(A1) | 2013-06-01 | Cawlfield et al. | Applicant |
| US2013/0162753(A1) | 2013-06-01 | Hendrickson et al. | Applicant |
| US2013/0169666(A1) | 2013-07-01 | Pacheco et al. | Applicant |
| US2013/0179941(A1) | 2013-07-01 | McGloin et al. | Applicant |
| US2013/0182712(A1) | 2013-07-01 | Aguayo et al. | Applicant |
| US2013/0185433(A1) | 2013-07-01 | Zhu et al. | Applicant |
| US2013/0191106(A1) | 2013-07-01 | Kephart et al. | Applicant |
| US2013/0198374(A1) | 2013-08-01 | Zalmanovitch et al. | Applicant |
| US2013/0201989(A1) | 2013-08-01 | Hu et al. | Applicant |
| US2013/0204849(A1) | 2013-08-01 | Chacko | Applicant |
| US2013/0232491(A1) | 2013-09-01 | Radhakrishnan et al. | Applicant |
| US2013/0246588(A1) | 2013-09-01 | Borowicz et al. | Applicant |
| US2013/0250770(A1) | 2013-09-01 | Zou et al. | Applicant |
| US2013/0254415(A1) | 2013-09-01 | Fullen et al. | Applicant |
| US2013/0262347(A1) | 2013-10-01 | Dodson | Applicant |
| US2013/0283364(A1) | 2013-10-01 | Chang et al. | Applicant |
| US2013/0297769(A1) | 2013-11-01 | Chang et al. | Applicant |
| US2013/0318240(A1) | 2013-11-01 | Hebert et al. | Applicant |
| US2013/0318546(A1) | 2013-11-01 | Kothuri et al. | Applicant |
| US2013/0339949(A1) | 2013-12-01 | Spiers et al. | Applicant |
| US2014/0006481(A1) | 2014-01-01 | Frey et al. | Applicant |
| US2014/0006535(A1) | 2014-01-01 | Reddy | Applicant |
| US2014/0006585(A1) | 2014-01-01 | Dunbar et al. | Applicant |
| US2014/0040473(A1) | 2014-02-01 | Ho et al. | Applicant |
| US2014/0040883(A1) | 2014-02-01 | Tompkins | Applicant |
| US2014/0052877(A1) | 2014-02-01 | Mao | Applicant |
| US2014/0056146(A1) | 2014-02-01 | Hu et al. | Applicant |
| US2014/0059310(A1) | 2014-02-01 | Du et al. | Applicant |
| US2014/0074850(A1) | 2014-03-01 | Noel et al. | Applicant |
| US2014/0075048(A1) | 2014-03-01 | Yuksel et al. | Applicant |
| US2014/0075108(A1) | 2014-03-01 | Dong et al. | Applicant |
| US2014/0075357(A1) | 2014-03-01 | Flores et al. | Applicant |
| US2014/0075501(A1) | 2014-03-01 | Srinivasan et al. | Applicant |
| US2014/0089727(A1) | 2014-03-01 | Cherkasova et al. | Applicant |
| US2014/0098762(A1) | 2014-04-01 | Ghai et al. | Applicant |
| US2014/0108985(A1) | 2014-04-01 | Scott et al. | Applicant |
| US2014/0122560(A1) | 2014-05-01 | Ramey et al. | Applicant |
| US2014/0136779(A1) | 2014-05-01 | Guha et al. | Applicant |
| US2014/0140211(A1) | 2014-05-01 | Chandrasekaran et al. | Applicant |
| US2014/0141720(A1) | 2014-05-01 | Princen et al. | Applicant |
| US2014/0156557(A1) | 2014-06-01 | Zeng et al. | Applicant |
| US2014/0164486(A1) | 2014-06-01 | Ravichandran et al. | Applicant |
| US2014/0188825(A1) | 2014-07-01 | Muthukkaruppan et al. | Applicant |
| US2014/0189095(A1) | 2014-07-01 | Lindberg et al. | Applicant |
| US2014/0189125(A1) | 2014-07-01 | Amies et al. | Applicant |
| US2014/0215471(A1) | 2014-07-01 | Cherkasova | Applicant |
| US2014/0222953(A1) | 2014-08-01 | Karve et al. | Applicant |
| US2014/0244851(A1) | 2014-08-01 | Lee | Applicant |
| US2014/0245298(A1) | 2014-08-01 | Zhou et al. | Applicant |
| US2014/0281173(A1) | 2014-09-01 | Im et al. | Applicant |
| US2014/0282536(A1) | 2014-09-01 | Dave et al. | Applicant |
| US2014/0282611(A1) | 2014-09-01 | Campbell et al. | Applicant |
| US2014/0282889(A1) | 2014-09-01 | Ishaya et al. | Applicant |
| US2014/0289200(A1) | 2014-09-01 | Kato | Applicant |
| US2014/0295831(A1) | 2014-10-01 | Karra et al. | Applicant |
| US2014/0297569(A1) | 2014-10-01 | Clark et al. | Applicant |
| US2014/0297835(A1) | 2014-10-01 | Buys | Applicant |
| US2014/0310391(A1) | 2014-10-01 | Sorenson, III et al. | Applicant |
| US2014/0310417(A1) | 2014-10-01 | Sorenson, III et al. | Applicant |
| US2014/0310418(A1) | 2014-10-01 | Sorenson, III et al. | Applicant |
| US2014/0314078(A1) | 2014-10-01 | Jilani | Applicant |
| US2014/0317261(A1) | 2014-10-01 | Shatzkamer et al. | Applicant |
| US2014/0321278(A1) | 2014-10-01 | Cafarelli et al. | Applicant |
| US2014/0330976(A1) | 2014-11-01 | van Bemmel | Applicant |
| US2014/0330977(A1) | 2014-11-01 | van Bemmel | Applicant |
| US2014/0334488(A1) | 2014-11-01 | Guichard et al. | Applicant |
| US2014/0362682(A1) | 2014-12-01 | Guichard et al. | Applicant |
| US2014/0365680(A1) | 2014-12-01 | van Bemmel | Applicant |
| US2014/0366155(A1) | 2014-12-01 | Chang et al. | Applicant |
| US2014/0369204(A1) | 2014-12-01 | Anand et al. | Applicant |
| US2014/0372567(A1) | 2014-12-01 | Ganesh et al. | Applicant |
| US2014/0379938(A1) | 2014-12-01 | Bosch et al. | Applicant |
| US2015/0033086(A1) | 2015-01-01 | Sasturkar et al. | Applicant |
| US2015/0043576(A1) | 2015-02-01 | Dixon et al. | Applicant |
| US2015/0052247(A1) | 2015-02-01 | Threefoot et al. | Applicant |
| US2015/0052517(A1) | 2015-02-01 | Raghu et al. | Applicant |
| US2015/0058382(A1) | 2015-02-01 | St Laurent et al. | Applicant |
| US2015/0058459(A1) | 2015-02-01 | Amendjian et al. | Applicant |
| US2015/0071285(A1) | 2015-03-01 | Kumar et al. | Applicant |
| US2015/0085870(A1) | 2015-03-01 | Narasimha et al. | Applicant |
| US2015/0089082(A1) | 2015-03-01 | Patwardhan et al. | Applicant |
| US2015/0100471(A1) | 2015-04-01 | Curry, Jr. et al. | Applicant |
| US2015/0103827(A1) | 2015-04-01 | Quinn et al. | Applicant |
| US2015/0106802(A1) | 2015-04-01 | Ivanov et al. | Applicant |
| US2015/0106805(A1) | 2015-04-01 | Melander et al. | Applicant |
| US2015/0117199(A1) | 2015-04-01 | Chinnaiah Sankaran et al. | Applicant |
| US2015/0117458(A1) | 2015-04-01 | Gurkan et al. | Applicant |
| US2015/0120914(A1) | 2015-04-01 | Wada et al. | Applicant |
| US2015/0124622(A1) | 2015-05-01 | Kovvali et al. | Applicant |
| US2015/0127834(A1) | 2015-05-01 | Udupi et al. | Applicant |
| US2015/0138973(A1) | 2015-05-01 | Kumar et al. | Applicant |
| US2015/0178133(A1) | 2015-06-01 | Phelan et al. | Applicant |
| US2015/0189009(A1) | 2015-07-01 | van Bemmel | Applicant |
| US2015/0215819(A1) | 2015-07-01 | Bosch et al. | Applicant |
| US2015/0227405(A1) | 2015-08-01 | Jan et al. | Applicant |
| US2015/0242204(A1) | 2015-08-01 | Hassine et al. | Applicant |
| US2015/0249709(A1) | 2015-09-01 | Teng et al. | Applicant |
| US2015/0263901(A1) | 2015-09-01 | Kumar et al. | Applicant |
| US2015/0280980(A1) | 2015-10-01 | Bitar | Applicant |
| US2015/0281067(A1) | 2015-10-01 | Wu | Applicant |
| US2015/0281113(A1) | 2015-10-01 | Siciliano et al. | Applicant |
| US2015/0309908(A1) | 2015-10-01 | Pearson et al. | Applicant |
| US2015/0319063(A1) | 2015-11-01 | Zourzouvillys et al. | Applicant |
| US2015/0326524(A1) | 2015-11-01 | Tankala et al. | Applicant |
| US2015/0339210(A1) | 2015-11-01 | Kopp et al. | Applicant |
| US2015/0358850(A1) | 2015-12-01 | La Roche, Jr. et al. | Applicant |
| US2015/0365324(A1) | 2015-12-01 | Kumar et al. | Applicant |
| US2015/0373108(A1) | 2015-12-01 | Fleming et al. | Applicant |
| US2016/0011925(A1) | 2016-01-01 | Kulkarni et al. | Applicant |
| US2016/0013990(A1) | 2016-01-01 | Kulkarni et al. | Applicant |
| US2016/0026684(A1) | 2016-01-01 | Mukherjee et al. | Applicant |
| US2016/0062786(A1) | 2016-03-01 | Meng et al. | Applicant |
| US2016/0094389(A1) | 2016-03-01 | Jain et al. | Applicant |
| US2016/0094398(A1) | 2016-03-01 | Choudhury et al. | Applicant |
| US2016/0094453(A1) | 2016-03-01 | Jain et al. | Applicant |
| US2016/0094454(A1) | 2016-03-01 | Jain et al. | Applicant |
| US2016/0094455(A1) | 2016-03-01 | Jain et al. | Applicant |
| US2016/0094456(A1) | 2016-03-01 | Jain et al. | Applicant |
| US2016/0094480(A1) | 2016-03-01 | Kulkarni et al. | Applicant |
| US2016/0094643(A1) | 2016-03-01 | Jain et al. | Applicant |
| US2016/0099847(A1) | 2016-04-01 | Melander et al. | Applicant |
| US2016/0099853(A1) | 2016-04-01 | Nedeltchev et al. | Applicant |
| US2016/0099864(A1) | 2016-04-01 | Akiya et al. | Applicant |
| US2016/0105393(A1) | 2016-04-01 | Thakkar et al. | Applicant |
| US2016/0127184(A1) | 2016-05-01 | Bursell | Applicant |
| US2016/0134557(A1) | 2016-05-01 | Steinder et al. | Applicant |
| US2016/0156708(A1) | 2016-06-01 | Jalan et al. | Applicant |
| US2016/0164780(A1) | 2016-06-01 | Timmons et al. | Applicant |
| US2016/0164914(A1) | 2016-06-01 | Madhav et al. | Applicant |
| US2016/0182378(A1) | 2016-06-01 | Basavaraja et al. | Applicant |
| US2016/0188355(A1) | 2016-06-01 | Cao | Examiner |
| US2016/0188527(A1) | 2016-06-01 | Cherian et al. | Applicant |
| US2016/0234071(A1) | 2016-08-01 | Nambiar et al. | Applicant |
| US2016/0239399(A1) | 2016-08-01 | Babu et al. | Applicant |
| US2016/0253078(A1) | 2016-09-01 | Ebtekar et al. | Applicant |
| US2016/0254968(A1) | 2016-09-01 | Ebtekar et al. | Applicant |
| US2016/0261564(A1) | 2016-09-01 | Foxhoven et al. | Applicant |
| US2016/0277368(A1) | 2016-09-01 | Narayanaswamy et al. | Applicant |
| US2017/0005948(A1) | 2017-01-01 | Melander et al. | Applicant |
| US2017/0024260(A1) | 2017-01-01 | Chandrasekaran et al. | Applicant |
| US2017/0026294(A1) | 2017-01-01 | Basavaraja et al. | Applicant |
| US2017/0026470(A1) | 2017-01-01 | Bhargava et al. | Applicant |
| US2017/0041342(A1) | 2017-02-01 | Efremov et al. | Applicant |
| US2017/0054659(A1) | 2017-02-01 | Ergin et al. | Applicant |
| US2017/0097841(A1) | 2017-04-01 | Chang et al. | Applicant |
| US2017/0099188(A1) | 2017-04-01 | Chang et al. | Applicant |
| US2017/0104755(A1) | 2017-04-01 | Arregoces et al. | Applicant |
| US2017/0147297(A1) | 2017-05-01 | Krishnamurthy et al. | Applicant |
| US2017/0149878(A1) | 2017-05-01 | Mutnuru | Applicant |
| US2017/0163531(A1) | 2017-06-01 | Kumar et al. | Applicant |
| US2017/0171158(A1) | 2017-06-01 | Hoy et al. | Applicant |
| US2017/0264663(A1) | 2017-09-01 | Bicket et al. | Applicant |
| US2017/0339070(A1) | 2017-11-01 | Chang et al. | Applicant |
| CN101719930 | 2010-06-01 | Applicant | |
| CN101394360 | 2011-07-01 | Applicant | |
| CN102164091 | 2011-08-01 | Applicant | |
| CN104320342 | 2015-01-01 | Applicant | |
| CN105740084 | 2016-07-01 | Applicant | |
| EP2228719 | 2010-09-01 | Applicant | |
| EP2439637 | 2012-04-01 | Applicant | |
| EP2645253 | 2014-11-01 | Applicant | |
| KR10-2015-0070676 | 2015-05-01 | Applicant | |
| TWM394537 | 2010-12-01 | Applicant | |
| WO2009/155574 | 2009-12-01 | Applicant | |
| WO2010/030915 | 2010-03-01 | Applicant | |
| WO2013/158707 | 2013-10-01 | Applicant |
Non-Patent Literature (73)
- “OpenStack Cloud Administrator Guide,” Cloud Administrator Guide, Jul. 20, 2015, 333 pages.Applicant
- Al-Haj, et al., “A Formal Approach for Virtual Machine Migration Planning,” 9th International Conference on Network and Service Management (CNSM), Oct. 14-18, 2013, Zurich, Switzerland; 8 pages.Applicant
- Changbin, et al., “Cologne: A Declarative Distributed Constraint Optimization Platform,” University of Pennsylvania, Department of Computer and Information Science, Jan. 1, 2012, 14 pages.Applicant
- Bari, et al., “CQNCR: Optimal VM Migration Planning in Cloud Data Centers,” David R. Cheriton School of Computer Science, IEEE, Jun. 2, 2014, 9 pages.Applicant
- Jain, et al., “Topology-Aware VM Migration in Bandwidth Oversubscribed Datacenter Networks,” 39th International Colloquium Conference on Automata, Languages, and Programming (ICALP'12) , Riga, Latvia, Jul. 9-13, 2012, 36 pages.Applicant
- Wang, et al., “Virtual Machine Migration Planning in Software-Defined Networks,” Tsinghua National Laboratory for Information Science and Technology Department of Electronic Engineering, Tsinghua University, Dec. 16, 2014, 10 pages.Applicant
- “VMware vSphere Training: Migration Hot & Cold Migration-DRS,” 2015, first published on or about Aug. 2, 2014; 5 pages.Applicant
- Amedro, Brian, et al., “An Efficient Framework for Running Applications on Clusters, Grids and Cloud,” 2010, 17 pages.Applicant
- Author Unknown, “5 Benefits of a Storage Gateway in the Cloud,” Blog, TwinStrata, Inc., Jul. 25, 2012, XP055141645, 4 pages, https://web.archive.org/web/20120725092619/http://blog.twinstrata.com/2012/07/10//5-benefits-of-a-storage-gateway-in-the-cloud.Applicant
- Author Unknown, “Joint Cisco and VMWare Solution for Optimizing Virtual Desktop Delivery: Data Center 3.0: Solutions to Accelerate Data Center Virtualization,” Cisco Systems, Inc. and VMware, Inc., Sep. 2008, 10 pages.Applicant
- Author Unknown, “A Look at DeltaCloud: The Multi-Cloud API,” Feb. 17, 2012, 4 pages.Applicant
- Author Unknown, “About Deltacloud,” Apache Software Foundation, Aug. 18, 2013, 1 page.Applicant
- Author Unknown, “Architecture for Managing Clouds, A White Paper from the Open Cloud Standards Incubator,” Version 1.0.0, Document No. DSP-IS0102, Jun. 18, 2010, 57 pages.Applicant
- Author Unknown, “Cloud Infrastructure Management Interface—Common Information Model (CIMI-CIM),” Document No. DSP0264, Version 1.0.0, Dec. 14, 2012, 21 pages.Applicant
- Author Unknown, “Cloud Infrastructure Management Interface (CIMI) Primer,” Document No. DSP2027, Version 1.0.1, Sep. 12, 2012, 30 pages.Applicant
- Author Unknown, “cloudControl Documentation,” Aug. 25, 2013, 14 pages.Applicant
- Author Unknown, “Interoperable Clouds, A White Paper from the Open Cloud Standards Incubator,” Version 1.0.0, Document No. DSP-IS0101, Nov. 11, 2009, 21 pages.Applicant
- Author Unknown, “Microsoft Cloud Edge Gateway (MCE) Series Appliance,” Iron Networks, Inc., 2014, 4 pages.Applicant
- Author Unknown, “Open Data Center Alliance Usage: Virtual Machine (VM) Interoperability in a Hybrid Cloud Environment Rev. 1.2,” Open Data Center Alliance, Inc., 2013, 18 pages.Applicant
- Author Unknown, “Real-Time Performance Monitoring on Juniper Networks Devices, Tips and Tools for Assessing and Analyzing Network Efficiency,” Juniper Networks, Inc., May 2010, 35 pages.Applicant
- Author Unknown, “Use Cases and Interactions for Managing Clouds, A White Paper from the Open Cloud Standards Incubator,” Version 1.0.0, Document No. DSP-ISO0103, Jun. 16, 2010, 75 pages.Applicant
- Author Unknown, “Apache Ambari Meetup What's New,” Hortonworks Inc., Sep. 2013, 28 pages.Applicant
- Author Unknown, “Introduction,” Apache Ambari project, Apache Software Foundation, 2014, 1 page.Applicant
- Baker, F., “Requirements for IP Version 4 Routers,” Jun. 1995, 175 pages, Network Working Group, Cisco Systems.Applicant
- Beyer, Steffen, “Module “Data::Locations?!”,” YAPC::Europe, London, UK,ICA, Sep. 22-24, 2000, XP002742700, 15 pages.Applicant
- Blanchet, M., “A Flexible Method for Managing the Assignment of Bits of an IPv6 Address Block,” Apr. 2003, 8 pages, Network Working Group, Viagnie.Applicant
- Borovick, Lucinda, et al., “Architecting the Network for the Cloud,” IDC White Paper, Jan. 2011, 8 pages.Applicant
- Bosch, Greg, “Virtualization,” last modified Apr. 2012 by B. Davison, 33 pages.Applicant
- Broadcasters Audience Research Board, “What's Next,” http://lwww.barb.co.uk/whats-next, accessed Jul. 22, 2015, 2 pages.Applicant
- Cisco Systems, Inc. “Best Practices in Deploying Cisco Nexus 1000V Series Switches on Cisco UCS B and C Series Cisco UCS Manager Servers,” Cisco White Paper, Apr. 2011, 36 pages, http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-558242.pdf.Applicant
- Cisco Systems, Inc., “Cisco Unified Network Services: Overcome Obstacles to Cloud-Ready Deployments,” Cisco White Paper, Jan. 2011, 6 pages.Applicant
- Cisco Systems, Inc., “Cisco Intercloud Fabric: Hybrid Cloud with Choice, Consistency, Control and Compliance,” Dec. 10, 2014, 22 pages.Applicant
- Cisco Technology, Inc., “Cisco Expands Videoscape TV Platform Into the Cloud,” Jan. 6, 2014, Las Vegas, Nevada, Press Release, 3 pages.Applicant
- Citrix, “Citrix StoreFront 2.0” White Paper, Proof of Concept Implementation Guide, Citrix Systems, Inc., 2013, 48 pages.Applicant
- Citrix, “CloudBridge for Microsoft Azure Deployment Guide,” 30 pages.Applicant
- Citrix, “Deployment Practices and Guidelines for NetScaler 10.5 on Amazon Web Services,” White Paper, citrix.com, 2014, 14 pages.Applicant
- CSS Corp, “Enterprise Cloud Gateway (ECG)—Policy driven framework for managing multi-cloud environments,” original published on or about Feb. 11, 2012; 1 page; http://www.css-cloud.com/platform/enterprise-cloud-gateway.php.Applicant
- Fang K., “LISP MAC-EID-TO-RLOC Mapping (LISP based L2VPN),” Network Working Group, Internet Draft, CISCO Systems, Jan. 2012, 12 pages.Applicant
- Ford, Bryan, et al., Peer-to-Peer Communication Across Network Address Translators, In USENIX Annual Technical Conference, 2005, pp. 179-192.Applicant
- Gedymin, Adam, “Cloud Computing with an emphasis on Google App Engine,” Sep. 2011, 146 pages.Applicant
- Good, Nathan A., “Use Apache Deltacloud to administer multiple instances with a single API,” Dec. 17, 2012, 7 pages.Applicant
- Herry, William, “Keep It Simple, Stupid: OpenStack nova-scheduler and its algorithm”, May 12, 2012, IBM, 12 pages.Applicant
- Hewlett-Packard Company, “Virtual context management on network devices”, Research Disclosure, vol. 564, No. 60, Apr. 1, 2011, Mason Publications, Hampshire, GB, Apr. 1, 2011, 524.Applicant
- Juniper Networks, Inc., “Recreating Real Application Traffic in Junosphere Lab,” Solution Brief, Dec. 2011, 3 pages.Applicant
- Kenhui, “Musings on Cloud Computing and IT-as-a-Service: [Updated for Havana] Openstack Computer for VSphere Admins, Part 2: Nova-Scheduler and DRS”, Jun. 26, 2013, Cloud Architect Musings, 12 pages.Applicant
- Kolyshkin, Kirill, “Virtualization in Linux,” Sep. 1, 2006, XP055141648, 5 pages, https://web.archive.org/web/20070120205111/http://download.openvz.org/doc/openvz-intro.pdf.Applicant
- Kumar, S., et al., “Infrastructure Service Forwarding for NSH,” Service Function Chaining Internet Draft, draft-kumar-sfc-nsh-forwarding-00, Dec. 5, 2015, 10 pages.Applicant
- Kunz, Thomas, et al., “OmniCloud—The Secure and Flexible Use of Cloud Storage Services,” 2014, 30 pages.Applicant
- Lerach, S.R.O., “Golem,” http://www.lerach.cz/en/products/golem, accessed Jul. 22, 2015, 2 pages.Applicant
- Linthicum, David, “VM Import could be a game changer for hybrid clouds”, InfoWorld, Dec. 23, 2010, 4 pages.Applicant
- Logan, Marcus, “Hybrid Cloud Application Architecture for Elastic Java-Based Web Applications,” F5 Deployment Guide Version 1.1, 2016, 65 pages.Applicant
- Lynch, Sean, “Monitoring cache with Claspin” Facebook Engineering, Sep. 19, 2012, 5 pages.Applicant
- Meireles, Fernando Miguel Dias, “Integrated Management of Cloud Computing Resources,” 2013-2014, 286 pages.Applicant
- Meraki, “meraki releases industry's first cloud-managed routers,” Jan. 13, 2011, 2 pages.Applicant
- Mu, Shuai, et al., “uLibCloud: Providing High Available and Uniform Accessing to Multiple Cloud Storages,” 2012 IEEE, 8 pages.Applicant
- Naik, Vijay K., et al., “Harmony: A Desktop Grid for Delivering Enterprise Computations,” Grid Computing, 2003, Fourth International Workshop on Proceedings, Nov. 17, 2003, pp. 1-11.Applicant
- Nair, Srijith K. et al., “Towards Secure Cloud Bursting, Brokerage and Aggregation,” 2012, 8 pages, www.flexiant.com.Applicant
- Nielsen, “SimMetry Audience Measurement—Technology”, http://www.nielsen-admosphere.eu/products-and-services/simmetry-audience-measurement-technology/, accessed Jul. 22, 2015, 6 pages.Applicant
- Nielsen, “Television,” http://www.nielsen.com/us/en/solutions/measurement/television.html, accessed Jul. 22, 2015, 4 pages.Applicant
- Open Stack, “Filter Scheduler,” updated Dec. 17, 2017, 5 pages, accessed on Dec. 18, 2017, https://docs.openstack.org/nova/latest/user/filter-scheduler.html.Applicant
- Quinn, P., et al., “Network Service Header,” Internet Engineering Task Force Draft, Jul. 3, 2014, 27 pages.Applicant
- Quinn, P., et al., “Service Function Chaining (SFC) Architecture,” Network Working Group, Internet Draft, draft-quinn-sfc-arch-03.txt, Jan. 22, 2014, 21 pages.Applicant
- Rabadan, J., et al., “Operational Aspects of Proxy-ARP/ND in EVPN Networks,” BESS Worksgroup Internet Draft, draft-snr-bess-evpn-proxy-arp-nd-02, Oct. 6, 2015, 22 pages.Applicant
- Saidi, Ali, et al., “Performance Validation of Network-Intensive Workloads on a Full-System Simulator,” Interaction between Operating System and Computer Architecture Workshop, (IOSCA 2005), Austin, Texas, Oct. 2005, 10 pages.Applicant
- Shunra, “Shunra for HP Software; Enabling Confidence in Application Performance Before Deployment,” 2010, 2 pages.Applicant
- Son, Jungmin, “Automatic decision system for efficient resource selection and allocation in inter-clouds,” Jun. 2013, 35 pages.Applicant
- Sun, Aobing, et al., “IaaS Public Cloud Computing Platform Scheduling Model and Optimization Analysis,” Int. J. Communications, Network and System Sciences, 2011, 4, 803-811, 9 pages.Applicant
- Szymaniak, Michal, et al., “Latency-Driven Replica Placement”, vol. 47 No. 8, IPSJ Journal, Aug. 2006, 12 pages.Applicant
- Toews, Everett, “Introduction to Apache jclouds,” Apr. 7, 2014, 23 pages.Applicant
- Von Laszewski, Gregor, et al., “Design of a Dynamic Provisioning System for a Federated Cloud and Bare-metal Environment,” 2012, 8 pages.Applicant
- Wikipedia, “Filter (software)”, Wikipedia, Feb. 8, 2014, 2 pages, https://en.wikipedia.org/w/index.php?title=Filter_%28software%29&oldid=594544359.Applicant
- Wikipedia; “Pipeline (Unix)”, Wikipedia, May 4, 2014, 4 pages, https://en.wikipedia.org/w/index.php?title=Pipeline2/028Unix%29&oldid=606980114.Applicant
- Ye, Xianglong, et al., “A Novel Blocks Placement Strategy for Hadoop,” 2012 IEEE/ACTS 11th International Conference on Computer and Information Science, 2012 IEEE, 5 pages.Applicant