DATA CENTER CUSTOMER COST DETERMINATION MECHANISMS

[0001] The disclosed technology relates to the field of data centers and, more particularly, to various techniques pertaining to data center customer cost determination mechanisms that may be implemented in connection with data center operations.

[0002] A typical data center serves hundreds or even thousands of different data center customers by leasing machine resources to the data center customers. Each data center customer generally has some level of data center resource requirements that has associated therewith a certain cost to the data center itself. Data center resource requirements typically include demand for computational resources such as memory, disk space, processors, and bandwidth. For example, a data center customer having a significantly larger amount of data to be stored than other data center customers of the same data center will typically require a significantly greater amount of data storage resources, such as memory, from the data center.

[0003] In order to more efficiently manage a data center, it is important that the data center be provided with such data center customer cost information as accurately as possible via a data center manager, for example. Such information may be used by the data center manager for making decisions concerning possible pricing changes, billing practices, and service level agreement (SLA) design. In situations where the data center experiences a greater than expected usage of data center resources by the data center customer, for example, the data center manager may decide to apportion the cost back to the data center customer by raising rates. Unfortunately, present notions of data center customer cost are difficult to compute.

[0004] Accordingly, there remains a need for efficient and scalable methods for practically computing the costs of a data center's customers to the data center itself.

[0013] FIG. 1 illustrates an example of a data center optimization system 100 in accordance with embodiments of the disclosed technology. In the example, the data center optimization system 100 includes a data center management module 102, a data center customer registration module 104, a data center resource optimization module 106, and a data center customer cost determination module 108. As used herein, the data center management module 102 may include the involvement of a human data center manager in certain decision-making processes such as those pertaining to pricing, billing, and SLA design tasks. In the example, the data center optimization system 100 also includes an operations center 110 such as a group of data center servers, for example, and a data center resource usage model update module 112.

[0014] In the example, the data center customer registration module 104 may be used to register each new data center customer by facilitating execution of a data center customer-specific service level agreement (SLA) with the data center and establishing a data center resource usage model for the data center customer, where the data center resource usage model generally includes a quantification of the specific data center resources requested by the data center customer. For example, the data center customer registration module 104 may query the data center customer as to how much of each particular data center resource such as memory, disk space, and CPU bandwidth that the data center customer would like to request. The data center optimization system 100 may then create a data center customer profile for the data center customer and store both the SLA and the data center resource usage model for the data center customer as part of the data center customer profile.

[0015] The data center resource optimization module 106 may determine an initial and/or optimal data center resource allocation for a given data center customer based on the data center customer's SLA and data center resource usage model, for example, and then assign the data center resource allocation to the operations center 110, which may include a group of data center servers, for execution by the operations center 110. In certain embodiments, the data center resource optimization module 106 may use statistical packing techniques such as those described in co-pending U.S. patent application Ser. No. 12/253,111, titled “STATISTICAL PACKING OF RESOURCE REQUIREMENTS IN DATA CENTERS” and filed on Oct. 16, 2008, which application is fully incorporated herein by reference. One having ordinary skill in the art will appreciate, however, that various other data center resource optimization techniques may be employed by the data center resource optimization module 106.

[0016] The initial data center resource allocation determined by a typical data center resource optimization module may be optimal in that it represents an optimal usage of the data center resources to meet the needs of multiple customers of the data center but it does not necessarily provide, however, information on the cost of provisioning individual customers in the data center. In embodiments of the disclosed technology, the data center resource optimization module 106 may interact with the data center customer cost determination module 108, which may determine or estimate the cost to the data center of servicing, such as providing memory and processing resources to the particular data center customer.

[0017] In certain embodiments, the data center management module 102 may send a request to the data center customer cost determination module 108 for a determination of the data center customer cost for a particular data center customer. Alternatively, the data center resource management module 102 may send a request to the data center customer cost determination module 108 for a data center customer cost determination corresponding to a particular group of data center customers.

[0018] The data center customer cost determination module 108 may use any of the techniques described herein to determine the cost to the data center of the particular data center customer or group of data center customers. The determination of a data center customer cost for a given data center customer or group of data center customers may provide various business feedback pertaining to pricing, billing, SLA design or re-design, etc. For example, a data center manager may wish to investigate pricing terms for a particular data center customer or group of data center customers whose cost is higher than expected, such as when the data center customer's usage of the data center resources exceeds a level of usage that the data center expected based on the data center customer's profile. Consequently, providing a sound and fair costing mechanism may enable data center managers and business divisions to do a better job of understanding their operating costs, pricing their services, and billing their data center customers in an equitable manner.

[0019] The techniques described herein may be used to generate and provide accurate cost information to a data center manager. There are scenarios in which a data center may use such cost information to directly bill customers, such as in an in-house data center with internal customers and budget centers, for example. Alternative scenarios that include external customers who need a simple way to understand their bills, for example, may involve implementations in which simpler billing formulas are based on one or more categories of SLAs and actual resources used, for example. In situations where such simple billing formulas are used, application of the accurate cost feedback techniques described herein may enable the data center manager to adjust the simple billing formulas to reflect an accurate understanding of the costs of servicing the data center customers.

[0020] The data center resource usage model update module 112 may monitor the operations center 110 and employ modeling techniques such as hidden Markov modeling or computing histograms. The system may use such modeling techniques to model customer resource needs as initially provided via the data center customer registration module 104 and provide dynamic load prediction to the data center resource optimization module 106. Based on the monitoring of the operations center 110, the data center resource usage model update module 112 may provide recommendations to the data center customer registration module 104. For example, the data center resource usage model update module 112 may recommend that the data center customer registration module 104 revise the data center customer profile for a particular data center customer given the data center customer's usage of the operations center 110 over a certain period of time. In alternative embodiments, the data center resource usage model update module 112 may either revise the data center customer profile directly or provide a newly created data center customer profile to replace the existing data center customer profile for the data center customer.

[0021] Individual Customer Cost

[0022] Computing the costs of individual data center customers in situations where the optimization algorithm applied has determined resource provisioning, e.g., the number of servers that must be running, for an ensemble of data center customers can be problematic. For example, at least some of the extra provisioning used to ensure that data center customers will have adequate resources for future loads may be shared among multiple data center customers. Accordingly, it is generally not possible to directly associate costs with individual data center customers in such scenarios.

[0023] A simple computation of the marginal cost of adding an individual data center customer to an existing group of data center customers will likely understate the cost of the individual data center customer because the resource provisioning for the group as a whole may already include a significant amount of reserved but unused resources to meet contingency requirements so that members to the group may have the necessary resources according to their SLAs. In such situations, the marginal cost for the additional data center customer is typically low. In certain situations where the SLA requirements of an additional customer are low, there may already be enough resources reserved for the group of data center customers and, as a result, the marginal cost of the additional data center customer to the data center may be zero.

[0024] To address the partitioning of costs across a jointly-optimized group of data center customers, implementations of the disclosed technology may include the application of techniques derived from cooperative games, such as the Shapley value. These techniques may provide a principled way of apportioning data center customer costs. In order to more equitably treat all of a group of data center customers, these techniques may involve computations over all subsets of the group of customers. Such techniques, while effective for small numbers of customers, may become computationally prohibitive for large numbers of customers.

[0025] Described herein are methods for accelerating and approximating data center cost computations that do not compute over all subsets of customers, but instead exploit the structure of the definition of costs and groupings of customers to systematically reduce the subsets of customers considered and, as a result, render the computation scalable and efficient. While the techniques described herein generally pertain to the Shapley value, such techniques may also be applied to other solution concepts for cooperative games such as the Nucleolus or the Core as well as other variations of these concepts that involve computations over subsets of data center customers.

[0026] Described herein are various scalable and efficient methods of determining data center customer cost. In certain embodiments, the determination of a data center customer cost for a given data center customer may be made by performing an approximate computation of the data center cost for the data center customer. In certain embodiments, randomization techniques such as shuffling may be used in determining the data center customer cost. In alternative embodiments, grouping techniques such as clustering may be used. Such techniques may include grouping data center customers into groups by SLAs, for example, which may greatly reduce the amount of computation to be performed for a large number of data center customers.

[0027] Data Center Customer Cost Expressed as a Shapley Value

[0028] As used herein, a data center customer cost generally refers to a determination or estimation that reflects the cost to a data center of servicing the data center customer. In certain embodiments of the disclosed technology, the data center customer cost for a given data center customer is expressed as a Shapley Value. Consider the following definition of the marginal cost for “a after S,” where a is a particular data center customer in the set A of all of the data center customers, e.g., a set of size n data center customers, where S denotes a subset of other data center customers within A, and where v(S) denotes a quantification of a total optimal data center resource usage for the subset S, e.g., a cost for the subset S:

m(S,a):=v(S∪{a})−v(S)  (Equation 1)

[0029] Accordingly, one having ordinary skill in the art will recognize that the measure of marginal cost for the particular data center customer “a after S” may be defined by subtracting the total optimal data center resource usage of the subset of other data center customers S from the total optimal data center resource usage of the subset S taken with the data center customer a.

[0030] Either or both of the total optimal data center resource usage values in Equation 1 may be computed by a data center resource optimizer such as the data center resource optimization module 106 of FIG. 1, for example, in response to requests for the determinations by a data center customer cost determination module such as the data center customer cost determination module 108 of FIG. 1. The system may request the data center resource optimization module 106 to compute values for hypothetical subsets S that are different from the entire set A that it is optimizing for the operations center 110 of FIG. 1. Alternatively, the system may request the data center resource optimization module 106 to directly compute the marginal cost m(S,a) in Equation 1. The determinations of the total optimal data center resource usage values are not, however, limited to a data center resource optimization module; rather, a data center resource optimization module is but one of various types of mechanisms that may determine total optimal data center resource usage values.

[0031] One having ordinary skill in the art will appreciate that the marginal or incremental cost of a given data center customer a depends on the particular subset of other data center customers S with which it is grouped as well as the size of S. For example, adding a data center customer a to a small subset S frequently requires the reserving of additional resources, while adding a data center customer to a large subset may be more easily accomplished due to the sharing of resource reservations. Moreover, a given set of heterogeneous data center customers in the set S may interact in complex ways with a in terms of their net effect on the total optimal resource usage. So if the marginal costs of all elements of A are computed by adding them one-by-one to S, the result may be significantly biased. The Shapley value addresses such a scenario by averaging over all possible ordering of elements as they are added to S to assembly A.

[0032] In the example, the Shapley value of the data center customer a is denoted by s(A, a), which is defined as the average marginal cost of the data center customer a taken over all possible groupings S of other data center customers within A. It follows that the Shapley value computation may be written as:

$\begin{array}{cc}s\left(A,a\right):=\sum _{S⋐A\backslash \left\{a\right\}}\frac{S!\left(n-1-S\right)!}{n!}m\left(S,a\right).& \left(\mathrm{Equation}.2\right)\end{array}$

[0033] One having ordinary skill in the art will note, however, that the summation over all subsets S of A involves 2^n-1 evaluations of the expression m(S, a), which can make prohibitive the practical evaluation of the data center resource cost for groups of data center customers that are greater than a certain size, e.g., groups of more than ten data center customers.

[0034] Data Center Customer Cost Determination Using Randomization Techniques

[0035] An alternative but equivalent definition of the Shapley value s (A, a) for the data center customer a may be written as follows:

$\begin{array}{cc}s\left(A,a\right):=\frac{1}{n}\sum _{i=1}^n{\left(\begin{array}{c}n-1\\ i\end{array}\right)}^{-1}\sum _{\underset{S=i}{S⋐A\backslash \left\{a\right\}}}m\left(S,a\right)& \left(\mathrm{Equation}.3\right)\end{array}$

[0036] One having ordinary skill in the art will note that the expression inside the first summation represents the average of marginal costs taken over all subsets of size i. This interpretation suggests a natural way of determining an approximation Ŝ(A, a) of the Shapley value for the data center customer a where, rather than evaluating a marginal cost m(S, a) for all sets of size i an average is taken over a collection (e.g., a “reasonably representative” collection) of subsets S of A.

[0037] In certain embodiments, a randomization technique may be used to generate m random sample sets S of size i, for example, where each of the other data center customers has an equal chance of being selected. The number of random samples of subsets each size i may be adjusted to improve the accuracy of the results. Such a technique may be applied as an approximation to the full sum in Equation 3. Also, because of the equal likelihood of choosing any data center customer for the subsets S, these techniques may be considered equitable in the treatment of any particular individual data center customer.

[0038] Because randomization may create variation in the results and, therefore, the sum of the Shapley values for all of the individual data center customers may not exactly equal the total cost v(A), a more structured pattern may be used to choose the collection of subsets S of A. These structured collections may reduce the computation by computing significantly fewer v( . . . ) evaluations in Equation 1 and, in some cases, those that are computed are used in more than one marginal computation.

[0039] In certain embodiments, the structured collection of sample sets S may be generated from a permutation n of the elements A. Hereafter a_j will be used to indicate that j is the index of the element a in the set A. Sets S may be defined as prefixes of a permutation:

S(π,j)={k|π(k)<π(j)}  (Equation 4)

[0040] By using this notation, the Shapley value for a_j may be expressed as an average over all permutations as follows:

$\begin{array}{cc}s\left(A,a_j\right)=\frac{1}{n!}\sum _{\pi }m\left(S\left(\pi ,j\right),a_j\right)& \left(\mathrm{Equation}5\right)\end{array}$

[0041] For large n it can be prohibitively expensive to compute all permutations. A single permutation, however, could be inaccurate. For example, customers appearing early in the permutation would typically have higher costs than those appearing at the end. The system may address these issues by approximating the Shapley value using a family of permutations π⁽¹⁾, π⁽²⁾, . . . that are cyclic permutations of a single permutation, so that each element appears equally frequently at different positions in the permutation. Other permutations, such as a shuffle permutation, may be used in addition to or in replacement of the cyclic permutation to generate a family of permutations that are useful in approximating the Shapley value.

[0042] In situations where a small structured family of permutations is used to approximate the Shapley value, the normalization factor 1/n! in Equation 5 may be adjusted to the number of permutations used. As used herein, this will be referred to as a renormalized Equation 5. The structured family of permutations may also correspond to a structured family of subsets according to Equation 4 and, if these subsets are used to approximate the Shapley value, then the normalization factors in Equation 3 may be adjusted for the actual numbers of subsets (and sizes i) used. As used herein, this will be referred to as a renormalized Equation 3. While these approaches are similar with respect to their use of a small number of structured samples to approximate an exponential number of summands, one having ordinary skill in the art will appreciate that each approach to normalization is different.

[0043] While some of these computations describe above are described as computing the cost for a single data center customer, other implementations may be organized to compute the costs for many or all data center customers simultaneously. For example, when enumerating a structured family of permutations, the system may use the renormalized Equation 5 to compute the approximate Shapley value for all data center customers.

[0044] An effect that may bias the approximation of the Shapley value is that computing the marginal costs m(S, a_j) for a customer a_j may be sensitive to the presence of another customer a_k in the set S and, for this reason, it may be beneficial to first construct family Π^(−j) of cyclic permutations of the elements excluding a_j and then extend these to the family of full permutations Π by inserting j in each possible position in each permutation in Π^(−j).

[0045] In certain embodiments, randomization techniques may be combined with the above-described structured approaches to generating subsets. For example, several random permutations may be chosen first. Then, each random permutation may be elaborated to a family of permutations according to any of the methods described above by elaborating all the cyclic permutations of each random permutation, for example.

[0046] Because a typical data center manager often has a good idea as to what a “reasonably representative” set of data center customers might look like, he or she could construct a small collection of representative customers and a collection of representative subsets of data center customers and use those to compute an approximation of the Shapley value. The smaller numbers of representative customers and representative subsets may be used to approximate more efficiently the expression inside the first sum in Equation 3. Certain implementations may use the structuring techniques described above, such as the cyclic permutations, to derive a family of representative subsets from a representative set of data center customers.

[0047] FIG. 2 is a flowchart illustrating a first example of a method 200 of determining a data center customer cost using a collection of data center customers derived using randomization, representative customers, or families of permutations of customers or representative customers.

[0048] At 202, a data center customer cost determination module such as the data center customer cost determination module 108 of FIG. 1, for example, identifies some or all of the data center customers. For example, the data center customer cost determination module may only identify active data center customers such as data center customers whose leases have not lapsed. The data center customers may have an initial arrangement or ordering. For example, the data center customers may be ordered by a date such as by the lease initiation or termination date, by size such as by the amount of initial or current data center resource allocation, alphabetically by data center customer name, or by virtually any other way of ordering a group of data center customers.

[0049] At 204, the data center customer cost determination module selects a collection of subsets of data center customers. The data center customer cost determination module may select the collection of subsets in any of a number of different ways. For example, the data center customer cost determination module may select the collection of subsets randomly or as a collection of representative subsets or as a collection of subsets derived from a family of permutations.

[0050] At 206, the data center customer cost determination module then determines an approximation of the Shapley value for a data center customer based on the selected collection of subsets in accordance with the techniques described herein.

[0051] At 208, the data center customer cost determination module transmits the generated approximate Shapley value for the data center customer to a data center management interface such as the data center management module 102 of FIG. 1, for example.

[0052] FIG. 3 is a flowchart illustrating a second example of a method 300 of determining a data center customer cost using data center customer randomization techniques. At 302, a data center customer cost determination module such as the data center customer cost determination module 108 of FIG. 1, for example, identifies some or all of the data center customers. This step is similar to step 202 of FIG. 2.

[0053] At 304, the data center customer cost determination module selects a collection of permutations of data center customers. At 306, the data center customer cost determination module determines an approximation of the Shapley value based on the collection of permutations as selected at 304.

[0054] At 308, the data center customer cost determination module transmits the generated approximate Shapley value for the data center customer to a data center management interface such as the data center management module 102 of FIG. 1, for example.

[0055] Data Center Customer Cost Determination Using Groupings of Homogeneous Data Center Customers

[0056] In certain situations, the total set of data center customers A may be partitioned into N disjoint groups A_i of size n_i, where the data center customers in each group A_i may be treated as identical for computational purposes. This can arise, for example, in situations where a data center is operated such that customers choose from a small number of categories of SLA agreements. The data center resource needs of customers sharing the same SLA are generally similar enough that the system may consider approximating the cost computation by assuming that the members of each group have identical characteristics that are representative of the group. The computation of the marginal cost m(S,a) is likely to be much more sensitive to the kinds of SLAs in S than to the specific customers in S.

[0057] Given a subset S of data center customers whose entries are the number of data center customers in S from each group, a corresponding vector i:=(i₁, . . . , i_N) may be defined such that i* represents a sum of all the entries of i (i.e., i*=|S|). Because the data center customers within each group are identical, any two sets S₁ and S₂ having the same number of data center customers from each group will have the same optimal costs. In other words, v(S₁), v(S₂), and v(i) are all equal. Therefore, the marginal costs will also be equal, i.e., m(S₁, a) and m(S₂, a) are equal. Without loss of generality, one may assume that a is in the group A₁. The Shapley value for the cost of data center customer a may be expressed as the following:

$\begin{array}{cc}s\left(A,a\in A_1\right)=\frac{1}{n}\sum _{i=\left(0,\dots ,0\right)}^{\left(n_1,\dots ,n_N\right)}{\left(\begin{array}{c}n-1\\ i^{*}\end{array}\right)}^{-1}\left(\begin{array}{c}{n}_1-1\\ i_1\end{array}\right)\left(\begin{array}{c}{n}_2\\ i_2\end{array}\right)\dots \left(\begin{array}{c}{n}_N\\ i_N\end{array}\right)m\left(i,a\right)& \left(\mathrm{Equation}6\right)\end{array}$

[0058] In the definition above, m(i, a):=v(i+e₁)−v(i), and e₁=(1, 0, . . . , 0). One having ordinary skill in the art will appreciate that this definition of the Shapley value involves only O(n₁× . . . ×n_N) evaluations of m(i, a). In other words, the Shapley cost depends on the number of data center customers and each subgroup of customers, where all of the data center customers in each subgroup A_k are identical in that they each present the same data center resource needs and reservation requirements to the optimization. Equation 6 allows for the marginal costs associated with element a, for which the Shapley value is being computed, to be different from the other members of its group. In other words, each group of customers may be assumed to be populated by identical, representative customers, whereas a is allowed to differ from the representative for its group.

[0059] The computational complexity is consequently reduced from being exponential in the number of data center customers to being exponential only in the number of groups, which is usually significantly smaller as a typical data center can have hundreds of data center customers but may have less than five data center customer groups. Also, the number of requests to be sent by a data center customer cost determination module to a data center resource optimization module may be significantly reduced as the data center customer cost determination module need only query the data center resource optimization module for a smaller number of combinations.

[0060] In certain embodiments, it is possible to further reduce the computation by recognizing that, where data center customers are homogenous within certain groups, it would be equitable to consider only those subsets with i:=(i₁, . . . , i_N) that are representative in the sense that they are proportional to the customer group sizes n:=(n₁, . . . , n_N). Ideally i=αn but, since these are discrete vectors, the system may constrain i to be close to being proportional to n. This may be described as a balanced shuffling of N distinct groups, where each group contains indistinguishable elements. In a balanced shuffle, each prefix has each group represented approximately proportional to the group sizes. There are several methods that may be used to generate balanced shuffles. In one method, for example, elements may be drawn from a group of customers that is most underrepresented in the prefix, e.g., the group k with the largest difference (i*/n*) n_k−i_k. In other methods, elements may be drawn according to a probability distribution that favors the more underrepresented groups.

[0061] Using a set H of one or more balanced shuffles, the system may determine a structured family of subsets for computing the cost for a, taking each balanced shuffle h_i in the set H and generating n₁ shuffles, each of which may be generated by placing a in a different location in h_i that is already occupied by an element of A₁. This creates an expanded set of shuffles H*, expanding H by a factor n₁. The structured family of subsets are the elements in the shuffles, e.g., permutations, proceeding a. To compute the approximate Shapley value for a, a renormalized version of Equation 5 may be applied to the structured family of permutations H*, or a renormalized version of Equation 3 may be applied to the structured family of subsets.

[0062] FIG. 4 is a flowchart illustrating a first example of a method 400 of determining a data center customer cost using data center customer grouping techniques. At 402, a data center customer cost determination module such as the data center customer cost determination module 108 of FIG. 1, for example, identifies two or more groups of data center customers. For example, the data center customer cost determination module may group together data center customers that are identical in that the data center resource usage models of the data center customers are at least substantially similar so that they may be replaced by identical representatives. For example, the data center customer cost determination module may group together data center customers that are identical, e.g., their service level agreements are at least substantially similar. FIG. 5 illustrates an example of a graphical representation of two sets of data center customer groups 502 and 504 that may be identified by the data center customer cost determination module.

[0063] At 404, the data center customer cost determination module performs a shuffle operation on the two groups of data center customers 502 and 504 by combining the data center customers of both groups into a single grouping 506 as illustrated in FIG. 5.

[0064] At 406, the data center customer cost determination module determines an approximation of the Shapley value for a data center customer by employing an approximation strategy based on shuffling. FIG. 6 illustrates an example of a graphical representation of determining an approximation of the Shapley value for data center customer a within one or more shuffles 506 by creating a family of permutations by inserting customer a systematically at all positions of the first group in a shuffle, and using the approximation techniques described herein such as a renormalized Equation 5, for example.

[0065] At 408, the data center customer cost determination module transmits the generated approximate Shapley value to a data center management interface such as the data center management module 102 of FIG. 1, for example.

[0066] Data Center Customer Cost Determination Using Groupings of Heterogeneous Data Center Customers

[0067] In certain situations, data center customers may be grouped into N disjoint groups {A_i} such that the data center customers within each group are similar but not identical. If the data center customers within a group are similar enough, the data center customer cost determination module may simply approximate the system by N homogeneous groups, in which each group has identical data center customers that are equal to the average of the actual non-identical data center customers in the original group. A “reasonable similarity” may be determined in a number of different ways. For example, a test template of a certain number of data center customers, such as a group of 10 data center customers, may be used to test a new data center customer against the test template by comparing the new data center customer's marginal cost or full Shapley cost to that of the test template.

[0068] If the data center customers within each group are sufficiently different, however, a definition of the Shapley value for a data center customer a within a partition A₁ may be written as:

$\begin{array}{cc}s\left(A,a\in A_1\right):=\frac{1}{n}\sum _{i=1}^n{\left(\begin{array}{c}n-1\\ i\end{array}\right)}^{-1}\sum _{\underset{\underset{S_1+\dots +S_N=i}{S_1⋐A_1\backslash \left\{a\right\},\dots ,S_N⋐A_N}}{S=S_1\bigcup \dots \bigcup S_N}}m\left(S,a\right).& \left(\mathrm{Equation}7\right)\end{array}$

[0069] The data center customer cost determination module may then use a randomized approach to choosing sets S. The following normalization factor:

$\begin{array}{cc}{\left(\begin{array}{c}n-1\\ i\end{array}\right)}^{-1}& \left(\mathrm{Equation}8\right)\end{array}$

may be adjusted to the actual number of sample sets of size i used in the approximation. The random sample may be improved by adjusting the number of sample sets chosen of each size i. The sample may also be improved by choosing sets S where the number chosen from each group is proportionally representative of the size of each group, e.g., |S₁|˜|A₁|−1|S|/(|A|−1) and for k>1, |S_k|˜|A_k∥S|/(|A|−1).

[0070] In another approach, the balanced shuffling approach described above may be combined with the structured families of permutations described above. One or more balanced shuffles may be used to determine the positions of each group within the permutation, as illustrated in FIG. 8. Then, for each balanced shuffle, the structured families of permutations of the groups of customers may be used to determine the position of individual members of the group within the group locations in the larger shuffle.

[0071] FIG. 7 is a flowchart illustrating a second example of a method 700 of determining a data center customer cost using data center customer grouping techniques, and FIG. 8 illustrates an example of a graphical representation of a shuffle of a collection of subsets of data center customer groups 802 as performed in connection with the data center customer cost determination method 700 illustrated in FIG. 7.

[0072] At 702, a data center customer cost determination module such as the data center customer cost determination module 108 of FIG. 1, for example, identifies two data center customer groups.

[0073] At 704, the data center customer cost determination module performs a shuffle permutation operation on the two groups of data center customers by combining the data center customers of both groups into a single grouping 802.

[0074] At 706, the data center customer cost determination module determines an approximation of the Shapley value for a data center customer using one or more balanced shuffles 802 combined with permutations within the groups of the shuffle 804. The resulting collection of permutations may be used to approximate the Shapley value using a renormalized Equation 5, or using Equation 4 to create a collection of subsets and using a renormalized Equation 7.

[0075] At 708, the data center customer cost determination module transmits the generated approximate Shapley value for the data center customer to a data center management interface such as the data center management module 102 of FIG. 1, for example.

[0076] The following discussion is intended to provide a brief, general description of a suitable machine in which certain embodiments of the disclosed technology can be implemented. As used herein, the term “machine” is intended to broadly encompass a single machine or a system of communicatively coupled machines or devices operating together. Exemplary machines can include computing devices such as personal computers, workstations, servers, portable computers, handheld devices, tablet devices, and the like.

[0077] Typically, a machine includes a system bus to which processors, memory (e.g., random access memory (RAM), read-only memory (ROM), and other state-preserving medium), storage devices, a video interface, and input/output interface ports can be attached. The machine can also include embedded controllers such as programmable or non-programmable logic devices or arrays, Application Specific Integrated Circuits (ASICs), embedded computers, smart cards, and the like. The machine can be controlled, at least in part, by input from conventional input devices (e.g., keyboards and mice), as well as by directives received from another machine, interaction with a virtual reality (VR) environment, biometric feedback, or other input signal.

[0078] The machine can utilize one or more connections to one or more remote machines, such as through a network interface, modem, or other communicative coupling. Machines can be interconnected by way of a physical and/or logical network, such as an intranet, the Internet, local area networks, wide area networks, etc. One having ordinary skill in the art will appreciate that network communication can utilize various wired and/or wireless short range or long range carriers and protocols, including radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) 545.11, Bluetooth, optical, infrared, cable, laser, etc.

[0079] Embodiments of the disclosed technology can be described by reference to or in conjunction with associated data including functions, procedures, data structures, application programs, instructions, etc. that, when accessed by a machine, can result in the machine performing tasks or defining abstract data types or low-level hardware contexts. Associated data can be stored in, for example, volatile and/or non-volatile memory (e.g., RAM and ROM) or in other storage devices and their associated storage media, which can include hard-drives, floppy-disks, optical storage, tapes, flash memory, memory sticks, digital video disks, biological storage, and other tangible, physical storage media.

[0080] Associated data can be delivered over transmission environments, including the physical and/or logical network, in the form of packets, serial data, parallel data, propagated signals, etc., and can be used in a compressed or encrypted format. Associated data can be used in a distributed environment, and stored locally and/or remotely for machine access.

[0081] It will be appreciated that various of the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations, or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.