Interpretability Framework for Differentially Private Deep Learning

[0001] The subject matter described herein relates to an interpretability framework for calculating confidence levels and expected membership advantages of an adversary in identifying members of a training dataset used with training machine learning models.

[0002] Machine learning models can leak sensitive information about training data. To address such situations, noise can be added during the training process via differential privacy (DP) to mitigate privacy risk. To apply differential privacy, data scientists choose DP parameters (ϵ, δ). However, interpreting and choosing DP privacy parameters (ϵ, δ), and communicating the factual guarantees with regard to re-identification risk and plausible deniability is still a cumbersome task for non-experts. Different approaches for justification and interpretation of DP privacy parameters have been introduced which stray from the original DP definition by offering an upper bound on privacy in face of an adversary with arbitrary auxiliary knowledge.

[0003] In a first aspect, data is received that specifies a bound for an adversarial posterior belief ρ_c that corresponds to a likelihood to re-identify data points from the dataset based on a differentially private function output. Privacy parameters ε, δ are then calculated based on the received data that govern a differential privacy (DP) algorithm to be ρ_c on a conditional probability of different possible datasets on a ratio of probabilities distributions of different observations, which are bound by the posterior belief ρ_c as applied to a dataset. The calculated privacy parameters are then used to apply the DP algorithm to the function over the dataset.

[0004] The probability distributions can be generated using a Gaussian mechanism with an (ε, δ) guarantee that perturbs the result of the function evaluated over the dataset, preventing a posterior belief greater than ρ_c on the dataset.

[0005] The probability distributions can be generated using a Laplacian mechanism with an ε guarantee that perturbs the result of the function evaluated over the dataset, preventing a posterior belief greater than ρ_c on the dataset.

[0006] The resulting dataset (i.e., the dataset after application of the DP algorithm to the function over the dataset) can be used for various applications including training a machine learning model. Such a trained machine learning model can be deployed and then classify data input therein.

[0007] Privacy parameter ϵ can equal log(ρ_c/(1−ρ_c) for a series of (ε, δ) or ε anonymized function evaluations with multidimensional data.

[0008] A resulting total posterior belief ρ_c can be calculated using a sequential composition or Rényi differential privacy (RDP) composition. The at least one machine learning model can be updated using the calculated resulting total posterior belief ρ_c.

[0009] In an interrelated aspect, data is received that specifies privacy parameters ε, δ which govern a differential privacy (DP) algorithm to be applied to a function to be evaluated over a dataset. The received data is then used to calculate an can be used when applying the DP algorithm to a function over the dataset.

[0010] The probability distributions can be generated using a Gaussian mechanism with an (ε, δ) guarantee that perturbs the result of the function evaluated over the dataset, ensuring that membership advantage is ρ_α on the dataset.

[0011] The probability distributions can be generated using a Laplacian mechanism with an ε guarantee that perturbs the result of the function evaluated over the dataset, ensuring that membership advantage is ρ_α on the dataset.

[0012] The resulting dataset (i.e., the dataset after application of the DP algorithm to the function over the dataset) can be used to train at least one machine learning. Such a trained machine learning model can be deployed so as to classify further data input therein.

[0013] The calculated expected membership advantage ρ_α for a series of (ε, δ) anonymized function evaluations with multidimensional data is equal to:

$\mathrm{CDF}\left(\frac{1}{\frac{2\sqrt{2\ln \left(\frac{1.25}{\delta }\right)}}{ϵ}}\right)-\mathrm{CDF}\left(\frac{-1}{\frac{2\sqrt{2\ln \left(\frac{1.25}{\delta }\right)}}{ϵ}}\right)$

wherein CDF is the cumulative distribution function of the standard normal distribution.

[0014] A resulting expected membership advantage ρ_α can be calculated using sequential composition or Rényi differential privacy (RDP) composition. The calculated resulting expected membership advantage ρ_α can be used to update the at least one machine learning model.

[0015] In a further interrelated aspect, data is received that specify privacy parameters ε, δ which govern a differential privacy (DP) algorithm to be applied to a function to be evaluated over a dataset. Thereafter, the received data is used to calculate an adversarial posterior belief bound ρ_c that corresponds to a likelihood to re-identify data points from the dataset based on a differentially private function output. Such calculating can be based on an overlap of two probability distributions. The DP algorithm can then be applied, using the calculated adversarial posterior belief bound ρ_c, to a function over the dataset to result in an anonymized function output (e.g., machine learning model, etc.).

[0016] Posterior belief bound ρ_c can equal 1/(1+e^−ϵ) for a series of (ε, δ) or ε anonymized function evaluations with multidimensional data.

[0017] Data can be received that specifies an expected adversarial posterior belief bound expected ρ_c such that ρ_c=expected ρ_c+δ*(1−expected ρ_c).

[0018] The probability distributions can be generated using a differential privacy mechanism either with an (ε, δ) guarantee or with an ε guarantee that perturbs the result of the function evaluated over the dataset, preventing a posterior belief greater than ρ_c on the dataset.

[0019] At least one machine learning model can be anonymously trained using the resulting dataset (i.e., the dataset after application of the DP algorithm to the function over the dataset). A resulting total posterior belief ρ_c can be calculated using a sequential composition or Rényi differential privacy (RDP) composition. The at least one machine learning model can be updated using the calculated resulting total posterior belief ρ_c.

[0020] In a still further interrelated aspect, a dataset is received. Thereafter, at least one first user-generated privacy parameter is received which governs a differential privacy (DP) algorithm to be applied to a function evaluated over the received dataset. Using the received at least one first user-generated privacy parameter, at least one second privacy parameter is calculated based on a ratio or overlap of probabilities of distributions of different observations. Thereafter, the DP algorithm is applied, using the at least one second privacy parameter, to the function over the received dataset to result in an anonymized function output (e.g., machine learning model, etc.). At least one machine learning model can be anonymously trained using the dataset which, when deployed, is configured to classify input data.

[0021] The machine learning model(s) can be deployed once trained to classify input data when received.

[0022] The at least one first user-generated privacy parameter can include a bound for an adversarial posterior belief ρ_c that corresponds to a likelihood to re-identify data points from the dataset based on a differentially private function output. With such an arrangement, the calculated at least one second privacy parameter can include privacy parameters ε, δ and the calculating can be based on a ratio of probabilities distributions of different observations which are bound by the posterior belief ρ_c as applied to the dataset.

[0023] In another variation, the at least one first user-generated privacy parameter includes privacy parameters ε, δ. With such an implementation, the calculated at least one second privacy parameter can include an expected membership advantage ρ_α that corresponds to a probability of an adversary successfully identifying a member in the dataset and the calculating can be based on an overlap of two probability distributions.

[0024] In still another variation, the at least one first user-generated privacy parameter can include privacy parameters ε, δ. With such an implementation, the that corresponds to a likelihood to re-identify data points from the dataset based on a differentially private function output and the calculating can be based on an overlap of two probability distributions.

[0025] Non-transitory computer program products (i.e., physically embodied computer program products) are also described that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and/or commands or other instructions or the like via one or more connections, including but not limited to a connection over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, etc.

[0026] The subject matter described herein provides many technical advantages. For example, the current framework provides enhanced techniques for selecting a privacy parameter ϵ based on the re-identification confidence ρ_c and expected membership advantage ρ_α. These advantages were demonstrated on synthetic data, reference data and real-world data in a machine learning and data analytics use case which show that the current framework is suited for multidimensional queries under composition. The current framework furthermore allows the optimization of the utility of differentially private queries at the same (ρ_c, ρ_α) by considering the sensitive range S(ƒ) instead of global sensitivity Δƒ. The framework allows data owners and data scientists to map their expectations of utility and privacy, and derive the consequent privacy parameters ϵ.

[0027] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims.

[0043] Like reference symbols in the various drawings indicate like elements.

[0044] Provided herein is an interpretability framework for calculating the confidence ρ_c and expected membership advantage ρ_α of an adversary in identifying members of training data used in connection with one or machine learning models. These metrics are derived a prióri for multidimensional, iterative computations, as found in machine learning. The framework is compatible with composition theorems and alternative differential privacy definitions like Rényi Differential Privacy, offering a tight upper bound on privacy. For illustration purposes, the framework and resulting utility is evaluated on synthetic data, in a deep learning reference task, and in a real-world electric load forecasting benchmark.

[0045] The current subject matter provides a generally applicable framework for interpretation of the DP guarantee in terms of an adversary's confidence and expected membership advantage for identifying the dataset on which a differentially private result was computed. The framework adapts to various DP mechanisms (e.g., Laplace, Gaussian, Exponential) for scalar and multidimensional outputs and is well-defined even under composition. The framework allows users to empirically analyze a worst-case adversary under DP, but also gives analytical bounds with regard to maximum confidence and expected membership advantage.

[0046] The current subject matter, in particular, can be used to generate anonymized function output within specific privacy parameter bounds which govern the difficulty of getting insight into the underlying input data. Such anonymous function evaluations can be used for various purposes including training of machine learning models which, when deployed, can classify future data input into such models.

[0047] Also provided herein, are illustrations of how different privacy regimes can be determined by the framework independent of a specific use case.

[0048] Still further, with the current subject matter, privacy parameters for abstract composition theorems such as Rényi Differential Privacy in deep learning can be inferred from the desired confidence and membership advantage in our framework.

[0049] Differential Privacy.

[0050] Generally, data analysis can be defined as the evaluation of a function ƒ: DOM→R on some dataset ∈DOM yielding a result r∈R. Differential privacy is a mathematical definition for anonymized analysis of datasets. In contrast to previous anonymization methods based on generalization (e.g., k-anonymity), DP perturbs the result of a function ƒ(⋅) over a dataset ={d₁, . . . , d_n} s.t. it is no longer possible to confidently determine whether ƒ(⋅) was evaluated on or some neighboring dataset ′ differing in one individual. The neighboring dataset ′ can be created by either removing one data point from (unbounded DP) or by replacing one data point in with another from DOM (bounded DP). Thus, privacy is provided to participants in the dataset since their impact of presence (absence) on the query result becomes negligible. To inject differentially private noise into the result of some arbitrary function ƒ(⋅), mechanisms fulfilling Definition 1 are utilized.

[0051] A mechanism gives (ϵ, δ)-Differential Privacy if for all ⊆DOM differing in at most one element, and all outputs S⊆R

Pr()∈S)≤e^−ϵ·Pr()∈S)+δ

[0052] ϵ-DP is defined as (ϵ, δ=0)-DP and refer to the application of a mechanism to a function ƒ(⋅) as output perturbation. DP holds if mechanisms are calibrated to the global sensitivity i.e., the largest influence a member of the dataset can cause to the outcome of any θ(⋅). Let and ′ be neighboring datasets, the global ₁-sensitivity of a function ƒ is defined as Δƒ=∥ƒ()−ƒ(′)∥₁. Similarly, Δƒ₂=∥ƒ()−ƒ(′)∥₂ can be referred to as global ₂-sensitivity.

[0053] A popular mechanism for perturbing the outcome of numerical query functions ƒ is the Laplace mechanism. Following Definition 1 the Laplace mechanism adds noise calibrated to Δƒ by drawing noise from the Laplace distribution with mean μ=0.

[0054] Theorem 1 (Laplace Mechanism).

[0055] Given a numerical query functions ƒ:DOM→^k, the Laplace mechanism

_Lap(,ƒ,ϵ):=ƒ(D)+(z₁, . . . ,z_k)

[0056] is an ϵ-differentially private mechanism when all z_i with 1≤i≤k are independently drawn from

$\text{~}\mathrm{Lap}\left(z,\lambda =\frac{\Delta f}{\text{?}}\mu \right)=0.$ $\text{?}\text{indicates text missing or illegible when filed}$

[0057] A second DP mechanism used for output perturbation within this work is the Gaussian mechanism of Definition 2. The Gaussian mechanism uses ₂-sensitivity.

[0058] Theorem 2 (Gaussian Mechanism).

[0059] Given a numerical query function ƒ:DOM→^k, it exists σ s. t. the Gaussian mechanism

_Gau(,ƒ,ϵ,δ):=ƒ()+(z₁, . . . z_k)

[0060] is an (ϵ, δ)-differentially private mechanism for a given pair of ϵ, δ∈(0, 1) when all z_i with 1≤i≤k are independently drawn from ˜N (0, σ²).

[0061] Prior work has analyzed the tails of the normal distributions and found that bounding σ>Δƒ₂√{square root over (2 ln(1.25/δ))}/ϵ fulfills Theorem 2. However, these bounds have been shown to be loose and result in overly pessimistic noise addition.

[0062] A mechanism gives (α, ϵ_RDP)-RDP if for any adjacent ⊆DOM and α>1

$D_{\alpha }(ℳ\left(𝒟\right)ℳ\left({𝒟}^{\prime }\right))=\frac{1}{\alpha -1}\ln {{𝔼}_{x~ℳ\left({𝒟}^{\prime }\right)}\left(\frac{ℳ\left(𝒟\right)}{ℳ\left({𝒟}^{\prime }\right)}\right)}^{\alpha }\le {ϵ}_{\mathrm{RDP}}$

[0063] Calibrating the Gaussian mechanism in terms of Rényi differential privacy (RDP) is straight forward due to the relation ϵ_RDP=α·Δƒ₂²/2σ². One option is to split σ=Δƒ₂*η where η is called noise multiplier which is the actual term dependent on ϵ_RDP as Δβ₂ is fixed. A (α, RDP)-RDP guarantee converts to

$\left({ϵ}_{\mathrm{RDP}}-\frac{\ln \delta }{\alpha -1},\delta \right)-\mathrm{DP}$

which is not trivially invertible as multiple (α, ϵ_RDP) yield the same (ϵ, δ)-DP guarantee. A natural choice is to search for a (α, ϵ_RDP) causing a to be as low as possible. Hence, it can be expanded as follows:

$ϵ={ϵ}_{\mathrm{RDP}}-\frac{\ln \delta }{\alpha -1}=\alpha ·\Delta f_2^2/2{\sigma }^2-\frac{\ln \delta }{\alpha -1}=\alpha /2{\eta }^2-\frac{\ln \delta }{\alpha -1}$

[0064] and minimize

$\eta =\underset{\alpha }{\min }\sqrt{\alpha /2\left(ϵ+\left(\ln \delta \right)/\left(\alpha -1\right)\right)}$

[0065] which provides a tight bound on η and thus on σ for given (ϵ, δ).

[0066] The standard approach to analyze the privacy decay over a series of ϵ-DP mechanisms is the sequential composition theorem.

[0067] Theorem 3 (Sequential Composition).

[0068] Let _i provide (ϵ_i,δ_i)-Differential Privacy. The sequence of _{1, . . . ,k}() provides (Σ_i ϵ_i, Σ_i δ_i)-DP.

[0069] Sequential composition is, again, loose for (δ, δ)-DP which has resulted in various advanced theorems for composition. Yet, tight composition bounds are also studied in the RDP domain which has the nice property of ϵ_RDP,i being summed up as well. So, for the sequence of k mechanisms executions each providing (α, ϵ_RDP,i)-RDP the total guarantee composes to (α, Σ_i ϵ_RDP,i)-RDP. Using the equations above, a tight per step η can be derived from this.

[0070] These aspects of DP build the foundations of private deep learning. In private deep learning, the tradeoff between privacy and utility becomes important because practical neural networks need to offer a certain accuracy. Although increasing privacy through the (ϵ, δ) guarantee always decreases utility, other factors also affect the accuracy of models, such as the quantity of training data and the value chosen for the clipping norm .

[0071] Various properties of affect its optimal value. Unfortunately, Δƒ₂ cannot be determined in advance for the size of gradients, so it has been proposed to clip each per example gradient to , bounding the influence of one example on an update. This parameter can be set to maximize model accuracy and offer the rule to set to “the median of the norms of the unclipped gradients over the course of training.” The following effects can be taken into account: the clipped gradient may point in a different direction from the original gradient if is too small, but if is too large, the large magnitude of noise added decreases utility. Since gradients change over the course of training, the optimal value of at the beginning of training may no longer be optimal toward the end of training. Adaptively setting the clipping norm may further improve utility by changing as training progresses or setting differently for each layer. To improve utility for a set privacy guarantee, the value of can be tuned and adapted.

TABLE 1
prob notations
Symbol	Description
	Data universe , i.e., a set of individuals that are possibly
	present in the original dataset .
	⊂  denotes the subset of records of whom  prob knows
	they have participated in   .
n	|D|.
	Mechanism  and parameters, e.g. (ϵ, δ, f) for   Lap
f	Data analysis function f(•).
r	Differentially private output r yielded by   .
pω(*)	probability density function of     given world ω.

[0072] Strong Probabilistic Adversary.

[0073] For interpretation of (ϵ, δ) the privacy guarantee ϵ with regard to a desired bound on the Bayesian belief of a probabilistic adversary _prob. _prob's knowledge is modeled as the tuple (, n, , ƒ, r) which is defined in Table 1. _prob seeks to identify by evaluating possible combinations of missing individuals drawn from , which can be formally denoted as possible worlds:

Ψ={∪{d₁, . . . ,d_n}|d₁, . . . ,d_n∈\}

[0074] _prob assigns a probability to each world ω∈P, reflecting the confidence that w was used as input to . This confidence can be referred to as belief β(ω). The posterior belief of _prob on world ω_i is defined as conditional probability:

$\begin{array}{cc}\begin{array}{c}\beta \left({\omega }_i\right)=\mathrm{Pr}\left({\omega }_i❘ℳ(·)=r\right)\\ =\frac{\mathrm{Pr}\left(ℳ(·)=r❘{\omega }_i\right)·\mathrm{Pr}\left({\omega }_i\right)}{\mathrm{Pr}\left(ℳ(·)=r\right)}\\ =\frac{\mathrm{Pr}\left(ℳ(·)=r❘{\omega }_i\right)·\mathrm{Pr}\left({\omega }_i\right)}{{\sum }_j\mathrm{Pr}\left(ℳ(·)=r❘{\omega }_j\right)·\mathrm{Pr}\left({\omega }_j\right)}\\ =\frac{\mathrm{Pr}\left(ℳ\left({\omega }_i\right)=r\right)·\mathrm{Pr}\left({\omega }_i\right)}{{\sum }_j\mathrm{Pr}\left(ℳ\left({\omega }_j\right)=r\right)·\mathrm{Pr}\left({\omega }_j\right)}\end{array}& \left(1\right)\\ =\frac{p_{{\omega }_i}\left(r\right)·\mathrm{Pr}\left({\omega }_i\right)}{{\sum }_j{p}_{{\omega }_j}\left(r\right)·\mathrm{Pr}\left({\omega }_j\right)}& \left(2\right)\end{array}$

[0075] Using the fact that represents a continuous random variable and the choice of worlds is discrete, Bayes theorem allows inserting 's probability density function (PDF) in step 2. The firmest guess of _prob is represented by the world co having the highest corresponding belief. However, it is not guaranteed that co represents the true world . From this point, the terms confidence and posterior belief are used interchangeably.

[0076] The initial distribution over w reflects the prior belief on each world by _prob. It is assumed that this is a discrete uniform distribution among worlds, thus

$\mathrm{Pr}\left(\omega \right)=\frac{1}{\Psi }.\forall \omega \in \Psi .$

By bounding the belief β({tilde over (ω)}) for the true world co by a chosen constant ρ, a desired level of privacy can be guaranteed. It is noted that bounding the belief for the true world implicitly also bounds the belief for any other world.

[0077] The noise added to hide ω can be scaled to the sensitivity of the result to a change in {tilde over (ω)}. Instead of basing this value on global sensitivity, the largest possible contribution of any individual can be quantified as the sensitive range S(ƒ).

[0078] The sensitive range of a query function ƒ is the range of ƒ:

S(ƒ)=max_ω1,ω2∈Ψ∥ƒ(ω₁)−ƒ(ω₂)∥

[0079] This approach resulted in the introduction of differential identifiability which is defined below in Definition 4.

[0080] Given a dataset D, a randomized mechanism satisfies ρ-Differential Identifiability if among all possible datasets ₁, ₂, . . . , _m differing in one individual w.r.t. the posterior belief β, after getting the response r∈R, is bounded by ρ:

β(_i|)=r)≤ρ.  (3)

[0081] The notation of possible world ω∈Ψ is replaces by possible datasets which is semantically the same. ρ-Differential Identifiability implies that after receiving a mechanism's output r the true dataset can be identified by _prob with confidence β()≤ρ.

[0082] DP and Differential Identifiability have been shown to be equal when |Ψ|=2, since DP considers two neighboring datasets , ′ by definition. Specifically, Differential Identifiability is equal to bounded DP in this case, since possible worlds each have the same number of records. Under this assumption, the sensitive range S(ƒ) represents a special case of local sensitivity in which both and are fixed. It can be assumed that Δƒ is equal to S(ƒ). If this condition is met, the relation ρ↔∈ for _Lap is:

$\begin{array}{cc}\frac{𝒮\left(f\right)}{\lambda }=ϵ-\ln \left(\frac{\rho }{1-\rho }\right)\leftrightarrow \rho =\frac{1}{1+e^{-\frac{𝒮\left(f\right)}{\lambda }}}=\frac{1}{1+e^{-ϵ}}>\frac{1}{2}.& \left(4\right)\end{array}$

[0083] Framework for Interpreting DP.

[0084] Based on the original strong probabilistic adversary _prob provided above, an interpretability framework is formulated that allows to translate formal (ϵ, δ) guarantees into concrete re-identification probabilities. First, the original confidence upper bound of Equation (3) can be extended to work with arbitrary DP mechanisms and a discussion is provided with regard to how δ is integrated into the confidence bound. Second, _prob is extended to behave adaptively with regard to a sequence of mechanisms. It is shown below that that the resulting adaptive adversary _adapt behaves as assumed by composition theorems. Third, expected membership advantage ρ_α is defined and suggested as a privacy measure complementing ρ, which we will refer to as ρ_c in the following.

[0085] General Adversarial Confidence Bound.

[0086] According to Equation (4) the probabilistic adversary with unbiased priors (i.e., 0.5) regarding neighboring datasets , has a maximum posterior belief of 1/(1+e^−ϵ) when the ϵ-differentially private Laplace mechanism (cf. Definition 1) is applied to ƒ having a scalar output. In the following, it is shown that this upper bound holds also for arbitrary ϵ-differentially private mechanisms with multidimensional output. Therefore, the general belief calculation of Equation (1) can be bound by the inequality of Definition 1.

$\begin{array}{c}\beta \left(𝒟\right)=\frac{\mathrm{Pr}\left(ℳ\left(𝒟\right)=r\right)}{\mathrm{Pr}\left(ℳ\left(𝒟\right)=r\right)+\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)}\\ \le \frac{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)·e^{ϵ}+\delta }{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)·e^{ϵ}+\delta +\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)}\\ =\frac{1}{1+\frac{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)}{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)·e^{ϵ}+\delta }}\end{array}$

[0087] For δ=0, the last equation simplifies to 1/(1+e^−ϵ) so it can be concluded:

[0088] Corollary 1.

[0089] For any ϵ-differentially private mechanism, the strong probabilistic adversary's confidence on either dataset , ′ is bounded by

$\rho \left(ϵ\right)=\frac{1}{1+e^{-ϵ}}$

[0090] For δ>0, however, it was observed that where Pr(′)=r) becomes very small, β() grows towards 1:

$\begin{array}{cc}\underset{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)\right)\to 0}{\lim }\frac{1}{1+\frac{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)}{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)·e^{ϵ}+\delta }}=1& \left(5\right)\end{array}$

[0091] Hence, if the Gaussian mechanism _Gαu samples a value at the tails of the distribution in the direction away from ƒ(′), the posterior belief for and ′ head to 1 and 0, respectively. If a value is sampled from the tails in the direction of ƒ(′), the posterior belief for and ′ go to 0 and 1, respectively. The difference in behavior between the Laplace and Gaussian mechanism when large values of noise are sampled is demonstrated. Fixes ƒ()=0, ƒ(′)=1 and Δƒ=Δƒ₂=1 can be utilized. In diagram 100b of FIG. 1(b), the effect of the output of _Lap on the posterior beliefs for and ′ when ϵ=1, δ_Gαu results in an upper bound of 1, as is visualized in diagram 100a of FIG. 1(a). Therefore, β() can only be bound for 1−δ_GαuGαu provides δ-P with probability 1−δ.

[0092] β is now extended to k-dimensional (ϵ, δ)-differentially private mechanisms where ƒ()→^k.

[0093] Theorem 4.

[0094] The general confidence bound of Corollary 1 holds for multidimensional (ϵ, δ)-differentially private mechanisms with probability 1−δ.

[0095] Proof.

[0096] Properties of RDP can be used to prove the confidence bound for multidimensional (ϵ, δ)-differentially private mechanisms.

$\begin{array}{cc}\begin{array}{c}\beta \left(𝒟\right)=\frac{\mathrm{Pr}\left(ℳ\left(𝒟\right)=r\right)}{\mathrm{Pr}\left(ℳ\left(𝒟\right)=r\right)+\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)}\\ =\frac{1}{1+\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)/\mathrm{Pr}\left(ℳ\left(𝒟\right)=r\right)}\end{array}& \left(6\right)\\ \le \frac{1}{1+\frac{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)}{{\left(e^{{ϵ}_{\mathrm{RDP}}}·\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)\right)}^{1-1/\alpha }}}& \left(7\right)\\ =\frac{1}{1+e^{-{ϵ}_{\mathrm{RDP}}\left(1-1/\alpha \right)}·{\mathrm{Pr}\left(ℳ\left({𝒟}^{\prime }\right)=r\right)}^{1/\alpha }}& \left(8\right)\end{array}$

[0097] In the step from Equation (6) to (7), probability preservation properties are used to prove that RDP guarantees can be converted to (ϵ, δ) guarantees. In the context of this proof, it is implied that ϵ-DP holds when e^−ϵ RDP Pr((′)=r)>δ^α/(^α-1), since otherwise Pr(M()=r)<δ. It can therefore be assumed that e^−ϵ RDP(′)=r)>δ^α/(α-1), which occurs at most in 1−δ cases, and continue from Equation (8):

$\begin{array}{cc}\begin{array}{c}\le \frac{1}{1+e^{-{ϵ}_{\mathrm{RDP}}\left(1-1/\alpha \right)}·{\left({\delta }^{\alpha /\left(\alpha -1\right)}·e^{-{ϵ}_{\mathrm{RDP}}}\right)}^{1/\alpha }}\\ =\frac{1}{1+e^{-{ϵ}_{\mathrm{RDP}}}·{\delta }^{1/\left(\alpha -1\right)}}\\ =\frac{1}{1+e^{-{ϵ}_{\mathrm{RDP}}}·e^{-1/\left(\alpha -1\right)\ln \left(1/\delta \right)}}\\ =\frac{1}{1+e^{-\left({ϵ}_{\mathrm{RDP}}+{\left(\alpha -1\right)}^{-1}\ln \left(1/\delta \right)\right)}}\end{array}& \left(9\right)\\ =\frac{1}{1+e^{-ϵ}}& \left(10\right)\end{array}$

[0098] In the step from Equation (9) to (10), it is noted the exponent perfectly matches the conversion from ϵ to ϵ_RCP.

[0099] Consequently, Corollary 1 holds with probability 1−δ for _Gαu. Hence, the general confidence upper bound for (ε, δ)-differentially private mechanisms can be defined as follows:

[0100] For any (ϵ, δ)-differentially private mechanism, the expected bound on the strong probabilistic adversary's confidence on either dataset , ′ is

ρ_c(ϵ,δ)=[ρ(ϵ)]=(1−δ)ρ(ϵ)+δ=ρ(ϵ)+δ(1−ρ(δ)).

[0101] Adaptive Posterior Belief Adversary.

[0102] _prob computes posterior beliefs β(⋅) for datasets and ′ and makes a guess arg maxβ(). Therefore, the strong _prob represents a naive Bayes classifier choosing an option w.r.t. to the highest posterior probability. The input features are the results r observed by _prob, which are independently sampled and thus fulfill the i.i.d. assumption. Also, the noise distributions are known to _prob, thus making the naive Bayes classifier the strongest probabilistic adversary in our scenario.

[0103] A universal adversary against DP observes multiple subsequent function results and adapts once a new result r is obtained. To extend _prob to an adaptive adversary _adapt, adaptive beliefs can be defined as provided below.

[0104] Let , ′ be neighboring datasets ₁, ₂ be ϵ₁, ϵ₂-differentially private mechanisms. If ₁() is executed first with posterior belief β₁(), the adaptive belief for after executing ₂() is:

${\beta }_2\left(𝒟,{\beta }_1\left(𝒟\right)\right)=\frac{\mathrm{Pr}\left({ℳ}_2\left(𝒟\right)=r\right)·{\beta }_1\left(𝒟\right)}{\mathrm{Pr}\left({ℳ}_2\left(𝒟\right)=r\right)·{\beta }_1\left(𝒟\right)+\mathrm{Pr}\left({ℳ}_2\left({𝒟}^{\prime }\right)=r\right)\left(1-{\beta }_1\left(𝒟\right)\right)}$

[0105] Given k iterative independent function evaluations, β_k() is written to mean β_k(, β_k−1(, . . . )). To compute β_k(), the adaptive adversary _adapt computes adaptive posterior beliefs as specified by Algorithm 1.

Algorithm 1 Strong Adaptive Adversary
Input: datasets ,   ′, mechanism outputs R = r1, . . . , rk,
	mechanisms  1, . . . ,    k
Output: βk(   ), βk(  ′)
1:	β0(  ), β0(  ′) ← 0.5
2:	for i ∈ {1, . . . , k} do
3:	p   ← Pr(  i(   ) = ri)
4:	p  ′ ← Pr(  i(  ′) = ri)
5:	βi(   ) ← βi−1(   ) · p  /(p    · βi−1(   ) + p   ′ · βi−1(  ′))
6:	βi(  ′) ← βi−1(  ′) · p  /(p    · βi−1(   ) + p   ′ · βi−1(  ′))
7:	end for

[0106] The calculation of β_k () and β_k (′) as presented in Algorithm 1 can also be expressed as a closed form calculation which can be used later to further analyze the attacker.

$\begin{array}{c}{\beta }_k\left(𝒟\right)=\frac{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left(𝒟\right)=r_i\right)}{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left(𝒟\right)=r_i\right)+{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)}\\ =\frac{1}{1+\frac{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)}{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left(𝒟\right)=r_i\right)}}\end{array}$

[0107] Aspects of the associated proof are provided below in which it is assumed that the attacker starts with uniform priors. Thus, β₁ () is calculated to be:

$\begin{array}{c}{\beta }_1\left(𝒟\right)=\frac{\mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)}{\mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)+\mathrm{Pr}\left({ℳ}_1\left({𝒟}^{\prime }\right)=r_1\right)}\\ =\frac{1}{1+\frac{\mathrm{Pr}\left({ℳ}_1\left({𝒟}^{\prime }\right)=r_1\right)}{\mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)}}\end{array}$

[0108] In the second step β₁ () is used as the prior, hence β₂ () is calculated as:

$\begin{array}{c}{\beta }_2\left(𝒟\right)=\frac{\mathrm{Pr}\left({ℳ}_2\left(𝒟\right)=r_2\right)·{\beta }_1\left(𝒟\right)}{\begin{array}{c}\mathrm{Pr}\left({ℳ}_2\left(𝒟\right)=r_2\right)·{\beta }_1\left(𝒟\right)+\\ \mathrm{Pr}\left({ℳ}_2\left({𝒟}^{\prime }\right)=r_2\right)·\left(1-{\beta }_1\left(𝒟\right)\right)\end{array}}\\ =\frac{1}{1+\frac{\begin{array}{c}\mathrm{Pr}\left({ℳ}_2\left({𝒟}^{\prime }\right)=r_2\right)-\\ \mathrm{Pr}\left({ℳ}_2\left({𝒟}^{\prime }\right)=r_2\right)·{\beta }_1\left(𝒟\right)\end{array}}{\mathrm{Pr}\left({ℳ}_2\left(𝒟\right)=r_2\right)·{\beta }_1\left(𝒟\right)}}\\ =\frac{1}{1+\frac{\mathrm{Pr}\left({ℳ}_2\left({𝒟}^{\prime }\right)=r_2\right)-\frac{\begin{array}{c}\mathrm{Pr}\left({ℳ}_2\left({𝒟}^{\prime }\right)=r_2\right)·\\ \mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)\end{array}}{\begin{array}{c}\mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)+\\ \mathrm{Pr}\left({ℳ}_1\left({𝒟}^{\prime }\right)=r_1\right)\end{array}}}{\frac{\mathrm{Pr}\left({ℳ}_2\left(𝒟\right)=r_2\right)·\mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)}{\mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)+\mathrm{Pr}\left({ℳ}_1\left({𝒟}^{\prime }\right)=r_1\right)}}}\\ =\frac{1}{1+\frac{\mathrm{Pr}\left({ℳ}_2\left({𝒟}^{\prime }\right)=r_2\right)·\mathrm{Pr}\left({ℳ}_1\left({𝒟}^{\prime }\right)=r_1\right)}{\mathrm{Pr}\left({ℳ}_2\left(𝒟\right)=r_2\right)·\mathrm{Pr}\left({ℳ}_1\left(𝒟\right)=r_1\right)}}\end{array}$

[0109] This scheme continues for all k iterations by induction.

[0110] Even though the closed form provides an efficient calculation scheme for β_k (), numerical issues can be experienced so Algorithm 1 can be used for practical simulation of _adapt. However, by applying the closed form, it can be shown that _adapt operates as assumed by the sequential composition theorem (cf. 3) which substantiates the strength of _adapt. It is also noted that β₁ () has the same form as β_k (), since the multiplication of two Gaussian distributions results in another Gaussian distribution. Therefore, the composition of several Gaussian mechanisms can be regarded as a single execution of a multidimensional mechanism with an adjusted privacy guarantee.

[0111] Theorem 5 (Composition of Adaptive Beliefs).

[0112] Let , ′ be neighboring datasets and ₁, . . . , _k be an arbitrary sequence of mechanisms providing ϵ₁, . . . , ϵ_k-Differential Privacy, then

$\begin{array}{cc}{\beta }_k\left(𝒟\right)\le \rho \left(\sum _{i=1}^j{ϵ}_i\right)& \left(11\right)\end{array}$

[0113] By using Definition 1 and δ=0, the following can be bound:

$\begin{array}{c}{\beta }_k\left(𝒟\right)=\frac{1}{1+\frac{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)}{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left(𝒟\right)=r_i\right)}}\le \\ \frac{1}{1+\frac{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)}{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)*e^{ϵ}i}}\\ =\frac{1}{1+{\prod }_{i=1}^k{e}^{ϵ}i}\\ =\frac{1}{1+e^{-{\sum }_{i=1}^k{e}_i}}\\ =\rho \left(\sum _{i=1}^k{ϵ}_i\right)\square \end{array}$

[0114] This demonstrates that in the worst case _adapt takes full advantage of the composition of ϵ. But what about the case where δ>0? The same a, can be had in all dimensions as if it is assumed that the privacy budget (ϵ, δ) is split equally s. t. ϵ_i=ϵ_j and δ₁=δ_j which, given previous assumptions, leads to σ_i=σ_j, jϵ{1, . . . , k}. The following can be transformed:

$\begin{array}{cc}{\beta }_k\left(𝒟\right)=\frac{1}{1+\frac{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)}{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left(𝒟\right)=r_i\right)}}\le & \left(12\right)\\ \frac{1}{1+{\prod }_{i=1}^k{e}^{-{ϵ}_i}}=\frac{1}{1+e^{-{\sum }_{i=1}^k{e}_i}}=\rho \left(\sum _{i=1}^k{ϵ}_i\right)& \left(13\right)\end{array}$

[0115] In the step from Equation (12) to (13), simplifications from Equations (6) to (10) in Theorem 4 are used. This short proof demonstrates that _adapt behaves as expected by sequential composition theorems also for the (ϵ, δ)-differentially private Gaussian mechanism.

[0116] To take advantage of RDP composition, simplifications from Equation (6) to (9) can be used. The following transformations can be utilized:

$\begin{array}{cc}\begin{array}{c}{\beta }_k\left(𝒟\right)=\frac{1}{1+\frac{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)}{{\prod }_{i=1}^k\mathrm{Pr}\left({ℳ}_i\left(𝒟\right)=r_i\right)}}\\ =\frac{1}{1+{\prod }_{i=1}^k\frac{\mathrm{Pr}\left({ℳ}_i\left({𝒟}^{\prime }\right)=r_i\right)}{\mathrm{Pr}\left({ℳ}_i\left(𝒟\right)=r_i\right)}}\le \end{array}& \left(14\right)\\ \begin{array}{c}\mathrm{ }\frac{1}{1+{\prod }_{i=1}^k{e}^{-\left(e_{\mathrm{RDP},i}+{\left(\alpha +1\right)}^{-1}\ln \left(1/\delta \right)\right)}}\\ =\frac{1}{1+e^{{k\left(\alpha -1\right)}^{-1}\ln \left(1/\delta \right)-{\sum }_{i=1}^k{ϵ}_{\mathrm{RDP},i}}}\\ =\frac{1}{1+e^{{\left(\alpha -1\right)}^{-1}\ln \left(1/\delta \right)-{\sum }_{i=1}^k{ϵ}_{\mathrm{RDP},i}}}\end{array}& \left(15\right)\\ =\rho \left(\sum _{i=1}^k{ϵ}_{\mathrm{RDP},i}-{\left(\alpha -1\right)}^{-1}\ln \left(1/{\delta }^k\right)\right)& \left(16\right)\end{array}$

[0117] Equation (16) implies that an RDP-composed bound can be achieved with a composed δ value of δ^k. It is known that sequential composition results in a composed δ value of kδ. Since δ^k<kδ, RDP offers a stronger (ϵ, δ) guarantee for the same ρ_c. This behavior can also be interpreted as the fact that holding the composed (ϵ, δ) guarantee constant, the value of ρ_c is greater when sequential composition is used compared to RDP. Therefore, RDP offers a tighter bound for ρ_c under composition.

[0118] Expected Membership Advantage.

[0119] The adaptive posterior belief adversary allows to transform the DP guarantee (ϵ, δ) into a scalar measure ρ_c indicating whether _adapt can confidently re-identify an individual's record in a dataset. From an individual's point of view, of interest is deniability, i.e., if _adapt has low confidence, an individual can plausibly deny that the hypothesis of _adapt is correct. A resulting question concerns how often a guess by _adapt about the presence of an individual is actually correct or what _adapt's advantage is. As described above, it can be assumed that _adapt operates as a naive Bayes classifier with known probability distributions. Looking at the decision boundary of the classifier (i.e., when to choose or _Gαu with different (ϵ, δ) guarantees, it is found that the decision boundary does not change as long as the PDFs are symmetric. For example, consider a scenario with given datasets , ′ and query ƒ: DOM→ that yields ƒ ()=0 and ƒ (′)=1. Furthermore, assuming w.l.o.g. that Δƒ₂=1.

[0120] FIGS. 2(a) and 2(b) are diagrams 200a, 200b, respectively illustrating decision boundaries based on PDFs and confidence. It is shown that the decision boundary of _adapt does not change when increasing the privacy guarantee since (ϵ, δ) causes the PDFs of and ′ to become squeezed. Thus, _adapt will exclusively choose if a value is sampled from the left, red region, and vice versa for ′ in the right blue region. Still, confidence towards either decision declines.

[0121] If a (6, 10⁻⁶)-P _Gaαu is applied to perturb the results off ƒ, _adapt has to choose between the two PDFs with solid lines in FIG. 2a based on the output M_Gαu(⋅)=r. FIG. 2b visualizes the resulting posterior beliefs for ,D′ (solid lines), the highest of which _adapt chooses given r. The regions where _adapt chooses are shaded red in both figures, and regions that result in the choice ′ are shaded blue. Increasing the privacy guarantee to (3, 10⁻⁶)-DP (dashed lines in the figures) squeezes the PDFs and confidence curves. However, the decision boundary of the regions at which _adapt chooses a certain dataset stay the same. Thus, it is important to note that holding r constant and reducing (ϵ, δ) solely affects the posterior beliefs of _adapt, not the choice (i.e., the order from most to least confident is maintained even while maximum posterior belief is lowered).

[0122] However, it is expected that the information “How likely is an adversary to guess the dataset in which I have participated?” to be a major point of interest when interpreting DP guarantees in iterative evaluations of ƒ, like those found in data science use cases such as machine learning. Expected membership advantage ρ_α can be defined as the difference between the probabilities of _adapt correctly identifying (true positive rate) and of _adapt misclassifying a member of ′ as belonging to (false negative rate), as in 40. The worst-case advantage ρ_α=1 occurs in the case in which always samples on that side of the decision boundary that belongs to the true dataset . In contrast to the analysis of ρ_c, ρ_α will not give a worst-case bound, but an average case estimation. Since _adapt is a naive Bayes classifier, the properties of normal distributions can be used. With the multidimensional region where _adapt chooses as ∫_c and where _adapt chooses ′ as ∫_i:

ρ_α=Pr(Succes:=Pr(_adapt=|))−Pr(Error:=Pr(_adapt=|))=∫_cPr(()=r)Pr()dr−∫_DiPr(()=r)Pr()dr.

[0123] The corresponding regions of error for the previous example are visualized in diagrams 300a, 300b of FIGS. 3(a) and 3(b). If _Gau is applied to achieve (ϵ, δ)-DP, the exact membership advantage of _adapt can be determined analytically. Two multidimensional Gaussian PDFs (i.e., _Gau (), _Gau (′)) with known covariance matrix Σ and known means μ₁=ƒ(), μ₂=ƒ(′) can be considered.

$\begin{array}{cc}\begin{array}{c}\mathrm{Pr}\left(\mathrm{Success}\right)=1-\mathrm{Pr}\left(\mathrm{Error}\right)\\ =1-\Phi \left(-\Delta /2\right),\end{array}& \left(17\right)\end{array}$

[0124] where ϕ is the cumulative distribution function (CDF) of the standard normal distribution and Δ=√{square root over ((μ₁−μ₂)^TΣ⁻¹(μ₁−μ₂))} the Mahalanobis distance. Adding independent noise in all dimensions Σ=σ², the Mahalanobis distance simplifies to

$\Delta =\frac{{{\mu }_1-{\mu }_2}_2}{\sigma }.$

[0125] For the (ϵ, δ)-differentially private Gaussian mechanism, the expected membership advantage of the strong probabilistic adversary on either dataset is

$\begin{array}{c}{\rho }_a\left(ϵ,\delta \right)=\Phi \left(\Delta /2\right)-\Phi \left(-\Delta /2\right)\\ =\Phi \left(\frac{{{\mu }_1-{\mu }_2}_2}{2{\sigma }_i}\right)-\Phi \left(-\frac{{{\mu }_1-{\mu }_2}_2}{2{\sigma }_i}\right)\\ =\Phi \left(\frac{{{\mu }_1-{\mu }_2}_2}{2\Delta f_2\left(\sqrt{2\ln \left(1.25/\delta \right)}/ϵ\right)}\right)-\\ \Phi \left(-\frac{{{\mu }_1-{\mu }_2}_2}{2\Delta f_2\left(\sqrt{2\ln \left(1.25/\delta \right)}/ϵ\right)}\right)\le \\ \Phi \left(\frac{1}{2\left(\sqrt{2\ln \left(1.25/\delta \right)}/ϵ\right)}\right)-\\ \Phi \left(-\frac{1}{2\left(\sqrt{2\ln \left(1.25/\delta \right)}/ϵ\right)}\right)\end{array}$

[0126] Again, the current framework can express (ϵ, δ) guarantees with δ>0 via a scalar value ρ_α. However, a specific membership advantage can be computed individually for different kinds of mechanisms .

[0127] Above it was evaluated how the confidence of _adapt changes under composition. A similar analysis of the membership advantage under composition is required. Again, the elucidations can be restricted to the Gaussian mechanism. As shown above, the k-fold composition of _Gαui, each step guaranteeing (α,ϵ_RDP,i)-RDP, can be represented by a single execution of _Gαu with k-dimensional output guaranteeing (α, ϵ_RDP=kϵ_RDP,i)-RDP. For this proof, it can be assumed that each of the composed mechanism executions has the same sensitivity ∥μ_1,i−μ_2,i∥=Δ∫₂. A single execution of _Gαu can be analyzed with the tools described above. Definition 7 yields

$\begin{array}{c}{\rho }_a=\Phi \left(\Delta /2\right)-\Phi \left(-\Delta /2\right)\\ =\Phi \left(\frac{{{\mu }_1-{\mu }_2}_2}{2{\sigma }_i}\right)-\Phi \left(-\frac{{{\mu }_1-{\mu }_2}_2}{2{\sigma }_i}\right)\\ =\Phi \left(\frac{\sqrt{k}{{\mu }_{1,i}-{\mu }_{2,i}}_2}{2\Delta f_2\sqrt{\alpha /\left(2{ϵ}_{\mathrm{RDP},i}\right)}}\right)-\left(-\frac{\sqrt{k}{{\mu }_{1,i}-{\mu }_{2,i}}_2}{2\Delta f_2\sqrt{\alpha /\left(2{ϵ}_{\mathrm{RDP},i}\right)}}\right)\\ =\Phi \left(\frac{\sqrt{k}}{2\sqrt{\alpha /\left(2{ϵ}_{\mathrm{RDP},i}\right)}}\right)-\Phi \left(-\frac{\sqrt{k}}{2\sqrt{\alpha /\left(2{ϵ}_{\mathrm{RDP},i}\right)}}\right)\\ =\Phi \left(\sqrt{\frac{k{ϵ}_{\mathrm{RDP},i}}{2\alpha }}\right)-\Phi \left(-\sqrt{\frac{k{ϵ}_{\mathrm{RDP},i}}{2\alpha }}\right)\\ =\Phi \left(\sqrt{\frac{{ϵ}_{\mathrm{RDP}}}{2\alpha }}\right)-\Phi \left(-\sqrt{\frac{{ϵ}_{\mathrm{RDP}}}{2\alpha }}\right)\end{array}$

[0128] The result shows that the strategy of _adapt fully takes advantage of the RDP composition properties of ϵ_RDP,i and α. As expected, ρ_α takes on the same value, regardless of whether k composition steps with ϵ_RDP,i or a single composition step with ϵ_RDP is carried out.

[0129] Privacy Regimes.

[0130] With confidence ρ_c and expected membership advantage ρ_α two measures were defined that taken together form the current framework for interpreting DP guarantees (ϵ, δ). While ρ_α indicates the likelihood with which _adapt is to discover any participant's data correctly, ρ_c complements this information with the plausibility with which any participant in the data can argue that _adapt guess is incorrect. Here it is demonstrated how the current framework can be applied to measure the level of protection independent of any particular dataset . Furthermore, several, allegedly secure (ϵ, δ) pairs suggested in literature are revisited and interpret their protection. Finally, general guidance is provided to realize high, mid or low to no privacy regimes.

[0131] The interpretability framework can be applied in two steps. First, the participants in the dataset receive a predefined (ϵ, δ) guarantee. This (ϵ, δ) guarantee is based on the maximum tolerable decrease in utility (e.g., accuracy) of a function ƒ evaluated by a data analyst. The participants interpret the resulting tuple (ρ_c, ρ_α) w.r.t. their protection. Each participant can either reject or accept the use of their data by the data analyst. Second, participants are free to suggest an (ϵ, δ) based on the corresponding adversarial confidence and membership advantage (ρ_c, ρ_α) which is in turn evaluated by the data analyst w.r.t. expected utility of ƒ. To enable participants to perform this matching, general curves of ρ_c and ρ_α are provided for different (ϵ, δ) as shown in diagrams 400a, 400b of FIGS. 4(a), 4(b). For ρ_α, the curves are specific for _Gαu. In contrast, ρ_c is independent of . To compute both measures, Definition 5 and Definition 7 can be used. It can also be assumed w.l.o.g. that ƒ()=(0₁, 0₂, . . . , 0_k) and ƒ(′)=(1₁, 1₂, . . . , 1_k) for all dimensions k. Thus, ƒ() and Δƒ₂=√{square root over (k)}(′) are maximally distinguishable, resulting in FIG. 4a illustrates that there is no significant difference between the expected worst-case confidence of _adapt for ϵ-DP and (ϵ, δ)-P for 0<δ<0.1. In contrast, ρ_α strongly depends on the choice of δ as depicted in FIG. 4b. For example, ρ_α is low for (2, 10⁻⁶)-DP indicating that the probability of _adapt choosing is similar to choosing ′. Yet, the corresponding ρ_c is high, which provides support that _adapt guesses is correct. With these implications in mind, data owners and data analysts are empowered to discuss about acceptable privacy guarantees.

[0132] Validation Over Synthetic Data.

[0133] The following demonstrates how the confidence and membership advantage of _adapt develop in an empirical example. The evaluation characterizes how well ρ_c and ρ_α actually model the expected membership advantage risk for data members and how effective _adapt behaves on synthetic data. As _adapt is assumed to know all data member except for one, the size of does not influence her. For this reason, the following tiny data universe U, true dataset D and alternative D′ presented in Tables 2, 3 and 4 was used. Let U represent a set of employees that were offered to participate in a survey about their hourly wage. Alice, Bob and Carol participated. Dan did not. Thus, the survey data consists of 3 entries. The survey owner allows data analysts to pose queries to until a DP budget of (ϵ=5, δ=0.01) is consumed. _adapt is the data analyst that queries . Aside from learning statistics about the wage, _adapt is also interested in knowing about who participated. So far, she knows that Alice and Carol participated for sure and there are three people in total. Thus, she has to decide between and ′, i.e., whether Bob or Dan is the missing entry. As a side information, she knows that the employer pays at least $1 and a maximum of $10. As a consequence, when _adapt is allowed to ask only the sum query function, S(ƒ)=Δƒ₂=9. Further, the Gaussian mechanism is known to be used for anonymization.

TABLE 2
	Name	Wage
	Alice	$5
	Bob	$10
	Carol	$2
	Dan	$1

TABLE 3
	Name	Wage
	Alice	$5
	Bob	$10
	Carol	$2

TABLE 4
	Name	Wage
	Alice	$5
	Dan	$1
	Carol	$2

[0134] Given this prior information, _adapt iteratively updates her belief on , ′ after each query. She makes a final guess after the whole (ϵ, δ)-budget has been used. By using the current framework (ρ_c, ρ_α), data members (especially Bob in this case) can compute their protection guarantee: What is the advantage of _adapt in disclosing a person's participation (i.e., ρ_α)? How plausibly can that person deny a revelation (i.e., (ρ_c) by _adapt? Referring to Definition 7 and Definition 5, ρ_α(ϵ=5, δ=0.01)=0.5 is computed under composition and ρ_c(ϵ=5, δ=0.01)=0.99 illustrating that the risk of re-identification is quite high and the deniability extremely low. However, to show if _adapt actually reaches those values her behavior can be empirically analyzed by iteratively querying and applying Algorithm 1 after each query. k=100 queries can be used and the experiment can be repeated 10,000 times to estimate membership advantage and show the distribution of confidence at the end of each run. As it is known that the adversary will compose k times, the RDP composition scheme can be used to determine what noise scale can be applied for each individual query. In diagram 500a of FIG. 5a the adaptive posterior beliefs for and ′ are depicted. After some fluctuation, the belief for starts growing up to 0.90. Consequently, the final guess of _adapt is which is correct. The guesses over all runs are summarized in diagram 500b of FIG. 5(b). Here, it is seen that about 75% of the guesses of _adapt are correct which corresponds exactly to the expected membership advantage of our threat model. However, the predicted upper bound of 0.99 was not reached in the sample run. In contrast to ρ_α, ρ_c is a worst case bound, not an expectation. Thus, the final belief of _adapt approaches ρ_c very rarely.

[0135] To illustrate this phenomenon, a a histogram over the beliefs at the end of each run for various choices of β_c in diagram 600 of FIG. 6. FIG. 6 illustrates that the predicted worst case bound was reached in a small proportion of runs. This effect becomes more visible when ϵ is lower.

[0136] A final note on δ which describes the probability of exceeding ρ_c. When looking closely at the histograms, one can see that there are some (small) columns for a range of values that are larger than the worst-case bound. Their proportion to all runs can be calculated, e.g., 0.0008 for ϵ=5 which is less than the expected 6.

[0137] Application to Deep Learning.

[0138] A natural question arising is how the introduced adversary _adapt behaves on real data and high dimensional, iterative differentially private function evaluations. Such characteristics are typically found in Deep Learning classification tasks. Here, a neural network (NN) is provided a training dataset to learn a prediction function ŷ=ƒ_nn(x) given (x, y) ϵ. Learning is achieved by means of an optimizer. Afterwards, the accuracy of the learned prediction function ƒ_nn(⋅) is tested on a dataset ^test.

[0139] A variety of differentially private optimizers for deep learning can be utilized. These optimizers represent a differentially private training mechanism _nn(ƒ_θ(⋅)) that updates the weights θ_t per training step tϵT with θ_t←θ_t−1−α·{tilde over (g)}, where α>0 is the learning rate and {tilde over (g)} denotes the Gaussian perturbed gradient (cf. Definition 2). After T update steps, where each update step itself an application of _Gau(ƒ_θ(⋅)), the algorithm outputs a differentially private weight matrix θ which is then used in the prediction function θ_nn(⋅). Considering the evaluation of ƒ_nn(⋅) given (x, y)ϵ as post-processing of the trained weights θ, it is found that prediction ŷ=ƒ_nn(x) is (ϵ, δ)-differentially private too.

[0140] It is assumed that _adapt desires to correctly identify the dataset that was utilized for training when having the choice between and ′. There are two variations of DP: bounded and unbounded. In bounded DP, it holds that ||=|′|. However, differentially private deep learning optimizers such as the one utilized herein consider unbounded DP as the standard case in which ||−|′|=1. Furthermore, it can be assumed that _adapt to possess the following information: the initial weights θ₀, the perturbed gradients {tilde over (g)} after every epoch, the values of privacy parameters (ϵ, δ), and sensitivity Δƒ₂= equal to the clipping norm. Here, Δƒ₂ refers to the sensitivity with which noise added by a mechanism is scaled, not necessarily global sensitivity. In some experiments, for example, Δƒ₂=S(ƒ), which expresses the true difference between ƒ() and ƒ(′), as in Definition 3. The assumptions are analogous to those of white-box membership inference attacks. The attack itself is based on calculating clipped gradients ĝ(, θ_t) and ĝ (′, θ_t) for each training step tϵT, and finding β() for that training step given and calculating a θ_t+1 by applying ĝ.

[0141] Above, sensitivity was set to Δƒ₂=S(ƒ)=∥ƒ()−ƒ(′)∥₂, the true difference between the sums of wages resulting from Bob's and Dan's participation in a survey. In order to create a comparable experiment for differentially private deep learning, the difference between gradients that can be obscured by noise for each epoch is S(ƒ_θt)=∥n·ĝ(, θ_t)−(n−1)·ĝ (′, θ_t))∥₂. bounds the influence of a single training example on training by clipping each per-example gradient to the chosen value of ; although this value bounds the influence of a single example on the gradient, this bound is loose. If S(ƒ_θt<<, adversary confidence β() would be very small in every case when Δƒ₂=, as is the case in most implementations of differentially private neural networks. This behavior is due to the fact that an assumption for Equation (4) does not hold, since Δβ₂≠S(ƒ_θt). To address this challenge in differentially private deep learning, Δƒ₂=S(ƒ_θt) can be adaptively set. Choosing Δθ₂ this way is analogous to using local sensitivity in differential privacy.

Algorithm 2 Strong Adaptive Adversary in Deep Learning
Require: Datasets  and ′ with n and n − 1 records i
and i′, respectively, training steps T, cost function
J(θ), perturbed gradients gt for each training step t ≤ T,
initial weights θ0, prior beliefs β0( ) = β0( ′) = 0.5,
learning rate α, clipping threshold C, and mechanism
Ensure: Adversary Confidence βT( )
1:	for t ∈ [T] do Compute gradients
2:	For each i ∈ , ′, compute gt( i) ← ∇θtJ(θt, i) and
	gt( i′) ← ∇θtJ(θt, i′)
3:	Clip gradients
4:	Clip each gt( i), gt( i′) for i ∈ , ′
	to have a maximum L2 norm C using
	g _ t ⁡ ( 𝒟 i ) ← g t ⁡ ( 𝒟 i ) ⁢ / ⁢ max ⁡ ( 1 ,  g t ⁡ ( 𝒟 i )  2 c ) ⁢ ⁢ and ⁢ ⁢ g _ t ⁡ ( 𝒟 i ′ ) ←
	g t ⁡ ( 𝒟 i ′ ) ⁢ / ⁢ max ⁡ ( 1 ,  g t ⁡ ( 𝒟 i ′ )  2 c )
5:	Calculate Batch gradients
6:	ĝt( ) ← avg(gi( i))
7:	ĝt( ′) ← avg(gi( i′))
8:	Calculate Sensitivity
9:	Δft ← n · ∥n · ĝt( ′) − (n − 1) · ĝt( )∥2
10:	Calculate Belief
11:	β t + 1 ⁡ ( 𝒟 ) ← β t ⁡ ( 𝒟 ) * Pr ⁡ [ ℳ ⁡ ( g ^ t ⁡ ( 𝒟 ) ) = g ~ t ] β t ⁡ ( 𝒟 ) * Pr ⁡ [ ℳ ⁡ ( g ^ t ⁡ ( 𝒟 ) ) = g ~ t ] + β t ⁡ ( 𝒟 ′ ) * Pr ⁡ [ ℳ ⁡ ( g ^ t ⁡ ( 𝒟 ′ ) ) = g ~ t ]
	Compute weights
12:	θt+1 ← θt − α{tilde over (g)}t
13:	end for

[0142] Based on the previously introduced assumptions and notations Algorithm 1 can be adapted to be bounded and unbounded, as well as global and S(ƒ_θ)-based, settings. The adapted Strong Adaptive Adversary for differentially private deep learning is stated in Algorithm 2 specifies _adapt in an unbounded environment with Δθ₂=S(ƒ_θ). For bounded differential privacy with Δƒ₂=S(ƒ_θ), Algorithm 2 can be adjusted, s.t. ′ is defined to contain n records, and Δƒ=S(ƒ_θt)=n·∥ĝ_t (′)−ĝ_t()∥₂. To implement global unbounded differential privacy, Δƒ₂= and ′ contains n−1 records. To implement global bounded differential privacy, ′ contains n−1 records and Δθ₂=2, since the maximum influence of one example on the sum of per-example gradients is . If one record is replaced with another, the lengths of the clipped gradients of these two records could each be and point in opposite directions, which results in n·∥ĝ_t(′)−n·∥ĝ_t()∥₂=2. It is also noted that the same value of Δƒ₂ used by _adapt can also be used by to add noise.

[0143] For practical evaluation a feed-forward NN for the MNIST dataset was built. For MNIST the utilized NN architecture consists of two repetitions of a convolutional layer with kernel size (3, 3), batch normalization and max pooling with pool size (2, 2) before being flattened for the output layer. Relu and softmax activation functions were used for the convolutional layers and the output layer, respectively.

[0144] One epoch represents the evaluation of all records in . Thus, it is important to highlight that the number of update steps T varies in practice depending on the number of records from used for calculating the DP gradient update ĝ. In mini-batch gradient descent a number of b records from is used for calculating an update and one epoch results in t=/b update steps. In contrast in batch gradient descent, all records in are used for calculating the update and each epoch consists of a single update step. While the approaches vary in their speed of convergence due to the gradient update behavior (i.e., many small updates vs. few large updates) none of the approaches has hard limitations w.r.t. convergence of accuracy and loss. With the current subject matter, batch gradient descent was utilized and given differentially private gradient updates {tilde over (g)} after any update step t the previously introduced adversary _adapt shall decide whether it was calculated on or ′. It was assumed that _adapt has equal prior beliefs of 0.5 on and ′. The prior belief of _adapt adapts at every step t according to (1).

[0145] In the experiments, relevant parameters were set as follows: training data |D|=100, epochs k=30, clipping norm =3.0, learning rate α=0.005, 5=0.01, and ρ_c=0.9. These values correspond to ρ_α=25.62%.

TABLE 5
Empirical (pa, δ)
			Δf2 = S(fθ)	Global Δf2
		Bounded DP	(0.240, 0.002)	(0.108, 0)
		Unbounded DP	(0.250, 0.001)	(0.266, 0.001)

[0146] The empirically calculated values (i.e., after the training) for ρ_α and δ are presented in Table 5. The belief distributions for the described experiments can be found in diagrams 700, 800 of FIGS. 7-8.

[0147] Note that δ indeed bounds the percentage of experiments for which β_T()>ρ_c. For all experiments with Δƒ₂=S(ƒ_θ) and for global, unbounded DP, the empirical and analytical values of ρ_α match the empirical values. However, in global, bounded differential privacy the difference of correct guesses and incorrect guesses by _adapt falls below ρ_α. In this experiment, the percentage of experiments for which β_T()>ρ_c is also far lower. This behavior confirms the hypothesis that is loose, so global sensitivity results in a lower value of β_T(), as is again confirmed by FIGS. 7b and 9a. It is also noted that the distributions in FIGS. 7a and 7c look identical to each other and to the distributions in FIG. 6 for the respective values of ρ_c and δ. This observation confirms that the strong adaptive adversary attack model is applicable to choose privacy parameter e in deep learning.

[0148] The following investigates the reason for the similarities between unbounded differential privacy with Δƒ₂=S(ƒ_θ) and Δƒ₂= and also for the differences between FIGS. 7(a) and 7(b) concerning bounded differential privacy with Δƒ₂=S(ƒ_θ) and Δƒ₂=2. In the unbounded case, the distributions seem identical in diagram 800 of FIG. 8, which occurs when Δƒ₂=S(ƒ_θ)=∥(n−1)·ĝ_t (′)−n·ĝ_t()∥₂=, so the clipped per example gradient of the differentiating example in should have the length 3, which is equal to . This hypothesis is confirmed with a glance at the development of ∥(n−1)·ĝ_t(′)−n·ĝ_t()∥₂ in diagram 900a of FIG. 9. This behavior is not surprising, since all per example gradients over the course of all epochs were greater than or close to =3. In the bounded differential privacy experiments, Δƒ₂=S(ƒ_θ)=n·∥ĝ_t()−ĝ_t ()∥₂≠2, since the corresponding distributions in FIGS. 7(a) and 7(b), as well as FIG. 8, do not look identical. This expectation is confirmed by the plot of n·∥ĝ_t(′)−ĝ_t()∥₂ in FIG. 9(a). This difference implies that the per example gradients of the differentiating examples in ′ and are less than 2 and do not point in opposite directions. It is also noted that the length of gradients tends to decrease over the course of training, a trend that can be observed in diagram 900a of FIG. 9(a), so if training converges to a point in which gradients are shorter than the chosen value of , globally differentially private deep learning inherently offers a stronger privacy guarantee than was originally chosen.

[0149] Diagram 900(b) of FIG. 9(b) confirms that the differentially trained models in these models do, indeed, yield some utility. It was also observed that test accuracy is directly affected by the value of sensitivity Δƒ₂ chosen for noise addition. Since gradients in all four scenarios are clipped to the same value of <, the only differences between training the neural networks is Δƒ₂. As visualized in FIG. 9(a), sensitivities for unbounded DP with Δƒ₂=S(ƒ_θ) and Δƒ₂= were identical, so the nearly identical corresponding distributions in FIG. 9(b) do not come as a surprise.

[0150] Similarly, it is observed that Δƒ₂ is greater for global, bounded DP in FIG. 9(a), so utility is also lower for this case in FIG. 9(b). The unbounded DP case with Δƒ₂=S(ƒ_θ) yields the highest utility, which can be explained by the low value of Δƒ₂ that can be read from FIG. 9(b).

[0151] Relation to Membership Inference Threat Model.

[0152] The membership inference threat model and the analysis of _adapt herein exhibit clear commonalities. Namely, they exhibit the same overarching goal: to intuitively quantify the privacy offered by DP in a deep learning scenario. Both approaches aim to clarify the privacy risks associated with deep learning models.

[0153] Considering that _adapt desires to identify the dataset used for training a NN, _adapt is analogous to a membership inference adversary who desires to identify individual records in the training data. Furthermore, parallel to membership inference, _adapt operates in the whitebox model, observing the development of over all training steps t of a NN. In one approach, the adversary uses the training loss to infer membership of an example.

[0154] Although the general ideas overlap, _adapt is far stronger than a membership inference adversary. Membership advantage quantifies the effectiveness of the practical membership inference attack and therefore provides a lower bound on information leakage, which adversaries with auxiliary information quickly surpass. _adapt has access to arbitrary auxiliary information, including all data points in and ′, staying closer to the original DP guarantees. Using _adapt, what the best possible adversary is able to infer can be calculated and it can be seen that this adversary reaches the upper bound.

[0155] An adversarial game can be defined in which the adversary receives both datasets and ′ instead of only receiving one value z, the size n of the dataset, and the distribution from which the data points are drawn.

[0156] Experiment 1.

[0157] Let be an adversary, A be a learning algorithm, and ′ be neighboring datasets. The identifiability experiment proceeds as follows:

• • [0158] 1. Choose b←{0,1} uniformly at random
  • [0159] 2 Let = () if b=0 and = (′) if b=1
  • [0160] 3. Output 1 if (, ′, )=0, 0 otherwise. outputs 0 or 1.

[0161] Here, the expected value of membership advantage to quantify the accuracy of _adapt is calculated.

[0162] In the evaluation of _adapt in a deep learning setting, it was realized that _adapt did not reach the upper confidence bound until the sensitivity was adjusted. In differentially private deep learning, gradients decrease over the course of training until convergence and can fall below the sensitivity or clipping norm. This means that more noise is added than would have been necessary to obscure the difference made by a member of the dataset. Overall, the difference between the lower bound on privacy offered by membership advantage in a membership inference setting and the upper bound offered by maximum adversary confidence includes the auxiliary knowledge of _adapt and the inherent privacy offered in deep learning scenarios through decreasing gradients.

[0163] Application to Analytics.

[0164] The observed utility gains can also be realized on real-world data in an energy forecasting task that is relevant for German energy providers. In this energy forecasting problem, the energy transmission network is structured into disjoint virtual balancing groups under the responsibility of individual energy providers. Each balancing group consist of multiple zones and each zone consists of individual households. Energy providers have an incentive to forecast the demand in their balancing group with low error to schedule energy production accordingly. Currently, energy providers can utilize the overall aggregated energy consumption per balancing group to calculate a demand forecast, since they have to report these number to the transmission system operator. However, with the rollout of smart meters additional communication channels can be set up and the demand per zone could be computed. Note that, counterintuitively, forecasting on grouped household loads instead of individual households is beneficial for forecasting performance due to reduced stochasticity. Nonetheless, computing the demand per zone is reflecting the sum of individual household energy consumption and thus a sensitive task. Here, the use of differential privacy allows one to compute the anonymized energy consumption per zone and mitigate privacy concerns. The energy provider will only have an incentive to apply differential privacy, if the forecasting error based on differentially private energy consumption per zone is lower than the forecasting error based on the aggregated zonal loads. Vice versa the energy provider has to quantify and communicate the achieved privacy guarantee to the balancing group households to gather consent for processing the data.

[0165] This forecasting task was based on the dataset and benchmarking model of the 2012 Global Energy Forecast Competition (GEFCom). The GEFCom dataset consists of hourly energy consumption data of 4.5 years from 20 zones for training and one week of data from the same zones for testing. The GEFCom winning model computes the Forecast F for a zone z at time t by computing a linear regression over a p dimensional feature vector x (representing 11 weather stations):

$\begin{array}{cc}{F}_{z,t}={\beta }_0+\sum _{j=1}^p{\beta }_j·x_{t,j}+e_t.& \left(18\right)\end{array}$

[0166] The sensitive target attribute of the linear regression is zonal load consumption L_z,t, e.g., the sum of n household loads l_z,t,i, . . . , n. Differential privacy can be added to the sum computation by applying the Gaussian mechanism (cf. 2) yielding:

$\begin{array}{cc}{L}_{z.t^{\prime }}=\sum _{i=1}^{n_z}{l}_{z,i,t^{\prime }}+𝒩\left(0,{\sigma }^2\right)).& \left(19\right)\end{array}$

[0167] The energy provider will only have a benefit if the differentially private load forecasting is having a smaller error than the aggregated forecast. A suited error metric for energy forecasting is the Mean Absolute Error (MAE), i.e.:

$\begin{array}{cc}\mathrm{MAE}=\frac{1}{T}\sum _{t=1}^TF_t-L_t.& \left(20\right)\end{array}$

[0168] Diagram 1000a of FIG. 10(a) illustrates that the forecasting error over 10 independent forecast trainings over increasing privacy parameters c. Note that this illustration was limited to ϵ<0.4216 since the forecasting error is already exceeding the aggregate forecast error for this ϵ and continues to increase afterwards. Thus, from a utility perspective the energy provider will have a preference for ϵ>>0.4216. The privacy loss is again analyzed over composition (k=38, 070) with RDP for an additive privacy loss δ=10⁻⁹ and a global sensitivity of Δƒ=48 kW which is the maximum technical power demand fused in German residential homes. However, in practice households have been observed not to exceed power consumptions of up to 15.36 kW, which thus is used as an estimate for S(ƒ).

[0169] FIG. 10(b) illustrates the corresponding MAE when applying the Gaussian mechanism with Δƒ=S(ƒ)=15.36. Note that for both Gaussian mechanisms noise is sampled from a Gaussian distribution with σ=ηz·Δƒ, and that was used an equal z for both Gaussian mechanisms. FIG. 10 illustrates that the achievable utility is consistently higher.

[0170] In contrast to the energy provider, it is assumed that households have an interest in ρ_c<<1. This leads to the question whether the energy provider and the households have a intersection of their preferred c. FIG. 10(c) maps ρ_c and ρ_α to MAE over ϵ for δ=10⁻⁹ and either Δƒ=48 or Δƒ=S(ƒ)=15.36. It is observed for ρ_c≈0.65, which results in ϵ≈0.6 and ρ_α≈0.04, that the use of S(ƒ) instead of global sensitivity allows to reduce the maximum MAE by approximately 10 MW, which is significant when considering that the absolute difference between the aggregated forecast MAE and unperturbed forecast MAE is only ≈12 MW.

[0171] FIG. 11 is a process flow diagram 1100 in which, at 1110, data is received that specifies a bound for an adversarial posterior belief ρ_c that corresponds to a likelihood to re-identify data points from the dataset based on a differentially private function output. Privacy parameters ε, δ are then calculated, at 1120, based on the received data that govern a differential privacy (DP) algorithm to be applied to a function to be evaluated over a dataset. The calculating is based on a ratio of probabilities distributions of different observations, which are bound by the posterior belief ρ_c as applied to a dataset. The calculated privacy parameters are then used, at 1130, to apply the DP algorithm to the function over the dataset to result in an anonymized function output (e.g., a machine learning model, etc.).

[0172] FIG. 12 is a process flow diagram 1200 in which, at 1210, data is received that specifies privacy parameters ε, δ which govern a differential privacy (DP) algorithm to be applied to a function to be evaluated over a dataset. The received data is then used, at 1220, to calculate an expected membership advantage ρ_α that corresponds to a likelihood of an adversary successfully identifying a member in the dataset. Such calculating can be based on an overlap of two probability distributions. The calculated expected membership advantage ρ_α can be used, at 1230, when applying the DP algorithm to a function over the dataset to result in an anonymized function output (e.g., a machine learning model, etc.).

[0173] FIG. 13 is a process flow diagram 1300 in which, at 1310, data is received that specifies privacy parameters ε, δ which govern a differential privacy (DP) algorithm to be applied to a function to be evaluated over a dataset. Thereafter, at 1320, the received data is used to calculate an adversarial posterior belief bound ρ_c that ρ_c to a function over the dataset to result in an anonymized function output (e.g., machine learning model, etc.).

[0174] FIG. 14 is a process flow diagram in which, at 1410, a dataset is received. Thereafter, at 1420, at least one first user-generated privacy parameter is received which governs a differential privacy (DP) algorithm to be applied to a function evaluated over the received dataset. Using the received at least one first user-generated privacy parameter, at least one second privacy parameter is calculated, at 1430, based on a ratio or overlap of probabilities of distributions of different observations. Subsequently, at 1440, the DP algorithm is applied, using the at least one second privacy parameter, to the function over the received dataset. At least one machine learning model can be trained, at 1450, using the dataset which, when deployed, is configured to classify input data.

[0175] The machine learning model(s) can be deployed once trained to classify input data when received.

[0176] The at least one first user-generated privacy parameter can include a bound for an adversarial posterior belief ρ_c that corresponds to a likelihood to re-identify data points from the dataset based on a differentially private function output. With such an arrangement, the calculated at least one second privacy parameter can include privacy parameters ε, δ and the calculating can be based on ρ_c which are bound by the posterior belief ρ_c as applied to the dataset.

[0177] In another variation, the at least one first user-generated privacy parameter includes privacy parameters ε, δ. With such an implementation, the calculated at least one second privacy parameter can include an expected membership advantage ρ_α that corresponds to a likelihood of an adversary successfully identifying a member in the dataset and the calculating can be based on an overlap of two probability distributions.

[0178] In still another variation, the at least one first user-generated privacy parameter can include privacy parameters ε, δ. With such an implementation, the calculated at least one second privacy parameter can include an adversarial posterior belief bound ρ_c that corresponds to a likelihood to re-identify data points from the dataset based on a differentially private function output and the calculating can be based on a conditional probability of different possible datasets.

[0179] FIG. 15 is a diagram 1500 illustrating a sample computing device architecture for implementing various aspects described herein. A bus 1504 can serve as the information highway interconnecting the other illustrated components of the hardware. A processing system 1508 labeled CPU (central processing unit) (e.g., one or more computer processors/data processors at a given computer or at multiple computers), can perform calculations and logic operations required to execute a program. A non-transitory processor-readable storage medium, such as read only memory (ROM) 1512 and random access memory (RAM) 1516, can be in communication with the processing system 1508 and can include one or more programming instructions for the operations specified here. Optionally, program instructions can be stored on a non-transitory computer-readable storage medium such as a magnetic disk, optical disk, recordable memory device, flash memory, or other physical storage medium.

[0180] In one example, a disk controller 1548 can interface with one or more optional disk drives to the system bus 1504. These disk drives can be external or internal floppy disk drives such as 1560, external or internal CD-ROM, CD-R, CD-RW, or DVD, or solid state drives such as 1552, or external or internal hard drives 1556. As indicated previously, these various disk drives 1552, 1556, 1560 and disk controllers are optional devices. The system bus 1504 can also include at least one communication port 1520 to allow for communication with external devices either physically connected to the computing system or available externally through a wired or wireless network. In some cases, the at least one communication port 1520 includes or otherwise comprises a network interface.

[0181] To provide for interaction with a user, the subject matter described herein can be implemented on a computing device having a display device 1540 (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information obtained from the bus 1504 via a display interface 1514 to the user and an input device 1532 such as keyboard and/or a pointing device (e.g., a mouse or a trackball) and/or a touchscreen by which the user can provide input to the computer. Other kinds of input devices 1532 can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback by way of a microphone 1536, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input. The input device 1532 and the microphone 1536 can be coupled to and convey information via the bus 1504 by way of an input device interface 1528. Other computing devices, such as dedicated servers, can omit one or more of the display 1540 and display interface 1514, the input device 1532, the microphone 1536, and input device interface 1528.

[0182] One or more aspects or features of the subject matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and/or combinations thereof. These various aspects or features can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0183] These computer programs, which can also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logical programming language, and/or in assembly/machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus and/or device, such as for example magnetic discs, optical disks, memory, and Programmable Logic Devices (PLDs), used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. The machine-readable medium can store such machine instructions non-transitorily, such as for example as would a non-transient solid-state memory or a magnetic hard drive or any equivalent storage medium. The machine-readable medium can alternatively or additionally store such machine instructions in a transient manner, such as for example as would a processor cache or other random access memory associated with one or more physical processor cores.

[0184] In the descriptions above and in the claims, phrases such as “at least one of” or “one or more of” may occur followed by a conjunctive list of elements or features. The term “and/or” may also occur in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, such a phrase is intended to mean any of the listed elements or features individually or any of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases “at least one of A and B;” “one or more of A and B;” and “A and/or B” are each intended to mean “A alone, B alone, or A and B together.” A similar interpretation is also intended for lists including three or more items. For example, the phrases “at least one of A, B, and C;” “one or more of A, B, and C;” and “A, B, and/or C” are each intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together.” In addition, use of the term “based on,” above and in the claims is intended to mean, “based at least in part on,” such that an unrecited feature or element is also permissible.

[0185] The subject matter described herein can be embodied in systems, apparatus, methods, and/or articles depending on the desired configuration. The implementations set forth in the foregoing description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects related to the described subject matter. Although a few variations have been described in detail above, other modifications or additions are possible. In particular, further features and/or variations can be provided in addition to those set forth herein. For example, the implementations described above can be directed to various combinations and subcombinations of the disclosed features and/or combinations and subcombinations of several further features disclosed above. In addition, the logic flows depicted in the accompanying figures and/or described herein do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Other implementations may be within the scope of the following claims.