Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Laplace and Gaussian: Exploring the Generalized Gaussian Mechanism for Private Machine Learning

Roy Rinberg, Ilia Shumailov, Vikrant Singhal, Rachel Cummings, Nicolas Papernot

Abstract

Differential privacy (DP) is obtained by randomizing a data analysis algorithm, which necessarily introduces a tradeoff between its utility and privacy. Many DP mechanisms are built upon one of two underlying tools: Laplace and Gaussian additive noise mechanisms. We expand the search space of algorithms by investigating the Generalized Gaussian (GG) mechanism, which samples the additive noise term $x$ with probability proportional to $e^{-\frac{| x |}σ^β }$ for some $β\geq 1$ (denoted $GG_{β, σ}(f,D)$). The Laplace and Gaussian mechanisms are special cases of GG for $β=1$ and $β=2$, respectively. We prove that the full GG family satisfies differential privacy and extend the PRV accountant to support privacy loss computation for these mechanisms. We then instantiate the GG mechanism in two canonical private learning pipelines, PATE and DP-SGD. Empirically, we explore PATE and DP-SGD with the GG mechanism across the computationally feasible values of $β$: $β\in [1,2]$ for DP-SGD and $β\in [1,4]$ for PATE. For both mechanisms, we find that $β=2$ (Gaussian) performs as well as or better than other values in their computational tractable domains.This provides justification for the widespread adoption of the Gaussian mechanism in DP learning.

Beyond Laplace and Gaussian: Exploring the Generalized Gaussian Mechanism for Private Machine Learning

Abstract

Differential privacy (DP) is obtained by randomizing a data analysis algorithm, which necessarily introduces a tradeoff between its utility and privacy. Many DP mechanisms are built upon one of two underlying tools: Laplace and Gaussian additive noise mechanisms. We expand the search space of algorithms by investigating the Generalized Gaussian (GG) mechanism, which samples the additive noise term with probability proportional to for some (denoted ). The Laplace and Gaussian mechanisms are special cases of GG for and , respectively. We prove that the full GG family satisfies differential privacy and extend the PRV accountant to support privacy loss computation for these mechanisms. We then instantiate the GG mechanism in two canonical private learning pipelines, PATE and DP-SGD. Empirically, we explore PATE and DP-SGD with the GG mechanism across the computationally feasible values of : for DP-SGD and for PATE. For both mechanisms, we find that (Gaussian) performs as well as or better than other values in their computational tractable domains.This provides justification for the widespread adoption of the Gaussian mechanism in DP learning.

Paper Structure

This paper contains 39 sections, 20 theorems, 70 equations, 16 figures, 1 table, 6 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Contributions
  4. Preliminaries
  5. Differential Privacy
  6. Rényi Differential Privacy
  7. PRV Accountant
  8. Generalized Gaussian Mechanism and Privacy Guarantees
  9. Generalized Gaussian Mechanism
  10. Privacy Accounting for GG Mechanisms
  11. GG Mechanism for PATE and Private Argmax
  12. Private Argmax and the GGNMax Mechanism
  13. PATE Experiments
  14. GG Mechanism for Differentially Private Stochastic Gradient Descent
  15. DP-SGD Experiments
  16. ...and 24 more sections

Key Result

Theorem 1

For any $\beta \geq 1$, $\sigma >0$, $\delta > 0$ there exists a finite value $\epsilon$ such that $GG_{\beta, \sigma}(\cdot,\cdot)$ satisfies $(\epsilon, \delta)$-DP.

Figures (16)

  • Figure 1: Linear (left) and log-scale (right) PDFs of the Generalized Gaussian distribution for GG mechanisms of varying $\beta$, which all satisfy the same $(\epsilon, \delta)$-DP privacy guarantee.
  • Figure 2: DP parameter $\epsilon$ as a function of noise parameter $\sigma$ with fixed $\delta = 10^{-5}$ and $\Delta f=1$, calculated using the PRV accountant. Mechanisms with equivalent DP guarantees can be identified by computing the privacy curve's intersection with a horizontal line, illustrated here with a red line for (arbitrarily chosen) $\epsilon=4$.
  • Figure 3: Average Label Accuracy of GGNMax mechanisms with equivalent privacy guarantees and varying values of $\beta$, evaluated on histograms which were generated by pate_2017 as an intermediate state produced by 250 teachers trained on MNIST (left) and SVHN (right).
  • Figure 4: $\beta$-DP-SGD results for the corresponding architectures trained on CIFAR-10, SVHN, Adult, and IMDB, for $\delta = 10^{-6}$. The test-accuracy is reported for five values of $\epsilon$, computed for each architecture-dataset pair. A vertical dashed line denotes the Gaussian mechanism. Note: Some values are not presented (for lower $\epsilon$), because larger $\beta$ tends to consume more privacy per step, and the model's privacy budget exceeds the target in the first step.
  • Figure 5: Likelihood of outliers for the GG Mechanism (left) and SGG Mechanism with Poisson sampling probability $q=0.01$ composed 100 rounds (right), both that satisfy $(1.5, 10^{-5})$-DP . In the legend, tail refers to the $\tau$ tail cutoff parameter. The jaggedness in the plotting comes from the contributions of sampling error methods and integration error for very small weights; values are slightly smoothed with a Savitzky-Golay filter with polynomial order 2 and window size 5.
  • ...and 11 more figures

Theorems & Definitions (40)

  • Definition 1: Differential privacy original_DP
  • Definition 2: $\ell_\beta$ sensitivity original_DP
  • Definition 3
  • Definition 4: Laplace Mechanism original_DP
  • Definition 5: Gaussian Mechanism privacy_book
  • Definition 6: Rényi Divergence
  • Definition 7
  • Definition 8: analytic_ggd
  • Theorem 1
  • Theorem 2
  • ...and 30 more