Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tight Group-Level DP Guarantees for DP-SGD with Sampling via Mixture of Gaussians Mechanisms

Arun Ganesh

TL;DR

This work gives a procedure for computing group-level DP guarantees for DP-SGD, when using Poisson sampling or fixed batch size sampling, when using Poisson sampling or fixed batch size sampling.

Abstract

We give a procedure for computing group-level $(ε, δ)$-DP guarantees for DP-SGD, when using Poisson sampling or fixed batch size sampling. Up to discretization errors in the implementation, the DP guarantees computed by this procedure are tight (assuming we release every intermediate iterate).

Tight Group-Level DP Guarantees for DP-SGD with Sampling via Mixture of Gaussians Mechanisms

TL;DR

This work gives a procedure for computing group-level DP guarantees for DP-SGD, when using Poisson sampling or fixed batch size sampling, when using Poisson sampling or fixed batch size sampling.

Abstract

We give a procedure for computing group-level -DP guarantees for DP-SGD, when using Poisson sampling or fixed batch size sampling. Up to discretization errors in the implementation, the DP guarantees computed by this procedure are tight (assuming we release every intermediate iterate).
Paper Structure (10 sections, 6 theorems, 14 equations, 2 figures)

This paper contains 10 sections, 6 theorems, 14 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Privacy Loss Distributions
  3. Definitions and Lemmas
  4. High-Level Strategy
  5. Mixture of Gaussians Mechanisms
  6. Reducing DP-SGD with Sampling to MoG Mechanisms
  7. Poisson Sampling
  8. Fixed Batch Size Sampling
  9. Tightness
  10. Empirical Results

Key Result

Lemma 1.1

If a mechanism satisfies $(\varepsilon, \delta)$-DP with respect to examples, it satisfies $(k \varepsilon, k e^{k \varepsilon} \delta)$-DP with respect to groups of size $k$.

Figures (2)

  • Figure 1: $\varepsilon$ as a function of $k$ for Poisson sampling. We use $T = 2000, q = 1/100$. "Our analysis" is $\varepsilon$ computed using the PLD of the dominating pair in \ref{['thm:poisson']}. "Vadhan analysis" computes an example-level DP guarantee and uses \ref{['lem:vadhan']}. "Lower bound" takes the example-level $(\varepsilon_1, \delta)$-DP guarantee and multiplies $\varepsilon_1$ by $k$ to get a lower bound on the true $\varepsilon$.
  • Figure 2: $\varepsilon$ as a function of $k$ for fixed batch size sampling. We use $T = 2000, B = 500, n = 50000$. The labels are defined analogously to \ref{['fig:poisson']}.

Theorems & Definitions (15)

  • Lemma 1.1: Lemma 2.2 in vadhan2017complexity
  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4: Definition 7 of zhucharacteristic2022
  • Definition 2.5
  • Lemma 2.6: Theorem 10 in zhucharacteristic2022
  • Definition 2.8
  • Lemma 2.9
  • Theorem 3.1
  • ...and 5 more