Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation

Ossi Räisä, Joonas Jälkö, Antti Honkela

TL;DR

This paper analyzes how batch size affects gradient variance in DP-SGD, decomposing it into subsampling- and noise-induced components. It proves that in the limit of many iterations, the DP-noise scale grows linearly with the subsampling rate, making the effective noise variance invariant to batch size, while the subsampling variance decreases with larger batches, reducing the total variance. Empirical results show the asymptotic regime is reached quickly and that, even outside it, large batches can further reduce DP variance, providing a theoretical explanation for the observed benefits of large batch sizes in DP-SGD. The findings advance the theoretical understanding of DP-SGD privacy-utility trade-offs and guide practical batch-size choices under differential privacy constraints.

Abstract

We study how the batch size affects the total gradient variance in differentially private stochastic gradient descent (DP-SGD), seeking a theoretical explanation for the usefulness of large batch sizes. As DP-SGD is the basis of modern DP deep learning, its properties have been widely studied, and recent works have empirically found large batch sizes to be beneficial. However, theoretical explanations of this benefit are currently heuristic at best. We first observe that the total gradient variance in DP-SGD can be decomposed into subsampling-induced and noise-induced variances. We then prove that in the limit of an infinite number of iterations, the effective noise-induced variance is invariant to the batch size. The remaining subsampling-induced variance decreases with larger batch sizes, so large batches reduce the effective total gradient variance. We confirm numerically that the asymptotic regime is relevant in practical settings when the batch size is not small, and find that outside the asymptotic regime, the total gradient variance decreases even more with large batch sizes. We also find a sufficient condition that implies that large batch sizes similarly reduce effective DP noise variance for one iteration of DP-SGD.

Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation

TL;DR

This paper analyzes how batch size affects gradient variance in DP-SGD, decomposing it into subsampling- and noise-induced components. It proves that in the limit of many iterations, the DP-noise scale grows linearly with the subsampling rate, making the effective noise variance invariant to batch size, while the subsampling variance decreases with larger batches, reducing the total variance. Empirical results show the asymptotic regime is reached quickly and that, even outside it, large batches can further reduce DP variance, providing a theoretical explanation for the observed benefits of large batch sizes in DP-SGD. The findings advance the theoretical understanding of DP-SGD privacy-utility trade-offs and guide practical batch-size choices under differential privacy constraints.

Abstract

We study how the batch size affects the total gradient variance in differentially private stochastic gradient descent (DP-SGD), seeking a theoretical explanation for the usefulness of large batch sizes. As DP-SGD is the basis of modern DP deep learning, its properties have been widely studied, and recent works have empirically found large batch sizes to be beneficial. However, theoretical explanations of this benefit are currently heuristic at best. We first observe that the total gradient variance in DP-SGD can be decomposed into subsampling-induced and noise-induced variances. We then prove that in the limit of an infinite number of iterations, the effective noise-induced variance is invariant to the batch size. The remaining subsampling-induced variance decreases with larger batch sizes, so large batches reduce the effective total gradient variance. We confirm numerically that the asymptotic regime is relevant in practical settings when the batch size is not small, and find that outside the asymptotic regime, the total gradient variance decreases even more with large batch sizes. We also find a sufficient condition that implies that large batch sizes similarly reduce effective DP noise variance for one iteration of DP-SGD.
Paper Structure (22 sections, 28 theorems, 119 equations, 2 figures)

This paper contains 22 sections, 28 theorems, 119 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Differential privacy
  5. Composition of Differential Privacy
  6. Differentially private SGD
  7. Kullback-Leibler Divergence and Total Variation Distance
  8. Accounting Oracles
  9. DP-SGD noise decomposition
  10. Subsampled Gaussian Mechanism in the Limit of Many Compositions
  11. Empirical results
  12. Subsampled Gaussian Mechanism without Compositions
  13. Limitations
  14. Discussion
  15. Conclusion
  16. ...and 7 more sections

Key Result

Theorem 2.2

Let $\mathcal{M}$ be an $(\epsilon, \delta)$-DP mechanism and let $f$ be any randomised algorithm. Then $f\circ \mathcal{M}$ is $(\epsilon, \delta)$-DP.

Figures (2)

  • Figure 1: The $\sigma_{\text{eff}}:=\sigma(q, T)/q$ decreases as $q$ grows for all the $\epsilon$ and $T$ values. As $T$ grows, $\sigma_{\text{eff}}$ approaches the $\sigma(1, T)$, as the asymptotic theory predicts. The privacy parameter $\delta$ was set to $10^{-5}$ when computing the $\sigma(q, T)$.
  • Figure 2: The largest $a-b$ computed for multiple $(q,\sigma)$ pairs stays negative for a broad range of $\epsilon$ values. The $a$ and $b$ were selected so that the corresponding $(q, \sigma)$ pair satisfies $(\epsilon, \delta)$-DP with $\delta \leq 10^{-5}$.

Theorems & Definitions (62)

  • Definition 2.1
  • Theorem 2.2
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5
  • Definition 2.6: zhuOptimalAccountingDifferential2022
  • Definition 2.7: sommerPrivacyLossClasses2019
  • Theorem 2.8: sommerPrivacyLossClasses2019
  • Definition 2.9
  • Lemma 2.10: kelbertSurveyDistancesMost2023
  • ...and 52 more