Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGD

TL;DR

This work addresses the gap between DP-SGD's data-independent privacy guarantees and observed privacy on real datasets by introducing per-instance Rényi-DP analysis. It defines sensitivity distributions that capture how similar updates arise across minibatches and proves a generalized, data-dependent composition that bounds total privacy leakage by the expected, rather than worst-case, per-step leakage. Empirical results on MNIST and CIFAR-10 show many datapoints enjoy substantially tighter per-instance privacy than the baseline, with correctly classified points and higher sampling rates often yielding the strongest gains. The results have practical implications for privacy auditing, unlearning, memorization, and the design of private learning systems that exploit data-dependent privacy to achieve stronger protections in realistic settings.

Abstract

Differentially private stochastic gradient descent (DP-SGD) is the canonical approach to private deep learning. While the current privacy analysis of DP-SGD is known to be tight in some settings, several empirical results suggest that models trained on common benchmark datasets leak significantly less privacy for many datapoints. Yet, despite past attempts, a rigorous explanation for why this is the case has not been reached. Is it because there exist tighter privacy upper bounds when restricted to these dataset settings, or are our attacks not strong enough for certain datapoints? In this paper, we provide the first per-instance (i.e., ``data-dependent") DP analysis of DP-SGD. Our analysis captures the intuition that points with similar neighbors in the dataset enjoy better data-dependent privacy than outliers. Formally, this is done by modifying the per-step privacy analysis of DP-SGD to introduce a dependence on the distribution of model updates computed from a training dataset. We further develop a new composition theorem to effectively use this new per-step analysis to reason about an entire training run. Put all together, our evaluation shows that this novel DP-SGD analysis allows us to now formally show that DP-SGD leaks significantly less privacy for many datapoints (when trained on common benchmarks) than the current data-independent guarantee. This implies privacy attacks will necessarily fail against many datapoints if the adversary does not have sufficient control over the possible training datasets.

Figures (35)

• Figure 1: Per-step privacy contribution from our composition theorem (Theorem \ref{['thm:better_composition']}) using the per-step guarantee for the sum update rule (Theorem \ref{['thm:easy_renyi_dp']}) as needed for DP-SGD, plotted as a fraction of the baseline data-independent per-step DP-SGD guarantee (Section 3.3 in mironov2019r). The x-axis represents the release of the intermediate models up to a given step in training. The y-axis represents the per-instance privacy leakage for a point given by the release of the model at that training step (relative to the data-independent guarantee); summing all the steps gives the overall privacy leakage of training. The different lines represent changing the Gaussian noise to train with different data-independent $\epsilon,\delta$-DP values. The expectations for Theorem \ref{['thm:better_composition']} are computed over 10 trials. Figure \ref{['fig:compo_1_more']} plots the average relative per-step contribution of 100 random points in MNIST for different strengths of the DP guarantee (i.e., different upper bounds $\varepsilon$) used when training on $X' = X \cup \{x^*\}$. The $10^{th} percentile$ is plotted in Figure \ref{['fig:compo_1_more_10per']}. Figure \ref{['fig:compo_1_less']} plots expectation over 10 random points in MNIST when training on $X'$ and $X$. We see from both subfigures our per-step contribution more tightly captures the per-instance privacy than the baseline as training progresses: using Theorem \ref{['thm:better_composition']} one can conclude that many datapoints have better overall data-dependent privacy guarantees than expected by classical analysis. Note our analysis does worse at the first few steps of training as our composition theorem has a blow up in the order of the Rényi divergence for the per-step guarantee for early steps of training; if the sensitivity does not drop quickly enough our composition theorem accounts higher privacy leakage to early steps than the data-independent bounds.
• Figure 2: Distribution plots of our per-step guarantees for the sum update rule given by Theorem \ref{['thm:easy_renyi_dp']} for $500$ datapoints in CIFAR10 with respect to: (a) different stages of training, and (b) varying mini-batch size. The x-axis represents the per-instance guarantee relative to the data-independent guarantee: i.e., the further the mass is to the left, the more our data-dependent guarantees improves upon the data-independent baseline. The purple dashed line represents the data-independent baseline. We observe a "long tail" of datapoints with magnitudes better privacy than expected in both plots, illustrated by the log scale on the x-axis.
• Figure 3: Per-step guarantees given by Theorem \ref{['thm:easy_renyi_dp']} for $500$ datapoints in CIFAR10 across training stages with respect to correct or incorrect classifications. It can be seen that correctly classified datapoints are on average more private than incorrectly classified ones.
• Figure 4: Distribution plots (log scale) of our per-step guarantees for the mean update rule (Theorem \ref{['thm:renyi_dp_sens']}) for $500$ datapoints in CIFAR10 with respect to different training stages and mini-batch sizes. Bounds on both $D_{\alpha}(M(X)||M(X'))$ and $D_{\alpha}(M(X')||M(X))$ are shown (the maximum of both is the per-instance Rényi-DP guarantee) for an expected mini-batch size of 128. From Figures \ref{['fig:hard_renyi_training_stage']},\ref{['fig:hard_renyi_reverse']}, we conclude our per-step guarantees for the mean update rule (Theorem \ref{['thm:renyi_dp_sens']}) gives better data-dependent guarantees for the mean update rule than classical analysis, and from Figure \ref{['fig:hard_renyi_vary_bs']} that increasing the expected mini-batch size decreases our bound for this update rule (counter-intuitive to privacy amplification by subsampling).
• Figure 5: The expected reweighted per-step contributions which are summed for our composition theorem (Theorem \ref{['thm:better_composition']}) using Theorem \ref{['thm:easy_renyi_dp']} for the unweighted per-step guarantee for 10 different points in MNIST. The guarantees are computed once each epoch when training with the datapoint (i.e., on $X' = X \cup \{x^*\}$). The shaded region is the $95\%$ confidence interval over 10 trials. As seen by the confidence intervals having a width a small fraction of the baseline value, with just 10 trials we are very confident in the estimates of the per-step contributions for most points.

Theorems & Definitions (18)

• Definition 2.1: $(\epsilon,\delta)$-DP
• Definition 2.2: Per-Instance Rényi DP
• Corollary 3.1
• Theorem 3.2
• Theorem 3.3
• proof
• proof
• proof
• Theorem 3.6
• proof