Empirical Privacy Variance

TL;DR

It is shown that models calibrated to the same $(\varepsilon, \delta)$-DP guarantee using DP-SGD with different hyperparameter configurations can exhibit significant variations in empirical privacy, which is quantified through the lens of memorization.

Abstract

We propose the notion of empirical privacy variance and study it in the context of differentially private fine-tuning of language models. Specifically, we show that models calibrated to the same $(\varepsilon, δ)$-DP guarantee using DP-SGD with different hyperparameter configurations can exhibit significant variations in empirical privacy, which we quantify through the lens of memorization. We investigate the generality of this phenomenon across multiple dimensions and discuss why it is surprising and relevant. Through regression analysis, we examine how individual and composite hyperparameters influence empirical privacy. The results reveal a no-free-lunch trade-off: existing practices of hyperparameter tuning in DP-SGD, which focus on optimizing utility under a fixed privacy budget, often come at the expense of empirical privacy. To address this, we propose refined heuristics for hyperparameter selection that explicitly account for empirical privacy, showing that they are both precise and practically useful. Finally, we take preliminary steps to understand empirical privacy variance. We propose two hypotheses, identify limitations in existing techniques like privacy auditing, and outline open questions for future research.

Figures (34)

• Figure 1: Empirical privacy variance: Starting from the same pre-trained model and fine-tuning on the same dataset (to achieve decent utility), DP-SGD with different hyperparameter configurations---each calibrated to the same$(\varepsilon, \delta)$-DP guarantee---produces models with drastically different privacy behaviors.
• Figure 2: Empirical privacy variance: ubiquitous, substantial, and revealing intriguing trends. Each subfigure presents jitter plots of empirical privacy scores (ACR or AIR) obtained by models trained under a given $(\varepsilon,\delta)$-DP guarantee. Higher $y$-axis scores indicate worse empirical privacy, while the $x$-axis contrasts different groups (e.g., models of varying sizes in (a)), represented by different colors. Within each group, scattered points correspond to unique hyperparameter configurations $(b, T, \eta)$, averaged over training randomness (we show the impact of training randomness is much smaller than that of hyperparameters in \ref{['adxsec:results-randomness']}). Each group's standard deviation is labeled at the top of its cluster. The subfigures demonstrate that empirical privacy variance increases with (a)model size, (b)dataset size, (c)secret density, and (a/b)privacy budget $\varepsilon$.
• Figure 3: Generality of empirical privacy variance. Across (a)secret subsets (subset 0 vs. 1) and (b)empirical privacy measures (ACR vs. VMR), we observe consistent trends as in \ref{['fig:var-generality']}: empirical privacy variance increases with $\varepsilon$ ($\rightarrow$ in each subfigure), dataset size ($\downarrow$ in column (a)), and model size ($\downarrow$ in column (b)).
• Figure 4: A conceptual illustration of classic mechanism vs. DP-SGD. In classic mechanisms, the monotonic relationship between privacy risk and privacy budget $\varepsilon$ allows any $\varepsilon \le \varepsilon^*$ to be certified if $\varepsilon^*$ satisfies the desired privacy risk. In DP-SGD, however, variance introduces an achievable region of privacy risk, reflected by the upper and lower bound. A measured configuration meeting the privacy requirements does not safeguard the corresponding $\varepsilon^*$; identifying the truly reliable threshold, $\varepsilon^{\text{gold}}$, requires testing a wide range of configurations to account for the full spectrum of privacy risks. While a conservative theoretical upper bound yeom2018privacyma2019datahayes2023bounding could aid in standardization by identifying $\varepsilon^{\text{strict}}$, such bounds are generally unavailable for empirical privacy measures like ACR.
• Figure 5: Effect of individual and composite hyperparameters (setting: GPT-2-S, Enron, ACR, $\varepsilon=8$). We show the empirical privacy and utility of the DP fine-tuned models using different hyperparameters. (a-c): Varying one hyperparameter while holding the others fixed. (d): Holding compute ($C=b\cdot T$) fixed and varying $(b,T)$; (e): Holding updates ($U=C\cdot \eta$) fixed and varying $(C,\eta)$.

Theorems & Definitions (1)

• Definition 2.1: dwork2006calibrating