Fine-Tuning Large Language Models with User-Level Differential Privacy

TL;DR

This work tackles the challenge of privately fine-tuning large language models under user-level differential privacy by comparing two scalable DP-SGD variants: DP-SGD-ELS, which operates at the example level with per-example clipping, and DP-SGD-ULS, which samples at the user level and clips per-user gradients. A key contribution is a tight Mixture-of-Gaussians based DP accountant that yields provably tight user-level privacy bounds for ELS, enabling fair comparisons with ULS. Theoretical analysis under Lipschitz losses, plus synthetic mean-estimation experiments, illuminate when ULS provides advantages—especially with diverse user data—and the empirical results on a 350M-parameter transformer show ULS often outperforms ELS under strict privacy or larger compute budgets, with practical heuristics for choosing group sizes. Overall, the paper demonstrates scalable, privacy-preserving DP fine-tuning of large models and offers actionable guidelines for deploying ELS and ULS in data-center settings.

Abstract

We investigate practical and scalable algorithms for training large language models (LLMs) with user-level differential privacy (DP) in order to provably safeguard all the examples contributed by each user. We study two variants of DP-SGD with: (1) example-level sampling (ELS) and per-example gradient clipping, and (2) user-level sampling (ULS) and per-user gradient clipping. We derive a novel user-level DP accountant that allows us to compute provably tight privacy guarantees for ELS. Using this, we show that while ELS can outperform ULS in specific settings, ULS generally yields better results when each user has a diverse collection of examples. We validate our findings through experiments in synthetic mean estimation and LLM fine-tuning tasks under fixed compute budgets. We find that ULS is significantly better in settings where either (1) strong privacy guarantees are required, or (2) the compute budget is large. Notably, our focus on LLM-compatible training algorithms allows us to scale to models with hundreds of millions of parameters and datasets with hundreds of thousands of users.

Figures (18)

• Figure 1: Upper bound on $\varepsilon$ for ELS using our Mixture-of-Gaussians (MoG) accountant and the prior state-of-the-art black-box accounting from vadhan2017complexity. We set $T = 2000$ iterations, sampling probability $p = 10^{-2}$, $\delta = 10^{-6}$, and vary $\sigma_{\text{ELS}}$ and $G_{\text{ELS}}$. For $\sigma_{\text{ELS}} \in \{0.5, 1\}$, the black-box accountant diverged for $G_\text{ELS}$ sufficiently large.
• Figure 2: A plot of the per-iteration variances $\mathop{\mathrm{var}}\nolimits(\zeta_{\text{ELS}})$ and $\mathop{\mathrm{var}}\nolimits(\zeta_{\text{ULS}})$ of ELS and ULS. We set $K = 32$, $T=1000$, and $N = 1024$. In (a) we fix a compute budget of $B = 64$ for $\texttt{ELS}\xspace$ and vary $G_\text{ELS}$. In (b) For ELS, we fix $G_\text{ELS} = K$ and the sampling probability $p = B / K N$ as we vary the compute budget $B$. For ULS, we fix the cohort size $M = 16$ and vary $G_\text{ULS} = B / M$ with $B$. We compare two settings, one in which $L_\text{ULS} = L_\text{ELS}$ (no intra-user gradient diversity), and one in which $L_\text{ULS} = L_\text{ELS} / \sqrt{G_\text{ULS}}$ (maximal intra-user diversity).
• Figure 3: Performance of ELS and ULS on a synthetic mean estimation task with (a) varying $\varepsilon$, (b) varying compute budget, and (c) varying within-user variance $\sigma_2$.
• Figure 4: Validation loss of ELS on Stack Overflow (left) and CC-News (right) for $\varepsilon = 4$ with varying compute budget and group size. The median user dataset size is plotted vertically in red.
• Figure 5: Loss of ULS on Stack Overflow for varying cohort and group sizes, with $\varepsilon = 4$. Both plots are based on views of the same data, with the lines of (b) corresponding to the diagonals of (a).

Theorems & Definitions (11)

• Theorem 1: Informal version of Theorem \ref{['thm:dp-sgd-accounting']}
• Theorem 2
• Conjecture 1
• Lemma 1
• Theorem 3
• Lemma 2: Lemma 4.5 of choquettechoo2024privacy
• Lemma 3
• proof
• proof
• proof : Proof of Theorem \ref{['thm:dp-sgd-accounting']}