Shifted Interpolation for Differential Privacy

TL;DR

This work tackles the fundamental challenge of tightly quantifying privacy leakage for private optimization under differential privacy, focusing on convex and strongly convex losses. It introduces shifted interpolation within the $f$-DP framework to obtain convergent, exact privacy bounds for the final iterate, enabling analysis that persists as the number of iterations grows. The authors prove a first exact $f$-DP privacy analysis for strongly convex problems and extend the technique to diverse settings (constrained/unconstrained, full/cyclic/stochastic batches), while showing how these bounds translate losslessly to $( extvarepsilon, extdelta)$-DP and Rényi DP. They further connect these bounds to the exponential mechanism via Langevin Monte Carlo, obtaining optimal $f$-DP guarantees for both strongly and weakly convex/log-concave targets, and demonstrate practical improvements with numerical experiments on MNIST. Overall, the shifted-interpolation approach yields tighter, convergent privacy bounds that enable longer training under fixed privacy budgets and provide algorithmic insights across private optimization and sampling contexts.

Abstract

Noisy gradient descent and its variants are the predominant algorithms for differentially private machine learning. It is a fundamental question to quantify their privacy leakage, yet tight characterizations remain open even in the foundational setting of convex losses. This paper improves over previous analyses by establishing (and refining) the "privacy amplification by iteration" phenomenon in the unifying framework of $f$-differential privacy--which tightly captures all aspects of the privacy loss and immediately implies tighter privacy accounting in other notions of differential privacy, e.g., $(\varepsilon,δ)$-DP and Rényi DP. Our key technical insight is the construction of shifted interpolated processes that unravel the popular shifted-divergences argument, enabling generalizations beyond divergence-based relaxations of DP. Notably, this leads to the first exact privacy analysis in the foundational setting of strongly convex optimization. Our techniques extend to many settings: convex/strongly convex, constrained/unconstrained, full/cyclic/stochastic batches, and all combinations thereof. As an immediate corollary, we recover the $f$-DP characterization of the exponential mechanism for strongly convex optimization in Gopi et al. (2022), and moreover extend this result to more general settings.

Figures (10)

• Figure 1: Left: improved $f$-DP versus the standard composition analysis. Right: improved $(\varepsilon,\delta)$-DP by losslessly converting from $f$-DP. Our privacy bound is optimal in all parameters, here for $\texttt{NoisyGD}$ on strongly convex losses; see \ref{['app:num']} for the parameter choices and other settings. Our $f$-DP analysis also implies optimal bounds for the Rényi DP framework (previously unknown), but $f$-DP is strictly better since it captures all aspects of the privacy leakage, whereas Rényi DP is intrinsically lossy.
• Figure 2: Illustration of $f$-DP and GDP. Gaussian tradeoff functions $G(\mu)$ are less private as $\mu$ increases from $0$ (full privacy) to $\infty$ (no privacy). The closer to $\text{Id}$, the more private. Here $\mathcal{A}$ is $1$-GDP but not $0.5$-GDP because its tradeoff function is pointwise above $G(1)$ but not pointwise above $G(0.5)$.
• Figure 3: Illustration of the shifted interpolated process $\{\widetilde{X}_k\}$ defined in \ref{['eq:shift-simple']}. It starts from one process ($\widetilde{X}_\tau = X'_\tau$) and ends at the other ($\widetilde{X}_t = X_t$). The intermediate time $\tau$ is an analysis parameter that we optimize to get the best final privacy bound.
• Figure 4: Illustration of $C_p(G(\frac{L}{b\sigma}))$ and $f^{(\lambda)}$, for $\lambda \in \{0, 0.5, 1\}$ with $p = 0.25, L/(b\sigma) = 2.5$.
• Figure 5: Illustration of shifted interpolated processes in the proofs of \ref{['thm:sgd-sc']} (left) and \ref{['thm:sgd-proj']} (right). The solid lines denote the updates based on the realized values of $\{V_k\}$, and the dashed lines denote the alternative updates based on their unrealized values; each interpolated process uses the same (coupled) values of $\{V_k\}$ as expressed in the figure. In \ref{['thm:sgd-sc']}, we build two processes, each of which tracks its corresponding original process. In \ref{['thm:sgd-proj']}, only one process is built and it inherits the identical deviation based on the realizations of $\{V_k\}$.

Theorems & Definitions (89)

• Definition 2.1: $(\varepsilon,\delta)$-DP
• Definition 2.2: $f$-DP
• Lemma 2.3: Characterization of tradeoff functions
• Definition 2.4: GDP
• Lemma 2.5: Post-processing
• Definition 2.6: Composition
• Lemma 2.7: Strong composition
• Lemma 2.8
• Lemma 2.9
• Definition 2.10: Gradient sensitivity