Optimal Rates for $O(1)$-Smooth DP-SCO with a Single Epoch and Large Batches
Christopher A. Choquette-Choo, Arun Ganesh, Abhradeep Thakurta
TL;DR
This work resolves a long-standing trade-off in DP stochastic convex optimization by designing Accelerated-DP-SRGD, a single-epoch algorithm that achieves near-optimal DP-SCO rates with sublinear batch-gradient complexity. By marrying stochastic recursive gradients, Nesterov-style acceleration, and DP continual counting, the method attains $T= ilde{O}(n^{1/4})$ batch steps (with $B=n/T$) when the unconstrained minimizer lies in the constraint set, and $T= ilde{O}( oot n)$ with $B= oot n$ in the general case, all while preserving $( ext{ε}, ext{δ})$-DP and requiring only $2$ gradient evaluations per data point. The analysis jointly bounds utility and privacy via a potential-based approach, derives tight sensitivity bounds for gradient differences, and leverages the binary-tree mechanism to control cumulative DP noise. The results improve prior DP-SCO guarantees in the important single-epoch setting, with practical implications for privacy-preserving federated and streaming learning. The paper also discusses unaccelerated variants, removing common convexity assumptions, and extensions to non-convex losses via clipping, highlighting the method’s broad applicability.
Abstract
In this paper we revisit the DP stochastic convex optimization (SCO) problem. For convex smooth losses, it is well-known that the canonical DP-SGD (stochastic gradient descent) achieves the optimal rate of $O\left(\frac{LR}{\sqrt{n}} + \frac{LR \sqrt{p \log(1/δ)}}{εn}\right)$ under $(ε, δ)$-DP, and also well-known that variants of DP-SGD can achieve the optimal rate in a single epoch. However, the batch gradient complexity (i.e., number of adaptive optimization steps), which is important in applications like federated learning, is less well-understood. In particular, all prior work on DP-SCO requires $Ω(n)$ batch gradient steps, multiple epochs, or convexity for privacy. We propose an algorithm, Accelerated-DP-SRGD (stochastic recursive gradient descent), which bypasses the limitations of past work: it achieves the optimal rate for DP-SCO (up to polylog factors), in a single epoch using $\sqrt{n}$ batch gradient steps with batch size $\sqrt{n}$, and can be made private for arbitrary (non-convex) losses via clipping. If the global minimizer is in the constraint set, we can further improve this to $n^{1/4}$ batch gradient steps with batch size $n^{3/4}$. To achieve this, our algorithm combines three key ingredients, a variant of stochastic recursive gradients (SRG), accelerated gradient descent, and correlated noise generation from DP continual counting.