Mirror Descent Algorithms with Nearly Dimension-Independent Rates for Differentially-Private Stochastic Saddle-Point Problems
Tomás González, Cristóbal Guzmán, Courtney Paquette
TL;DR
The paper addresses DP stochastic saddle-point problems in the $\ell_1$ setting and proposes two stochastic mirror-descent–based algorithms that achieve nearly dimension-independent convergence rates for the duality gap. It leverages Maurey sparsification, vertex sampling, and bias-reduction techniques to control dimension effects and gradient bias, obtaining rates such as $\mathbb{E}[\operatorname{Gap}(\tilde{x},\tilde{y})] \lesssim (L_0+L_1)\left[\sqrt{\frac{\ell}{n}} + \left(\frac{\ell^{3/2}\sqrt{\log(1/\delta)}}{n\varepsilon}\right)^{1/3}\right]$ for first-order-smooth objectives, and enhanced rates under second-order-smoothness. The work extends to DP-SCO in the polyhedral setting, delivering convex-excess-risk bounds not based on Frank-Wolfe, and introduces high-probability guarantees via bias-reduction and boosting techniques. The results demonstrate near-optimality up to polylogarithmic factors and provide practical, $O(n)$-gradient-complexity algorithms with concrete DP guarantees. Overall, the paper advances DP methods for high-dimensional, non-Euclidean geometries and offers new tools (Maurey-function-value approximations, bias-reduction, Anytime online-to-batch) with potential impact on privacy-preserving learning in $\ell_1$-restricted domains.
Abstract
We study the problem of differentially-private (DP) stochastic (convex-concave) saddle-points in the $\ell_1$ setting. We propose $(\varepsilon, δ)$-DP algorithms based on stochastic mirror descent that attain nearly dimension-independent convergence rates for the expected duality gap, a type of guarantee that was known before only for bilinear objectives. For convex-concave and first-order-smooth stochastic objectives, our algorithms attain a rate of $\sqrt{\log(d)/n} + (\log(d)^{3/2}/[n\varepsilon])^{1/3}$, where $d$ is the dimension of the problem and $n$ the dataset size. Under an additional second-order-smoothness assumption, we show that the duality gap is bounded by $\sqrt{\log(d)/n} + \log(d)/\sqrt{n\varepsilon}$ with high probability, by using bias-reduced gradient estimators. This rate provides evidence of the near-optimality of our approach, since a lower bound of $\sqrt{\log(d)/n} + \log(d)^{3/4}/\sqrt{n\varepsilon}$ exists. Finally, we show that combining our methods with acceleration techniques from online learning leads to the first algorithm for DP Stochastic Convex Optimization in the $\ell_1$ setting that is not based on Frank-Wolfe methods. For convex and first-order-smooth stochastic objectives, our algorithms attain an excess risk of $\sqrt{\log(d)/n} + \log(d)^{7/10}/[n\varepsilon]^{2/5}$, and when additionally assuming second-order-smoothness, we improve the rate to $\sqrt{\log(d)/n} + \log(d)/\sqrt{n\varepsilon}$. Instrumental to all of these results are various extensions of the classical Maurey Sparsification Lemma \cite{Pisier:1980}, which may be of independent interest.