Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finding Differentially Private Second Order Stationary Points in Stochastic Minimax Optimization

Difei Xu, Youming Tao, Meng Ding, Chenglin Fan, Di Wang

TL;DR

The paper tackles the problem of privately finding second-order stationary points in stochastic nonconvex minimax optimization. It introduces DP RGDA, a first-order method that fuses SPIDER variance reduction, a value-function perspective that tracks the inner maximizer $y^*(x)$, and a perturb‑and‑monitor saddle-escape mechanism to certify SOSP without Hessian computations. The authors prove high-probability SOSP guarantees for both empirical and population losses, achieving $\alpha = \widetilde{O}\left(\left(\frac{\sqrt{d}}{n\varepsilon}\right)^{2/3}\right)$ in ERM and $\alpha = \widetilde{O}\left(\frac{1}{n^{1/3}} + \left(\frac{\sqrt{d}}{n\varepsilon}\right)^{1/2}\right)$ in the population setting, matching the best known private first-order rates. Empirical results on synthetic matrix sensing corroborate the method’s robustness under DP and its advantage over DP-SPIDER baselines, illustrating the practical viability of DP-SOSP in minimax models.

Abstract

We provide the first study of the problem of finding differentially private (DP) second-order stationary points (SOSP) in stochastic (non-convex) minimax optimization. Existing literature either focuses only on first-order stationary points for minimax problems or on SOSP for classical stochastic minimization problems. This work provides, for the first time, a unified and detailed treatment of both empirical and population risks. Specifically, we propose a purely first-order method that combines a nested gradient descent--ascent scheme with SPIDER-style variance reduction and Gaussian perturbations to ensure privacy. A key technical device is a block-wise ($q$-period) analysis that controls the accumulation of stochastic variance and privacy noise without summing over the full iteration horizon, yielding a unified treatment of both empirical-risk and population formulations. Under standard smoothness, Hessian-Lipschitzness, and strong concavity assumptions, we establish high-probability guarantees for reaching an $(α,\sqrt{ρ_Φα})$-approximate second-order stationary point with $α= \mathcal{O}( (\frac{\sqrt{d}}{n\varepsilon})^{2/3})$ for empirical risk objectives and $\mathcal{O}(\frac{1}{n^{1/3}} + (\frac{\sqrt{d}}{n\varepsilon})^{1/2})$ for population objectives, matching the best known rates for private first-order stationarity.

Finding Differentially Private Second Order Stationary Points in Stochastic Minimax Optimization

TL;DR

The paper tackles the problem of privately finding second-order stationary points in stochastic nonconvex minimax optimization. It introduces DP RGDA, a first-order method that fuses SPIDER variance reduction, a value-function perspective that tracks the inner maximizer , and a perturb‑and‑monitor saddle-escape mechanism to certify SOSP without Hessian computations. The authors prove high-probability SOSP guarantees for both empirical and population losses, achieving in ERM and in the population setting, matching the best known private first-order rates. Empirical results on synthetic matrix sensing corroborate the method’s robustness under DP and its advantage over DP-SPIDER baselines, illustrating the practical viability of DP-SOSP in minimax models.

Abstract

We provide the first study of the problem of finding differentially private (DP) second-order stationary points (SOSP) in stochastic (non-convex) minimax optimization. Existing literature either focuses only on first-order stationary points for minimax problems or on SOSP for classical stochastic minimization problems. This work provides, for the first time, a unified and detailed treatment of both empirical and population risks. Specifically, we propose a purely first-order method that combines a nested gradient descent--ascent scheme with SPIDER-style variance reduction and Gaussian perturbations to ensure privacy. A key technical device is a block-wise (-period) analysis that controls the accumulation of stochastic variance and privacy noise without summing over the full iteration horizon, yielding a unified treatment of both empirical-risk and population formulations. Under standard smoothness, Hessian-Lipschitzness, and strong concavity assumptions, we establish high-probability guarantees for reaching an -approximate second-order stationary point with for empirical risk objectives and for population objectives, matching the best known rates for private first-order stationarity.
Paper Structure (30 sections, 15 theorems, 102 equations, 1 figure, 2 tables, 3 algorithms)

This paper contains 30 sections, 15 theorems, 102 equations, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Differential Privacy
  4. Second Order Stationary Points for Minimax Optimization
  5. Challenges
  6. Main Theory
  7. Key Idea and Proof Strategy
  8. (i) Enforce a value-function viewpoint by tracking $y^\star(x)$.
  9. (ii) Control the recursive estimator locally (period-wise) rather than globally.
  10. (iii) Avoid private model selection via a movement-based escape test.
  11. SOSP with Empirical Loss
  12. SOSP with Population Loss
  13. Experiements
  14. Experimental setup
  15. Problem.
  16. ...and 15 more sections

Key Result

Lemma 1

Consider Algorithm innerloop, and for any $t \in \{0, ..., T\}$ let $t_0 = \left\lfloor\frac{t}{q}\right\rfloor q$. If each $\nabla_t$ computed defined above is an unbiased estimate of $\nabla F(w_t; S)$ satisfying with probability $1-\delta_1$, and each $\Delta_t$ is an unbiased estimate of the gradient variation satisfying Then for any $t \geqslant t_0 + 1$, the iterates of Algorithm innerloop

Figures (1)

  • Figure 1: Trajectories on the synthetic matrix sensing minimax instance. DP RGDA implements our SPIDER recursion and (enabled) escape mechanism, while DP-SGDA is a single-loop baseline. We report $\Phi(x_t)$, the (non-private) gradient norm of the induced objective, and an estimated minimum Hessian eigenvalue.

Theorems & Definitions (30)

  • Definition 1: Differential Privacy dwork2006calibrating
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Lemma 1
  • Theorem 1
  • Remark 1
  • Lemma 2
  • Theorem 2
  • ...and 20 more