Private Zeroth-Order Nonsmooth Nonconvex Optimization

TL;DR

The paper addresses private stochastic optimization for nonconvex and nonsmooth objectives using a zeroth-order, gradient-free approach. It builds on the Online-to-Nonconvex (O2NC) framework by introducing a variance-reduced, private gradient oracle and employing the tree mechanism to amplify privacy efficiently. The main result is a data complexity of $\tilde{O}\big(d\delta^{-1}\epsilon^{-3}+d^{3/2}\rho^{-1}\delta^{-1}\epsilon^{-2}\big)$ to obtain a $(\delta,\epsilon)$-stationary point while guaranteeing $(\alpha,\alpha\rho^2/2)$-Rényi DP, with privacy becoming effectively free when $\rho\ge\sqrt{d}\epsilon$. This work is the first to achieve private nonsmooth nonconvex optimization in a zeroth-order setting, matching nonprivate rates and highlighting the efficacy of variance reduction and tree-based privacy in high-dimensional regimes.

Abstract

We introduce a new zeroth-order algorithm for private stochastic optimization on nonconvex and nonsmooth objectives. Given a dataset of size $M$, our algorithm ensures $(α,αρ^2/2)$-Rényi differential privacy and finds a $(δ,ε)$-stationary point so long as $M=\tildeΩ\left(\frac{d}{δε^3} + \frac{d^{3/2}}{ρδε^2}\right)$. This matches the optimal complexity of its non-private zeroth-order analog. Notably, although the objective is not smooth, we have privacy ``for free'' whenever $ρ\ge \sqrt{d}ε$.

Theorems & Definitions (41)

• Definition 2.1
• Lemma 2.1
• Corollary 2.1
• Lemma 3.0
• Lemma 3.0
• Remark 4.1
• Remark 4.2
• Theorem 4.3
• Corollary 4.3
• Corollary 4.3