Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Optimization with Sparse Gradients

Badih Ghazi, Cristóbal Guzmán, Pritish Kamath, Ravi Kumar, Pasin Manurangsi

Abstract

Motivated by applications of large embedding models, we study differentially private (DP) optimization problems under sparsity of individual gradients. We start with new near-optimal bounds for the classic mean estimation problem but with sparse data, improving upon existing algorithms particularly for the high-dimensional regime. Building on this, we obtain pure- and approximate-DP algorithms with almost optimal rates for stochastic convex optimization with sparse gradients; the former represents the first nearly dimension-independent rates for this problem. Finally, we study the approximation of stationary points for the empirical loss in approximate-DP optimization and obtain rates that depend on sparsity instead of dimension, modulo polylogarithmic factors.

Differentially Private Optimization with Sparse Gradients

Abstract

Motivated by applications of large embedding models, we study differentially private (DP) optimization problems under sparsity of individual gradients. We start with new near-optimal bounds for the classic mean estimation problem but with sparse data, improving upon existing algorithms particularly for the high-dimensional regime. Building on this, we obtain pure- and approximate-DP algorithms with almost optimal rates for stochastic convex optimization with sparse gradients; the former represents the first nearly dimension-independent rates for this problem. Finally, we study the approximation of stationary points for the empirical loss in approximate-DP optimization and obtain rates that depend on sparsity instead of dimension, modulo polylogarithmic factors.
Paper Structure (26 sections, 25 theorems, 99 equations, 2 tables, 6 algorithms)

This paper contains 26 sections, 25 theorems, 99 equations, 2 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Our Results
  3. Related Work
  4. Future Directions
  5. Notation and Preliminaries
  6. Upper Bounds for DP Mean Estimation with Sparse Data
  7. Sharper Approximate DP Mean Estimation via Compressed Sensing
  8. Lower Bounds for DP Mean Estimation with Sparse Data
  9. Lower Bounds for Pure-DP
  10. Lower Bounds for Approximate-DP
  11. Bias-Reduction Method for DP-ERM with Sparse Gradients
  12. Subsampled Bias-Reduced Gradient Estimator for DP-ERM
  13. Bias and Moment Estimates for the Debiased Gradient Estimator
  14. Accuracy Guarantees for Subsampled Bias-Reduced Sparse SGD
  15. Lower Bounds on the Stopping Time
  16. ...and 11 more sections

Key Result

Lemma 3.1

[lemma]lem:proj-main In alg:proj, it holds that $\|\hat{z} - \bar{z}(S)\|_2 \leq \sqrt{2L \|\xi\|_{\infty} \sqrt{s}}$, almost surely.

Theorems & Definitions (58)

  • Remark 2.1
  • Definition 2.2: Differential Privacy
  • Lemma 3.1
  • proof
  • Theorem 3.2
  • proof
  • Theorem 3.3
  • proof
  • Theorem 3.4
  • proof
  • ...and 48 more