Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Output Perturbation for Differentially Private Convex Optimization: Faster and More General

Andrew Lowy, Meisam Razaviyayn

TL;DR

The power of this novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Abstract

Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private stochastic convex optimization (SCO), which corresponds to population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM/SCO setups. Further, most recent work in private SCO has focused on $(\varepsilon, δ)$-DP ($δ> 0$), whereas proving tight excess risk and runtime bounds for $(\varepsilon, 0)$-differential privacy remains a challenging open problem. Our first contribution is to provide the tightest known $(\varepsilon, 0)$-differentially private expected population loss bounds and fastest runtimes for smooth and strongly convex loss functions. In particular, for SCO with well-conditioned smooth and strongly convex loss functions, we provide a linear-time algorithm with optimal excess risk. For our second contribution, we study DP optimization for a broad class of tilted loss functions-which can be used to promote fairness or robustness, and are not necessarily of ERM form. We establish the first known DP excess risk and runtime bounds for optimizing this class; under smoothness and strong convexity assumptions, our bounds are near optimal. For our third contribution, we specialize our theory to DP adversarial training. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Output Perturbation for Differentially Private Convex Optimization: Faster and More General

TL;DR

The power of this novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Abstract

Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private stochastic convex optimization (SCO), which corresponds to population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM/SCO setups. Further, most recent work in private SCO has focused on -DP (), whereas proving tight excess risk and runtime bounds for -differential privacy remains a challenging open problem. Our first contribution is to provide the tightest known -differentially private expected population loss bounds and fastest runtimes for smooth and strongly convex loss functions. In particular, for SCO with well-conditioned smooth and strongly convex loss functions, we provide a linear-time algorithm with optimal excess risk. For our second contribution, we study DP optimization for a broad class of tilted loss functions-which can be used to promote fairness or robustness, and are not necessarily of ERM form. We establish the first known DP excess risk and runtime bounds for optimizing this class; under smoothness and strong convexity assumptions, our bounds are near optimal. For our third contribution, we specialize our theory to DP adversarial training. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Paper Structure

This paper contains 69 sections, 49 theorems, 203 equations, 6 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Output Perturbation.
  3. Excess Risk Bounds for Conceptual Output Perturbation Method
  4. Conceptual output perturbation algorithm
  5. Choice of Noise
  6. Excess risk bounds for strongly convex, Lipschitz functions
  7. Excess risk bounds for smooth, strongly convex, Lipschitz functions.
  8. Excess risk bounds for convex, Lipschitz functions
  9. Excess risk bounds for smooth, convex, Lipschitz functions.
  10. Efficient Implementation of Output Perturbation
  11. Strongly convex, Lipschitz functions
  12. Smooth, strongly convex Lipschitz functions.
  13. From Empirical Loss to Population Loss
  14. Strongly convex, Lipschitz functions
  15. Smooth, strongly convex, Lipschitz functions.
  16. ...and 54 more sections

Key Result

Proposition 2.1

[proposition]conceptual sc alg is private Let $\varepsilon > 0, \delta \in [0, \frac{1}{2}).$ Let $F: \mathbb{R}^d \times \mathcal{X}^{n} \to \mathbb{R}$ be a function with $L_2$ sensitivity $\Delta_{F}$ and unique minimizer $w^{*}(X) = \mathop{\mathrm{argmin}}\limits_{w \in \mathbb{R}^d} F(w,X)$ fo are $(\varepsilon, \delta)$-differentially private. Here $c_{\delta} := \sqrt{\log\left(\frac{2}{\s

Theorems & Definitions (67)

  • Definition 1
  • Proposition 2.1
  • Proposition 2.2
  • Proposition 2.3
  • Corollary 2.1
  • Remark 2.1
  • Remark 2.2: Suboptimality of output perturbation for non-smooth strongly convex ERM
  • Proposition 2.4
  • Corollary 2.2
  • Remark 2.3
  • ...and 57 more