Optimal Differentially Private Model Training with Public Data

TL;DR

This work proves tight lower and upper bounds that characterize the optimal error rates of three fundamental problems: mean estimation, empirical risk minimization, and stochastic convex optimization and develops novel algorithms that are even more optimal than the asymptotically optimal approaches described above.

Abstract

Differential privacy (DP) ensures that training a machine learning model does not leak private data. In practice, we may have access to auxiliary public data that is free of privacy concerns. In this work, we assume access to a given amount of public data and settle the following fundamental open questions: 1. What is the optimal (worst-case) error of a DP model trained over a private data set while having access to side public data? 2. How can we harness public data to improve DP model training in practice? We consider these questions in both the local and central models of pure and approximate DP. To answer the first question, we prove tight (up to log factors) lower and upper bounds that characterize the optimal error rates of three fundamental problems: mean estimation, empirical risk minimization, and stochastic convex optimization. We show that the optimal error rates can be attained (up to log factors) by either discarding private data and training a public model, or treating public data like it is private and using an optimal DP algorithm. To address the second question, we develop novel algorithms that are "even more optimal" (i.e. better constants) than the asymptotically optimal approaches described above. For local DP mean estimation, our algorithm is optimal including constants. Empirically, our algorithms show benefits over the state-of-the-art.

Figures (20)

• Figure 1: Minimax optimal error rates for central $(\varepsilon, \delta)$-semi-DP (up to logs) and (local) $(\varepsilon, \delta)$-semi-LDP. $n = n_{\text{priv}} + n_{\text{pub}}$, where $n_{\text{priv}}$ ($n_{\text{pub}}$) denotes the number of private (public) samples. Dependence on $\delta$, range and Lipschitz parameters, constraint set diameter omitted. See Appendix for strongly convex SCO results.
• Figure 2: Minimax optimal error rates for $\varepsilon$-semi-DP ERM. See Table \ref{['app table: summary of pure rates']} in Appendix for more $\varepsilon$-semi-(L)DP results (e.g. SCO).
• Figure 3: Test loss vs. $n_{\text{pub}}/n$. $\varepsilon=2$.
• Figure 4: Test loss vs. $n_{\text{pub}}/n$. $\varepsilon=4$.
• Figure 5: Test loss vs. $n_{\text{pub}}/n$. $\varepsilon=4$, without warm-start.

Theorems & Definitions (71)

• Definition 1: Differential Privacy dwork2006calibrating
• Definition 2: Zero-Concentrated Differential Privacy (zCDP) bun16
• Definition 3: Semi-Differential Privacy beimel2013privatealon2019limits
• Theorem 4
• Remark 5
• Definition 6
• Lemma 7
• Proposition 8
• Definition 9
• Theorem 10