Public-data Assisted Private Stochastic Optimization: Power and Limitations
Enayat Ullah, Michael Menart, Raef Bassily, Cristóbal Guzmán, Raman Arora
TL;DR
This paper characterizes the power and limits of public-data aided differential privacy in two core tasks: stochastic convex optimization and private supervised learning. It proves tight lower bounds for PA-DP SCO with complete public data, showing that public data provides limited improvement beyond trivial strategies, especially in high-dimensions. In contrast, it demonstrates that unlabeled public data can meaningfully aid private learning, delivering dimension-independent rates for GLMs and extending to fat-shattering classes, neural networks, and non-Euclidean geometries via principled subspace reduction and Cover-based private selection. The results illuminate when public data helps, quantify sample-size trade-offs, and provide practical PA-DP algorithms with provable guarantees. Overall, public data can bolster private learning under unlabeled settings but offers limited gains for private SCO when public data is labeled.
Abstract
We study the limits and capability of public-data assisted differentially private (PA-DP) algorithms. Specifically, we focus on the problem of stochastic convex optimization (SCO) with either labeled or unlabeled public data. For complete/labeled public data, we show that any $(ε,δ)$-PA-DP has excess risk $\tildeΩ\big(\min\big\{\frac{1}{\sqrt{n_{\text{pub}}}},\frac{1}{\sqrt{n}}+\frac{\sqrt{d}}{nε} \big\} \big)$, where $d$ is the dimension, ${n_{\text{pub}}}$ is the number of public samples, ${n_{\text{priv}}}$ is the number of private samples, and $n={n_{\text{pub}}}+{n_{\text{priv}}}$. These lower bounds are established via our new lower bounds for PA-DP mean estimation, which are of a similar form. Up to constant factors, these lower bounds show that the simple strategy of either treating all data as private or discarding the private data, is optimal. We also study PA-DP supervised learning with \textit{unlabeled} public samples. In contrast to our previous result, we here show novel methods for leveraging public data in private supervised learning. For generalized linear models (GLM) with unlabeled public data, we show an efficient algorithm which, given $\tilde{O}({n_{\text{priv}}}ε)$ unlabeled public samples, achieves the dimension independent rate $\tilde{O}\big(\frac{1}{\sqrt{n_{\text{priv}}}} + \frac{1}{\sqrt{n_{\text{priv}}ε}}\big)$. We develop new lower bounds for this setting which shows that this rate cannot be improved with more public samples, and any fewer public samples leads to a worse rate. Finally, we provide extensions of this result to general hypothesis classes with finite fat-shattering dimension with applications to neural networks and non-Euclidean geometries.