Differentially Private Generalized Linear Models Revisited

TL;DR

The paper develops near-optimal differential privacy rates for learning generalized linear models with convex losses, separating the analysis into smooth non-negative and Lipschitz loss regimes. It introduces two DP-SCO strategies—constrained ERM with Gaussian output perturbation and Johnson-Lindenstrauss-based dimensionality reduction—together with private model selection to adapt to the unknown minimizer norm $\|w^*\|$. For smooth non-negative GLMs it derives a refined rate $\tilde{O}\left(\frac{\|w^*\|}{\sqrt{n}} + \min\left\{\frac{\|w^*\|^2}{(nε)^{2/3}},\frac{\sqrt{d}\|w^*\|^2}{nε}\right\}\right)$, with a low-/high-dimensional transition at $d \approx (\|w^*\| nε)^{2/3}$, and matching lower bounds. In the Lipschitz GLM setting, it tightens the rate to $\tilde{O}\left(\frac{\|w^*\|}{\sqrt{n}} + \min\left\{\frac{\|w^*\|}{\sqrt{nε}},\frac{\sqrt{\text{rank}}\|w^*\|}{nε}\right\}\right)$, improving in the high-privacy regime; it also extends corresponding lower bounds. The paper further presents non-Euclidean DP-SCO lower bounds and non-private insights, and provides a private model-selection mechanism with confidence boosting to eliminate the need for prior knowledge of $\|w^*\|$. These contributions advance understanding of DP learning for GLMs under smooth/non-Lipschitz losses and broaden the toolkit for private high-dimensional statistical learning.

Abstract

We study the problem of $(ε,δ)$-differentially private learning of linear predictors with convex losses. We provide results for two subclasses of loss functions. The first case is when the loss is smooth and non-negative but not necessarily Lipschitz (such as the squared loss). For this case, we establish an upper bound on the excess population risk of $\tilde{O}\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^* \Vert^2}{(nε)^{2/3}},\frac{\sqrt{d}\Vert w^*\Vert^2}{nε}\right\}\right)$, where $n$ is the number of samples, $d$ is the dimension of the problem, and $w^*$ is the minimizer of the population risk. Apart from the dependence on $\Vert w^\ast\Vert$, our bound is essentially tight in all parameters. In particular, we show a lower bound of $\tildeΩ\left(\frac{1}{\sqrt{n}} + {\min\left\{\frac{\Vert w^*\Vert^{4/3}}{(nε)^{2/3}}, \frac{\sqrt{d}\Vert w^*\Vert}{nε}\right\}}\right)$. We also revisit the previously studied case of Lipschitz losses [SSTT20]. For this case, we close the gap in the existing work and show that the optimal rate is (up to log factors) $Θ\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^*\Vert}{\sqrt{nε}},\frac{\sqrt{\text{rank}}\Vert w^*\Vert}{nε}\right\}\right)$, where $\text{rank}$ is the rank of the design matrix. This improves over existing work in the high privacy regime. Finally, our algorithms involve a private model selection approach that we develop to enable attaining the stated rates without a-priori knowledge of $\Vert w^*\Vert$.

Theorems & Definitions (54)

• Lemma 1
• Definition 1: $(\alpha,\beta)$-JL property
• Lemma 2
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6
• Theorem 7