Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adapting to noise tails in private linear regression

Jinyuan Chang, Lin Yang, Mengyue Zha, Wen-Xin Zhou

TL;DR

This paper develops differentially private tail-robust methods for linear regression that extend classical perspectives on privacy-preserving robust regression by quantifying the interplay among bias, privacy, and robustness.

Abstract

While the traditional goal of statistics is to infer population parameters, modern practice increasingly demands protection of individual privacy. One way to address this need is to adapt classical statistical procedures into privacy-preserving algorithms. In this paper, we develop differentially private tail-robust methods for linear regression. The trade-off among bias, privacy, and robustness is controlled by a tunable robustification parameter in the Huber loss. We implement noisy clipped gradient descent for low-dimensional settings and noisy iterative hard thresholding for high-dimensional sparse models. Under sub-Gaussian errors, our method achieves near-optimal convergence rates while relaxing several assumptions required in earlier work. For heavy-tailed errors, we explicitly characterize how the non-asymptotic convergence rate depends on the moment index, privacy parameters, sample size, and intrinsic dimension. Our analysis shows how the moment index influences the choice of robustification parameters and, in turn, the resulting statistical error and privacy cost. By quantifying the interplay among bias, privacy, and robustness, we extend classical perspectives on privacy-preserving robust regression. The proposed methods are evaluated through simulations and two real datasets.

Adapting to noise tails in private linear regression

TL;DR

This paper develops differentially private tail-robust methods for linear regression that extend classical perspectives on privacy-preserving robust regression by quantifying the interplay among bias, privacy, and robustness.

Abstract

While the traditional goal of statistics is to infer population parameters, modern practice increasingly demands protection of individual privacy. One way to address this need is to adapt classical statistical procedures into privacy-preserving algorithms. In this paper, we develop differentially private tail-robust methods for linear regression. The trade-off among bias, privacy, and robustness is controlled by a tunable robustification parameter in the Huber loss. We implement noisy clipped gradient descent for low-dimensional settings and noisy iterative hard thresholding for high-dimensional sparse models. Under sub-Gaussian errors, our method achieves near-optimal convergence rates while relaxing several assumptions required in earlier work. For heavy-tailed errors, we explicitly characterize how the non-asymptotic convergence rate depends on the moment index, privacy parameters, sample size, and intrinsic dimension. Our analysis shows how the moment index influences the choice of robustification parameters and, in turn, the resulting statistical error and privacy cost. By quantifying the interplay among bias, privacy, and robustness, we extend classical perspectives on privacy-preserving robust regression. The proposed methods are evaluated through simulations and two real datasets.
Paper Structure (69 sections, 37 theorems, 296 equations, 6 figures, 5 tables, 5 algorithms)

This paper contains 69 sections, 37 theorems, 296 equations, 6 figures, 5 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Background on Differential Privacy
  3. Differentially Private Huber Regression
  4. Huber regression
  5. Randomized estimator via noisy gradient descent
  6. Noisy iterative hard thresholding for sparse Huber regression
  7. Statistical Analysis of Private Huber Regression
  8. Low-dimensional setting
  9. High-dimensional setting
  10. Numerical Studies
  11. Selection of tuning parameters
  12. Selection of tuning parameters in Algorithm \ref{['alg:DPHuberlow']}
  13. Selection of tuning parameters in Algorithm \ref{['alg:DPHuberhigh']}
  14. Low-dimensional setting
  15. High-dimensional setting
  16. ...and 54 more sections

Key Result

Lemma 1

(Gaussian mechanism) Assume sens$_2(\mathcal{M})\leq B$ for some $B>0$. Then $\mathcal{M}({\mathbf X})+{\mathbf g}$, with ${\mathbf g}\sim \mathcal{N}(0,\sigma^2{\mathbf I}_{d})$ and $\sigma = B \epsilon^{-1}\sqrt{2 \log(1.25/\delta)}$, is $(\epsilon,\delta)$-DP.

Figures (6)

  • Figure 1: Plots of the average logarithmic relative $\ell_2$-error over 300 repetitions as a function of the sample size under different design settings. The top two panels correspond to Gaussian covariates, and the bottom two panels to uniform covariates, with $\epsilon=0.9$, $p=10$, $a=2$, and $b=0.5$. The noise distribution is either $\mathcal{N}(0,1)$ or $t_{2.25}$.
  • Figure 2: Boxplots of the logarithmic relative $\ell_2$-error, based on 300 repetitions, across three sample sizes under four design settings. The top two panels correspond to Gaussian covariates, and the bottom two to uniform covariates, with $p=10$, $a=2$, and $b=0.5$. The noise distribution is either $\mathcal{N}(0,1)$ or $t_{2.25}$.
  • Figure 3: Boxplots of the logarithmic relative $\ell_2$-error over 300 repetitions for private and non-private sparse Huber estimators across sample sizes, with $p = 10000$ and $s^* = 10$.
  • Figure S1: Boxplots of the MSPE over 300 replications, comparing the $(\epsilon,\delta_m)$-DP Huber estimator at different privacy levels with its non-private counterpart.
  • Figure S2: Plots of the average MSPE, based on 300 replications, comparing the sparse DP Huber estimator, the sparse DP LS estimator, and the non-private sparse Huber estimator under privacy levels $(0.9, \delta_m)$ and $s=5$, as $m$ increases from 500 to 1900.
  • ...and 1 more figures

Theorems & Definitions (47)

  • Definition 1: dwork2006calibrating
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Definition 2: Trade-off function
  • Definition 3: $f$-DP and GDP
  • Lemma 5
  • Remark 1
  • Remark 2
  • ...and 37 more