Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy-Accuracy Trade-offs in High-Dimensional LASSO under Perturbation Mechanisms

Ayaka Sakata, Haruka Tanzawa

Abstract

We study privacy-preserving sparse linear regression in the high-dimensional regime, focusing on the LASSO estimator. We analyze two widely used mechanisms for differential privacy: output perturbation, which injects noise into the estimator, and objective perturbation, which adds a random linear term to the loss function. Using approximate message passing (AMP), we characterize the typical behavior of these estimators under random design and privacy noise. To quantify privacy, we adopt typical-case measures, including the on-average KL divergence, which admits a hypothesis-testing interpretation in terms of distinguishability between neighboring datasets. Our analysis reveals that sparsity plays a central role in shaping the privacy-accuracy trade-off: stronger regularization can improve privacy by stabilizing the estimator against single-point data changes. We further show that the two mechanisms exhibit qualitatively different behaviors. In particular, for objective perturbation, increasing the noise level can have non-monotonic effects, and excessive noise may destabilize the estimator, leading to increased sensitivity to data perturbations. Our results demonstrate that AMP provides a powerful framework for analyzing privacy-accuracy trade-offs in high-dimensional sparse models.

Privacy-Accuracy Trade-offs in High-Dimensional LASSO under Perturbation Mechanisms

Abstract

We study privacy-preserving sparse linear regression in the high-dimensional regime, focusing on the LASSO estimator. We analyze two widely used mechanisms for differential privacy: output perturbation, which injects noise into the estimator, and objective perturbation, which adds a random linear term to the loss function. Using approximate message passing (AMP), we characterize the typical behavior of these estimators under random design and privacy noise. To quantify privacy, we adopt typical-case measures, including the on-average KL divergence, which admits a hypothesis-testing interpretation in terms of distinguishability between neighboring datasets. Our analysis reveals that sparsity plays a central role in shaping the privacy-accuracy trade-off: stronger regularization can improve privacy by stabilizing the estimator against single-point data changes. We further show that the two mechanisms exhibit qualitatively different behaviors. In particular, for objective perturbation, increasing the noise level can have non-monotonic effects, and excessive noise may destabilize the estimator, leading to increased sensitivity to data perturbations. Our results demonstrate that AMP provides a powerful framework for analyzing privacy-accuracy trade-offs in high-dimensional sparse models.

Paper Structure

This paper contains 41 sections, 2 theorems, 93 equations, 11 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Our contribution
  4. Problem setting
  5. Private LASSO via Objective Perturbation
  6. Private LASSO via Output Perturbation
  7. Evaluation Criteria: Accuracy and Privacy
  8. (1) Prediction Accuracy.
  9. (2) Privacy Measure: On-Average KL-Privacy.
  10. Approximate Message Passing algorithm
  11. Bayesian formulation and MAP estimator
  12. Assumptions on data generative process and privacy noise
  13. Setting D (Data model)
  14. Setting P (Privacy noise model)
  15. Approximate Message Passing Algorithm
  16. ...and 26 more sections

Key Result

Theorem 4.1

Consider a component of the AMP estimator at step $t$, $\widehat{\bm{\beta}}_{\mathrm{AMP}}^{(t)}$, with assigned privacy noise $\eta$ and ground truth $\beta^{(0)}$. Denote the component by $\widehat{\beta}^{(t)}_{\mathrm{AMP}}(X,\beta^{(0)},\eta)$. If the privacy noise satisfies Assumption P, then where $\sigma_z^{(t)}=\sqrt{E^{(t)}/\alpha}$, $\Sigma^{(t)}=(1+V^{(t)})/\alpha$, and $z\sim{\cal N}

Figures (11)

  • Figure 1: $\sigma_\eta$-dependence of (a) the fraction of nonzero components in the estimator and (b) the generalization error for various regularization parameters $\lambda$ under objective perturbation. Circles ($\circ$) denote AMP results and solid lines show the corresponding analysis by replica method. In the AMP experiments, we set $p=1000$, $\alpha=0.5$, $\rho=0.1$, and $\sigma_\xi=0.1$. AMP results are averaged over 100 datasets; error bars indicate the standard error. For each dataset, a single realization of privacy noise $\bm{\eta}$ is generated. The dashed lines in (b) shows the generalization error under output perturbation.
  • Figure 2: (a) $\lambda$-dependence of the generalization error for different privacy noise levels ($\sigma_\eta=0.1$, $0.3$) compared with the noiseless case. (b) Generalization error as a function of the resulting $\widehat{\rho}$. The experimental setup and plotting style are the same as in Fig. \ref{['fig:vs_sigma_eta']}.
  • Figure 3: (a) Dependence of the ratio between generalization and training errors on $\widehat{\rho}$ at $\rho=0.1$, $\sigma_\xi=1$, and $\sigma_\eta=0.1$, for $\alpha=0.5$ and $\alpha=0.3$, as predicted by state evolution in the high-dimensional limit. Circles represent the AMP results. (b) Parameter region where AMP does not converge at $\alpha=0.5$, $\rho=0.5$ and $\sigma_\xi=0.1$. The solid line denotes the boundary predicted by the replica method, circles and squares indicate the empirical convergence limit of AMP, and squares indicate that of coordinate descent. For both AMP and coordinate descent, convergence was evaluated over 100 independent runs, and a point is marked as non-convergent if at least one run fails to converge.
  • Figure 4: Comparison of distribution of the estimators with respect to data by AMP ($\circ$) and DE (boxes) at $\alpha=0.5$, $\rho=0.3$, $\sigma_\xi=0.1$, $\lambda=0.5$ and $\sigma_\eta=0.1$. Distributions for different $\beta^{(0)}$ and $\eta$ are shown: (a) $\beta^{(0)}=1.0629,~\eta = 0.0460$, (b) $\beta^{(0)}=-1.0412,~ \eta = -0.0238$, (c) $\beta^{(0)}=-2.1796,~\eta = -0.0079$, (d) $\beta^{(0)}=2.1290,~\eta = -0.0317$, (e) $\beta^{(0)}=3.2188,~\eta = 0.0556$, (f) $\beta=-3.2333,~\eta = -0.0321$, (g) $\beta^{(0)}=-1.0095,~\eta = 0.2907$, (h) $\beta^{(0)}=0.1017,~\eta = -0.3261$, (i) $\beta^{(0)}=0,~\eta = 0.3579$, (j) $\beta^{(0)} = 0, \eta = -0.3233$.
  • Figure 5: Examples of the distributions of the AMP estimates induced by the injected privacy noise (solid lines) and the corresponding SE predictions (dashed lines) at $\alpha=0.5$, $\rho=0.5$, $\sigma_y=0.1$, $\lambda=1$, and $\sigma_\eta=0.5$. For AMP, we set $N=500$, and 1000 realizations of the objective noise are used in the computation. Panels (a)-(f) correspond to $\beta^{(0)}=0$, $0.3554$, $1.308$, $-1.279$, $2.191$, and $-2.432$, respectively. The circle at $\beta=0$ indicates the height of the peak of the AMP distribution.
  • ...and 6 more figures

Theorems & Definitions (7)

  • Theorem 4.1: AMP decoupling with privacy noise
  • Claim 4.2: Proportionality between two errors
  • Claim 5.1: Equivalent distribution induced by objective privacy noise
  • Claim 6.1: Distribution of estimate under One-Point-Mutant data
  • Corollary 6.2
  • Claim S.4.1: Self-averaging of the privacy-noise-dependent free energy
  • Remark S.4.2