Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Linear Regression and Synthetic Data Generation with Statistical Guarantees

Shurong Lin, Aleksandra Slavković, Deekshith Reddy Bhoomireddy

TL;DR

This work tackles differential privacy for linear regression with uncertainty quantification and for synthetic data generation in small to mid-scale social science data. It introduces BinAgg, a binning-aggregation framework that uses PrivTree to form a private partition and aggregates statistics within bins to support a weighted LR and a DP SDG mechanism, enabling replication without extra privacy cost. A CLT-based analysis yields asymptotically valid DP confidence intervals for regression coefficients via a debiased estimator, and the synthetic data generation preserves the same DP-consistent distribution as the private model. Empirical results across simulations and real data show improved accuracy, valid CIs, and faster performance compared with existing DP LR and SDG methods, supporting reproducible, privacy-preserving analyses on small datasets. The approach offers practical pathways for DP-based inference and data sharing in contexts where standard deep-learning–based SDG is impractical.

Abstract

In social sciences, small- to medium-scale datasets are common and linear regression (LR) is canonical. In privacy-aware settings, much work has focused on differentially private (DP) LR, but mostly on point estimation with limited attention to uncertainty quantification. Meanwhile, synthetic data generation (SDG) is increasingly important for reproducibility studies, yet current DP LR methods do not readily support it. Mainstream SDG approaches are either tailored to discretized data, making them less suitable for continuous regression, or rely on deep models that require large datasets, limiting their use for the smaller, continuous data typical in social science. We propose a method for LR with valid inference under Gaussian DP: a DP bias-corrected estimator with asymptotic confidence intervals (CIs) and a general SDG procedure in which regression on the synthetic data matches our DP regression. Our binning-aggregation strategy is effective in small- to moderate-dimensional settings. Experiments show our method (1) improves accuracy over existing methods, (2) provides valid CIs, and (3) produces more reliable synthetic data for downstream ML tasks than current DP SDGs.

Differentially Private Linear Regression and Synthetic Data Generation with Statistical Guarantees

TL;DR

This work tackles differential privacy for linear regression with uncertainty quantification and for synthetic data generation in small to mid-scale social science data. It introduces BinAgg, a binning-aggregation framework that uses PrivTree to form a private partition and aggregates statistics within bins to support a weighted LR and a DP SDG mechanism, enabling replication without extra privacy cost. A CLT-based analysis yields asymptotically valid DP confidence intervals for regression coefficients via a debiased estimator, and the synthetic data generation preserves the same DP-consistent distribution as the private model. Empirical results across simulations and real data show improved accuracy, valid CIs, and faster performance compared with existing DP LR and SDG methods, supporting reproducible, privacy-preserving analyses on small datasets. The approach offers practical pathways for DP-based inference and data sharing in contexts where standard deep-learning–based SDG is impractical.

Abstract

In social sciences, small- to medium-scale datasets are common and linear regression (LR) is canonical. In privacy-aware settings, much work has focused on differentially private (DP) LR, but mostly on point estimation with limited attention to uncertainty quantification. Meanwhile, synthetic data generation (SDG) is increasingly important for reproducibility studies, yet current DP LR methods do not readily support it. Mainstream SDG approaches are either tailored to discretized data, making them less suitable for continuous regression, or rely on deep models that require large datasets, limiting their use for the smaller, continuous data typical in social science. We propose a method for LR with valid inference under Gaussian DP: a DP bias-corrected estimator with asymptotic confidence intervals (CIs) and a general SDG procedure in which regression on the synthetic data matches our DP regression. Our binning-aggregation strategy is effective in small- to moderate-dimensional settings. Experiments show our method (1) improves accuracy over existing methods, (2) provides valid CIs, and (3) produces more reliable synthetic data for downstream ML tasks than current DP SDGs.
Paper Structure (27 sections, 8 theorems, 40 equations, 2 figures, 8 tables, 4 algorithms)

This paper contains 27 sections, 8 theorems, 40 equations, 2 figures, 8 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Related Work
  4. Preliminaries
  5. Differential Privacy
  6. A DP Binning Algorithm
  7. Methodology
  8. Aggregated Linear Model
  9. DP Binning-Aggregation Algorithms
  10. Sensitivity of $\bm x$ and $y$
  11. Remark on binning dependence.
  12. Theoretical Results
  13. Privacy Guarantees
  14. Statistical Inference
  15. Experiments
  16. ...and 12 more sections

Key Result

Proposition 2.1

Let $\Phi(\cdot)$ be the standard Gaussian cumulative distribution function.

Figures (2)

  • Figure 1: Coefficient estimation error across different $(n, d)$ over 100 repetitions with $\mu=1$
  • Figure 2: Coefficient estimation error using different feature bounds with $d=5$ and $\mu=1$.

Theorems & Definitions (16)

  • Definition 1: Sensitivity
  • Definition 2: $(\varepsilon,\delta)$-DP, DworkR14
  • Definition 3: Gaussian DP, dong2022gaussian
  • Proposition 2.1: Conversion
  • Proposition 2.2: Composition
  • Proposition 2.3: Post-processing
  • Proposition 2.4: Gaussian mechanism
  • Remark 3.1
  • Corollary 3.1: Equivalence of Two Algorithms
  • Theorem 4.1: Gaussian DP Guarantees
  • ...and 6 more