Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Two-Stage Gradient Descent for Instrumental Variable Regression

Haodong Liang, Yanhao Jin, Krishnakumar Balasubramanian, Lifeng Lai

TL;DR

The paper tackles private linear instrumental variable regression (IVaR) by proposing DP-2S-GD, a two-stage gradient-descent method with per-sample gradient clipping and Gaussian noise that provides $\rho$-zCDP guarantees. It delivers non-asymptotic convergence bounds that decompose error into optimization, privacy, and sampling components and prescribes how iteration count $T$ should scale with sample size $n$ and privacy budgets $(\rho_1,\rho_2)$. Theoretical results are complemented by synthetic and real-data experiments (Angrist and Card datasets) demonstrating accurate private estimates that align with the classical 2SLS benchmark as privacy is relaxed. The work advances privacy-preserving causal inference by delivering end-to-end DP guarantees for IVaR with explicit trade-offs and practical guidance for implementation in privacy-sensitive settings.

Abstract

We study instrumental variable regression (IVaR) under differential privacy constraints. Classical IVaR methods (like two-stage least squares regression) rely on solving moment equations that directly use sensitive covariates and instruments, creating significant risks of privacy leakage and posing challenges in designing algorithms that are both statistically efficient and differentially private. We propose a noisy two-state gradient descent algorithm that ensures $ρ$-zero-concentrated differential privacy by injecting carefully calibrated noise into the gradient updates. Our analysis establishes finite-sample convergence rates for the proposed method, showing that the algorithm achieves consistency while preserving privacy. In particular, we derive precise bounds quantifying the trade-off among optimization, privacy, and sampling error. To the best of our knowledge, this is the first work to provide both privacy guarantees and provable convergence rates for instrumental variable regression in linear models. We further validate our theoretical findings with experiments on both synthetic and real datasets, demonstrating that our method offers practical accuracy-privacy trade-offs.

Differentially Private Two-Stage Gradient Descent for Instrumental Variable Regression

TL;DR

The paper tackles private linear instrumental variable regression (IVaR) by proposing DP-2S-GD, a two-stage gradient-descent method with per-sample gradient clipping and Gaussian noise that provides -zCDP guarantees. It delivers non-asymptotic convergence bounds that decompose error into optimization, privacy, and sampling components and prescribes how iteration count should scale with sample size and privacy budgets . Theoretical results are complemented by synthetic and real-data experiments (Angrist and Card datasets) demonstrating accurate private estimates that align with the classical 2SLS benchmark as privacy is relaxed. The work advances privacy-preserving causal inference by delivering end-to-end DP guarantees for IVaR with explicit trade-offs and practical guidance for implementation in privacy-sensitive settings.

Abstract

We study instrumental variable regression (IVaR) under differential privacy constraints. Classical IVaR methods (like two-stage least squares regression) rely on solving moment equations that directly use sensitive covariates and instruments, creating significant risks of privacy leakage and posing challenges in designing algorithms that are both statistically efficient and differentially private. We propose a noisy two-state gradient descent algorithm that ensures -zero-concentrated differential privacy by injecting carefully calibrated noise into the gradient updates. Our analysis establishes finite-sample convergence rates for the proposed method, showing that the algorithm achieves consistency while preserving privacy. In particular, we derive precise bounds quantifying the trade-off among optimization, privacy, and sampling error. To the best of our knowledge, this is the first work to provide both privacy guarantees and provable convergence rates for instrumental variable regression in linear models. We further validate our theoretical findings with experiments on both synthetic and real datasets, demonstrating that our method offers practical accuracy-privacy trade-offs.

Paper Structure

This paper contains 33 sections, 15 theorems, 175 equations, 12 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Privacy notions
  5. IVaR Model and Assumptions
  6. Algorithm and Theoretical Guarantees
  7. Experiments
  8. Synthetic Data Simulations
  9. Real-Data Experiments
  10. Conclusion
  11. Additional Definitions
  12. Proof of Proposition \ref{['lem: privacy II - main']}
  13. Proof of Theorem \ref{['thm: main result II']}
  14. Supporting Lemmas
  15. Proof of Supporting Lemmas
  16. ...and 18 more sections

Key Result

Proposition 3.1

If we set $\lambda_1 = \frac{2\gamma_{1}}{n}\sqrt{\frac{T}{\rho_1}}$ and $\lambda_2 = \frac{2\gamma_{2}}{n}\sqrt{\frac{T}{\rho_2}}$, Algorithm alg: DP-2S-GD-II is $\rho$-zCDP, where $\rho:=\rho_1+\rho_2=\frac{2 T}{n^2}\left(\frac{\gamma_1^2}{\lambda_1^2}+\frac{\gamma_2^2}{\lambda_2^2}\right)$.

Figures (12)

  • Figure 1: IVaR model: Instrument $\mathbf{z}$ is correlated with the endogenous regressor $\mathbf{x}$ and influences the outcome $y$ only indirectly through $\mathbf{x}$, while an unobserved confounder $\mathbf{u}$ affects both $\mathbf{x}$ and $y$.
  • Figure 2: Qualitative trend of the error bound \ref{['eq: e_beta(T) bound thm2']} as a function of $T$.
  • Figure 3: Comparison of Algorithm \ref{['alg: DP-2S-GD-II']}'s performance versus $n$. We set $T=20$, (a) $p=q=5$, (b) $p=q=50$. Note that the $T$ condition \ref{['eq: parameter settings']} is not satisfied in (b). We set the total budget $\rho=10$ and compare three regimes: (i) $\rho_1=1, \rho_2=9$, (ii) $\rho_1=5, \rho_2=5$, (iii) $\rho_1=9, \rho_2=1$. The curves are averaged over 100 runs, with vertical bars representing the standard errors.
  • Figure 4: Comparison of Algorithm \ref{['alg: DP-2S-GD-II']}'s performance versus number of iterations $T$. We fix $n=1000$, $p=q=5$, (a) keep $\rho_2$ large and vary $\rho_1$, (b) keep $\rho_1$ large and vary $\rho_2$. The curves are averaged over 100 runs, with vertical bars representing the standard errors.
  • Figure 5: Results on the Angrist dataset with $T=20, \rho_1=1, \rho_2=1$. (a) Boxplot of estimated $\boldsymbol{\beta}^{(T)}$, over 1000 runs. (b) Learning paths of parameters $\boldsymbol{\beta}^{(t)}, \boldsymbol{\Theta}^{(t)}$, over 1000 runs. The shaded area represents the standard error.
  • ...and 7 more figures

Theorems & Definitions (43)

  • Definition 2.1: $(\varepsilon,\delta)$-Differential Privacy dwork2006calibrating
  • Definition 2.2: Zero-Concentrated Differential Privacy (zCDP) dwork2016concentratedbun2016concentrated
  • Definition 2.3: 2SLS estimator
  • Proposition 3.1
  • Remark 3.1
  • Theorem 3.1
  • Remark 3.2
  • Remark 3.3
  • Remark 3.4
  • Remark 3.5
  • ...and 33 more