Near-Optimal Private Linear Regression via Iterative Hessian Mixing

TL;DR

This work advances private linear regression by introducing Iterative Hessian Mixing (IHM), a DP-OLS algorithm that leverages Gaussian sketches within a Hessian-sketch framework. By sketching only the design matrix’s Hessian and employing a DP Gaussian mechanism across iterative updates, IHM achieves DP guarantees while delivering improved excess empirical risk compared to AdaSSP and prior Gaussian-sketch methods. The authors provide a detailed utility analysis, compare fundamental tradeoffs (leading constants, residual sensitivity, eigenvalue dependence, and scaling), and demonstrate strong empirical gains across 33 real datasets. The approach offers a practical, near-optimal DP-OLS solution suitable for bounded-data regimes and federated-like settings, with potential extensions to structured sketches and private second-order optimization.

Abstract

We study differentially private ordinary least squares (DP-OLS) with bounded data. The dominant approach, adaptive sufficient-statistics perturbation (AdaSSP), adds an adaptively chosen perturbation to the sufficient statistics, namely, the matrix $X^{\top}X$ and the vector $X^{\top}Y$, and is known to achieve near-optimal accuracy and to have strong empirical performance. In contrast, methods that rely on Gaussian-sketching, which ensure differential privacy by pre-multiplying the data with a random Gaussian matrix, are widely used in federated and distributed regression, yet remain relatively uncommon for DP-OLS. In this work, we introduce the iterative Hessian mixing, a novel DP-OLS algorithm that relies on Gaussian sketches and is inspired by the iterative Hessian sketch algorithm. We provide utility analysis for the iterative Hessian mixing as well as a new analysis for the previous methods that rely on Gaussian sketches. Then, we show that our new approach circumvents the intrinsic limitations of the prior methods and provides non-trivial improvements over AdaSSP. We conclude by running an extensive set of experiments across standard benchmarks to demonstrate further that our approach consistently outperforms these prior baselines.

Figures (8)

• Figure 1: Performance of the proposed iterative Hessian mixing method compared with state-of-the-art DP-OLS baselines—AdaSSP wang_adassp and methods that rely on using a Gaussian-sketch sheffet2017differentiallysheffet2019oldlev2025gaussianmix—on four representative datasets spanning diverse problem regimes. Across these datasets, and consistently over the full experimental suite (Appendix \ref{['app:additional_exps']}---Appendix \ref{['app:additional_exps_trigger_clipping']}), the IHM matches or outperforms all baselines.
• Figure 2: Overall performance comparison.
• Figure 3: Overall performance comparison.
• Figure 4: Overall performance comparison: together with DP-GD baseline.
• Figure 5: Overall performance comparison: together with DP-GD baseline.

Theorems & Definitions (28)

• Definition 1: $(\varepsilon,\delta)$-DP dwork2006calibrating
• Theorem 1
• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Theorem 2
• Remark 1
• Proposition 5: lev2025gaussianmix
• Lemma 1: lev2025gaussianmix