High-Dimensional Privacy-Utility Dynamics of Noisy Stochastic Gradient Descent on Least Squares

TL;DR

This work develops a diffusion-based framework to analyze high-dimensional noisy DP-SGD for ℓ2-regularized least squares, introducing the noisy Homogenized SGD (HSGD) diffusion as a precise surrogate for both statistical risk and privacy loss. The authors derive Volterra integral equations that capture the deterministic risk trajectory and provide exact Gaussian laws for the privacy loss, enabling closed-form Rényi-DP bounds under last-iterate, intermediate, and averaged releases. They also propose a clipping-free variant and validate the theory with simulations in a high-dimensional setting, showing strong risk concentration around the Volterra trajectory and a clear privacy–utility tradeoff controlled by the noise scale. The results offer a rigorous, high-dimensional DP analysis that informs privacy-utility choices in streaming/one-pass training and suggests diffusion-based proxies as practical tools for DP-SGD assessment. The research lays groundwork for extending to multi-pass training and broader loss families, potentially guiding deployment of differential privacy in large-scale, overparameterized models.

Abstract

The interplay between optimization and privacy has become a central theme in privacy-preserving machine learning. Noisy stochastic gradient descent (SGD) has emerged as a cornerstone algorithm, particularly in large-scale settings. These variants of gradient methods inject carefully calibrated noise into each update to achieve differential privacy, the gold standard notion of rigorous privacy guarantees. Prior work primarily provides various bounds on statistical risk and privacy loss for noisy SGD, yet the \textit{exact} behavior of the process remains unclear, particularly in high-dimensional settings. This work leverages a diffusion approach to analyze noisy SGD precisely, providing a continuous-time perspective that captures both statistical risk evolution and privacy loss dynamics in high dimensions. Moreover, we study a variant of noisy SGD that does not require explicit knowledge of gradient sensitivity, unlike existing work that assumes or enforces sensitivity through gradient clipping. Specifically, we focus on the least squares problem with $\ell_2$ regularization.

Figures (3)

• Figure 1: Diagnostics of HSGD approximation for privacy loss ($\sigma=1.5$).
• Figure 2: Statistical risk and privacy loss of noisy SGD via noisy HSGD.
• Figure 3: Statistical risk and privacy loss across different clipping thresholds ($\sigma=1.5$).

Theorems & Definitions (32)

• Definition 1: Mironov2017RnyiDP Rényi Differential Privacy
• Definition 2: Quadratic population risk
• Theorem 3.1: Quadratic Risk Equivalence
• Conjecture 3.1: Distributional Equivalence
• Theorem 4.1: Risk Concentration
• Theorem 4.2: Risk Trajectory
• Corollary 4.1
• Theorem 4.3: RDP for last iterate
• Theorem 4.4
• Theorem 4.5: RDP for releasing the average