Information Hidden in Gradients of Regression with Target Noise

TL;DR

This work addresses how to extract second-order information, specifically the Hessian/ data covariance Σ, when only gradient information is accessible. By injecting Gaussian target noise whose total variance matches the batch size $n$, the empirical gradient covariance $S_g(w)$ aligns with Σ under mild sub-Gaussian assumptions, providing non-asymptotic operator-norm guarantees. The authors prove the calibration is necessary (without it, recovery can fail by an $oldsymbol{Ω}(1)$ factor) and demonstrate practical gradient-only applications, including preconditioning for faster convergence and adversarial-risk estimation, with extensions to non-linear models. Experiments on synthetic and real data support the theory, showing calibrated gradient covariances closely approximate Σ and enable robust gradient-only optimization across batch regimes and model types.

Abstract

Second-order information -- such as curvature or data covariance -- is critical for optimisation, diagnostics, and robustness. However, in many modern settings, only the gradients are observable. We show that the gradients alone can reveal the Hessian, equalling the data covariance $Σ$ for the linear regression. Our key insight is a simple variance calibration: injecting Gaussian noise so that the total target noise variance equals the batch size ensures that the empirical gradient covariance closely approximates the Hessian, even when evaluated far from the optimum. We provide non-asymptotic operator-norm guarantees under sub-Gaussian inputs. We also show that without such calibration, recovery can fail by an $Ω(1)$ factor. The proposed method is practical (a "set target-noise variance to $n$" rule) and robust (variance $\mathcal{O}(n)$ suffices to recover $Σ$ up to scale). Applications include preconditioning for faster optimisation, adversarial risk estimation, and gradient-only training, for example, in distributed systems. We support our theoretical results with experiments on synthetic and real data.

Figures (11)

• Figure 1: Gradient covariance calibrated with target noise matches the Hessian (Insight 1) and preserves the mean gradient (Insight 2). Blue thin lines: batch gradients; red thick line: their average; black solid ellipse: empirical covariance $S_g(w)$; orange dashed ellipse: data covariance $\Sigma$. Left: without calibration, $S_g(w)$ deviates from $\Sigma$. Right: after adding noise to the targets, the gradients spread out and $S_g(w)$ aligns closely with $\Sigma$, while the mean gradient remains accurate.
• Figure 2: Comparison of true data covariance and its estimates for the data generated from a Gaussian distribution. Data covariance (left), its estimates by gradient covariance matrices with clean targets (middle), and with target noise of variance equal to batch size, $n=256$ (right). We compare the results using the relative operator norm $r$ (Eq. \ref{['eq:norm_r']}).
• Figure 3: Sensitivity of Hessian estimates. Dependency of the estimates on (a) number of batches, (b) batch size, (c) number of features, (d) proximity of model weights to optimum, and (e) standard deviation of added target noise. We report norm $r$ (Eq. \ref{['eq:norm_r']}). The estimate of the covariance matrix improves with a larger number of batches and larger batch sizes, and remains stable in the vicinity of the true model weights. The optimal added noise variance is equal to the batch size $n=256$. Here, the model is randomly initialised except in plot (d), the optimal weights are distorted by a random vector with norm $c$.
• Figure 4: Hessian estimate quality during model training.Top: the relative operator norm $r$ (Eq. \ref{['eq:norm_r']} for the four studied datasets. Middle: the corresponding distance between the analytical solution and the weight on the current epoch. Bottom: Absolute error for the quadratic form approximation (Lemma \ref{['lemma:advrisk']}) used in adversarial risk calculation (Lemma \ref{['lemma:advriskref']}). The results correspond to the runs in Table \ref{['tab:lr_real']} and are averaged across 10 random seeds.
• Figure 5: Hessian estimate during MLP training with gradient covariance matrices.Top: the relative operator norm for the four studied datasets with noise-free and noisy gradients. Bottom: Test MSE metric as the measure of convergence. The results correspond to the runs in Table \ref{['tab:lr_real']} with an MLP.

Theorems & Definitions (40)

• Theorem 3.1
• Theorem 4.1
• Lemma 4.2
• Lemma 5.1
• Corollary 5.1.1
• Lemma 5.2
• Theorem 6.1
• Lemma 6.2: Informal
• Definition 6.2.1: Adversarial Risk
• Lemma 6.3: scetbon2023robust Closed-Form Expression for Adversarial Risk