Uncertainty quantification for iterative algorithms in linear models with application to early stopping
Pierre C. Bellec, Kai Tan
TL;DR
The paper develops a finite-iteration uncertainty quantification framework for iterates produced by GD-type algorithms in high-dimensional linear models with $p$ comparable to $n$. By modeling the update as a Lipschitz map and introducing memory matrices ${\mathcal J},{\mathcal D}$ and the memory matrix ${\widehat{\mathbf A}}$, it derives a data-driven generalization-error estimator $\hat r_t$ that is $\sqrt n$-consistent under Gaussian designs, enabling early stopping via $\hat t = \arg\min_t \hat r_t$. It further provides a debiased, coordinate-wise inference scheme with asymptotic normality and practical confidence intervals for the true coefficients at any finite iteration, along with an oracle inequality for stopping rules. The approach is instantiated for GD, AGD, ISTA, FISTA, and LQA/MCP, and supported by efficient computation strategies (forward-substitution and Hutchinson trace approximation) to scale to large problems. Overall, the method offers a scalable, data-driven path to quantify predictive risk and perform inference along algorithmic trajectories, without requiring knowledge of the design covariance or noise level."
Abstract
This paper investigates the iterates $\hbb^1,\dots,\hbb^T$ obtained from iterative algorithms in high-dimensional linear regression problems, in the regime where the feature dimension $p$ is comparable with the sample size $n$, i.e., $p \asymp n$. The analysis and proposed estimators are applicable to Gradient Descent (GD), proximal GD and their accelerated variants such as Fast Iterative Soft-Thresholding (FISTA). The paper proposes novel estimators for the generalization error of the iterate $\hbb^t$ for any fixed iteration $t$ along the trajectory. These estimators are proved to be $\sqrt n$-consistent under Gaussian designs. Applications to early-stopping are provided: when the generalization error of the iterates is a U-shape function of the iteration $t$, the estimates allow to select from the data an iteration $\hat t$ that achieves the smallest generalization error along the trajectory. Additionally, we provide a technique for developing debiasing corrections and valid confidence intervals for the components of the true coefficient vector from the iterate $\hbb^t$ at any finite iteration $t$. Extensive simulations on synthetic data illustrate the theoretical results.