Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$
Bhavya Agrawalla, Krishnakumar Balasubramanian, Promit Ghosal
TL;DR
The paper tackles uncertainty quantification for online least-squares SGD in growing-dimensional settings by proving non-asymptotic Berry–Esseen bounds for linear functionals of SGD iterates, achieving a Gaussian CLT in the regime $t \gtrsim d^{1+\delta}$ with a practical online variance estimator. It introduces an online, assumption-lean framework with step sizes $\eta_i = \frac{\eta}{\sqrt{d}\, i^{\alpha}}$, $\alpha \in (\tfrac{1}{2},1)$, and milder moment conditions than prior covariance-inversion methods; importantly, it quantifies the finite-sample distributional distance $d_K$ and provides a bias-corrected variant $d^{true}_K$. The paper additionally develops a fully online variance estimator with high-probability deviation bounds, enabling data-driven confidence intervals for SGD projections in near-optimal dimensional scaling, and demonstrates end-to-end online inference with $\mathcal{O}(td)$ time and $\mathcal{O}(d)$ space. Collectively, these results offer a principled, scalable approach to online inference in high-dimensional linear models and pave the way for Wald-type tests and predictive intervals in streaming settings.
Abstract
Stochastic Gradient Descent (SGD) has become a cornerstone method in modern data science. However, deploying SGD in high-stakes applications necessitates rigorous quantification of its inherent uncertainty. In this work, we establish \emph{non-asymptotic Berry--Esseen bounds} for linear functionals of online least-squares SGD, thereby providing a Gaussian Central Limit Theorem (CLT) in a \emph{growing-dimensional regime}. Existing approaches to high-dimensional inference for projection parameters, such as~\cite{chang2023inference}, rely on inverting empirical covariance matrices and require at least $t \gtrsim d^{3/2}$ iterations to achieve finite-sample Berry--Esseen guarantees, rendering them computationally expensive and restrictive in the allowable dimensional scaling. In contrast, we show that a CLT holds for SGD iterates when the number of iterations grows as $t \gtrsim d^{1+δ}$ for any $δ> 0$, significantly extending the dimensional regime permitted by prior works while improving computational efficiency. The proposed online SGD-based procedure operates in $\mathcal{O}(td)$ time and requires only $\mathcal{O}(d)$ memory, in contrast to the $\mathcal{O}(td^2 + d^3)$ runtime of covariance-inversion methods. To render the theory practically applicable, we further develop an \emph{online variance estimator} for the asymptotic variance appearing in the CLT and establish \emph{high-probability deviation bounds} for this estimator. Collectively, these results yield the first fully online and data-driven framework for constructing confidence intervals for SGD iterates in the near-optimal scaling regime $t \gtrsim d^{1+δ}$.