Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis
Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, Yaoqing Yang
TL;DR
This work analyzes how SGD behaves on ill-conditioned loss landscapes with a dominant Hessian subspace and a bulk subspace. It introduces an adaptive step-size condition, revealing two alignment regimes and a stable intermediate interval; in high-alignment regimes, updates in the dominant subspace self-correct while bulk projections drive loss reduction, explaining empirical observations. A constant-step SGD analysis shows a two-phase dynamic where alignment first decreases and then stabilizes near unity as the spectrum gap grows, with the long-run alignment $ heta_ ext{∞}$ determined by the spectrum and noise via $ heta_ ext{∞} = rac{ extstyle\sum_{i otin ext{B}} oldsymbol{beta}_i}{ extstyle\sum_{i} oldsymbol{beta}_i}$ (in the high-dimensional limit). The results illuminate the trade-offs in step-size design under ill-conditioning and motivate preconditioning to improve optimization efficiency. Overall, the paper provides a rigorous, adaptive framework for understanding SGD alignment, two-phase dynamics, and the limitations of dominant-space projections in reducing loss.
Abstract
This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. This phenomenon describes the behavior of gradient alignment in SGD updates. Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease. Subsequently, it enters a rising phase and eventually stabilizes in a high-alignment phase. The alignment is considered ``suspicious'' because, paradoxically, the projected gradient update along this highly-aligned dominant subspace proves ineffective at reducing the loss. The focus of this work is to give a fine-grained analysis in a high-dimensional quadratic setup about how step size selection produces this phenomenon. Our main contribution can be summarized as follows: We propose a step-size condition revealing that in low-alignment regimes, an adaptive critical step size $η_t^*$ separates alignment-decreasing ($η_t < η_t^*$) from alignment-increasing ($η_t > η_t^*$) regimes, whereas in high-alignment regimes, the alignment is self-correcting and decreases regardless of the step size. We further show that under sufficient ill-conditioning, a step size interval exists where projecting the SGD updates to the bulk space decreases the loss while projecting them to the dominant space increases the loss, which explains a recent empirical observation that projecting gradient updates to the dominant subspace is ineffective. Finally, based on this adaptive step-size theory, we prove that for a constant step size and large initialization, SGD exhibits this distinct two-phase behavior: an initial alignment-decreasing phase, followed by stabilization at high alignment.