Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
Aaron Mishkin, Mert Pilanci, Mark Schmidt
TL;DR
This work addresses accelerating stochastic gradient methods under the interpolation setting by developing a generalized stochastic AGD framework built on estimating sequences. The key contribution is proving that any progression-guaranteeing primal update can be accelerated, yielding a reduced dependence on the strong growth constant from $\rho$ to $\sqrt{\rho}$ and, in the strongly convex case, an iteration complexity of $O\left(\frac{\sqrt{L L_{\max}}}{\mu}\log\frac{1}{\epsilon}\right)$. The results specialize to accelerated SGD under the strong growth condition and extend to preconditioned variants, offering faster convergence than SGD under favorable conditioning and addressing criticisms about stochastic acceleration guarantees. Comparisons with existing rates show improved robustness to noise and broader applicability beyond quadratics. The work also identifies future directions, including relaxing growth conditions and exploring fully stochastic estimating-sequence formulations.
Abstract
We prove new convergence rates for a generalized version of stochastic Nesterov acceleration under interpolation conditions. Unlike previous analyses, our approach accelerates any stochastic gradient method which makes sufficient progress in expectation. The proof, which proceeds using the estimating sequences framework, applies to both convex and strongly convex functions and is easily specialized to accelerated SGD under the strong growth condition. In this special case, our analysis reduces the dependence on the strong growth constant from $ρ$ to $\sqrtρ$ as compared to prior work. This improvement is comparable to a square-root of the condition number in the worst case and address criticism that guarantees for stochastic acceleration could be worse than those for SGD.