Gradient correlation is a key ingredient to accelerate SGD with momentum
Julien Hermant, Marien Renaud, Jean-François Aujol, Charles Dossal, Aude Rondepierre
TL;DR
The paper tackles whether stochastic momentum (SNAG) can accelerate SGD in convex finite-sum problems under interpolation. It introduces RACOGA, a gradient-correlation condition, and shows how it governs the Strong Growth Condition constant ρ_K, thereby determining when SNAG can beat SGD. The work provides both finite-time and almost-sure convergence results under RACOGA and its relaxations, plus a theory for how batch size interacts with gradient correlation. Empirical results on linear regression and neural networks corroborate that higher gradient correlation along the optimization path yields faster SNAG performance, validating the proposed framework and its practical implications for choosing batch sizes and hyperparameters.
Abstract
Empirically, it has been observed that adding momentum to Stochastic Gradient Descent (SGD) accelerates the convergence of the algorithm. However, the literature has been rather pessimistic, even in the case of convex functions, about the possibility of theoretically proving this observation. We investigate the possibility of obtaining accelerated convergence of the Stochastic Nesterov Accelerated Gradient (SNAG), a momentum-based version of SGD, when minimizing a sum of functions in a convex setting. We demonstrate that the average correlation between gradients allows to verify the strong growth condition, which is the key ingredient to obtain acceleration with SNAG. Numerical experiments, both in linear regression and deep neural network optimization, confirm in practice our theoretical results.