On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
Dongruo Zhou, Jinghui Chen, Yuan Cao, Ziyan Yang, Quanquan Gu
TL;DR
This work offers a thorough convergence analysis of adaptive gradient methods—AMSGrad, RMSProp, and AdaGrad—in stochastic nonconvex optimization. It establishes convergence in expectation with an improved rate that scales favorably with dimension, demonstrating a worst-case bound of O(sqrt(d/T) + d/T) under a growth-rate constraint on per-coordinate gradients. The authors further derive the first high-probability bounds for these methods in the nonconvex setting, reinforcing their practical reliability. By introducing a refined analytical framework and an auxiliary sequence to handle momentum and adaptive steps, the paper illuminates the mechanisms behind adaptive gradients in nonconvex objectives and provides actionable guidance on parameter scaling.
Abstract
Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly studied. In this paper, we provide a fine-grained convergence analysis for a general class of adaptive gradient methods including AMSGrad, RMSProp and AdaGrad. For smooth nonconvex functions, we prove that adaptive gradient methods in expectation converge to a first-order stationary point. Our convergence rate is better than existing results for adaptive gradient methods in terms of dimension. In addition, we also prove high probability bounds on the convergence rates of AMSGrad, RMSProp as well as AdaGrad, which have not been established before. Our analyses shed light on better understanding the mechanism behind adaptive gradient methods in optimizing nonconvex objectives.