Stability and convergence analysis of AdaGrad for non-convex optimization via novel stopping time-based techniques

TL;DR

This work delivers a comprehensive stability and convergence theory for AdaGrad-Norm in smooth non-convex optimization, introducing a novel stopping-time technique to prove stability in expectation and then establish almost-sure and mean-square convergence. It further derives a near-optimal non-asymptotic rate in expectation, showing $\frac{1}{T}\sum_{n=1}^T \mathbb{E}[\|\nabla g(\theta_n)\|^2] = \mathcal{O}(\frac{\log T}{\sqrt{T}})$ under affine-noise variance, without requiring gradient-boundedness. The analysis is extended to RMSProp, yielding similar stability and convergence results under suitable hyperparameters, highlighting the broader utility of the stopping-time approach for adaptive stochastic methods. Collectively, the results bridge gaps in the theory of AdaGrad and related adaptive optimizers for non-convex problems, with implications for deriving sharper convergence guarantees in practice.

Abstract

Adaptive gradient optimizers (AdaGrad), which dynamically adjust the learning rate based on iterative gradients, have emerged as powerful tools in deep learning. These adaptive methods have significantly succeeded in various deep learning tasks, outperforming stochastic gradient descent. However, despite AdaGrad's status as a cornerstone of adaptive optimization, its theoretical analysis has not adequately addressed key aspects such as asymptotic convergence and non-asymptotic convergence rates in non-convex optimization scenarios. This study aims to provide a comprehensive analysis of AdaGrad and bridge the existing gaps in the literature. We introduce a new stopping time technique from probability theory, which allows us to establish the stability of AdaGrad under mild conditions. We further derive the asymptotically almost sure and mean-square convergence for AdaGrad. In addition, we demonstrate the near-optimal non-asymptotic convergence rate measured by the average-squared gradients in expectation, which is stronger than the existing high-probability results. The techniques developed in this work are potentially of independent interest for future research on other adaptive stochastic algorithms.

Figures (3)

• Figure 1: The structure of the proof of \ref{['stable']}
• Figure 2: The proof structure of AdaGrad-Norm
• Figure 3: The proof structure of RMSProp

Theorems & Definitions (69)

• Lemma 1.1
• Lemma 1.2
• Remark 1
• Lemma 3.1
• proof
• Theorem 3.1
• Lemma 3.2
• proof
• proof
• proof