Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration

Li Shen, Congliang Chen, Fangyu Zou, Zequn Jie, Ju Sun, Wei Liu

TL;DR

The paper tackles the theoretical gap in convergences for adaptive SGD methods in non-convex stochastic settings by introducing AdaUSM, which fuses Weighted AdaGrad with a Unified Stochastic Momentum that spans Heavy Ball and Nesterov momentum. It develops a general weighted, coordinate-wise learning-rate scheme and proves an $O(\log(T)/\sqrt{T})$ non-asymptotic convergence rate under polynomially growing weights, with broader implications that Adam and RMSProp correspond to specific exponential-weight choices within AdaUSM. The framework unifies existing adaptive methods, provides high-probability guarantees, and yields practical acceleration validated by experiments on multiple deep learning models and datasets. Overall, AdaUSM offers both a theoretical and practical bridge between momentum-based acceleration and adaptive step-size strategies in non-convex stochastic optimization.

Abstract

Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, especially in the difficult non-convex stochastic setting. To fill this gap, we propose \emph{weighted AdaGrad with unified momentum}, dubbed AdaUSM, which has the main characteristics that (1) it incorporates a unified momentum scheme which covers both the heavy ball momentum and the Nesterov accelerated gradient momentum; (2) it adopts a novel weighted adaptive learning rate that can unify the learning rates of AdaGrad, AccAdaGrad, Adam, and RMSProp. Moreover, when we take polynomially growing weights in AdaUSM, we obtain its $\mathcal{O}(\log(T)/\sqrt{T})$ convergence rate in the non-convex stochastic setting. We also show that the adaptive learning rates of Adam and RMSProp correspond to taking exponentially growing weights in AdaUSM, thereby providing a new perspective for understanding Adam and RMSProp. Lastly, comparative experiments of AdaUSM against SGD with momentum, AdaGrad, AdaEMA, Adam, and AMSGrad on various deep learning models and datasets are also carried out.

A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration

TL;DR

The paper tackles the theoretical gap in convergences for adaptive SGD methods in non-convex stochastic settings by introducing AdaUSM, which fuses Weighted AdaGrad with a Unified Stochastic Momentum that spans Heavy Ball and Nesterov momentum. It develops a general weighted, coordinate-wise learning-rate scheme and proves an non-asymptotic convergence rate under polynomially growing weights, with broader implications that Adam and RMSProp correspond to specific exponential-weight choices within AdaUSM. The framework unifies existing adaptive methods, provides high-probability guarantees, and yields practical acceleration validated by experiments on multiple deep learning models and datasets. Overall, AdaUSM offers both a theoretical and practical bridge between momentum-based acceleration and adaptive step-size strategies in non-convex stochastic optimization.

Abstract

Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, especially in the difficult non-convex stochastic setting. To fill this gap, we propose \emph{weighted AdaGrad with unified momentum}, dubbed AdaUSM, which has the main characteristics that (1) it incorporates a unified momentum scheme which covers both the heavy ball momentum and the Nesterov accelerated gradient momentum; (2) it adopts a novel weighted adaptive learning rate that can unify the learning rates of AdaGrad, AccAdaGrad, Adam, and RMSProp. Moreover, when we take polynomially growing weights in AdaUSM, we obtain its convergence rate in the non-convex stochastic setting. We also show that the adaptive learning rates of Adam and RMSProp correspond to taking exponentially growing weights in AdaUSM, thereby providing a new perspective for understanding Adam and RMSProp. Lastly, comparative experiments of AdaUSM against SGD with momentum, AdaGrad, AdaEMA, Adam, and AMSGrad on various deep learning models and datasets are also carried out.

Paper Structure

This paper contains 17 sections, 11 theorems, 90 equations, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations.
  4. Assumptions.
  5. Weighted AdaGrad with Unified Momentum
  6. Unified Stochastic Momentum (USM)
  7. Weighted Adaptive Learning Rate
  8. AdaUSM: Weighted AdaGrad with USM
  9. Convergence Results
  10. Relationships with Adam and RMSProp
  11. Adam
  12. RMSProp
  13. Experiments
  14. Conclusions
  15. Detailed Proof
  16. ...and 2 more sections

Key Result

Theorem 1

Let $\{x_t\}$ be a sequence generated by AdaUSM. Assume that the noisy gradient $g_{t}$ satisfies assumptions (A1)- (A2), and the weight $\{a_t\}$ is non-decreasing in $t$. Let $\tau$ be uniformly randomly drawn from $\{1, 2, \ldots T\}$. Then where with $C_1 = \frac{2(f(x_1) -f^*)}{(1 +\lambda\mu)\eta}$ and $C_2 =\frac{2\eta(1+2\lambda)^2 L d}{(1+\lambda\mu)(1-\mu)^3}+\frac{4\sigma d}{1-\mu}$,

Figures (1)

  • Figure 1: The first two and the last two figures illustrate the performance profiles of various optimizers for training LeNet on MNIST and training GoogLeNet on CIFAR10, respectively.

Theorems & Definitions (24)

  • Remark 1
  • Theorem 1
  • Remark 2
  • Corollary 1
  • Remark 3
  • Corollary 2
  • Remark 4
  • Remark 5
  • Lemma 1
  • proof
  • ...and 14 more