Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration

Soham De, Anirbit Mukherjee, Enayat Ullah

TL;DR

This work advances the understanding of adaptive gradient methods by providing the first convergence guarantees for RMSProp and ADAM in non-convex optimization, with explicit runtime bounds. It couples theory with a comprehensive empirical comparison against Nesterov acceleration across controlled autoencoder and CNN experiments, highlighting the pivotal roles of the shift parameter $\xi$ and the momentum parameter $\beta_1$ (notably values near 1) for generalization. The findings reveal that high $\beta_1$ can enable ADAM to match or surpass carefully tuned NAG on large networks, while NAG more effectively reduces gradient norms and progresses toward favorable Hessian regions. Overall, the results emphasize nuanced interactions between adaptivity and momentum, and they point to future work needed to extend theory to the high-$\beta_1$ regime.

Abstract

RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the running time. Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov's Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter $β_1$. We show that at very high values of the momentum parameter ($β_1 = 0.99$) ADAM outperforms a carefully tuned NAG on most of our experiments, in terms of getting lower training and test losses. On the other hand, NAG can sometimes do better when ADAM's $β_1$ is set to the most commonly used value: $β_1 = 0.9$, indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance. We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms, and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates.

Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration

TL;DR

This work advances the understanding of adaptive gradient methods by providing the first convergence guarantees for RMSProp and ADAM in non-convex optimization, with explicit runtime bounds. It couples theory with a comprehensive empirical comparison against Nesterov acceleration across controlled autoencoder and CNN experiments, highlighting the pivotal roles of the shift parameter and the momentum parameter (notably values near 1) for generalization. The findings reveal that high can enable ADAM to match or surpass carefully tuned NAG on large networks, while NAG more effectively reduces gradient norms and progresses toward favorable Hessian regions. Overall, the results emphasize nuanced interactions between adaptivity and momentum, and they point to future work needed to extend theory to the high- regime.

Abstract

RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the running time. Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov's Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter . We show that at very high values of the momentum parameter () ADAM outperforms a carefully tuned NAG on most of our experiments, in terms of getting lower training and test losses. On the other hand, NAG can sometimes do better when ADAM's is set to the most commonly used value: , indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance. We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms, and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates.

Paper Structure

This paper contains 36 sections, 5 theorems, 60 equations, 12 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. A summary of our contributions
  3. Notations and Pseudocodes
  4. Convergence Guarantees for ADAM and RMSProp
  5. Experimental setup
  6. Full-batch experiments
  7. Mini-batch experiments
  8. Experimental Results
  9. RMSProp and ADAM are sensitive to choice of $\xi$
  10. Tracking $\lambda_{min}(\text{Hessian})$ of the loss function
  11. Comparing performance in the full-batch setting
  12. Conclusions from the full-batch experiments of training autoencoders on mini-MNIST:
  13. Corroborating the full-batch behaviors in the mini-batch setting
  14. Conclusions from the mini-batch experiments of training autoencoders on full MNIST dataset:
  15. Conclusion
  16. ...and 21 more sections

Key Result

Theorem 3.1

Stochastic RMSProp converges to criticality (Proof in subsection sec:supp_rmspropS) Let $f : \mathbb R ^d \rightarrow \mathbb R$ be $L-$smooth and be of the form $f = \frac{1}{k} \sum_{p=1}^k f_p$ s.t. (a) each $f_i$ is at least once differentiable, $(b)$ the gradients are s.t $\forall {\hbox{\boldm

Figures (12)

  • Figure 1: Optimally tuned parameters for different $\xi$ values. 1 hidden layer network of 1000 nodes; Left: Loss on training set; Middle: Loss on test set; Right: Gradient norm on training set
  • Figure 2: Tracking the smallest eigenvalue of the Hessian on a 1 hidden layer network of size 300. Left: Minimum Hessian eigenvalue. Right: Gradient norm on training set.
  • Figure 3: Full-batch experiments on a 3 hidden layer network with 1000 nodes in each layer; Left: Loss on training set; Middle: Loss on test set; Right: Gradient norm on training set
  • Figure 4: Mini-batch experiments on a network with 5 hidden layers of 1000 nodes each; Left: Loss on training set; Middle: Loss on test set; Right: Gradient norm on training set
  • Figure 5: Fixed parameters with changing $\xi$ values. 1 hidden layer network of 1000 nodes
  • ...and 7 more figures

Theorems & Definitions (13)

  • Definition 1
  • Definition 2
  • Definition 3
  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • proof
  • Lemma A.1
  • proof
  • ...and 3 more