Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, Quanquan Gu

TL;DR

The paper addresses the generalization gap observed when using adaptive gradient methods (e.g., Adam, Amsgrad) versus SGD with momentum in training deep networks. It introduces Padam, a partially adaptive momentum method with a parameter $p \in [0,1/2]$ that blends Adam/Amsgrad with SGD-Momentum, recovering SGD-Momentum at $p=0$ and Amsgrad at $p=1/2$, thereby mitigating the small learning rate dilemma. The authors provide a convergence guarantee for Padam in stochastic nonconvex optimization and derive a rate bound that yields $\mathbb{E}[\|\nabla f(\mathbf{x}_{\text{out}})\|_2^2] = O\left(\frac{d^{1/2}}{T^{3/4 - s/2}} + \frac{d}{T}\right)$ under suitable conditions, with $s$ capturing gradient sparsity. Empirically, Padam demonstrates fast convergence comparable to Adam while achieving generalization comparable to SGD-Momentum across CNNs and language models, outperforming several modern adaptive methods and suggesting practitioners can safely employ adaptive methods for faster training without sacrificing generalization.

Abstract

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter $p$, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain a fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

TL;DR

The paper addresses the generalization gap observed when using adaptive gradient methods (e.g., Adam, Amsgrad) versus SGD with momentum in training deep networks. It introduces Padam, a partially adaptive momentum method with a parameter that blends Adam/Amsgrad with SGD-Momentum, recovering SGD-Momentum at and Amsgrad at , thereby mitigating the small learning rate dilemma. The authors provide a convergence guarantee for Padam in stochastic nonconvex optimization and derive a rate bound that yields under suitable conditions, with capturing gradient sparsity. Empirically, Padam demonstrates fast convergence comparable to Adam while achieving generalization comparable to SGD-Momentum across CNNs and language models, outperforming several modern adaptive methods and suggesting practitioners can safely employ adaptive methods for faster training without sacrificing generalization.

Abstract

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter , to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain a fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

Paper Structure

This paper contains 22 sections, 7 theorems, 53 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Additional Related Work
  3. Notation:
  4. Review of Adaptive Gradient Methods
  5. The Proposed Algorithm
  6. Convergence Analysis of the Proposed Algorithm
  7. Experiments
  8. Experimental Results
  9. Conclusions and Future Work
  10. Proof of the Main Theory
  11. Proof of Theorem \ref{['eq:Gu0000']}
  12. Proof of Corollary \ref{['cl:1']}
  13. Proof of Technical Lemmas
  14. Proof of Lemma \ref{['lm:3']}
  15. Proof of Lemma \ref{['lm:4']}
  16. ...and 7 more sections

Key Result

Theorem 4.3

In Algorithm alg:Padam, suppose that $p\in[0,1/2]$, $\beta_1 < \beta_2^{2p}$, $\alpha_t = \alpha$ and $\|\mathbf{g}_{1:T,i}\|_2 \leq G_{\infty}T^s$ for $t=1,\ldots,T$, $0 \leq s \leq 1/2$, under Assumptions as:1 and as:2, let $\Delta f=f(\mathbf{x}_1) - \inf_{\mathbf{x}} f(\mathbf{x})$, for any $q\ where

Figures (4)

  • Figure 1: Performance comparison of Padam with different choices of $p$ for training ResNet on (a) CIFAR-10 and (b) CIFAR-100 datasets.
  • Figure 2: Train loss and test error (top-$1$) on the CIFAR-10 dataset.
  • Figure 3: Test error on the ImageNet dataset (left and middle columns), and test perplexity on the Penn Treebank dataset (right column).
  • Figure 4: Train loss and test error (top-$1$ error) of three CNN architectures on CIFAR-100. In all cases, Padam achieves the fastest training procedure among all methods and generalizes as well as SGD with momentum.

Theorems & Definitions (9)

  • Theorem 4.3
  • Remark 4.4
  • Corollary 4.5
  • Remark 4.6
  • Lemma A.1
  • Lemma A.2
  • Lemma A.3
  • Lemma A.4
  • Lemma A.5