Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MADA: Meta-Adaptive Optimizers through hyper-gradient Descent

Kaan Ozkara, Can Karakus, Parameswaran Raman, Mingyi Hong, Shoham Sabach, Branislav Kveton, Volkan Cevher

TL;DR

This work introduces Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training, and proposes AVGrad, a modification of AMSGrad that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization.

Abstract

Following the introduction of Adam, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Adam uniformly across all tasks. In this work, we introduce Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training. The key idea in MADA is to parameterize the space of optimizers and dynamically search through it using hyper-gradient descent during training. We empirically compare MADA to other popular optimizers on vision and language tasks, and find that MADA consistently outperforms Adam and other popular optimizers, and is robust against sub-optimally tuned hyper-parameters. MADA achieves a greater validation performance improvement over Adam compared to other popular optimizers during GPT-2 training and fine-tuning. We also propose AVGrad, a modification of AMSGrad that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization. Finally, we provide a convergence analysis to show that parameterized interpolations of optimizers can improve their error bounds (up to constants), hinting at an advantage for meta-optimizers.

MADA: Meta-Adaptive Optimizers through hyper-gradient Descent

TL;DR

This work introduces Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training, and proposes AVGrad, a modification of AMSGrad that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization.

Abstract

Following the introduction of Adam, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Adam uniformly across all tasks. In this work, we introduce Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training. The key idea in MADA is to parameterize the space of optimizers and dynamically search through it using hyper-gradient descent during training. We empirically compare MADA to other popular optimizers on vision and language tasks, and find that MADA consistently outperforms Adam and other popular optimizers, and is robust against sub-optimally tuned hyper-parameters. MADA achieves a greater validation performance improvement over Adam compared to other popular optimizers during GPT-2 training and fine-tuning. We also propose AVGrad, a modification of AMSGrad that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization. Finally, we provide a convergence analysis to show that parameterized interpolations of optimizers can improve their error bounds (up to constants), hinting at an advantage for meta-optimizers.
Paper Structure (21 sections, 8 theorems, 86 equations, 9 figures, 7 tables, 1 algorithm)

This paper contains 21 sections, 8 theorems, 86 equations, 9 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Parameterized Optimizers
  3. Meta-Adaptive Optimizers
  4. On Convergence of Interpolated Optimizers
  5. AVGrad and its interpolation with Adam
  6. Convergence of the interpolated optimizer
  7. Experiments
  8. A concrete parameterized optimizer
  9. Experimental setting
  10. Results
  11. Conclusion
  12. Setup, Details, and Proof for Theorem \ref{['thm3']}
  13. Proof of Theorem \ref{['thm3']}
  14. Convergence of AVGrad
  15. Proof of Theorem \ref{['thm1']}
  16. ...and 6 more sections

Key Result

Proposition 1

The following inequality is a sufficient condition for non-increasing effective learning rate:

Figures (9)

  • Figure 1: Value of the bound in Theorem \ref{['thm3']} for a representative case where $\beta_2=0.9, T= 10,000$.
  • Figure 2: Validation loss of MADA on OpenWebText vs other adaptive optimizer baselines.
  • Figure 3: Validation losses of competing methods on OpenWebText for GPT-2 (125M) model using the same random seed.
  • Figure 4: Training losses after 2 epochs of fine-tuning on Shakespeare dataset.
  • Figure 5: Parameter evolution for $\beta_{1,0}=0.9,\beta_{2,0}=0.95,\beta_{3,0}=0$.
  • ...and 4 more figures

Theorems & Definitions (12)

  • Proposition 1
  • Theorem 1: Convergence of interpolation of AVGrad and Adam without momentum
  • Lemma 1: Updates approximately follow a descent direction
  • proof
  • Lemma 2
  • proof
  • Theorem 2: Convergence of AVGrad without momentum
  • Theorem 3: Convergence of AVGrad with momentum
  • Lemma 3
  • Lemma 4
  • ...and 2 more