Disentangling Adaptive Gradient Methods from Learning Rates

TL;DR

The paper tackles the challenge of accurately evaluating optimization algorithms for deep learning by disentangling adaptive preconditioning from the learning-rate schedule. It introduces AdaGraft, a learning-rate grafting framework that decouples update magnitude from direction, revealing that implicit step-size schedules largely drive training dynamics. Through large-scale experiments on ImageNet and WMT14, the authors show that many purported advantages of adaptive methods hinge on how the learning rate is scheduled, and they demonstrate how to bootstrap improved schedules for certain optimizers. The work also provides a critical examination of prior evidence, discusses replication challenges, and offers practical guidance for more robust optimizer evaluation and future research into adaptive methods and their role in NLP and computer vision.

Abstract

We investigate several confounding factors in the evaluation of optimization algorithms for deep learning. Primarily, we take a deeper look at how adaptive gradient methods interact with the learning rate schedule, a notoriously difficult-to-tune hyperparameter which has dramatic effects on the convergence and generalization of neural network training. We introduce a "grafting" experiment which decouples an update's magnitude from its direction, finding that many existing beliefs in the literature may have arisen from insufficient isolation of the implicit schedule of step sizes. Alongside this contribution, we present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods, aimed at bringing more clarity to this space.

Figures (14)

• Figure 1: Training curves for grafted optimizers on ResNet-50 for ImageNet classification. To first order, the shape of the training curve is determined by the step size schedule; most dramatically, AdaGrad's poor convergence appears to be entirely due to its implicit step size schedule.
• Figure 2: Training curves for grafted optimizers on Transformer for WMT14 English-French translation. Convergence and solution quality are very sensitive to implicit step size schedule.
• Figure 3: Global grafting experiments to fix AdaGrad. Bootstrapping with a well-tuned $\mathcal{M}$=SGD, we find an offline correction for AdaGrad's implicit step size schedule; corrected AdaGrad competes with SGD or Adam. Grafting is denoted by $\mathcal{M}\#\mathcal{D}$.
• Figure 4: Empirical discovery of linear learning rate correction from Figure \ref{['fig:imagenet-transfer']}. Norms of proposed steps are measured in a global (SGD, AdaGrad) grafting experiment. The ratio is found empirically to scale linearly with $t$; this adjustment can be applied directly to AdaGrad, without any grafting.
• Figure 5: Training curves for global grafting and learning rate schedule correction, as in Figure \ref{['fig:imagenet-transfer']}, for the Transformer/WMT setup, layer-wise grafting with $\mathcal{M}$=AdaGrad, $\mathcal{D}$=SGD. Unlike the analogous experiment for ImageNet, we could not get corrected SGD to perform as well as AdaGrad.

Theorems & Definitions (3)

• Proposition 1: wilson2017marginal; informal
• Proposition 2: Counter-counterexample; informal
• Theorem 3