On Empirical Comparisons of Optimizers for Deep Learning

TL;DR

This paper argues that empirical comparisons of optimizers in deep learning are dominated by hyperparameter tuning protocols rather than intrinsic algorithmic differences. It formalizes an inclusion-based taxonomy showing that more general optimizers can replicate the behavior of their special cases, and demonstrates this across diverse image and language workloads with optimizer-specific search spaces. The key finding is that, when hyperparameters are thoroughly tuned, adaptive methods like Adam and NAdam never underperform Momentum/SGD, and the observed rankings align with the inclusion hierarchy. The work urges practitioners to thoroughly report tuning procedures and cautions against drawing conclusions from under-tuned comparisons, offering practical tips for fair benchmarking and optimizer selection.

Abstract

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

Figures (10)

• Figure 1: The relative performance of optimizers is consistent with the inclusion relationships, regardless of whether we compare final validation error (top) or test error (bottom). For all workloads, we tuned the hyperparameters of each optimizer separately, and selected the trial that achieved the lowest final validation error. Optimizers appear in the same order as the legend in all plots in this paper.
• Figure 2: The relative training speed of optimizers is consistent with the inclusion relationships. We measured (idealized) training speed as the number of training steps required to reach a target validation error (see Table \ref{['table:workloads']} for the error targets).
• Figure 3: Tuning more hyperparameters removes the differences in test error between optimizers observed by wilson2017marginal. Tuning a subset of optimizer hyperparameters and the initial learning rate is sufficient to equalize performance between all optimizers (left). More extensive hyperparameter tuning in our setup, including the learning rate schedule, improves results for all optimizers and still does not produce any differences between optimizer performances (right).
• Figure 4: Tuning more hyperparameters changes optimizer rankings from schneider2019deepobs to rankings that are consistent with the inclusion relationships. The leftmost columns for each workload reproduce the rankings from schneider2019deepobs, while the remaining columns tune over increasingly general search spaces. All columns use our random search tuning protocol.
• Figure 5: Example plot of final validation error projected onto the axes of the hyperparameter space. We consider this search space to be appropriate because the optimal values are away from the search space boundaries.

Theorems & Definitions (1)

• Definition 1: Inclusion relationship