The Marginal Value of Adaptive Gradient Methods in Machine Learning

TL;DR

Adaptive gradient methods can lead to qualitatively different minimizers than non-adaptive methods on overparameterized problems. The paper provides a convex construction where AdaGrad/Adam/RMSProp converge to a poor generalizing solution, while SGD achieves perfect generalization. It then corroborates these findings with extensive deep learning experiments across vision, language, and parsing tasks, showing adaptive methods generalize worse and often plateau early. The authors also discuss tuning strategies that mitigate these issues and argue for careful reconsideration of adaptive methods in practical neural network training.

Abstract

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

Figures (8)

• Figure 1: CIFAR-10 (Train)
• Figure 2: CIFAR-10 (Test)
• Figure 4: War and Peace (Training Set)
• Figure 5: War and Peace (Test Set)
• Figure 6: Discriminative Parsing (Training Set)

Theorems & Definitions (1)

• Lemma 3.1