Adam with model exponential moving average is effective for nonconvex optimization

TL;DR

This work demonstrates that a clipped version of Adam with model EMA achieves the optimal convergence rates in various nonconvex optimization settings, both smooth and nonsmooth.

Abstract

In this work, we offer a theoretical analysis of two modern optimization techniques for training large and complex models: (i) adaptive optimization algorithms, such as Adam, and (ii) the model exponential moving average (EMA). Specifically, we demonstrate that a clipped version of Adam with model EMA achieves the optimal convergence rates in various nonconvex optimization settings, both smooth and nonsmooth. Moreover, when the scale varies significantly across different coordinates, we demonstrate that the coordinate-wise adaptivity of Adam is provably advantageous. Notably, unlike previous analyses of Adam, our analysis crucially relies on its core elements -- momentum and discounting factors -- as well as model EMA, motivating their wide applications in practice.

Theorems & Definitions (19)

• Theorem 1: Informal
• definition 3: $(\lambda,\varepsilon)$-stationary point
• Lemma 4
• Lemma 5
• definition 6: Discounted regret
• Lemma 7: Discounted-to-nonconvex conversion
• Lemma 8: Gradient-adaptive regret bound
• Theorem 9: Discounted regret bound
• Lemma 10: Variance bound
• proof