A Simple Convergence Proof of Adam and Adagrad

TL;DR

This work provides a simple, unified convergence proof for Adagrad and Adam when optimizing smooth, possibly non-convex objectives with bounded gradients. By analyzing per-coordinate updates and a momentum-augmented scheme, it derives explicit convergence bounds and shows that, with suitable hyper-parameters, Adam attains the same $O(d\ln(N)/\sqrt{N})$ rate as Adagrad, effectively making Adam a finite-horizon twin of Adagrad. A key theoretical advance is improving the dependence on the heavy-ball parameter $\beta_1$ from $O((1-\beta_1)^{-3})$ (or worse) to $O((1-\beta_1)^{-1})$, aligning theory more closely with observed practical momentum benefits. The paper also clarifies when Adam fails to converge under default settings and demonstrates, via toy and CIFAR-10 experiments, the relative impact of the Adam corrective terms on training dynamics. Overall, the results provide a clearer understanding of when adaptive methods converge and how momentum influences their behavior in non-convex stochastic optimization.

Abstract

We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients. We show that in expectation, the squared norm of the objective gradient averaged over the trajectory has an upper-bound which is explicit in the constants of the problem, parameters of the optimizer, the dimension $d$, and the total number of iterations $N$. This bound can be made arbitrarily small, and with the right hyper-parameters, Adam can be shown to converge with the same rate of convergence $O(d\ln(N)/\sqrt{N})$. When used with the default parameters, Adam doesn't converge, however, and just like constant step-size SGD, it moves away from the initialization point faster than Adagrad, which might explain its practical success. Finally, we obtain the tightest dependency on the heavy ball momentum decay rate $β_1$ among all previous convergence bounds for non-convex Adam and Adagrad, improving from $O((1-β_1)^{-3})$ to $O((1-β_1)^{-1})$.

Figures (2)

• Figure 1: Observed average squared norm of the objective gradients after a fixed number of iterations when varying a single parameter out of $\alpha$, $1 - \beta_1$ and $1 - \beta_2$, on a toy task (left, $10^6$ iterations) and on CIFAR-10 (right, 600 epochs with a batch size 128). All curves are averaged over 3 runs, error bars are negligible except for small values of $\alpha$ on CIFAR-10. See Section \ref{['sec:experiments']} for details.
• Figure 2: Training trajectories for varying values of $\alpha\in\{10^{-4}, 10^{-3}\}$, $\beta_1 \in \{0., 0.5, 0.8, 0.9, 0.99\}$ and $\beta_2\in\{0.9, 0.99, 0.999, 0.9999\}$. The top row (resp. bottom) gives the training loss (resp. squared norm of the expected gradient). The left column uses all corrective terms in the original Adam algorithm, the middle column drops the corrective term on $m_n$ (equivalent to our proof setup), and the right column drops the corrective term on $v_n$. We notice a limited impact when dropping the corrective term on $m_n$, but dropping the corrective term on $v_n$ has a much stronger impact.

Theorems & Definitions (26)

• Theorem 1: Convergence of Adagrad without momentum
• Theorem 2: Convergence of Adam without momentum
• Theorem 3: Convergence of Adagrad with momentum
• Theorem 4: Convergence of Adam with momentum
• Lemma 5.1: adaptive update approximately follow a descent direction
• proof
• Lemma 5.2: sum of ratios with the denominator being the sum of past numerators
• proof
• Theorem A.1: Convergence of Adagrad with momentum
• Theorem A.1: Convergence of Adam with momentum