Towards Understanding Adam Convergence on Highly Degenerate Polynomials

TL;DR

This work identifies a class of highly degenerate polynomials where Adam converges automatically without additional schedulers and derives theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrates strong alignment between theoretical bounds and experimental results.

Abstract

Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $β_2$ near 1 for convergence, this work investigates the "natural" auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.

Figures (8)

• Figure 1: Convergence behavior differs between strongly convex and degenerate polynomials. Adam uses $\beta_1=0.9, \beta_2=0.99$.
• Figure 2: (a) Training loss of Adam ($\beta_1=0.9, \beta_2=0.93$) for different $L(x)$. The text online shows the slope of the exponential fit. (b) Evolution of effective curvature $\lambda_t$. The red shaded region highlights instability where $\lambda_t$ exceeds the theoretical stability threshold $\frac{2}{\eta}\frac{1+\beta_1}{1-\beta_1}$cohen2023adaptivebai2025adaptive.
• Figure 3: (a) Theoretical convergence phase diagram for $k=4$, partitioned according to the stability conditions derived in Eq. \ref{['eq:stability_condition_1']} and Eq. \ref{['eq:stability_condition_2']}. (b) Experimental validation using Adam on $L(x)=\frac{1}{4}x^4$ with $x_0=1.0, \eta=0.001$. The heatmap displays the final training loss after $100,000$ steps. Additional results for $k=6$ can be found in Fig. \ref{['fig:convergence_phase_k_6']}. For a detailed discussion and a magnified view of the lower-left corner region, please refer to Appendix \ref{['app:zoomin']}.
• Figure 4: Bifurcation diagram of effective sharpness $u_t$ for $k=4$. The system Eq: \ref{['eq:sharpness_map']} exhibits three regimes: (I) Stable Fixed Point ($u_t \to u^*$), leading to exponential convergence of $x_t$ with a fixed rate; (II) Period-Doubling ($u_t$ oscillates within $(0, 2)$), preserving convergence albeit with oscillating rates; (III) Divergence ($u_t > 2$), where numerical instability occurs.
• Figure 5: Phase diagram for $k=4$ according to Thm. \ref{['thm:fixed_point_property']}.

Theorems & Definitions (13)

• Theorem 4.1: Local Stability and Convergence Rate of Adam
• proof
• Remark 4.2
• Theorem 5.1: Power-Law Convergence of Gradient Flow
• Remark 5.2: The Curse of Degeneracy
• Theorem 5.3: Power-Law Convergence of Momentum
• Lemma 5.4: Asymptotic Behavior of Second Moments
• proof
• Lemma 5.5: Linear Convergence via Exponential Learning Rate
• Proposition 5.6: Stability of the Acceleration Map