Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold

Xinghan Li, Haodong Wen, Kaifeng Lyu

TL;DR

The paper investigates Adam's implicit bias beyond SGD by developing a slow SDE framework that tracks adaptive preconditioning near the minimizer manifold $\Gamma$ over long time horizons. It proves that Adam implicitly minimizes a unique sharpness measure $\mathrm{tr}(\mathrm{Diag}(\mathbf{H})^{1/2})$, distinct from SGD's Hessian-trace goal, and shows this bias can improve sparse ground-truth recovery in diagonal-net settings but may hurt generalization in matrix-factorization tasks with label noise. The authors extend the slow SDE approach to a broad class of adaptive gradient methods (AGMs) and provide high-probability convergence guarantees, a principled interpretation of the preconditioner’s role, and practical variants like AdamE to tune the implicit bias. Empirical results on diagonal nets and matrix factorization illustrate the distinct trajectories and generalization outcomes under label noise, while theoretical results suggest careful optimizer design is needed to leverage adaptive sharpness reduction for generalization benefits. Overall, the work offers a unified perspective on how adaptive optimizers reduce sharpness and opens avenues for designing AGMs with targeted implicit regularizers.

Abstract

Despite the popularity of the Adam optimizer in practice, most theoretical analyses study Stochastic Gradient Descent (SGD) as a proxy for Adam, and little is known about how the solutions found by Adam differ. In this paper, we show that Adam implicitly reduces a unique form of sharpness measure shaped by its adaptive updates, leading to qualitatively different solutions from SGD. More specifically, when the training loss is small, Adam wanders around the manifold of minimizers and takes semi-gradients to minimize this sharpness measure in an adaptive manner, a behavior we rigorously characterize through a continuous-time approximation using stochastic differential equations. We further demonstrate how this behavior differs from that of SGD in a well-studied setting: when training overparameterized models with label noise, SGD has been shown to minimize the trace of the Hessian matrix, $\tr(\mH)$, whereas we prove that Adam minimizes $\tr(\Diag(\mH)^{1/2})$ instead. In solving sparse linear regression with diagonal linear networks, this distinction enables Adam to achieve better sparsity and generalization than SGD. Finally, our analysis framework extends beyond Adam to a broad class of adaptive gradient methods, including RMSProp, Adam-mini, Adalayer and Shampoo, and provides a unified perspective on how these adaptive optimizers reduce sharpness, which we hope will offer insights for future optimizer design.

Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold

TL;DR

The paper investigates Adam's implicit bias beyond SGD by developing a slow SDE framework that tracks adaptive preconditioning near the minimizer manifold over long time horizons. It proves that Adam implicitly minimizes a unique sharpness measure , distinct from SGD's Hessian-trace goal, and shows this bias can improve sparse ground-truth recovery in diagonal-net settings but may hurt generalization in matrix-factorization tasks with label noise. The authors extend the slow SDE approach to a broad class of adaptive gradient methods (AGMs) and provide high-probability convergence guarantees, a principled interpretation of the preconditioner’s role, and practical variants like AdamE to tune the implicit bias. Empirical results on diagonal nets and matrix factorization illustrate the distinct trajectories and generalization outcomes under label noise, while theoretical results suggest careful optimizer design is needed to leverage adaptive sharpness reduction for generalization benefits. Overall, the work offers a unified perspective on how adaptive optimizers reduce sharpness and opens avenues for designing AGMs with targeted implicit regularizers.

Abstract

Despite the popularity of the Adam optimizer in practice, most theoretical analyses study Stochastic Gradient Descent (SGD) as a proxy for Adam, and little is known about how the solutions found by Adam differ. In this paper, we show that Adam implicitly reduces a unique form of sharpness measure shaped by its adaptive updates, leading to qualitatively different solutions from SGD. More specifically, when the training loss is small, Adam wanders around the manifold of minimizers and takes semi-gradients to minimize this sharpness measure in an adaptive manner, a behavior we rigorously characterize through a continuous-time approximation using stochastic differential equations. We further demonstrate how this behavior differs from that of SGD in a well-studied setting: when training overparameterized models with label noise, SGD has been shown to minimize the trace of the Hessian matrix, , whereas we prove that Adam minimizes instead. In solving sparse linear regression with diagonal linear networks, this distinction enables Adam to achieve better sparsity and generalization than SGD. Finally, our analysis framework extends beyond Adam to a broad class of adaptive gradient methods, including RMSProp, Adam-mini, Adalayer and Shampoo, and provides a unified perspective on how these adaptive optimizers reduce sharpness, which we hope will offer insights for future optimizer design.

Paper Structure

This paper contains 73 sections, 45 theorems, 259 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contributions.
  3. Related Work
  4. Implicit Bias of SGD.
  5. Implicit Bias of Adam.
  6. Approximation of Stochastic First-Order Optimizers with SDE.
  7. Adaptive Gradient Methods.
  8. Convergence of Adam.
  9. Preliminaries
  10. Notations.
  11. Derivatives.
  12. Loss Functions.
  13. Smoothness Assumptions.
  14. SGD and Adam.
  15. SDE First-Order Approximation For SGD.
  16. ...and 58 more sections

Key Result

Theorem 4.1

Let $T>0$ and suppose amp:basic--amp:beta1-bound hold. There exist constant $\epsilon_0,C>0$ such that the following statement holds for all sufficiently small learning rates $\eta$. Define step $K_0 := \lfloor C\log(1/\eta)\rfloor$ and $K := \lfloor T\,\eta^{-2}\rfloor$. Run an AGM for $K_0+K$ iter For any $k \in \{0, 1, \dots, K\}$, let $\bar{{\bm{X}}}_{k} := \left(\Phi_{{\bm{S}}({\bm{v}}_{K_0 +

Figures (5)

  • Figure 1: (a): Coutour of the elliptical loss, from which we can see the two tips as the flattest minima. (b): SGD implicitly minimizes $\text{tr}({\bm{H}})$ and converges to the flattest minima. (c): Adam reduces sharpness too but converges to a different and sparser minimizer.
  • Figure 2: Final test loss as a function of the training data size with $d=10000$, $\kappa=50$. Each plotted point is the final test loss after both the training and test losses have converged; its x-coordinate is the training data size and the curve denotes the optimizer and configuration. (a) Loss comparison between SGD with different learning rates, and Adam with different learning rates and $\beta_2$ values. (b) Loss comparison between AdamE with $\lambda=0.01, 0.1,0.25,0.75,0.9$, Adam, and SGD.
  • Figure 3: Deep matrix factorization with label noise with deepth $L =2$. Adam and SGD are trained on identical data and noise realizations. Top: evolution of $\text{tr}({\bm{H}})$ and $\text{tr}(\mathrm{Diag}({\bm{H}})^{1/2})$. Bottom: training and test MSE. Adam converges to a point with larger $\text{tr}({\bm{H}})$ but smaller $\text{tr}(\mathrm{Diag}({\bm{H}})^{1/2})$, and exhibits higher test error.
  • Figure 4: Deep matrix factorization with label noise with deepth $L =5$.
  • Figure 5: Comparison of conventional SDE and slow SDE.

Theorems & Definitions (106)

  • Definition 3.1: Slow SDE for SGD
  • Remark 4.1
  • Definition 4.1: Preconditioner Flow Projection
  • Definition 4.2
  • Definition 4.3: Slow SDE for AGMs
  • Theorem 4.1
  • Theorem 4.2: Convergence Bound of AGMs, Stated Informally
  • Theorem 5.1: Slow ODE for AGMs with Label Noise
  • Lemma 5.1: Adam's Implicit Bias under Label Noise
  • Lemma 5.2: AdamE's Implicit Bias with Label Noise
  • ...and 96 more