Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case

Meixuan He, Yuqing Liang, Jinlan Liu, Dongpo Xu

TL;DR

This work analyzes Adam in non-convex stochastic optimization by introducing precise ergodic and non-ergodic convergence notions and showing non-ergodic results better reflect practical outcomes. It presents a relaxed sufficient condition, SC-Adam, that broadens permissible hyperparameters beyond prior criteria, and proves an almost-sure ergodic rate arbitrarily close to $o\left(1/\sqrt{K}\right)$. Crucially, it establishes the first last-iterate convergence of Adam to a stationary point for non-convex objectives and, under the Polyak-Lojasiewicz condition, a non-ergodic rate of $O(1/K)$ for function values. Together, these results provide a solid theoretical foundation for applying Adam to non-convex stochastic problems and offer actionable guidance on hyperparameter choices.

Abstract

Adam is a commonly used stochastic optimization algorithm in machine learning. However, its convergence is still not fully understood, especially in the non-convex setting. This paper focuses on exploring hyperparameter settings for the convergence of vanilla Adam and tackling the challenges of non-ergodic convergence related to practical application. The primary contributions are summarized as follows: firstly, we introduce precise definitions of ergodic and non-ergodic convergence, which cover nearly all forms of convergence for stochastic optimization algorithms. Meanwhile, we emphasize the superiority of non-ergodic convergence over ergodic convergence. Secondly, we establish a weaker sufficient condition for the ergodic convergence guarantee of Adam, allowing a more relaxed choice of hyperparameters. On this basis, we achieve the almost sure ergodic convergence rate of Adam, which is arbitrarily close to $o(1/\sqrt{K})$. More importantly, we prove, for the first time, that the last iterate of Adam converges to a stationary point for non-convex objectives. Finally, we obtain the non-ergodic convergence rate of $O(1/K)$ for function values under the Polyak-Lojasiewicz (PL) condition. These findings build a solid theoretical foundation for Adam to solve non-convex stochastic optimization problems.

Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case

TL;DR

This work analyzes Adam in non-convex stochastic optimization by introducing precise ergodic and non-ergodic convergence notions and showing non-ergodic results better reflect practical outcomes. It presents a relaxed sufficient condition, SC-Adam, that broadens permissible hyperparameters beyond prior criteria, and proves an almost-sure ergodic rate arbitrarily close to . Crucially, it establishes the first last-iterate convergence of Adam to a stationary point for non-convex objectives and, under the Polyak-Lojasiewicz condition, a non-ergodic rate of for function values. Together, these results provide a solid theoretical foundation for applying Adam to non-convex stochastic problems and offer actionable guidance on hyperparameter choices.

Abstract

Adam is a commonly used stochastic optimization algorithm in machine learning. However, its convergence is still not fully understood, especially in the non-convex setting. This paper focuses on exploring hyperparameter settings for the convergence of vanilla Adam and tackling the challenges of non-ergodic convergence related to practical application. The primary contributions are summarized as follows: firstly, we introduce precise definitions of ergodic and non-ergodic convergence, which cover nearly all forms of convergence for stochastic optimization algorithms. Meanwhile, we emphasize the superiority of non-ergodic convergence over ergodic convergence. Secondly, we establish a weaker sufficient condition for the ergodic convergence guarantee of Adam, allowing a more relaxed choice of hyperparameters. On this basis, we achieve the almost sure ergodic convergence rate of Adam, which is arbitrarily close to . More importantly, we prove, for the first time, that the last iterate of Adam converges to a stationary point for non-convex objectives. Finally, we obtain the non-ergodic convergence rate of for function values under the Polyak-Lojasiewicz (PL) condition. These findings build a solid theoretical foundation for Adam to solve non-convex stochastic optimization problems.
Paper Structure (30 sections, 21 theorems, 110 equations, 1 table, 1 algorithm)

This paper contains 30 sections, 21 theorems, 110 equations, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations
  4. Definitions
  5. Assumptions
  6. Adam Algorithm
  7. Ergodic Convergence of Adam
  8. Minimum Convergence
  9. Uniform Convergence and Sufficient Condition
  10. Almost Sure Convergence
  11. Non-ergodic Convergence of Adam
  12. Non-ergodic Convergence for Gradient Sequences
  13. Non-ergodic Convergence for Function Values
  14. Conclusion
  15. Technical Lemmas
  16. ...and 15 more sections

Key Result

Proposition 3

Suppose that $(x_{n})_{n\geq 1}$ is a sequence of real numbers and $(\omega_{n,k})_{n\geq 1, 1\leq k\leq n}$ is a double array of real numbers such that $\lim_{n\to\infty}\omega_{n,k}=0$ for any $1\leq k\leq n$. Then if $\lim_{n\to\infty}x_{n}=x$, we have where $\bar{x}_{n}$ is the ergodic sequence of $x_{k}$ with respect to $\omega_{n,k}$, that is, $\bar{x}_{n}:=\sum_{k=1}^{n}\omega_{n,k}x_{k}$.

Theorems & Definitions (27)

  • Definition 1: Ergodic convergence
  • Definition 2: Non-ergodic convergence
  • Example 1
  • Proposition 3
  • Example 2
  • Example 3
  • Theorem 4: Minimum convergence
  • Corollary 5
  • Corollary 6
  • Theorem 7: Uniform convergence
  • ...and 17 more