Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

Steffen Dereich, Thang Do, Arnulf Jentzen

Abstract

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.

Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

Abstract

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.
Paper Structure (13 sections, 18 theorems, 691 equations)

This paper contains 13 sections, 18 theorems, 691 equations.

Table of Contents

  1. Introduction
  2. Main result: Unconditional error analysis for Adam
  3. Convergence rates for concrete example stochastic optimization problems
  4. Literature review
  5. Structure of this article
  6. A priori bounds for strongly convex stochastic optimization problems (SOPs)
  7. Quantitative a priori bounds for Adam for general strongly convex SOPs
  8. Non-uniform a priori bounds for Adam for quadratic example SOPs
  9. Qualitative a priori bounds for Adam for general strongly convex SOPs
  10. Uniform a priori bounds for Adam for quadratic example SOPs
  11. Convergence rates for strongly convex SOPs
  12. Convergence rates for Adam without assuming L-smoothness
  13. Convergence rates for Adam with assuming L-smoothness

Key Result

Theorem 1.1

Let $( \Omega, \mathcal{F},\mathbb{P} )$ be a probability space, let $d, \mathscr{d} \in \mathbb{N}$, $\varepsilon,p \in (0,\infty)$, $q,r\in (0,1/2)$, $\xi\in \mathbb{R}^d$, let $(\gamma_n)_{n\in \mathbb{N}}\subseteq (0,\infty)$ be non-increasing, let $U\subseteq \mathbb{R}^\mathscr{d}$ be compact assume $\limsup_{n\to\infty}( ( \gamma_n )^{ - 2 } ( \gamma_n- \gamma_{ n + 1 } ) +\sum_{ m = n }^{

Theorems & Definitions (35)

  • Theorem 1.1
  • Lemma 2.1
  • proof : Proof of \ref{['V estimate']}
  • Lemma 2.2
  • proof : Proof of \ref{['lem: bounded increament Adam']}
  • Proposition 2.3
  • proof : Proof of \ref{['lem: priori bound stochastic Adam non explosion']}
  • Lemma 2.4
  • proof : Proof of \ref{['lem: verify 1']}
  • Corollary 2.5
  • ...and 25 more