Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of ADAM for Lipschitz Objective Functions

Juan Ferrera, Javier Gómez Gil

Abstract

The aim of this paper is to prove the exponential convergence, local and global, of Adam algorithm under precise conditions on the parameters, when the objective function lacks differentiability. More precisely, we require Lipschitz continuity, and control on the gradient whenever it exists. We provide also examples of interesting functions that satisfies the required restrictions.

Convergence of ADAM for Lipschitz Objective Functions

Abstract

The aim of this paper is to prove the exponential convergence, local and global, of Adam algorithm under precise conditions on the parameters, when the objective function lacks differentiability. More precisely, we require Lipschitz continuity, and control on the gradient whenever it exists. We provide also examples of interesting functions that satisfies the required restrictions.
Paper Structure (4 sections, 16 theorems, 99 equations)

This paper contains 4 sections, 16 theorems, 99 equations.

Table of Contents

  1. Introduction
  2. Local Convergence
  3. Approximation to Convergence Basin
  4. Global convergence

Key Result

Theorem 1.4

If $\zeta_{w^*}=0$ then $x^*=(0,0,w^*)$ is a fixed point of $\Gamma$, and $\Omega (n,x^*)=0$ for every $n\in \mathbb{N}$, and consequently $\Theta(n,x^*)=x^*$ for every $n\in \mathbb{N}$.

Theorems & Definitions (42)

  • Definition 1.1
  • Definition 1.2
  • Remark 1.3
  • Theorem 1.4
  • Theorem 2.1
  • Lemma 2.2
  • proof
  • Lemma 2.3
  • proof
  • Proposition 2.4
  • ...and 32 more