Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Stability and Step Size Robustness of RMSProp

Naum Dimitrieski, Maria Christine Honecker, Carsten Scherer, Christian Ebenbauer

Abstract

In this paper, an input-to-state Lyapunov function for the RMSProp optimization algorithm is introduced. Global asymptotic stability of the RMSProp algorithm for constant step sizes and robustness properties with respect to arbitrary bounded time-varying step size rules are established.

Global Stability and Step Size Robustness of RMSProp

Abstract

In this paper, an input-to-state Lyapunov function for the RMSProp optimization algorithm is introduced. Global asymptotic stability of the RMSProp algorithm for constant step sizes and robustness properties with respect to arbitrary bounded time-varying step size rules are established.
Paper Structure (4 sections, 2 theorems, 147 equations)

This paper contains 4 sections, 2 theorems, 147 equations.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Main Results
  4. Conclusions and Outlook

Key Result

Theorem 1

Consider algorithm sys:rmsprop and let Assumption assump be satisfied with ${\mu > 0}$, ${x^{*}\in \mathbb{R}^d}$ and ${L > 0}$. Let ${\beta \in (0,1)}$, ${\varepsilon > 0}$, and ${\eta(u)\coloneqq \eta_0 + u}$, where ${\eta_0 \in \left(0, \tfrac{2\varepsilon}{L}\right)}$ and ${u(\cdot) \in \math

Theorems & Definitions (8)

  • Theorem 1
  • lemma 1
  • proof
  • proof
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4