Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms

Eric Lu

TL;DR

The paper addresses the dynamics of momentum-based stochastic gradient descent with time-varying learning rates by deriving continuous-time approximations. It develops $ODE$ and $SDE$ limits using a 2d augmented state that encodes momentum and provides explicit weak-error expansions for smooth test functions, including a second-order extension. The main contributions are the precise statements and proofs of the $ODE$ and $SDE$ approximation theorems under weak assumptions, plus the handling of diagonal learning-rate schedules and time-dependent objectives. This framework enables rigorous understanding of how momentum and learning-rate schedules shape SGD dynamics, with potential applications to accelerated methods such as Nesterov-type schemes.

Abstract

Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due to empirical information rather than rigorous proof. While the dynamics of gradient descent methods can be studied through continuous approximations, existing works only cover scenarios with constant learning rates or SGD without momentum terms. We present approximation results under weak assumptions for SGD that allow learning rates and momentum parameters to vary with respect to time.

First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms

TL;DR

The paper addresses the dynamics of momentum-based stochastic gradient descent with time-varying learning rates by deriving continuous-time approximations. It develops and limits using a 2d augmented state that encodes momentum and provides explicit weak-error expansions for smooth test functions, including a second-order extension. The main contributions are the precise statements and proofs of the and approximation theorems under weak assumptions, plus the handling of diagonal learning-rate schedules and time-dependent objectives. This framework enables rigorous understanding of how momentum and learning-rate schedules shape SGD dynamics, with potential applications to accelerated methods such as Nesterov-type schemes.

Abstract

Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due to empirical information rather than rigorous proof. While the dynamics of gradient descent methods can be studied through continuous approximations, existing works only cover scenarios with constant learning rates or SGD without momentum terms. We present approximation results under weak assumptions for SGD that allow learning rates and momentum parameters to vary with respect to time.

Paper Structure

This paper contains 14 sections, 21 theorems, 152 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Spaces
  4. Feynman-Kac and Itô's Lemma
  5. Assumptions
  6. Main Theorem
  7. ODE Approximation
  8. Proof of Theorem \ref{['sgdodeapproxmain']}
  9. Proof of Theorem \ref{['sgdmomentumapproxmain']}, ODE Case
  10. SDE Approximation
  11. Proof of Theorem \ref{['momentumsderesult']}
  12. Proof of Theorem \ref{['sgdmomentumapproxmain']}, SDE Case
  13. Second Order SDE Approximation
  14. Conclusion

Key Result

Theorem 1

Let $f: \mathbb{R}^d \rightarrow \mathbb{R}$, $k:\mathbb{R}^d \rightarrow [0,\infty]$, and $g: [0,T] \times \mathbb{R}^d \rightarrow \mathbb{R}$ be continuous functions satisfying, for a constant $\lambda \geq 1$, and Suppose a continuous function $y: [0,T] \times \mathbb{R}^d \rightarrow \mathbb{R}$ is the expectation of a stochastic process Then $u$ satisfies the partial differential equati

Figures (2)

  • Figure 1: Flow of the proof to the Ordinary Differential Equation case
  • Figure 2: Flow of the proof to the Stochastic Differential Equation case

Theorems & Definitions (39)

  • Theorem 1: Feynman-Kac
  • Definition 1: Quadratic Forms
  • Lemma 1: Itô's Lemma
  • Corollary 1
  • Definition 2: Weak solutions
  • Lemma 2
  • proof
  • Theorem 2
  • Remark 1
  • Theorem 3
  • ...and 29 more