Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modified Equations for Stochastic Optimization

Stefan Perko

TL;DR

The thesis extends the theory of stochastic modified equations (SMEs) to stochastic gradient optimization, bridging numerical ODE methods with stochastic analysis to study SGD and its variants. It develops time-inhomogeneous SDEs with step-size expansions, establishing first- and second-order weak approximation properties and explicit linear error terms, then instantiates these results for SGD with linear regression as a key example. A novel diffusion model for SGD without replacement (SGDo) is introduced via epoched Brownian motion (EBM) and Young differential equations, with almost sure convergence in the strongly convex case and sharp convergence-rate bounds; a weak-convergence theory links EBMs to scaling limits governed by deterministic permutons. The work further compares first-order SMEs (gradient flow, NCC-SGF, CC-SGF) in linear regression, deriving explicit linear errors that depend on batch size and kurtosis, and confirms the predictions with numerical experiments. Collectively, the results provide a rigorous toolkit for predicting SGD behavior under finite data regimes and for guiding the choice of SME approximations in practice, with implications for hyperparameter tuning and convergence analysis in large-scale learning systems.

Abstract

In this thesis, we extend the recently introduced theory of stochastic modified equations (SMEs) for stochastic gradient optimization algorithms. In Ch. 3 we study time-inhomogeneous SDEs driven by Brownian motion. For certain SDEs we prove a 1st and 2nd-order weak approximation properties, and we compute their linear error terms explicitly, under certain regularity conditions. In Ch. 4 we instantiate our results for SGD, working out the example of linear regression explicitly. We use this example to compare the linear error terms of gradient flow and two commonly used 1st-order SMEs for SGD in Ch. 5. In the second part of the thesis we introduce and study a novel diffusion approximation for SGD without replacement (SGDo) in the finite-data setting. In Ch. 6 we motivate and define the notion of an epoched Brownian motion (EBM). We argue that Young differential equations (YDEs) driven by EBMs serve as continuous-time models for SGDo for any shuffling scheme whose induced permutations converge to a det. permuton. Further, we prove a.s. convergence for these YDEs in the strongly convex setting. Moreover, we compute an upper asymptotic bound on the convergence rate which is as sharp as, or better than previous results for SGDo. In Ch. 7 we study scaling limits of families of random walks (RW) that share the same increments up to a random permutation. We show weak convergence under the assumption that the sequence of permutations converges to a det. (higher-dimensional) permuton. This permuton determines the covariance function of the limiting Gaussian process. Conversely, we show that every Gaussian process with a covariance function determined by a permuton in this way arises as a weak scaling limit of families of RW with shared increments. Finally, we apply our weak convergence theory to show that EBMs arise as scaling limits of RW with finitely many distinct increments.

Modified Equations for Stochastic Optimization

TL;DR

The thesis extends the theory of stochastic modified equations (SMEs) to stochastic gradient optimization, bridging numerical ODE methods with stochastic analysis to study SGD and its variants. It develops time-inhomogeneous SDEs with step-size expansions, establishing first- and second-order weak approximation properties and explicit linear error terms, then instantiates these results for SGD with linear regression as a key example. A novel diffusion model for SGD without replacement (SGDo) is introduced via epoched Brownian motion (EBM) and Young differential equations, with almost sure convergence in the strongly convex case and sharp convergence-rate bounds; a weak-convergence theory links EBMs to scaling limits governed by deterministic permutons. The work further compares first-order SMEs (gradient flow, NCC-SGF, CC-SGF) in linear regression, deriving explicit linear errors that depend on batch size and kurtosis, and confirms the predictions with numerical experiments. Collectively, the results provide a rigorous toolkit for predicting SGD behavior under finite data regimes and for guiding the choice of SME approximations in practice, with implications for hyperparameter tuning and convergence analysis in large-scale learning systems.

Abstract

In this thesis, we extend the recently introduced theory of stochastic modified equations (SMEs) for stochastic gradient optimization algorithms. In Ch. 3 we study time-inhomogeneous SDEs driven by Brownian motion. For certain SDEs we prove a 1st and 2nd-order weak approximation properties, and we compute their linear error terms explicitly, under certain regularity conditions. In Ch. 4 we instantiate our results for SGD, working out the example of linear regression explicitly. We use this example to compare the linear error terms of gradient flow and two commonly used 1st-order SMEs for SGD in Ch. 5. In the second part of the thesis we introduce and study a novel diffusion approximation for SGD without replacement (SGDo) in the finite-data setting. In Ch. 6 we motivate and define the notion of an epoched Brownian motion (EBM). We argue that Young differential equations (YDEs) driven by EBMs serve as continuous-time models for SGDo for any shuffling scheme whose induced permutations converge to a det. permuton. Further, we prove a.s. convergence for these YDEs in the strongly convex setting. Moreover, we compute an upper asymptotic bound on the convergence rate which is as sharp as, or better than previous results for SGDo. In Ch. 7 we study scaling limits of families of random walks (RW) that share the same increments up to a random permutation. We show weak convergence under the assumption that the sequence of permutations converges to a det. (higher-dimensional) permuton. This permuton determines the covariance function of the limiting Gaussian process. Conversely, we show that every Gaussian process with a covariance function determined by a permuton in this way arises as a weak scaling limit of families of RW with shared increments. Finally, we apply our weak convergence theory to show that EBMs arise as scaling limits of RW with finitely many distinct increments.

Paper Structure

This paper contains 60 sections, 84 theorems, 696 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Modified equations
  4. Introduction
  5. Preliminaries
  6. Growth conditions
  7. General assumptions on the one-step method
  8. Convergence of the noisy Euler method
  9. Differentiation and regularity
  10. Weighted Hölder spaces
  11. Regularity of random fields
  12. Differentiating stochastic differential equation s
  13. Estimating $\psi$:
  14. Estimating $\int\langle \nabla_y f(X), \partial^\alpha X\rangle$:
  15. Finishing the inductive step
  16. ...and 45 more sections

Key Result

Lemma 3.2.1

Let $l, \kappa, \lambda\in \mathbb{N}_0$. Then the following properties hold true:

Figures (3)

  • Figure 1: The best continuous-time approximation of SGD for linear regression in dimension $1$ in terms of the kurtosis of the features and the batch size. Here $\kappa = 1$, $(\theta-\theta^*)^2 = 1$ and $T = 0.5$ ($T = 2.0$) in the left (right) plot. In the lower part of the middle region, where CC-SGF is the best approximation, gradient flow is worse than NCC-SGF. In the upper part of the middle region, gradient flow is better than NCC-SGF.
  • Figure 2: The weak error's dependence on the learning rate for several continuous-time approximations to SGD, in various settings. The plots (1)-(5) correspond to the settings (i)-(v) in Theorem \ref{['thm:linRegBS']}. Further, (4) and (6) also correspond to (vi). Finally, (6) depicts a situation where $X^{\operatorname{CC}} \not\asymp X^{\operatorname{NCC}}$, but the weak errors are close to each other since the common initial value is close to the minimum.
  • Figure :

Theorems & Definitions (170)

  • Lemma 3.2.1
  • proof
  • Lemma 3.2.2
  • proof
  • Lemma 3.2.3
  • proof
  • Lemma 3.2.4
  • proof
  • Lemma 3.2.5
  • proof
  • ...and 160 more