Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence Analysis of SGD under Expected Smoothness

Yuta Kawamoto, Hideaki Iiduka

TL;DR

This work studies stochastic gradient descent under the Expected Smoothness framework, linking the second moment of stochastic gradients to suboptimality, gradient norm, and data heterogeneity via constants $(A,B,C)$. It provides a self-contained descent analysis with explicit, sampling-aware bounds for several step-size schedules (constant, harmonic, polynomial, cosine) and derives a global convergence theorem under ES, along with closed-form ES constants for common sampling schemes. The authors validate the ES model empirically on CIFAR-100 with ResNet-18, showing that ES accurately captures gradient variance and reveals a two-phase microscopic noise structure: an initial suboptimality-dominated phase followed by a geometry-dominated phase. The results offer practical guidance for designing step-size and batch-size schedules in large-scale nonconvex optimization and deepen the theoretical understanding of variance–descent tradeoffs under realistic stochastic noise models.

Abstract

Stochastic gradient descent (SGD) is the workhorse of large-scale learning, yet classical analyses rely on assumptions that can be either too strong (bounded variance) or too coarse (uniform noise). The expected smoothness (ES) condition has emerged as a flexible alternative that ties the second moment of stochastic gradients to the objective value and the full gradient. This paper presents a self-contained convergence analysis of SGD under ES. We (i) refine ES with interpretations and sampling-dependent constants; (ii) derive bounds of the expectation of squared full gradient norm; and (iii) prove $O(1/K)$ rates with explicit residual errors for various step-size schedules. All proofs are given in full detail in the appendix. Our treatment unifies and extends recent threads (Khaled and Richtárik, 2020; Umeda and Iiduka, 2025).

Convergence Analysis of SGD under Expected Smoothness

TL;DR

This work studies stochastic gradient descent under the Expected Smoothness framework, linking the second moment of stochastic gradients to suboptimality, gradient norm, and data heterogeneity via constants . It provides a self-contained descent analysis with explicit, sampling-aware bounds for several step-size schedules (constant, harmonic, polynomial, cosine) and derives a global convergence theorem under ES, along with closed-form ES constants for common sampling schemes. The authors validate the ES model empirically on CIFAR-100 with ResNet-18, showing that ES accurately captures gradient variance and reveals a two-phase microscopic noise structure: an initial suboptimality-dominated phase followed by a geometry-dominated phase. The results offer practical guidance for designing step-size and batch-size schedules in large-scale nonconvex optimization and deepen the theoretical understanding of variance–descent tradeoffs under realistic stochastic noise models.

Abstract

Stochastic gradient descent (SGD) is the workhorse of large-scale learning, yet classical analyses rely on assumptions that can be either too strong (bounded variance) or too coarse (uniform noise). The expected smoothness (ES) condition has emerged as a flexible alternative that ties the second moment of stochastic gradients to the objective value and the full gradient. This paper presents a self-contained convergence analysis of SGD under ES. We (i) refine ES with interpretations and sampling-dependent constants; (ii) derive bounds of the expectation of squared full gradient norm; and (iii) prove rates with explicit residual errors for various step-size schedules. All proofs are given in full detail in the appendix. Our treatment unifies and extends recent threads (Khaled and Richtárik, 2020; Umeda and Iiduka, 2025).
Paper Structure (56 sections, 5 theorems, 112 equations, 10 figures, 2 tables)

This paper contains 56 sections, 5 theorems, 112 equations, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Historical foundations.
  4. Classical bounded variance assumption.
  5. Variance growth models.
  6. Strong growth condition.
  7. Expected Smoothness: a unifying framework.
  8. Modern developments.
  9. Motivation
  10. Limitation 1 of khaled2020better:
  11. Limitation 2 of umeda2024accelerates:
  12. Our motivation.
  13. Contributions
  14. Stochastic Minimization Problem and Assumptions
  15. Stochastic Minimization Problem
  16. ...and 41 more sections

Key Result

Proposition 1

Consider Problem stochastic_minimization and suppose that Assumption as:unbiased holds and that each component $f_i$ is $L_i$-smooth and bounded below, which implies Assumptions as:L-smooth and as:lower hold. Let $\Delta^{\inf}=\frac{1}{n}\sum_{i=1}^n(f^{\star}-f_i^{\inf})$, where $f^{\inf}$ (resp.

Figures (10)

  • Figure 2: Validation of the ES model for SGD. The ES model's predictions (RHS) accurately match the empirical gradient variance (LHS) throughout the training for all samplers.
  • Figure 3: Residuals of the ES model fit. The residuals are centered at zero, indicating an unbiased and accurate model fit. This validates the use of the ES framework for our subsequent analysis.
  • Figure : Training loss
  • Figure : Coefficient $A$ (Suboptimality-related)
  • Figure : Training loss
  • ...and 5 more figures

Theorems & Definitions (10)

  • Proposition 1: Validity of ES for common sampling
  • Theorem 1: Convergence guarantee of SGD under ES
  • Corollary 1: Convergence rates of SGD with specific step sizes
  • Proposition 2: Variance decomposition identity
  • proof
  • Lemma 1: Gradient–function gap inequality
  • proof
  • proof
  • proof : Proof of Theorem \ref{['thm:1']}
  • proof