Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis

Yuichi Kondo, Hideaki Iiduka

TL;DR

The paper tackles convergence of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel Lyapunov function. It develops a unified framework that covers both mini-batch SHB and NSHB and establishes convergence bounds on the expected gradient norm for several scheduling strategies, including constant batch with decaying LR, increasing batch with decaying LR, and increasing batch with increasing LR. The key finding is a hierarchy: constant batch size may fail to guarantee convergence, increasing batch size ensures convergence, and increasing both batch size and learning rate can yield faster decay (with exponential rates under certain schedules). Empirical results on CIFAR-100 with ResNet-18 validate the theory, showing dynamically scheduled SGDM markedly accelerates convergence, with warm-up strategies often performing best and aligning with the predicted orderings of gradient norms across schedules.

Abstract

We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warm-up schedule in experiments, which empirically outperformed all other strategies in convergence behavior.

Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis

TL;DR

The paper tackles convergence of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel Lyapunov function. It develops a unified framework that covers both mini-batch SHB and NSHB and establishes convergence bounds on the expected gradient norm for several scheduling strategies, including constant batch with decaying LR, increasing batch with decaying LR, and increasing batch with increasing LR. The key finding is a hierarchy: constant batch size may fail to guarantee convergence, increasing batch size ensures convergence, and increasing both batch size and learning rate can yield faster decay (with exponential rates under certain schedules). Empirical results on CIFAR-100 with ResNet-18 validate the theory, showing dynamically scheduled SGDM markedly accelerates convergence, with warm-up strategies often performing best and aligning with the predicted orderings of gradient norms across schedules.

Abstract

We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warm-up schedule in experiments, which empirically outperformed all other strategies in convergence behavior.

Paper Structure

This paper contains 32 sections, 6 theorems, 94 equations, 36 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Empirical Risk Minimization
  4. Mini-batch SHB and NSHB Optimizers
  5. Convergence Analysis of Mini-batch SGDM
  6. A Novel Lyapunov Function
  7. General Convergence Bound
  8. Constant Batch Size and Decaying Learning Rate Scheduler
  9. Increasing Batch Size and Decaying Learning Rate Scheduler
  10. Increasing Batch Size and Increasing Learning Rate Scheduler
  11. Experiments
  12. Conclusion
  13. Increasing Batch Size with Warm-Up Learning Rate Scheduler
  14. Proof
  15. Proposition
  16. ...and 17 more sections

Key Result

Theorem 1

Suppose Assumption assum:1 holds. Let $\{\bm{\theta}_t\}$ be the sequence generated by either Algorithm algo:nshb (NSHB) or Algorithm algo:shb (SHB) with learning rates $\lambda_t$ and batch sizes $b_t$. Assume and $\sum_{t=0}^{T-1} \lambda_t \neq 0$. Define and Then, for any $T \in \mathbb{N}$, the following bound holds: where $\mathbb{E}$ denotes the expectation over all randomness up to ite

Figures (36)

  • Figure 4: (i) Constant BS + decay LR
  • Figure 5: (ii) Increasing BS + decay LR
  • Figure 6: (iii) Increasing BS + increasing LR
  • Figure 7: (iv) Increasing BS + warm-up LR
  • Figure 12: Results of SHB under (i): constant batch size and decaying learning rate. Top row: learning rate = high (same as NSHB); bottom row: learning rate = low (one-tenth of NSHB).
  • ...and 31 more figures

Theorems & Definitions (12)

  • Theorem 1: General and unified convergence bound for NSHB and SHB
  • Remark 1: Translation from squared gradient bound to gradient norm
  • Corollary 1: Convergence rates under schedule \ref{['scheduler_1']}
  • Corollary 2: Convergence rates under schedule \ref{['scheduler_2']}
  • Corollary 3: Convergence under schedule \ref{['scheduler_3']}
  • Corollary 4: Convergence under schedule \ref{['scheduler_4']}
  • Proposition 1
  • proof
  • proof
  • proof
  • ...and 2 more