Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Iteration and Stochastic First-order Oracle Complexities of Stochastic Gradient Descent using Constant and Decaying Learning Rates

Kento Imaizumi, Hideaki Iiduka

TL;DR

The paper analyzes stochastic gradient descent for nonconvex deep learning under constant and decaying learning rates, focusing on how batch size affects both iteration and stochastic first-order oracle (SFO) complexity. It derives tight upper bounds on the gradient-norm objective, establishes that the SFO cost N(b)=Kb is convex in the batch size and identifies regime-specific critical batch sizes b*, which minimize N. The results yield explicit iteration and SFO complexity rates for each learning-rate schedule, and are validated by neural network experiments showing that SGD with a critical batch size often outperforms standard optimizers and that measured critical sizes align with theory. Together, these findings provide principled guidance for selecting batch sizes to minimize stochastic gradient computations in nonconvex SGD training of deep nets.

Abstract

The performance of stochastic gradient descent (SGD), which is the simplest first-order optimizer for training deep neural networks, depends on not only the learning rate but also the batch size. They both affect the number of iterations and the stochastic first-order oracle (SFO) complexity needed for training. In particular, the previous numerical results indicated that, for SGD using a constant learning rate, the number of iterations needed for training decreases when the batch size increases, and the SFO complexity needed for training is minimized at a critical batch size and that it increases once the batch size exceeds that size. Here, we study the relationship between batch size and the iteration and SFO complexities needed for nonconvex optimization in deep learning with SGD using constant or decaying learning rates and show that SGD using the critical batch size minimizes the SFO complexity. We also provide numerical comparisons of SGD with the existing first-order optimizers and show the usefulness of SGD using a critical batch size. Moreover, we show that measured critical batch sizes are close to the sizes estimated from our theoretical results.

Iteration and Stochastic First-order Oracle Complexities of Stochastic Gradient Descent using Constant and Decaying Learning Rates

TL;DR

The paper analyzes stochastic gradient descent for nonconvex deep learning under constant and decaying learning rates, focusing on how batch size affects both iteration and stochastic first-order oracle (SFO) complexity. It derives tight upper bounds on the gradient-norm objective, establishes that the SFO cost N(b)=Kb is convex in the batch size and identifies regime-specific critical batch sizes b*, which minimize N. The results yield explicit iteration and SFO complexity rates for each learning-rate schedule, and are validated by neural network experiments showing that SGD with a critical batch size often outperforms standard optimizers and that measured critical sizes align with theory. Together, these findings provide principled guidance for selecting batch sizes to minimize stochastic gradient computations in nonconvex SGD training of deep nets.

Abstract

The performance of stochastic gradient descent (SGD), which is the simplest first-order optimizer for training deep neural networks, depends on not only the learning rate but also the batch size. They both affect the number of iterations and the stochastic first-order oracle (SFO) complexity needed for training. In particular, the previous numerical results indicated that, for SGD using a constant learning rate, the number of iterations needed for training decreases when the batch size increases, and the SFO complexity needed for training is minimized at a critical batch size and that it increases once the batch size exceeds that size. Here, we study the relationship between batch size and the iteration and SFO complexities needed for nonconvex optimization in deep learning with SGD using constant or decaying learning rates and show that SGD using the critical batch size minimizes the SFO complexity. We also provide numerical comparisons of SGD with the existing first-order optimizers and show the usefulness of SGD using a critical batch size. Moreover, we show that measured critical batch sizes are close to the sizes estimated from our theoretical results.
Paper Structure (28 sections, 5 theorems, 46 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 5 theorems, 46 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Motivation
  4. Contribution
  5. Upper bound of theoretical performance measure
  6. Critical batch size to reduce SFO complexity
  7. Iteration and SFO complexities
  8. Nonconvex Optimization and SGD
  9. Nonconvex optimization in deep learning
  10. SGD
  11. Conditions and algorithm
  12. Learning rates
  13. Our Results
  14. Upper bound of the squared norm of the full gradient
  15. Number of iterations needed for SGD to be an $\epsilon$--approximation
  16. ...and 13 more sections

Key Result

Theorem 3.1

The sequence $(\bm{\theta}_k)_{k\in \mathbb{N}}$ generated by Algorithm algo:1 under (C1)--(C3) satisfies that, for all $K \geq 1$, where

Figures (6)

  • Figure 1:
  • Figure 3:
  • Figure 5:
  • Figure 7:
  • Figure 9:
  • ...and 1 more figures

Theorems & Definitions (5)

  • Theorem 3.1: Upper bound of the squared norm of the full gradient
  • Theorem 3.2: Numbers of iterations needed for nonconvex optimization using SGD
  • Theorem 3.3: SFO complexity needed for nonconvex optimization of SGD
  • Theorem 3.4: Iteration and SFO complexities of SGD
  • Lemma A.1