Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Bootstrap Perspective on Stochastic Gradient Descent

Hongjian Lan, Yucong Liu, Florian Schäfer

TL;DR

SGD often generalizes better than GD in overparameterized models. This work proposes a bootstrap-based view in which mini-batch gradient variability acts as a proxy for the solution's sensitivity to data resampling, linking SGD to generalization. It derives a Hessian-weighted decomposition of the generalization gap, shows SGD implicitly regularizes the gradient-covariance trace, and introduces explicit regularizers Reg1 and Reg2 that further reduce generalization error; these ideas are validated in both synthetic diagonal-linear-network settings and practical CNNs. The results offer a principled mechanism for SGD generalization and practical tools to enhance it, while highlighting open challenges in automatic regularizer tuning.

Abstract

Machine learning models trained with \emph{stochastic} gradient descent (SGD) can generalize better than those trained with deterministic gradient descent (GD). In this work, we study SGD's impact on generalization through the lens of the statistical bootstrap: SGD uses gradient variability under batch sampling as a proxy for solution variability under the randomness of the data collection process. We use empirical results and theoretical analysis to substantiate this claim. In idealized experiments on empirical risk minimization, we show that SGD is drawn to parameter choices that are robust under resampling and thus avoids spurious solutions even if they lie in wider and deeper minima of the training loss. We prove rigorously that by implicitly regularizing the trace of the gradient covariance matrix, SGD controls the algorithmic variability. This regularization leads to solutions that are less sensitive to sampling noise, thereby improving generalization. Numerical experiments on neural network training show that explicitly incorporating the estimate of the algorithmic variability as a regularizer improves test performance. This fact supports our claim that bootstrap estimation underpins SGD's generalization advantages.

A Bootstrap Perspective on Stochastic Gradient Descent

TL;DR

SGD often generalizes better than GD in overparameterized models. This work proposes a bootstrap-based view in which mini-batch gradient variability acts as a proxy for the solution's sensitivity to data resampling, linking SGD to generalization. It derives a Hessian-weighted decomposition of the generalization gap, shows SGD implicitly regularizes the gradient-covariance trace, and introduces explicit regularizers Reg1 and Reg2 that further reduce generalization error; these ideas are validated in both synthetic diagonal-linear-network settings and practical CNNs. The results offer a principled mechanism for SGD generalization and practical tools to enhance it, while highlighting open challenges in automatic regularizer tuning.

Abstract

Machine learning models trained with \emph{stochastic} gradient descent (SGD) can generalize better than those trained with deterministic gradient descent (GD). In this work, we study SGD's impact on generalization through the lens of the statistical bootstrap: SGD uses gradient variability under batch sampling as a proxy for solution variability under the randomness of the data collection process. We use empirical results and theoretical analysis to substantiate this claim. In idealized experiments on empirical risk minimization, we show that SGD is drawn to parameter choices that are robust under resampling and thus avoids spurious solutions even if they lie in wider and deeper minima of the training loss. We prove rigorously that by implicitly regularizing the trace of the gradient covariance matrix, SGD controls the algorithmic variability. This regularization leads to solutions that are less sensitive to sampling noise, thereby improving generalization. Numerical experiments on neural network training show that explicitly incorporating the estimate of the algorithmic variability as a regularizer improves test performance. This fact supports our claim that bootstrap estimation underpins SGD's generalization advantages.

Paper Structure

This paper contains 31 sections, 6 theorems, 44 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Our Contributions
  4. Paper Outline
  5. Preliminaries
  6. Empirical Risk Minimization and Generalization Gap
  7. Stochastic Gradient Descent
  8. Bootstrap Estimation
  9. An Idealized Experiment
  10. Theoretical Results
  11. Notations
  12. Decomposition of the Expected Generalization Gap
  13. Bootstrap Estimation of the Algorithmic Variability
  14. Implicit Regularizer and Generalization
  15. Empirical Validation
  16. ...and 16 more sections

Key Result

Lemma 1

Consider a loss function $L$ whose value is bounded by $U_L$, with batch gradient $\ell_2$-norm bounded by $U_G$ and all third-order partial derivatives bounded by $U_J$. Assume the parameters are bounded as $\|\theta\|_2\leq U_F$. If Assumptions assum1 and assum2 hold for $L$, the expected generali

Figures (5)

  • Figure 1: Construction of loss functions in the idealized experiment.
  • Figure 2: Heat maps of algorithm trajectory densities, population and training losses, gradient norms, and gradient covariance traces from the idealized experiment, with representative trajectories overlaid.
  • Figure 3: Trajectories of the generalization gap and the algorithmic variability trace versus iteration for GD, SGD, NoisyGD, and SGDwReg2, shown for two initializations in the idealized experiment.
  • Figure 4: Average test loss ratios of SGD with regularizers $1$ and $2$ relative to the vanilla SGD benchmarks. Shaded areas represent standard deviations of the test loss ratios across different initializations.
  • Figure 5: Average test losses of SGD with regularizer $2$. Shaded areas represent standard deviations across three runs with different random seeds.

Theorems & Definitions (9)

  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Lemma 2
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • proof