Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization

Roi Livni

TL;DR

This paper determines the fundamental limits of full-batch gradient descent in non-smooth stochastic convex optimization by proving a dimension-dependent lower bound on the generalization gap of GD, showing a rate of $\tilde{\Theta}\left(\frac{d}{m}+\frac{1}{\sqrt{m}}\right)$ that matches worst-case ERMs. The authors introduce a novel analysis framework with a sample-dependent oracle and a reduction to differentiable extensions, constructing adversarial targets using Feldman’s and Nemirovski-like components to force GD to overfit unless the iteration budget scales appropriately with dimension. The results establish that, in regimes where $d$ is large relative to $m$ or when iterations are limited, GD does not surpass worst-case ERM performance and, in particular, may require $T=\Omega\left(\frac{1}{\varepsilon^4}\right)$ iterations to avoid overfitting. This work resolves open problems about the necessity of dimension-dependent sample complexity for GD and clarifies when late stopping or stability-based strategies can improve generalization. Overall, the findings sharpen our understanding of first-order methods in SCO and their limitations compared to empirical risk minimization.

Abstract

We analyze the sample complexity of full-batch Gradient Descent (GD) in the setup of non-smooth Stochastic Convex Optimization. We show that the generalization error of GD, with common choice of hyper-parameters, can be $\tilde Θ(d/m + 1/\sqrt{m})$, where $d$ is the dimension and $m$ is the sample size. This matches the sample complexity of \emph{worst-case} empirical risk minimizers. That means that, in contrast with other algorithms, GD has no advantage over naive ERMs. Our bound follows from a new generalization bound that depends on both the dimension as well as the learning rate and number of iterations. Our bound also shows that, for general hyper-parameters, when the dimension is strictly larger than number of samples, $T=Ω(1/ε^4)$ iterations are necessary to avoid overfitting. This resolves an open problem by Schlisserman et al.23 and Amir er Al.21, and improves over previous lower bounds that demonstrated that the sample size must be at least square root of the dimension.

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization

TL;DR

This paper determines the fundamental limits of full-batch gradient descent in non-smooth stochastic convex optimization by proving a dimension-dependent lower bound on the generalization gap of GD, showing a rate of that matches worst-case ERMs. The authors introduce a novel analysis framework with a sample-dependent oracle and a reduction to differentiable extensions, constructing adversarial targets using Feldman’s and Nemirovski-like components to force GD to overfit unless the iteration budget scales appropriately with dimension. The results establish that, in regimes where is large relative to or when iterations are limited, GD does not surpass worst-case ERM performance and, in particular, may require iterations to avoid overfitting. This work resolves open problems about the necessity of dimension-dependent sample complexity for GD and clarifies when late stopping or stability-based strategies can improve generalization. Overall, the findings sharpen our understanding of first-order methods in SCO and their limitations compared to empirical risk minimization.

Abstract

We analyze the sample complexity of full-batch Gradient Descent (GD) in the setup of non-smooth Stochastic Convex Optimization. We show that the generalization error of GD, with common choice of hyper-parameters, can be , where is the dimension and is the sample size. This matches the sample complexity of \emph{worst-case} empirical risk minimizers. That means that, in contrast with other algorithms, GD has no advantage over naive ERMs. Our bound follows from a new generalization bound that depends on both the dimension as well as the learning rate and number of iterations. Our bound also shows that, for general hyper-parameters, when the dimension is strictly larger than number of samples, iterations are necessary to avoid overfitting. This resolves an open problem by Schlisserman et al.23 and Amir er Al.21, and improves over previous lower bounds that demonstrated that the sample size must be at least square root of the dimension.
Paper Structure (29 sections, 7 theorems, 114 equations, 1 figure)

This paper contains 29 sections, 7 theorems, 114 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. First order optimization
  4. Empirical Risk Minimization
  5. Main Result
  6. Discussion
  7. Stability in low dimension
  8. Late stopping
  9. Technical Overview
  10. When $d\ll T$
  11. When $T \ll d$,
  12. Working with any first order oracle
  13. Sample-dependent Oracle
  14. Reduction to sample-dependent oracles
  15. Proof of \ref{['thm:main']}
  16. ...and 14 more sections

Key Result

theorem 1

For every $d\ge 4096, T\ge 10, m\ge 1$ and $\eta>0$, there exists a distribution $D$, and a $4$-Lipschitz convex function $f(w,z)$ in $\mathbb{R}^{d+1}$, such that for any first order oracle of $f(w,z)$, with probability $1/2$, if we run GD with $\eta$ as a learning rate then:

Figures (1)

  • Figure 1: Depiction of the dynamics induced by \ref{['eq:nemonster']} and our choice of sub-differentials

Theorems & Definitions (10)

  • theorem 1
  • corollary 2
  • corollary 3
  • lemma 7
  • lemma 8
  • lemma 9
  • theorem 10
  • claim 1
  • claim 2
  • Proof