Rapid Overfitting of Multi-Pass Stochastic Gradient Descent in Stochastic Convex Optimization

TL;DR

This work analyzes the generalization behavior of multi-pass SGD in stochastic convex optimization. It proves tight population-risk bounds of the form $Θ\left(\frac{1}{ηT} + η\sqrt{T}\right)$ for multi-pass SGD and shows a rapid deterioration in out-of-sample performance after the first epoch, with similar phenomena for with-replacement SGD after $O(n\log n)$ steps. It also establishes a $Ω(η\sqrt{n})$ lower bound on the empirical risk of one-pass SGD in near-linear dimension, indicating a fundamental separation between population risk and empirical risk in early training. The results highlight a phase transition between the first and subsequent passes and extend to both shuffled and replacement sampling, offering sharp insights into when and how SGD overfits in SCO. These findings have implications for practical SGD usage and theoretical understanding of generalization in convex optimization under finite-sample regimes.

Abstract

We study the out-of-sample performance of multi-pass stochastic gradient descent (SGD) in the fundamental stochastic convex optimization (SCO) model. While one-pass SGD is known to achieve an optimal $Θ(1/\sqrt{n})$ excess population loss given a sample of size $n$, much less is understood about the multi-pass version of the algorithm which is widely used in practice. Somewhat surprisingly, we show that in the general non-smooth case of SCO, just a few epochs of SGD can already hurt its out-of-sample performance significantly and lead to overfitting. In particular, using a step size $η= Θ(1/\sqrt{n})$, which gives the optimal rate after one pass, can lead to population loss as large as $Ω(1)$ after just one additional pass. More generally, we show that the population loss from the second pass onward is of the order $Θ(1/(ηT) + η\sqrt{T})$, where $T$ is the total number of steps. These results reveal a certain phase-transition in the out-of-sample behavior of SGD after the first epoch, as well as a sharp separation between the rates of overfitting in the smooth and non-smooth cases of SCO. Additionally, we extend our results to with-replacement SGD, proving that the same asymptotic bounds hold after $O(n \log n)$ steps. Finally, we also prove a lower bound of $Ω(η\sqrt{n})$ on the generalization gap of one-pass SGD in dimension $d = \smash{\widetilde O}(n)$, improving on recent results of Koren et al.(2022) and Schliserman et al.(2024).

Figures (1)

• Figure 1: An illustration of the minmax rates for the population loss of multi-pass SGD established in \ref{['thm:main_result', 'thm:upper_bound']}, through $K=5$ epochs and for different stepsizes $\eta$.

Theorems & Definitions (35)

• Theorem 3.1
• proof : Proof sketch
• Theorem 3.2
• proof : Proof sketch
• Theorem 3.3
• Theorem 3.4
• Theorem 4.1
• proof : Proof sketch
• Lemma 5.1
• proof