A Classical View on Benign Overfitting: The Role of Sample Size

TL;DR

This work challenges the traditional view that overfitting necessarily harms generalization by showing that benign overfitting can occur within the classical risk–capacity regime when sample size is properly scaled with model complexity. It establishes non-asymptotic, high-probability guarantees for two foundational settings: kernel ridge regression and two-layer ReLU networks trained in the neural tangent kernel (NTK) regime. Central to the results is a decomposition of excess risk into approximation and estimation errors and the interpretation of gradient flow as an implicit regularizer, which helps avoid uniform convergence traps. The findings suggest that larger models can simultaneously achieve small training and test errors without strong assumptions on the regression function or noise, though the bounds are upper bounds and the analysis is currently limited to these settings. The work provides a framework and techniques that may inform future analyses of benign overfitting beyond NTK and kernel methods, with potential implications for understanding overparameterized neural networks in practice.

Abstract

Benign overfitting is a phenomenon in machine learning where a model perfectly fits (interpolates) the training data, including noisy examples, yet still generalizes well to unseen data. Understanding this phenomenon has attracted considerable attention in recent years. In this work, we introduce a conceptual shift, by focusing on almost benign overfitting, where models simultaneously achieve both arbitrarily small training and test errors. This behavior is characteristic of neural networks, which often achieve low (but non-zero) training error while still generalizing well. We hypothesize that this almost benign overfitting can emerge even in classical regimes, by analyzing how the interaction between sample size and model complexity enables larger models to achieve both good training fit but still approach Bayes-optimal generalization. We substantiate this hypothesis with theoretical evidence from two case studies: (i) kernel ridge regression, and (ii) least-squares regression using a two-layer fully connected ReLU neural network trained via gradient flow. In both cases, we overcome the strong assumptions often required in prior work on benign overfitting. Our results on neural networks also provide the first generalization result in this setting that does not rely on any assumptions about the underlying regression function or noise, beyond boundedness. Our analysis introduces a novel proof technique based on decomposing the excess risk into estimation and approximation errors, interpreting gradient flow as an implicit regularizer, that helps avoid uniform convergence traps. This analysis idea could be of independent interest.

Figures (10)

• Figure 1: Dashed and solid lines show empirical and excess risk, respectively. On plots (b), (c) and (d), black, blue and red curves are in order of increasing sample size. The vertical dotted lines represent the model complexity of the model under consideration, and the points where the empirical and excess risk curves cross and stay over are marked with $\star$ (which may not necessarily happen at the troughs of the U-curves). In (a) and (c), the model is taken at trough of the stationary U-curve, and in (b), the model is taken at the troughs of the moving U-curve. In (d), the model is taken in the interpolation regime.
• Figure 2: Risk vs. model complexity plot for Abalone dataset with Gaussian noise (mean-zero, std. dev $0.2$) added to the target variable (age) during the training process. We use $m=100000$.
• Figure 4: Synthetic Data Experiment: Risk vs. model complexity plot on synthetic data. Increasing both the sample size $n$ and the number of training iterations simultaneously allows for reduction of both empirical and excess risks.
• Figure 5: Synthetic Data Experiment: The average iteration at which the excess risk crosses and stays over the empirical evaluated over $10$ runs with different random initializations to the neural network. The bars indicate the standard deviation on the iteration number. Note the clear shift to the right and down.
• Figure 7: Abalone Data Experiment: The average iteration at which the excess risk crosses and stays over the empirical evaluated over $10$ runs with different random initializations. This is for the setting discussed in Section \ref{['sec:experiments']}, with Gaussian noise (mean-zero, std. dev $0.2$). Again notice the shift to the right and down of where the crossing occurs.

Theorems & Definitions (53)

• Definition 1: Benign Overfitting
• Theorem 2: Overfitting
• Theorem 3: Approximation
• Theorem 4: Estimation
• Theorem 5: Generalization
• Theorem 6: Benign Overfitting
• Theorem 7: Overfitting
• Theorem 8: Approximation Error
• Theorem 9: Estimation Error
• Theorem 10: Generalization