Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benign Overfitting in Linear Regression

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, Alexander Tsigler

TL;DR

This work characterizes when a perfect fit to training data in linear regression remains predictive, by connecting the excess risk of the minimum-norm interpolator to two notions of covariance effective rank, r_k(Σ) and R_k(Σ). The results show that substantial overparameterization—many small-variance directions—is crucial for benign overfitting, with precise finite-sample bounds that depend on the tail behavior of the covariance spectrum. The analysis highlights fundamental differences between infinite- and finite-dimensional settings and draws connections to neural networks via the NTK framework, suggesting finite-dimensional approximations may be key to understanding benign overfitting in practice. The findings provide a rigorous lens for when interpolating predictions can achieve near-optimal accuracy and point to rich future directions, including extensions beyond linear models and relaxed distributional assumptions.

Abstract

The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lies in an infinite dimensional space versus when the data lies in a finite dimensional space whose dimension grows faster than the sample size.

Benign Overfitting in Linear Regression

TL;DR

This work characterizes when a perfect fit to training data in linear regression remains predictive, by connecting the excess risk of the minimum-norm interpolator to two notions of covariance effective rank, r_k(Σ) and R_k(Σ). The results show that substantial overparameterization—many small-variance directions—is crucial for benign overfitting, with precise finite-sample bounds that depend on the tail behavior of the covariance spectrum. The analysis highlights fundamental differences between infinite- and finite-dimensional settings and draws connections to neural networks via the NTK framework, suggesting finite-dimensional approximations may be key to understanding benign overfitting in practice. The findings provide a rigorous lens for when interpolating predictions can achieve near-optimal accuracy and point to rich future directions, including extensions beyond linear models and relaxed distributional assumptions.

Abstract

The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lies in an infinite dimensional space versus when the data lies in a finite dimensional space whose dimension grows faster than the sample size.

Paper Structure

This paper contains 18 sections, 39 theorems, 183 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Definitions and Notation
  3. Main Results
  4. Effective Ranks and Overparameterization
  5. Deep neural networks
  6. Proof
  7. Conclusions and Further Work
  8. Proof of Lemma \ref{['lemma:bv']}
  9. An Algebraic Property
  10. Proof of concentration inequalities
  11. Proof of Lemma \ref{['lemma:singletermlower']}
  12. Proof of Lemma \ref{['lemma::sum_of_pos']}
  13. Proof of Lemma \ref{['lemma:bestk']}
  14. Eigenvalue monotonicity
  15. Rank facts
  16. ...and 3 more sections

Key Result

Theorem 4

For any $\sigma_x$ there are $b,c, c_1>1$ for which the following holds. Consider a linear regression problem from Definition def:linreg. Define where the minimum of the empty set is defined as $\infty$. Suppose $\delta<1$ with $\log(1/\delta)<n/c$. If $k^* \geq n/c_1$, then $\mathbb{E} R(\hat{\theta})\ge \sigma^2/c$. Otherwise, with probability at least $1-\delta$, and Moreover, there are univ

Figures (1)

  • Figure 1: A diagram illustrating the definition of Algorithm $C$.

Theorems & Definitions (64)

  • Definition 1: Linear regression
  • Definition 2: Minimum norm estimator
  • Definition 3: Effective Ranks
  • Theorem 4
  • Lemma 5
  • Theorem 6
  • Lemma 7
  • Lemma 8
  • proof
  • Lemma 9
  • ...and 54 more