Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harmful Overfitting in Sobolev Spaces

Kedar Karhadkar, Alexander Sietsema, Deanna Needell, Guido Montufar

TL;DR

This work analyzes interpolation with noisy data in Sobolev spaces $W^{k,p}(\mathbb{R}^d)$ in fixed dimension. It proves that approximately norm-minimizing interpolants (ANM) cannot yield benign overfitting: under mild noise and regularity assumptions, any interpolant with Sobolev norm within a constant factor of the minimum incurs a constant excess risk, independent of the sample size $n$, for $kp>d$ and $k\in(d/p, 1.5d/p)$. The results extend prior $p=2$ Hilbert-space analyses to general $p\in[1,\infty)$ by constructing explicit bump-function interpolants, locating a noisy, well-separated subset, and applying Sobolev/Morrey-type inequalities to relate local oscillation to global generalization. In the Gaussian heteroskedastic noise setting with squared loss, the paper provides a concrete $L^2(\mu)$ lower bound on the error of ANM interpolants. Overall, the findings show that norm minimization combined with smoothness does not guarantee benign interpolation in fixed-dimensional Sobolev spaces, highlighting the need for alternative inductive biases or undertraining in such regimes.

Abstract

Motivated by recent work on benign overfitting in overparameterized machine learning, we study the generalization behavior of functions in Sobolev spaces $W^{k, p}(\mathbb{R}^d)$ that perfectly fit a noisy training data set. Under assumptions of label noise and sufficient regularity in the data distribution, we show that approximately norm-minimizing interpolators, which are canonical solutions selected by smoothness bias, exhibit harmful overfitting: even as the training sample size $n \to \infty$, the generalization error remains bounded below by a positive constant with high probability. Our results hold for arbitrary values of $p \in [1, \infty)$, in contrast to prior results studying the Hilbert space case ($p = 2$) using kernel methods. Our proof uses a geometric argument which identifies harmful neighborhoods of the training data using Sobolev inequalities.

Harmful Overfitting in Sobolev Spaces

TL;DR

This work analyzes interpolation with noisy data in Sobolev spaces in fixed dimension. It proves that approximately norm-minimizing interpolants (ANM) cannot yield benign overfitting: under mild noise and regularity assumptions, any interpolant with Sobolev norm within a constant factor of the minimum incurs a constant excess risk, independent of the sample size , for and . The results extend prior Hilbert-space analyses to general by constructing explicit bump-function interpolants, locating a noisy, well-separated subset, and applying Sobolev/Morrey-type inequalities to relate local oscillation to global generalization. In the Gaussian heteroskedastic noise setting with squared loss, the paper provides a concrete lower bound on the error of ANM interpolants. Overall, the findings show that norm minimization combined with smoothness does not guarantee benign interpolation in fixed-dimensional Sobolev spaces, highlighting the need for alternative inductive biases or undertraining in such regimes.

Abstract

Motivated by recent work on benign overfitting in overparameterized machine learning, we study the generalization behavior of functions in Sobolev spaces that perfectly fit a noisy training data set. Under assumptions of label noise and sufficient regularity in the data distribution, we show that approximately norm-minimizing interpolators, which are canonical solutions selected by smoothness bias, exhibit harmful overfitting: even as the training sample size , the generalization error remains bounded below by a positive constant with high probability. Our results hold for arbitrary values of , in contrast to prior results studying the Hilbert space case () using kernel methods. Our proof uses a geometric argument which identifies harmful neighborhoods of the training data using Sobolev inequalities.
Paper Structure (18 sections, 23 theorems, 146 equations)

This paper contains 18 sections, 23 theorems, 146 equations.

Table of Contents

  1. Introduction
  2. Related work
  3. Setup and main result
  4. Assumptions on loss function and data distribution
  5. Main theorem
  6. Proof overview and supporting results
  7. Explicit interpolating solution
  8. Existence of noisy, separated subset
  9. Smoothness of the $\gamma$-ANM solution
  10. Proof of the main theorem
  11. Conclusion
  12. Background on Sobolev spaces
  13. Proof of Corollary \ref{['corr:square-loss']}
  14. Proofs from Section \ref{['sec:proof-overview']}
  15. Properties of the conditional loss
  16. ...and 3 more sections

Key Result

Theorem 3.7

Let $\epsilon \in (0,1)$, let $k \in (d/p, 1.5d/p)$, and let $n \gtrsim \rho^{-2} \log(\epsilon^{-1}) + \mathop{\mathrm{Poly}}\nolimits_{k, p, d}(\epsilon^{-1})$. Let $(\mathbf{x}, y) \sim \mu$ be a test point chosen independently from the training set $(\mathbf{X}, \mathbf{y})$. Then with probabili where $C \in (0, \infty)$ is a constant depending on $k, d, p, \mu$, and $\ell$.

Theorems & Definitions (41)

  • Theorem 3.7
  • Corollary 3.8
  • Lemma 4.1
  • Lemma 4.2
  • Corollary 4.3
  • Lemma 4.4
  • Corollary 4.5
  • Lemma 4.6
  • proof : Proof of Theorem \ref{['thm:tempered-overfitting-bayes-v2']}
  • Theorem 1.1: Sobolev embedding
  • ...and 31 more