Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noisy Interpolation Learning with Shallow Univariate ReLU Networks

Nirmit Joshi, Gal Vardi, Nathan Srebro

TL;DR

This work provides the first rigorous analysis of the overfitting behavior of regression with minimum norm of weights, focusing on univariate two-layer ReLU networks and shows overfitting is tempered (with high probability) when measured with respect to the $L_1$ loss, but also shows that the situation is more complex than suggested.

Abstract

Understanding how overparameterized neural networks generalize despite perfect interpolation of noisy training data is a fundamental question. Mallinar et. al. 2022 noted that neural networks seem to often exhibit ``tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error, but neither does it approach infinity, yielding non-trivial generalization. However, this has not been studied rigorously. We provide the first rigorous analysis of the overfitting behavior of regression with minimum norm ($\ell_2$ of weights), focusing on univariate two-layer ReLU networks. We show overfitting is tempered (with high probability) when measured with respect to the $L_1$ loss, but also show that the situation is more complex than suggested by Mallinar et. al., and overfitting is catastrophic with respect to the $L_2$ loss, or when taking an expectation over the training set.

Noisy Interpolation Learning with Shallow Univariate ReLU Networks

TL;DR

This work provides the first rigorous analysis of the overfitting behavior of regression with minimum norm of weights, focusing on univariate two-layer ReLU networks and shows overfitting is tempered (with high probability) when measured with respect to the loss, but also shows that the situation is more complex than suggested.

Abstract

Understanding how overparameterized neural networks generalize despite perfect interpolation of noisy training data is a fundamental question. Mallinar et. al. 2022 noted that neural networks seem to often exhibit ``tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error, but neither does it approach infinity, yielding non-trivial generalization. However, this has not been studied rigorously. We provide the first rigorous analysis of the overfitting behavior of regression with minimum norm ( of weights), focusing on univariate two-layer ReLU networks. We show overfitting is tempered (with high probability) when measured with respect to the loss, but also show that the situation is more complex than suggested by Mallinar et. al., and overfitting is catastrophic with respect to the loss, or when taking an expectation over the training set.
Paper Structure (24 sections, 18 theorems, 131 equations, 6 figures)

This paper contains 24 sections, 18 theorems, 131 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Why min norm?
  3. Noisy interpolation learning.
  4. Linear Splines.
  5. From Splines to Min-Norm ReLU Nets.
  6. Effect of Overfitting on $L_p$ Risk.
  7. Convergence vs. Expectation.
  8. I.I.D. Samples vs. Samples on a Grid.
  9. Discussion and Takeaways.
  10. Comparison with concurrent work.
  11. Review: Min-Norm ReLU Networks
  12. Warm up: tempered overfitting in linear-spline interpolation
  13. Delicate behavior of linear splines.
  14. Min-norm interpolation with random data
  15. Characterizing the min-norm interpolator
  16. ...and 9 more sections

Key Result

Lemma 2.1

For $0 \leq x_1<x_2<\cdots<x_n$, the problem in equation eq:main_problem admits a unique minimizer of the form: where $\tau_i \in [x_i,x_{i+1})$ for every $i \in [n-1]$.

Figures (6)

  • Figure 1: Comparison between linear-spline (purple) and min-norm (green) interpolators.
  • Figure 2: The min-norm interpolator for $30$ random points with $f^* \equiv 0$ and $\mathcal{N}(0,1)$ label noise.
  • Figure 3: An illustration of the linear spline interpolator $\hat{g}_S$ (left), and of the variant $\hat{h}_S$ where linear pieces are extended beyond the endpoints (right).
  • Figure 4: An illustration of the characterization of $\hat{f}_S$ from Lemma \ref{['lem:complete-description-property-main_text']}.
  • Figure 5: An illustration of the spike formed by Lemma \ref{['lem:sparsity-main_text']}. Here, $x_2$ and $x_4$ are two consecutive special points with exactly one point in between. There must be exactly one kink in $(x_1,x_4)$. Thus, in $[x_2,x_3)$, the interpolator $\hat{f}_S$ must be $\min\{g_1(x),g_3(x)\}$.
  • ...and 1 more figures

Theorems & Definitions (40)

  • Lemma 2.1: boursier2023penalising
  • Theorem 1
  • Definition 4.1
  • Lemma 4.2
  • Definition 4.3
  • Lemma 4.4: boursier2023penalising
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • ...and 30 more