Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benign overfitting in leaky ReLU networks with moderate input dimension

Kedar Karhadkar, Erin George, Michael Murray, Guido Montúfar, Deanna Needell

TL;DR

This work describes conditions on the signal to noise ratio (SNR) of the model parameters giving rise to benign versus non-benign (or harmful) overfitting and attributes both benign and non-benign overfitting to an approximate margin maximization property.

Abstract

The problem of benign overfitting asks whether it is possible for a model to perfectly fit noisy training data and still generalize well. We study benign overfitting in two-layer leaky ReLU networks trained with the hinge loss on a binary classification task. We consider input data that can be decomposed into the sum of a common signal and a random noise component, that lie on subspaces orthogonal to one another. We characterize conditions on the signal to noise ratio (SNR) of the model parameters giving rise to benign versus non-benign (or harmful) overfitting: in particular, if the SNR is high then benign overfitting occurs, conversely if the SNR is low then harmful overfitting occurs. We attribute both benign and non-benign overfitting to an approximate margin maximization property and show that leaky ReLU networks trained on hinge loss with gradient descent (GD) satisfy this property. In contrast to prior work we do not require the training data to be nearly orthogonal. Notably, for input dimension $d$ and training sample size $n$, while results in prior work require $d = Ω(n^2 \log n)$, here we require only $d = Ω\left(n\right)$.

Benign overfitting in leaky ReLU networks with moderate input dimension

TL;DR

This work describes conditions on the signal to noise ratio (SNR) of the model parameters giving rise to benign versus non-benign (or harmful) overfitting and attributes both benign and non-benign overfitting to an approximate margin maximization property.

Abstract

The problem of benign overfitting asks whether it is possible for a model to perfectly fit noisy training data and still generalize well. We study benign overfitting in two-layer leaky ReLU networks trained with the hinge loss on a binary classification task. We consider input data that can be decomposed into the sum of a common signal and a random noise component, that lie on subspaces orthogonal to one another. We characterize conditions on the signal to noise ratio (SNR) of the model parameters giving rise to benign versus non-benign (or harmful) overfitting: in particular, if the SNR is high then benign overfitting occurs, conversely if the SNR is low then harmful overfitting occurs. We attribute both benign and non-benign overfitting to an approximate margin maximization property and show that leaky ReLU networks trained on hinge loss with gradient descent (GD) satisfy this property. In contrast to prior work we do not require the training data to be nearly orthogonal. Notably, for input dimension and training sample size , while results in prior work require , here we require only .
Paper Structure (25 sections, 23 theorems, 216 equations)

This paper contains 25 sections, 23 theorems, 216 equations.

Table of Contents

  1. Introduction
  2. Summary of contributions
  3. Related work
  4. Preliminaries
  5. Data model
  6. Architecture and learning algorithm
  7. Approximate margin maximization
  8. Main results
  9. Approximate margin maximization and generalization: insight from linear models
  10. 1. Generalization bounds based on the SNR:
  11. 2. Upper bound the norm of the max-margin classifier:
  12. 3. Lower bound the SNR using the approximate margin maximization property:
  13. 4. Upper bound the SNR using the zero loss condition:
  14. 5. Upper bound the SNR using the zero loss condition and maximum margin property:
  15. From linear models to leaky ReLU networks
  16. ...and 10 more sections

Key Result

Theorem 3.1

Let $f:\mathbb{R}^p \times \mathbb{R}^n \rightarrow \mathbb{R}$ be a leaky ReLU network with forward pass as defined by equation eq:architecture. Suppose the step size $\eta$ and initialization condition $\lambda$ satisfy Assumption assumption:stepsize-init. Then for any linearly separable data set Furthermore $\mathcal{A}_{GD}$ is approximately margin maximizing on $f$ (Definition def:approx-max

Theorems & Definitions (46)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Theorem 3.1
  • Theorem 3.2
  • Corollary 3.2.1
  • Theorem 3.3
  • Theorem 3.4
  • Lemma 4.0
  • Lemma A.1
  • ...and 36 more