Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Double Descent Behavior in Two Layer Neural Network for Binary Classification

Chathurika S Abeykoon, Aleksandr Beknazaryan, Hailin Sang

TL;DR

This work investigates the double descent phenomenon in binary classification using a two-layer ReLU network under an over- and under-parameterized regime characterized by the ratio $\alpha=\frac{n}{d}$. It avoids explicit training dynamics by employing the Convex Gaussian Min-Max Theorem (CGMT) and Legendre transforms to derive asymptotic generalization error curves, providing fixed-point equations for key quantities $r^*,s^*,b^*,\gamma^*$ in the square-loss setting. The authors show that, at low regularization, the test error exhibits a second descent near the interpolation threshold, while increasing $\lambda=\lambda(d)$ mitigates or eliminates this peak, yielding monotone improvements with more data. The results offer a theoretical account of ratio-wise double descent and demonstrate how appropriate regularization can align the model’s behavior with classical bias-variance intuition, with implications for high-dimensional binary classification and the design of regularization strategies.

Abstract

Recent studies observed a surprising concept on model test error called the double descent phenomenon, where the increasing model complexity decreases the test error first and then the error increases and decreases again. To observe this, we work on a two layer neural network model with a ReLU activation function designed for binary classification under supervised learning. Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes. We quantify the model size by the ratio of number of training samples to the dimension of the model. Due to the complexity of the empirical risk minimization procedure, we use the Convex Gaussian Min Max Theorem to find a suitable candidate for the global training loss.

The Double Descent Behavior in Two Layer Neural Network for Binary Classification

TL;DR

This work investigates the double descent phenomenon in binary classification using a two-layer ReLU network under an over- and under-parameterized regime characterized by the ratio . It avoids explicit training dynamics by employing the Convex Gaussian Min-Max Theorem (CGMT) and Legendre transforms to derive asymptotic generalization error curves, providing fixed-point equations for key quantities in the square-loss setting. The authors show that, at low regularization, the test error exhibits a second descent near the interpolation threshold, while increasing mitigates or eliminates this peak, yielding monotone improvements with more data. The results offer a theoretical account of ratio-wise double descent and demonstrate how appropriate regularization can align the model’s behavior with classical bias-variance intuition, with implications for high-dimensional binary classification and the design of regularization strategies.

Abstract

Recent studies observed a surprising concept on model test error called the double descent phenomenon, where the increasing model complexity decreases the test error first and then the error increases and decreases again. To observe this, we work on a two layer neural network model with a ReLU activation function designed for binary classification under supervised learning. Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes. We quantify the model size by the ratio of number of training samples to the dimension of the model. Due to the complexity of the empirical risk minimization procedure, we use the Convex Gaussian Min Max Theorem to find a suitable candidate for the global training loss.
Paper Structure (27 sections, 3 theorems, 100 equations, 8 figures)

This paper contains 27 sections, 3 theorems, 100 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Various forms of double descent
  3. Our contribution
  4. Visualizing the double descent in binary classification models
  5. Two layer neural network with ReLU activation function
  6. Problem setup
  7. Student model
  8. Data generation - teacher model
  9. Loss and risk
  10. Asymptotic setting and fixed quantities
  11. Generalization
  12. Regularized empirical risk
  13. Regularized empirical risk minimization procedure
  14. Theoretical generalization error curves for square loss
  15. Effect of regularization on double descent behavior
  16. ...and 12 more sections

Key Result

Theorem 4.1

The test/generalization error of the two layer neural network model defined in (student) under the fixed quantities $r$ and $s$ introduced in (r and s) is given by, where $\Phi$ is the cumulative distribution function of standard normal distribution, $b$ is the bias term and $\rho_{-1}$ and $\rho_{1}$ relate to the probabilities of getting 1 or -1 in classification.

Figures (8)

  • Figure 1: The double descent phenomenon in highly over-parameterized models.
  • Figure 2: Test error showing the double descent behavior when a two-layer ReLu model is used with very low $l_2$ regularization $\lambda=10^{-6}$ for binary classification in Wisconsin Breast Cancer dataset. The test error decreases first and shows a slight increase around $\alpha=1$ and decreases again in the under-parameterized region.
  • Figure 3: Test error not showing the double descent behavior when a two-layer ReLu model is used with high $l_2$ regularization $\lambda=0.1$ for binary classification in Wisconsin Breast Cancer dataset. The test error decreases monotonically in both over and under parameterized regions.
  • Figure 4: Test error of model (\ref{['student']}) with square loss as a function of $\alpha$ with low regularization ($\lambda=10^{-5}$) and with $\rho_1=0.5$.
  • Figure 5: Test error of model (\ref{['student']}) for square loss as a function of $\alpha$ with $\rho_1=0.5$ and with varying regularization.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Theorem 4.1
  • proof
  • Theorem 5.1
  • Theorem 8.1
  • proof