Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regression

Yingzhen Yang, Ping Li

TL;DR

This work addresses nonparametric regression when training an over-parameterized two-layer neural network with gradient descent and early stopping. By leveraging Neural Tangent Kernel (NTK) theory and local Rademacher complexity, it establishes a distribution-free sharp risk bound of $O(\varepsilon_n^2)$, where $\varepsilon_n$ is the NTK’s critical radius, and characterizes the stopping time $\widehat{T}=\Theta(\varepsilon_n^{-2})$ and width requirements. Under polynomial eigenvalue decay, the results imply minimax-optimal rates $O(n^{-2\alpha/(2\alpha+1)})$, with explicit width scaling $m\gtrsim n^{25\alpha/(2\alpha+1)}d^{5/2}$ and a constant learning rate. The analysis provides a conceptual and technical bridge between kernel regression and finite-width neural networks trained by GD, without distributional assumptions on the covariate beyond boundedness, with practical implications for early-stopping and network sizing. Overall, the paper delivers distribution-free, minimax-like guarantees for nonparametric regression using a practical, shallow NN architecture trained by gradient methods.

Abstract

We study nonparametric regression by an over-parameterized two-layer neural network trained by gradient descent (GD) in this paper. We show that, if the neural network is trained by GD with early stopping, then the trained network renders a sharp rate of the nonparametric regression risk of $\mathcal{O}(ε_n^2)$, which is the same rate as that for the classical kernel regression trained by GD with early stopping, where $ε_n$ is the critical population rate of the Neural Tangent Kernel (NTK) associated with the network and $n$ is the size of the training data. It is remarked that our result does not require distributional assumptions about the covariate as long as the covariate is bounded, in a strong contrast with many existing results which rely on specific distributions of the covariates such as the spherical uniform data distribution or distributions satisfying certain restrictive conditions. The rate $\mathcal{O}(ε_n^2)$ is known to be minimax optimal for specific cases, such as the case that the NTK has a polynomial eigenvalue decay rate which happens under certain distributional assumptions on the covariates. Our result formally fills the gap between training a classical kernel regression model and training an over-parameterized but finite-width neural network by GD for nonparametric regression without distributional assumptions on the bounded covariate. We also provide confirmative answers to certain open questions or address particular concerns in the literature of training over-parameterized neural networks by GD with early stopping for nonparametric regression, including the characterization of the stopping time, the lower bound for the network width, and the constant learning rate used in GD.

Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regression

TL;DR

This work addresses nonparametric regression when training an over-parameterized two-layer neural network with gradient descent and early stopping. By leveraging Neural Tangent Kernel (NTK) theory and local Rademacher complexity, it establishes a distribution-free sharp risk bound of , where is the NTK’s critical radius, and characterizes the stopping time and width requirements. Under polynomial eigenvalue decay, the results imply minimax-optimal rates , with explicit width scaling and a constant learning rate. The analysis provides a conceptual and technical bridge between kernel regression and finite-width neural networks trained by GD, without distributional assumptions on the covariate beyond boundedness, with practical implications for early-stopping and network sizing. Overall, the paper delivers distribution-free, minimax-like guarantees for nonparametric regression using a practical, shallow NN architecture trained by gradient methods.

Abstract

We study nonparametric regression by an over-parameterized two-layer neural network trained by gradient descent (GD) in this paper. We show that, if the neural network is trained by GD with early stopping, then the trained network renders a sharp rate of the nonparametric regression risk of , which is the same rate as that for the classical kernel regression trained by GD with early stopping, where is the critical population rate of the Neural Tangent Kernel (NTK) associated with the network and is the size of the training data. It is remarked that our result does not require distributional assumptions about the covariate as long as the covariate is bounded, in a strong contrast with many existing results which rely on specific distributions of the covariates such as the spherical uniform data distribution or distributions satisfying certain restrictive conditions. The rate is known to be minimax optimal for specific cases, such as the case that the NTK has a polynomial eigenvalue decay rate which happens under certain distributional assumptions on the covariates. Our result formally fills the gap between training a classical kernel regression model and training an over-parameterized but finite-width neural network by GD for nonparametric regression without distributional assumptions on the bounded covariate. We also provide confirmative answers to certain open questions or address particular concerns in the literature of training over-parameterized neural networks by GD with early stopping for nonparametric regression, including the characterization of the stopping time, the lower bound for the network width, and the constant learning rate used in GD.

Paper Structure

This paper contains 24 sections, 30 theorems, 18 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Two-Layer Neural Network
  4. Kernel and Kernel Regression for Nonparametric Regression
  5. Summary of Main Results
  6. Training by Gradient Descent and Preconditioned Gradient Descent
  7. Main Results
  8. Kernel Complexity
  9. Main Results
  10. Roadmap of Proofs
  11. Uniform Convergence to the NTK (\ref{['eq:kernel-two-layer']}) and More
  12. Basic Definitions
  13. Detailed Roadmap and Key Results
  14. Summary of the approaches and key technical results in the proofs.
  15. Key Technical Results
  16. ...and 9 more sections

Key Result

Theorem 5.1

Suppose that ${\cal X}$ is bounded and $n \gtrsim \max\left\{1/\lambda_1,\sigma_0^2 u_0^2/2\right\}$. Let $c_T, c_t \in (0,1]$ be arbitrary positive constants, and $c_T \widehat{T} \le T \le \widehat{T}$. Suppose $m$ satisfies m ≳ d^52ɛ_n^25, and the neural network $f(\mathbf{W}(t),\cdot)$ is trai

Figures (2)

  • Figure 1: Roadmap of key results leading to the main result, Theorem \ref{['theorem:LRC-population-NN-fixed-point']}. The uniform convergence results in Theorem \ref{['theorem:good-random-initialization']} are used in all the optimization results and Theorem \ref{['theorem:bounded-NN-class']}.
  • Figure 2: Illustration of the test loss by GD and the ratio of early stopping time.

Theorems & Definitions (64)

  • Theorem 5.1
  • Corollary 5.2: Applying Theorem \ref{['theorem:LRC-population-NN-fixed-point']} to the special case of polynomial EDR
  • Theorem 6.1
  • proof
  • Definition 6.1
  • Theorem 6.2
  • Theorem 6.3
  • Remark
  • Lemma 6.4
  • Theorem 6.5
  • ...and 54 more