Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One-Step Early Stopping Strategy using Neural Tangent Kernel Theory and Rademacher Complexity

Daniel Martin Xavier, Ludovic Chamoin, Jawher Jerray, Laurent Fribourg

TL;DR

The paper tackles how to stop training a neural network that imitates an MPC controller at an optimal time, by deriving a computable one-step stopping rule and a rigorous upper bound on the population loss. It fuses Neural Tangent Kernel (NTK) theory with Rademacher complexity to bound generalization and express the population loss after a single gradient-descent step in terms of the initial error and NTK eigenvalues, particularly in the underparameterized regime. The main contributions are explicit formulas for the one-step stopping time ${t_1=t_0+\eta_1}$ with ${\eta_1=\beta/\lambda_1^-}$, and an upper bound ${L_{\mathcal{D}}(t_1) \le \Omega_1 + 3M_1^2 \sqrt{\frac{\log(2/\delta)}{2n}}}$, where ${\Omega_1}$ is computable from NTK quantities; these results are illustrated on a Van der Pol oscillator MPC surrogate. The work provides a principled mechanism to mitigate overfitting and offers insights into benign overfitting in overparameterized regimes, while recognizing limitations to single-hidden-layer architectures and pointing to future extensions to deeper networks and multi-dimensional outputs.

Abstract

The early stopping strategy consists in stopping the training process of a neural network (NN) on a set $S$ of input data before training error is minimal. The advantage is that the NN then retains good generalization properties, i.e. it gives good predictions on data outside $S$, and a good estimate of the statistical error (``population loss'') is obtained. We give here an analytical estimation of the optimal stopping time involving basically the initial training error vector and the eigenvalues of the ``neural tangent kernel''. This yields an upper bound on the population loss which is well-suited to the underparameterized context (where the number of parameters is moderate compared with the number of data). Our method is illustrated on the example of an NN simulating the MPC control of a Van der Pol oscillator.

One-Step Early Stopping Strategy using Neural Tangent Kernel Theory and Rademacher Complexity

TL;DR

The paper tackles how to stop training a neural network that imitates an MPC controller at an optimal time, by deriving a computable one-step stopping rule and a rigorous upper bound on the population loss. It fuses Neural Tangent Kernel (NTK) theory with Rademacher complexity to bound generalization and express the population loss after a single gradient-descent step in terms of the initial error and NTK eigenvalues, particularly in the underparameterized regime. The main contributions are explicit formulas for the one-step stopping time with , and an upper bound , where is computable from NTK quantities; these results are illustrated on a Van der Pol oscillator MPC surrogate. The work provides a principled mechanism to mitigate overfitting and offers insights into benign overfitting in overparameterized regimes, while recognizing limitations to single-hidden-layer architectures and pointing to future extensions to deeper networks and multi-dimensional outputs.

Abstract

The early stopping strategy consists in stopping the training process of a neural network (NN) on a set of input data before training error is minimal. The advantage is that the NN then retains good generalization properties, i.e. it gives good predictions on data outside , and a good estimate of the statistical error (``population loss'') is obtained. We give here an analytical estimation of the optimal stopping time involving basically the initial training error vector and the eigenvalues of the ``neural tangent kernel''. This yields an upper bound on the population loss which is well-suited to the underparameterized context (where the number of parameters is moderate compared with the number of data). Our method is illustrated on the example of an NN simulating the MPC control of a Van der Pol oscillator.

Paper Structure

This paper contains 11 sections, 8 theorems, 64 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Gradient descent and training error
  4. Rademacher complexity and generalization error
  5. Sketch of the method
  6. Upper Bound on the Population Loss after One GD Step
  7. Final Remarks
  8. Appendix: Summary of Definitions
  9. Appendix: Proof of Proposition \ref{['lemma:1']}
  10. Appendix: Proof of Theorem \ref{['th:gap']}
  11. Appendix: Proof of Proposition \ref{['prop:2']}

Key Result

Proposition 1

(cf. Theorem 11.3, p. 270 of ChenM2019) With probability at least $1-\delta$ over a sample $S$ of size $n$: where

Figures (2)

  • Figure 1: Curves $\Omega$, $\frac{\|\boldsymbol{v}\|^2}{n}$, $\omega$, $\nu$, $L_G$ for $t\leq t_1=0.024$
  • Figure 2: Curves $\Omega$ (for $t\leq t_1=0.024$) and $L_{test}$

Theorems & Definitions (23)

  • Proposition 1
  • Proposition 2
  • proof
  • Proposition 3
  • Definition 1
  • Remark 1
  • Remark 2
  • Definition 2
  • Proposition 4
  • proof
  • ...and 13 more