Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

Hossein Taheri, Christos Thrampoulidis, Arya Mazumdar

TL;DR

This work tackles the challenge of providing tight, data-dependent guarantees for gradient-based learning of deep neural networks with smooth activations. It introduces an algorithmic-stability framework that leverages the Hessian along gradient-descent trajectories to bound both training and test losses, producing bounds that depend on the distance from initialization rather than width, and which can be polylogarithmic in the sample size. The authors show that under Neural Tangent Kernel (NTK) separability with margin $\gamma$, the test loss scales as $\widetilde{O}(e^{O(L)}/(\gamma^2 n))$ and, with a large step-size, can surpass NTK limitations, enabling efficient learning of the XOR distribution with a constant-width network in $\log(d)$ iterations. They also establish that, in the presence of label noise, gradient descent can still achieve the optimal excess risk with polynomial width, and provide concrete results for XOR that demonstrate substantial computational and sample efficiency gains. Overall, the paper delivers tighter, initialization-aware guarantees and practical guidance on step-size regimes that can dramatically improve learning efficiency beyond the traditional NTK regime.

Abstract

In this paper, we study the data-dependent convergence and generalization behavior of gradient methods for neural networks with smooth activation. Our first result is a novel bound on the excess risk of deep networks trained by the logistic loss, via an alogirthmic stability analysis. Compared to previous works, our results improve upon the shortcomings of the well-established Rademacher complexity-based bounds. Importantly, the bounds we derive in this paper are tighter, hold even for neural networks of small width, do not scale unfavorably with width, are algorithm-dependent, and consequently capture the role of initialization on the sample complexity of gradient descent for deep nets. Specialized to noiseless data separable with margin $γ$ by neural tangent kernel (NTK) features of a network of width $Ω(\text{poly}(\log(n)))$, we show the test-error rate to be $e^{O(L)}/{γ^2 n}$, where $n$ is the training set size and $L$ denotes the number of hidden layers. This is an improvement in the test loss bound compared to previous works while maintaining the poly-logarithmic width conditions. We further investigate excess risk bounds for deep nets trained with noisy data, establishing that under a polynomial condition on the network width, gradient descent can achieve the optimal excess risk. Finally, we show that a large step-size significantly improves upon the NTK regime's results in classifying the XOR distribution. In particular, we show for a one-hidden-layer neural network of constant width $m$ with quadratic activation and standard Gaussian initialization that mini-batch SGD with linear sample complexity and with a large step-size $η=m$ reaches the perfect test accuracy after only $\ceil{\log(d)}$ iterations, where $d$ is the data dimension.

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

TL;DR

This work tackles the challenge of providing tight, data-dependent guarantees for gradient-based learning of deep neural networks with smooth activations. It introduces an algorithmic-stability framework that leverages the Hessian along gradient-descent trajectories to bound both training and test losses, producing bounds that depend on the distance from initialization rather than width, and which can be polylogarithmic in the sample size. The authors show that under Neural Tangent Kernel (NTK) separability with margin , the test loss scales as and, with a large step-size, can surpass NTK limitations, enabling efficient learning of the XOR distribution with a constant-width network in iterations. They also establish that, in the presence of label noise, gradient descent can still achieve the optimal excess risk with polynomial width, and provide concrete results for XOR that demonstrate substantial computational and sample efficiency gains. Overall, the paper delivers tighter, initialization-aware guarantees and practical guidance on step-size regimes that can dramatically improve learning efficiency beyond the traditional NTK regime.

Abstract

In this paper, we study the data-dependent convergence and generalization behavior of gradient methods for neural networks with smooth activation. Our first result is a novel bound on the excess risk of deep networks trained by the logistic loss, via an alogirthmic stability analysis. Compared to previous works, our results improve upon the shortcomings of the well-established Rademacher complexity-based bounds. Importantly, the bounds we derive in this paper are tighter, hold even for neural networks of small width, do not scale unfavorably with width, are algorithm-dependent, and consequently capture the role of initialization on the sample complexity of gradient descent for deep nets. Specialized to noiseless data separable with margin by neural tangent kernel (NTK) features of a network of width , we show the test-error rate to be , where is the training set size and denotes the number of hidden layers. This is an improvement in the test loss bound compared to previous works while maintaining the poly-logarithmic width conditions. We further investigate excess risk bounds for deep nets trained with noisy data, establishing that under a polynomial condition on the network width, gradient descent can achieve the optimal excess risk. Finally, we show that a large step-size significantly improves upon the NTK regime's results in classifying the XOR distribution. In particular, we show for a one-hidden-layer neural network of constant width with quadratic activation and standard Gaussian initialization that mini-batch SGD with linear sample complexity and with a large step-size reaches the perfect test accuracy after only iterations, where is the data dimension.

Paper Structure

This paper contains 33 sections, 15 theorems, 140 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Overview
  3. Prior works
  4. Generalization of deep nets.
  5. Test rates under the NTK separability condition.
  6. Feature learning and the XOR distribution.
  7. Main results
  8. Train and test loss bounds in deep nets
  9. Specializing to the NTK-separablity condition
  10. Consistency of GD with noisy data
  11. Overcoming NTK limitations for XOR dataset
  12. Numerical Results
  13. Experiments on learning under NTK with small step-size.
  14. Experiments on learning the XOR distribution with large step-size.
  15. Conclusions and Future Directions
  16. ...and 18 more sections

Key Result

Theorem 2.1

Consider the $L$-layer neural network with width $m$ where $\Phi(w,x):= \frac{1}{\sqrt m} W_{L+1}^\top(\frac{1}{\sqrt m} \sigma(W_{L}^\top \cdots \frac{1}{\sqrt{m}}\sigma (W_1^\top x)\cdots)$ and $\sigma$ is a $1$-smooth and $1$-Lipschitz activation function such that $\sigma(0)=0$. Moreover, let $ where $\rho^\star:=\|w^\star-w_0\|$. Moreover, assume the width $m$ is large enough such that it sa

Figures (2)

  • Figure 1: Iteration-based distance from initialization ($\|w_t-w_0\|$), training loss, test loss and generalization gap (i.e., test loss -- train loss) for training a two hidden-layer neural network with FashionMNIST dataset and two choices of step-size. Here $n=12\times 10^3, m=500,$ and total number of parameters $p \approx 6 \times 10^5$.
  • Figure 4: Left: Misclassification error based on iteration in learning the $d-$dimensional XOR distribution with SGD. Right: Total number of SGD steps based on data dimension to reach approximately zero test error.

Theorems & Definitions (21)

  • Theorem 2.1: Train & Test loss of deep nets
  • Remark 2.2
  • Corollary 1: NTK results
  • Theorem 2.3: Test error for noisy data
  • Theorem 2.4: Improved guarantees for learning XOR
  • Remark 2.5
  • Lemma 1: Hessian's spectral norm liu2020linearity
  • Lemma 2
  • proof
  • Theorem B.1: Convergence of GD in deep nets
  • ...and 11 more