Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wide neural networks with general weights: convergence rate and explicit dependence on the hyper-parameters

Lucia Celli

TL;DR

This work develops quantitative central limit theorems for fully connected neural networks with Lipschitz activations under general (non-Gaussian) weight initialization. Using a discrete, copy-based Stein method, the authors derive explicit convergence bounds to Gaussian limits as hidden widths grow, both for a single input and for finite input sets, with rates of order $O\big(n^{-1/2}\big)$ under broad conditions. They also establish a joint infinite-width/depth regime, yielding explicit bounds that hold when depth grows slowly with width, and provide a non-degeneracy analysis via a lower bound on the limiting covariance determinant. The results generalize previous Gaussian-initialization findings to non-Gaussian weights, and they deliver fully explicit constants depending on the network's hyper-parameters, which is valuable for understanding training dynamics and NTK behavior in the wide/deep regime. Overall, the paper advances the theory of Gaussian fluctuations in deep networks beyond Gaussian initialization while offering practically computable error bounds and insights into the role of depth in the infinite-width limit.

Abstract

Using Stein's method techniques introduced by Chatterjee (2008) and further extended by Kasprzak and Peccati (2022) and by Lachièze-Rey and Peccati (2017), we derive novel quantitative bounds on the convergence in distribution of feed-forward fully connected neural networks (with Lipschitz activation functions) towards Gaussian processes, as the hidden layer width $n$ tends to infinity. We consider networks initialized with independent and identically distributed (i.i.d.) weights possessing sufficiently many finite moments, and i.i.d. Gaussian biases independent of the weights. Specifically, when the network is evaluated at a single input, we obtain convergence rates of order $O(n^{-1/2})$ in both total variation and Wasserstein distances. When evaluated at a general finite collection of inputs, we establish bounds of the same order in terms of the convex distance. All bounds are given in explicit and computable form. As a consequence of our estimates, we also deduce a novel convergence result in the regime where the depth of the neural network increases simultaneously with the width $n$, up to order $O\big((\log_2 n)^{1/3}\big)$. To the best of our knowledge, this is the first CLT in the infinite width/depth limit holding for general (nonlinear) Lipschitz activation functions and non-Gaussian weight distributions. Our analysis yields several results of independent interest, including: (i) an explicit lower bound on the determinant of the limiting covariance matrix and (ii) new advances in Stein's method, both for the one-dimensional Stein's equation associated with the square of a Lipschitz function and for the multivariate Stein's equation associated with the tensor product of a Lipschitz function with itself.

Wide neural networks with general weights: convergence rate and explicit dependence on the hyper-parameters

TL;DR

This work develops quantitative central limit theorems for fully connected neural networks with Lipschitz activations under general (non-Gaussian) weight initialization. Using a discrete, copy-based Stein method, the authors derive explicit convergence bounds to Gaussian limits as hidden widths grow, both for a single input and for finite input sets, with rates of order under broad conditions. They also establish a joint infinite-width/depth regime, yielding explicit bounds that hold when depth grows slowly with width, and provide a non-degeneracy analysis via a lower bound on the limiting covariance determinant. The results generalize previous Gaussian-initialization findings to non-Gaussian weights, and they deliver fully explicit constants depending on the network's hyper-parameters, which is valuable for understanding training dynamics and NTK behavior in the wide/deep regime. Overall, the paper advances the theory of Gaussian fluctuations in deep networks beyond Gaussian initialization while offering practically computable error bounds and insights into the role of depth in the infinite-width limit.

Abstract

Using Stein's method techniques introduced by Chatterjee (2008) and further extended by Kasprzak and Peccati (2022) and by Lachièze-Rey and Peccati (2017), we derive novel quantitative bounds on the convergence in distribution of feed-forward fully connected neural networks (with Lipschitz activation functions) towards Gaussian processes, as the hidden layer width tends to infinity. We consider networks initialized with independent and identically distributed (i.i.d.) weights possessing sufficiently many finite moments, and i.i.d. Gaussian biases independent of the weights. Specifically, when the network is evaluated at a single input, we obtain convergence rates of order in both total variation and Wasserstein distances. When evaluated at a general finite collection of inputs, we establish bounds of the same order in terms of the convex distance. All bounds are given in explicit and computable form. As a consequence of our estimates, we also deduce a novel convergence result in the regime where the depth of the neural network increases simultaneously with the width , up to order . To the best of our knowledge, this is the first CLT in the infinite width/depth limit holding for general (nonlinear) Lipschitz activation functions and non-Gaussian weight distributions. Our analysis yields several results of independent interest, including: (i) an explicit lower bound on the determinant of the limiting covariance matrix and (ii) new advances in Stein's method, both for the one-dimensional Stein's equation associated with the square of a Lipschitz function and for the multivariate Stein's equation associated with the tensor product of a Lipschitz function with itself.
Paper Structure (39 sections, 41 theorems, 413 equations)

This paper contains 39 sections, 41 theorems, 413 equations.

Table of Contents

  1. Introduction
  2. Overview
  3. Structure of the paper
  4. Preliminaries
  5. Random Fully Connected Neural Networks
  6. Probability metrics
  7. Stein's method
  8. Main results
  9. Literature review
  10. One-dimensional case (Gaussian initialization)
  11. Multidimensional Case (Gaussian initialization)
  12. Non-Gaussian initialization
  13. Infinite widths and depth
  14. Minimum eigenvalue of the limiting covariance matrix
  15. Proof's scheme
  16. ...and 24 more sections

Key Result

Theorem 1

Let $z^{(L+1)}(\mathcal{X})$ be the output of a fully connected feedforward neural network evaluated on a finite set of distinct inputs $\mathcal{X} = \{x^{(1)}, \dots, x^{(d)}\}$. Assume the network has i.i.d. Gaussian biases with variance $C_b \neq 0$, i.i.d. weights with sufficiently many finite Furthermore, the constants involved in the above bounds are explicitly given in terms of the neural

Theorems & Definitions (91)

  • Theorem 1: Informal version of Theorems \ref{['finale_uno_1']} and \ref{['pres_sec_prob']}
  • Remark 1
  • Corollary 1
  • Definition 1: Random Fully Connected Neural Network
  • Remark 2
  • Theorem 2: Theorem 1.2 in Han22
  • Definition 2
  • Definition 3: Multi-dimensional Kolmogorov distance
  • Definition 4: $p$-Wasserstein Distance NP12villani
  • Proposition 1: Kantorovich–Rubinstein Duality villani
  • ...and 81 more