Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large deviation principles and functional limit theorems in the deep limit of wide random neural networks

Simmaco Di Lillo, Claudio Macci, Barbara Pacchiarotti

TL;DR

This work extends large deviation and weak convergence theory to the recursive covariance structure arising from deep, wide random neural networks mapped to isotropic Gaussian fields on the sphere. By partitioning depth-by-activation dynamics into three regimes via κ'(1), it derives finite-dimensional LDPs under two natural centerings and clarifies when functional LDPs and functional convergence hold or fail. In the low-disorder regime, the authors obtain both functional LDPs and weak convergence to a continuous Gaussian limit, while in the sparse regime functional results fail due to covariance discontinuities, revealing a sharp depth-induced degeneration mechanism. The high-disorder regime is discussed to emphasize qualitatively different long-range behavior and the absence of corresponding functional limit theorems. Collectively, the results quantify how depth and initialization govern the probabilistic limits of deep, wide neural networks through the spectral properties of the iterated covariance kernel.

Abstract

This paper studies large deviation principles and weak convergence, both at the level of finite-dimensional distributions and in functional form, for a class of continuous, isotropic, centered Gaussian random fields defined on the unit sphere. The covariance functions of these fields evolve recursively through a nonlinear map induced by an activation function, reflecting the statistical dynamics of infinitely wide random neural networks as depth increases. We consider two types of centered fields, obtained by subtracting either the value at the North Pole or the spherical average. According to the behavior of the derivative at $t=1$ of the associated covariance function, we identify three regimes: low disorder, sparse, and high disorder. In the low-disorder regime, we establish functional large deviation principles and weak convergence results. In the sparse regime, we obtain large deviation principles and weak convergence for finite-dimensional distributions, while both properties fail at the functional level sense due to the emergence of discontinuities in the covariance recursion.

Large deviation principles and functional limit theorems in the deep limit of wide random neural networks

TL;DR

This work extends large deviation and weak convergence theory to the recursive covariance structure arising from deep, wide random neural networks mapped to isotropic Gaussian fields on the sphere. By partitioning depth-by-activation dynamics into three regimes via κ'(1), it derives finite-dimensional LDPs under two natural centerings and clarifies when functional LDPs and functional convergence hold or fail. In the low-disorder regime, the authors obtain both functional LDPs and weak convergence to a continuous Gaussian limit, while in the sparse regime functional results fail due to covariance discontinuities, revealing a sharp depth-induced degeneration mechanism. The high-disorder regime is discussed to emphasize qualitatively different long-range behavior and the absence of corresponding functional limit theorems. Collectively, the results quantify how depth and initialization govern the probabilistic limits of deep, wide neural networks through the spectral properties of the iterated covariance kernel.

Abstract

This paper studies large deviation principles and weak convergence, both at the level of finite-dimensional distributions and in functional form, for a class of continuous, isotropic, centered Gaussian random fields defined on the unit sphere. The covariance functions of these fields evolve recursively through a nonlinear map induced by an activation function, reflecting the statistical dynamics of infinitely wide random neural networks as depth increases. We consider two types of centered fields, obtained by subtracting either the value at the North Pole or the spherical average. According to the behavior of the derivative at of the associated covariance function, we identify three regimes: low disorder, sparse, and high disorder. In the low-disorder regime, we establish functional large deviation principles and weak convergence results. In the sparse regime, we obtain large deviation principles and weak convergence for finite-dimensional distributions, while both properties fail at the functional level sense due to the emergence of discontinuities in the covariance recursion.
Paper Structure (19 sections, 9 theorems, 104 equations)

This paper contains 19 sections, 9 theorems, 104 equations.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Large deviations
  4. Isotropic random fields on the sphere
  5. Random neural networks
  6. Three regimes of spectral complexity
  7. Results for finite-dimensional distributions
  8. A possible link between the sequences in \ref{['lem:statement']}
  9. Some examples
  10. Functional results
  11. Some remarks on the high-disorder regime kappa'(1)>1
  12. Proofs
  13. Proof of Lemma \ref{['lem:set-of-attractive-points-of-iterations']}
  14. Proof of Lemma \ref{['lem:low-disorder']}
  15. Proof of Lemma \ref{['lem:sparse']}
  16. ...and 4 more sections

Key Result

Lemma 2.1

If $\kappa^\prime(1)\le 1$, then the following sequences converge to zero in $L^2(\Omega)$, as $L\to\infty$, for each fixed $x\in\mathbb{S}^d$:

Theorems & Definitions (22)

  • Remark 2.1
  • Lemma 2.1
  • proof
  • Remark 3.1
  • Theorem 3.1
  • Theorem 3.2
  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Remark 3.2: Assumption \ref{['ass:A']} avoids triviality in the low-disorder case
  • ...and 12 more