Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Functional Large Deviations for Wide Deep Neural Networks with Gaussian Initialization and Lipschitz Activations

Claudio Macci, Barbara Pacchiarotti, Katerina Papagiannouli, Giovanni Luca Torrisi, Dario Trevisan

TL;DR

This work derives a functional large deviation principle for deep, wide neural networks with Gaussian (LeCun) initialization and Lipschitz activations, including ReLU, over any compact input set. The authors develop a layer-wise inductive framework: they first prove exponential tightness, then establish finite-dimensional LDPs for fixed input ensembles via Chaganty’s lemma and Cramér’s theorem, and finally lift these to a process-level LDP using the Dawson–Gärtner theorem. Central to the analysis are the layer-wise post-activation kernels $v^{(\ell)}(\mathcal{X})$ and a conditional Gaussian structure that enables a recursive rate-function formulation, culminating in a functional LDP for $\frac{1}{\sqrt{n}} f^{(L)}_{(n)}$ on $C(\mathcal{X}, \mathbb{R}^{d_{out}})$. The results extend prior finite-input and bounded-activation LDPs to a broad, practically relevant setting, providing a rigorous basis for rare-event analysis in deep Gaussian networks with ReLU-like activations. The framework opens avenues for studying large deviations during training and for extending to more general architectures via tensor-programming techniques.

Abstract

We establish a functional large deviation principle for fully connected multi-layer perceptrons with i.i.d. Gaussian weights (LeCun initialization) and general Lipschitz activation functions, including therefore the popular case of ReLU. The large deviation principle holds for the entire network output process on any compact input set. The proof combines exponential tightness for recursively defined processes, finite-dimensional large deviations, and the Dawson-Gärtner theorem, extending existing results beyond finite input sets and less general activations.

Functional Large Deviations for Wide Deep Neural Networks with Gaussian Initialization and Lipschitz Activations

TL;DR

This work derives a functional large deviation principle for deep, wide neural networks with Gaussian (LeCun) initialization and Lipschitz activations, including ReLU, over any compact input set. The authors develop a layer-wise inductive framework: they first prove exponential tightness, then establish finite-dimensional LDPs for fixed input ensembles via Chaganty’s lemma and Cramér’s theorem, and finally lift these to a process-level LDP using the Dawson–Gärtner theorem. Central to the analysis are the layer-wise post-activation kernels and a conditional Gaussian structure that enables a recursive rate-function formulation, culminating in a functional LDP for on . The results extend prior finite-input and bounded-activation LDPs to a broad, practically relevant setting, providing a rigorous basis for rare-event analysis in deep Gaussian networks with ReLU-like activations. The framework opens avenues for studying large deviations during training and for extending to more general architectures via tensor-programming techniques.

Abstract

We establish a functional large deviation principle for fully connected multi-layer perceptrons with i.i.d. Gaussian weights (LeCun initialization) and general Lipschitz activation functions, including therefore the popular case of ReLU. The large deviation principle holds for the entire network output process on any compact input set. The proof combines exponential tightness for recursively defined processes, finite-dimensional large deviations, and the Dawson-Gärtner theorem, extending existing results beyond finite input sets and less general activations.
Paper Structure (13 sections, 13 theorems, 111 equations)

This paper contains 13 sections, 13 theorems, 111 equations.

Table of Contents

  1. Introduction
  2. Main result and outline of its proof
  3. Relation to the literature.
  4. Perspectives
  5. Notation and basic facts
  6. General notation
  7. Convex analysis
  8. Neural networks
  9. Large deviations
  10. Exponential tightness
  11. Large deviations
  12. Acknowledgements
  13. Proof of Theorem \ref{['thm:schied']}

Key Result

Theorem 1.1

Fix a depth $L \ge 1$, input $d_{\operatorname{in}}$, and output $d_{\operatorname{out}}\ge 1$ dimensions, Lipschitz activation functions $(\sigma^{(\ell)})_{\ell=1,\ldots, L}$, and consider a sequence of outputs $f^{(L)}_{(n)}: \mathbb{R}^{d_{\operatorname{in}}} \to \mathbb{R}^{d_{\operatorname{out and i.i.d. Gaussian weights as in eq:lecun -- also known as LeCun initialization. Then, for every c

Theorems & Definitions (25)

  • Theorem 1.1
  • Theorem 2.1: epi-continuity of Legendre-Fenchel transform
  • Remark 2.2
  • Remark 2.3
  • Remark 2.4
  • Remark 2.5
  • Proposition 2.6: Contraction principle
  • Theorem 2.7: Cramér's theorem
  • Proposition 2.8: Exponential equivalence
  • Remark 2.9
  • ...and 15 more