Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian random field approximation via Stein's method with applications to wide random neural networks

Krishnakumar Balasubramanian, Larry Goldstein, Nathan Ross, Adil Salim

TL;DR

This work develops quantitative Gaussian random-field approximations for fields indexed by the sphere using Stein's method, focusing on the Wasserstein distance $W_1$ under the sup-norm. The authors introduce a novel Laplacian-based smoothing that yields an explicit Cameron–Martin space, enabling transfer from smooth metrics to the $W_1$ distance and extending prior one-dimensional smoothing techniques to sphere-indexed fields. They derive a master bound for the distance between a general random field $F$ and a Gaussian field $H$ on $\mathcal{S}^n$, and apply it to wide deep neural networks to obtain explicit, depth- and width-dependent $W_1$ bounds at the random-field level, under Lipschitz activations. They further show improved rates when activations are three-times differentiable and discuss comparisons to related works and potential future directions, including rate improvements and extensions to heavier-tailed weight distributions or other manifolds.

Abstract

We derive upper bounds on the Wasserstein distance ($W_1$), with respect to $\sup$-norm, between any continuous $\mathbb{R}^d$ valued random field indexed by the $n$-sphere and the Gaussian, based on Stein's method. We develop a novel Gaussian smoothing technique that allows us to transfer a bound in a smoother metric to the $W_1$ distance. The smoothing is based on covariance functions constructed using powers of Laplacian operators, designed so that the associated Gaussian process has a tractable Cameron-Martin or Reproducing Kernel Hilbert Space. This feature enables us to move beyond one dimensional interval-based index sets that were previously considered in the literature. Specializing our general result, we obtain the first bounds on the Gaussian random field approximation of wide random neural networks of any depth and Lipschitz activation functions at the random field level. Our bounds are explicitly expressed in terms of the widths of the network and moments of the random weights. We also obtain tighter bounds when the activation function has three bounded derivatives.

Gaussian random field approximation via Stein's method with applications to wide random neural networks

TL;DR

This work develops quantitative Gaussian random-field approximations for fields indexed by the sphere using Stein's method, focusing on the Wasserstein distance under the sup-norm. The authors introduce a novel Laplacian-based smoothing that yields an explicit Cameron–Martin space, enabling transfer from smooth metrics to the distance and extending prior one-dimensional smoothing techniques to sphere-indexed fields. They derive a master bound for the distance between a general random field and a Gaussian field on , and apply it to wide deep neural networks to obtain explicit, depth- and width-dependent bounds at the random-field level, under Lipschitz activations. They further show improved rates when activations are three-times differentiable and discuss comparisons to related works and potential future directions, including rate improvements and extensions to heavier-tailed weight distributions or other manifolds.

Abstract

We derive upper bounds on the Wasserstein distance (), with respect to -norm, between any continuous valued random field indexed by the -sphere and the Gaussian, based on Stein's method. We develop a novel Gaussian smoothing technique that allows us to transfer a bound in a smoother metric to the distance. The smoothing is based on covariance functions constructed using powers of Laplacian operators, designed so that the associated Gaussian process has a tractable Cameron-Martin or Reproducing Kernel Hilbert Space. This feature enables us to move beyond one dimensional interval-based index sets that were previously considered in the literature. Specializing our general result, we obtain the first bounds on the Gaussian random field approximation of wide random neural networks of any depth and Lipschitz activation functions at the random field level. Our bounds are explicitly expressed in terms of the widths of the network and moments of the random weights. We also obtain tighter bounds when the activation function has three bounded derivatives.
Paper Structure (15 sections, 16 theorems, 190 equations, 1 table)

This paper contains 15 sections, 16 theorems, 190 equations, 1 table.

Table of Contents

  1. Introduction
  2. Bounds for random field approximations
  3. Application to wide random neural networks
  4. Comparison to related works
  5. Future directions
  6. Gaussian smoothing for random fields indexed by the sphere
  7. Constructing a covariance and Cameron-Martin space from the Laplacian
  8. Regularization to the Cameron-Martin space
  9. Smoothing using $S$ and regularization
  10. Proof of the Wasserstein bound
  11. Properties of Solution to the Stein Equation
  12. Chaining arguments for modulus of continuity
  13. Proofs for wide random neural network approximations
  14. $W_1$ bounds for wide random neural networks: Proof of Theorem \ref{['thm:w1bound']}
  15. Improved $W_1$ bounds: Proof of Theorem \ref{['thm:rateimprovment']}

Key Result

Theorem 1.1

[Master Theorem] Let $F,H \in \mathrm{C}(\mathcal{S}^n;\mathbb{R}^d)$ be random fields, where $\mathcal{S}^n$ is the unit sphere in $\mathbb{R}^{n+1}$ for some finite integer $n$. Then for any $\varepsilon,\delta \in(0,1)$ and $\iota>0$, where $F_\varepsilon$ and $H_\varepsilon$ are $\varepsilon$-regularizations of $F$ and $H$ defined at eq:genrep below, and $C$ is a constant depending only on $n

Theorems & Definitions (36)

  • Theorem 1.1
  • Theorem 1.2
  • Remark 1.3
  • Theorem 1.4
  • Remark 1.5
  • Definition 2.1: The Smoothing Gaussian random field $S$ and its Cameron-Martin space
  • Proposition 2.2
  • proof
  • Theorem 2.3: Karhunen-Loeve Expansion for the Smoothing Gaussian random field $S$
  • Definition 2.4: Paley-Wiener integral
  • ...and 26 more