LDP for the covariance process in fully connected neural networks
Luisa Andreis, Federico Bassetti, Christian Hirsch
TL;DR
This work develops a functional large deviation framework for the covariance process in fully connected Gaussian neural networks by casting the covariance trajectory as a Markov process on the cone of non-negative symmetric trace-class operators. It establishes a law of large numbers and an LDP in both the trace-class operator space and its contraction to a space of kernels, and extends these results to posterior distributions under Gaussian likelihoods. A key contrast is that, in the infinite-width (lazy) regime, the posterior LDP matches the prior, while the mean-field regime yields a nontrivial rate function reflecting data influence and feature learning. Collectively, the results provide rigorous insights into the behavior of Bayesian neural networks in kernel-limit regimes and supply tools for analyzing posterior fluctuations in high-dimensional covariance structures.
Abstract
In this work, we study large deviation properties of the covariance process in fully connected Gaussian deep neural networks. More precisely, we establish a large deviation principle (LDP) for the covariance process in a functional framework, viewing it as a process in the space of continuous functions. As key applications of our main results, we obtain posterior LDPs under Gaussian likelihood in both the infinite-width and mean-field regimes. The proof is based on an LDP for the covariance process as a Markov process valued in the space of non-negative, symmetric trace-class operators equipped with the trace norm.