Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility
Hoil Lee, Fadhel Ayed, Paul Jung, Juho Lee, Hongseok Yang, François Caron
TL;DR
This work extends the classic infinite-width analysis of neural networks by introducing a structured dependence among outgoing weights via layer-wise per-node variance factors. When the sum of these variances converges to an infinitely divisible random variable, the infinite-width limit is either a Gaussian process (GP) if all Lévy measures are trivial, or a mixture of Gaussian processes (MoGP) if at least one layer has a non-trivial Lévy measure. The MoGP regime yields correlated, non-Gaussian outputs with possible heavy tails and enables weight compressibility and feature-learning through pruning, contrasting with the GP regime where weights vanish and dimensions decouple. The paper provides theoretical limits, pruning bounds, and a diverse set of examples (Bernoulli, Gamma, Beta, Horseshoe, generalised Gamma Pareto) and supports the theory with experiments on MNIST/Fashion-MNIST and CNNs, illustrating practical benefits in regularisation, Bayesian inference, and representation learning. Overall, the MoGP framework broadens the expressivity of infinite-width neural networks and links heavy-tailed priors with sparsity-aware compression and feature discovery in deep architectures.
Abstract
This article studies the infinite-width limit of deep feedforward neural networks whose weights are dependent, and modelled via a mixture of Gaussian distributions. Each hidden node of the network is assigned a nonnegative random variable that controls the variance of the outgoing weights of that node. We make minimal assumptions on these per-node random variables: they are iid and their sum, in each layer, converges to some finite random variable in the infinite-width limit. Under this model, we show that each layer of the infinite-width neural network can be characterised by two simple quantities: a non-negative scalar parameter and a Lévy measure on the positive reals. If the scalar parameters are strictly positive and the Lévy measures are trivial at all hidden layers, then one recovers the classical Gaussian process (GP) limit, obtained with iid Gaussian weights. More interestingly, if the Lévy measure of at least one layer is non-trivial, we obtain a mixture of Gaussian processes (MoGP) in the large-width limit. The behaviour of the neural network in this regime is very different from the GP regime. One obtains correlated outputs, with non-Gaussian distributions, possibly with heavy tails. Additionally, we show that, in this regime, the weights are compressible, and some nodes have asymptotically non-negligible contributions, therefore representing important hidden features. Many sparsity-promoting neural network models can be recast as special cases of our approach, and we discuss their infinite-width limits; we also present an asymptotic analysis of the pruning error. We illustrate some of the benefits of the MoGP regime over the GP regime in terms of representation learning and compressibility on simulated, MNIST and Fashion MNIST datasets.