Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime

Francesco Camilli, Daria Tieplova, Eleonora Bergamin, Jean Barbier

TL;DR

This work establishes an information-theoretic layer-wise Gaussian equivalence for fully trained deep networks in the proportional scaling regime, proving a deep Gaussian equivalence principle that reduces a deep Bayes-optimal learner to a generalized linear model with an effective noise structure. The core technique combines an interpolation framework with Nishimori identities and a second circular interpolation to show that each hidden layer contributes only an equivalent linear term plus Gaussian noise, enabling explicit mutual information and optimal generalization error calculations. The main contributions include a rigorous layer-reduction result, an explicit formula for the effective linear model coefficients, and a proof of mutual-information and generalization-error equivalence that holds in the limit as the smallest layer width $d_m$ grows, under suitable smoothness of the activation and readout. The findings imply that, in the strongly overparameterized proportional regime, deep networks effectively behave like GLMs, highlighting the need to explore beyond-proportional data scales to capture nonlinear feature learning and avoid trivialisation for practical performance.

Abstract

We rigorously analyse fully-trained neural networks of arbitrary depth in the Bayesian optimal setting in the so-called proportional scaling regime where the number of training samples and width of the input and all inner layers diverge proportionally. We prove an information-theoretic equivalence between the Bayesian deep neural network model trained from data generated by a teacher with matching architecture, and a simpler model of optimal inference in a generalized linear model. This equivalence enables us to compute the optimal generalization error for deep neural networks in this regime. We thus prove the "deep Gaussian equivalence principle" conjectured in Cui et al. (2023) (arXiv:2302.00375). Our result highlights that in order to escape this "trivialisation" of deep neural networks (in the sense of reduction to a linear model) happening in the strongly overparametrized proportional regime, models trained from much more data have to be considered.

Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime

TL;DR

This work establishes an information-theoretic layer-wise Gaussian equivalence for fully trained deep networks in the proportional scaling regime, proving a deep Gaussian equivalence principle that reduces a deep Bayes-optimal learner to a generalized linear model with an effective noise structure. The core technique combines an interpolation framework with Nishimori identities and a second circular interpolation to show that each hidden layer contributes only an equivalent linear term plus Gaussian noise, enabling explicit mutual information and optimal generalization error calculations. The main contributions include a rigorous layer-reduction result, an explicit formula for the effective linear model coefficients, and a proof of mutual-information and generalization-error equivalence that holds in the limit as the smallest layer width grows, under suitable smoothness of the activation and readout. The findings imply that, in the strongly overparameterized proportional regime, deep networks effectively behave like GLMs, highlighting the need to explore beyond-proportional data scales to capture nonlinear feature learning and avoid trivialisation for practical performance.

Abstract

We rigorously analyse fully-trained neural networks of arbitrary depth in the Bayesian optimal setting in the so-called proportional scaling regime where the number of training samples and width of the input and all inner layers diverge proportionally. We prove an information-theoretic equivalence between the Bayesian deep neural network model trained from data generated by a teacher with matching architecture, and a simpler model of optimal inference in a generalized linear model. This equivalence enables us to compute the optimal generalization error for deep neural networks in this regime. We thus prove the "deep Gaussian equivalence principle" conjectured in Cui et al. (2023) (arXiv:2302.00375). Our result highlights that in order to escape this "trivialisation" of deep neural networks (in the sense of reduction to a linear model) happening in the strongly overparametrized proportional regime, models trained from much more data have to be considered.
Paper Structure (14 sections, 24 theorems, 171 equations)

This paper contains 14 sections, 24 theorems, 171 equations.

Table of Contents

  1. Introduction
  2. Notations.
  3. Definitions and main results
  4. Outline of the proofs
  5. Proof of Theorem \ref{['thm:reduction']}
  6. Proof of Theorem \ref{['th:Z_k_concentr']}
  7. Conclusion and perspectives
  8. Nishimori identity
  9. Lipschitz functions of sub-Gaussian random variables
  10. Bounds for post-activations moments
  11. Properties of the output kernel
  12. Details for the proof of Theorem \ref{['thm:reduction']}
  13. Complete network reduction
  14. Concentration of free energy

Key Result

Theorem 1

Let the activation and readout functions $\varphi,f\in C^2(\mathbb{R})$ be odd and with bounded first and second derivatives. Let $Z$ be a standard Gaussian, assume $L\geq 1$ and define the coefficients for all $\ell\in[L]$. Denote by $d_m=\min\{(d_\ell)_{\ell=0}^L\}$. Then where ${I}_n^{(L - 1)}({\boldsymbol{\theta}}^{*(L-1)},{\boldsymbol{\xi}}^*;\mathcal{D}_{L-1})$ is the mutual information be

Theorems & Definitions (24)

  • Theorem 1: Layer reduction
  • Theorem 2: Free entropy concentration
  • Corollary 3: Mutual information equivalence
  • Theorem 4: Generalization error equivalence
  • Proposition 5: Nishimori identity
  • Lemma 6: Orthogonal approximation
  • Lemma 7: Concentration propagation
  • Lemma 8: Moments bounds
  • Lemma 9: Post-activations moments
  • Lemma 10: Quasi-orthogonality propagation
  • ...and 14 more