The Principles of Deep Learning Theory

TL;DR

The Principles of Deep Learning Theory develops an effective-theory framework for real-world deep networks by starting from microscopic network components and deriving macroscopic, layer-to-layer descriptions. It reveals that wide networks are governed by nearly-Gaussian statistics with finite-width corrections that grow with the depth-to-width ratio, enabling a tractable 1/n expansion and a controlled understanding of representation learning. A central construct is the RG-like Representation Group Flow, which tracks how preactivation distributions and their couplings run with depth, leading to notions of criticality and universality across activation functions. The work provides practical prescriptions for initializing networks at criticality, classifies activation functions into universality classes, and quantifies how depth and width jointly constrain expressivity and stability, offering a principled lens on inductive bias, optimization, and architectural design.

Abstract

This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.

Figures (7)

• Figure 1: A graph of a simple multilayer neural network, depicting how the input $x$ is transformed through a sequence of intermediate signals, $s^{(1)}$, $s^{(2)}$, and $s^{(3)}$, into the output $f(x;\theta)$. The white circles represent the neurons, the black dot at the top represents the network output, and the parameters $\theta$ are implicit; they weight the importance of the different arrows carrying the signals and bias the firing threshold of each neuron.
• Figure 2: Left: depiction of the neurons and connections for an example multilayer perceptron (MLP) architecture. This particular MLP has $L=4$ layers, defining a set of functions $f(x; \theta)$ with input dimension $n_0=4$ and output dimension $n_{4}=1$. The three hidden layers have five neurons each $n_1, n_2, n_3 = 5$, implying $P=91$ total model parameters. The graph describing the connections between neurons is a directed acyclic graph, meaning that signals only propagate in one direction and do not loop inside the network. For this reason, MLPs are also sometimes called feedforward networks. Right: the detailed structure of each neuron that (i) adds the bias and the weighted signals to produce the preactivation, (ii) generates the activation from the preactivation, and (iii) multiplies the activation by the next-layer weight.
• Figure 3: Commonly-used activation functions $\sigma(z)$. Grids are in units of one for both the preactivation $z$ and activation $\sigma$. (The $\texttt{leaky ReLU}$ is not shown.)
• Figure 4: Two algorithms to pin down a nontrivial fixed point, illustrated here for the $\texttt{tanh}$ activation function. Left: the lines defined by the conditions $\chi_{\perp}^{\star}=1$ (solid) and $\chi_{\parallel}^{\star}=1$ (dashed) are shown in the hyperparameter plane $(C_W,C_b)$ for the $\texttt{tanh}$ activation function. The intersection of these two lines gives the critical initialization hyperparameters $(C_W,C_b)=(1,0)$. Right: the left-hand side of the condition \ref{['easycriticality']} is plotted as a function of $K_{00}^{\star}$. The plotted line hits unity as $K_{00}^{\star}\rightarrow 0$.
• Figure 5: The left-hand side of the condition \ref{['easycriticality']} is plotted as a function of $K_{00}^{\star}$ for the $\texttt{sigmoid}$ activation function (left) and the $\texttt{softplus}$ activation function (right). For the $\texttt{sigmoid}$, the plotted line hits unity as $K_{00}^{\star}\rightarrow 0$, but the associated critical initialization hyperparameters $(C_b,C_W)$ are unphysical because $C_b<0$. For the $\texttt{softplus}$, the plotted line does not hit unity. These activation functions cannot be tuned to criticality.