Why and When Deep is Better than Shallow: An Implementation-Agnostic State-Transition View of Depth Supremacy

TL;DR

This work provides an implementation-agnostic theory of depth in deep learning by modeling networks as state-transition semigroups on a metric space and deriving a bias-variance decomposition in which variance depends solely on the abstract depth-k class $\mathcal{H}_k = H\circ B(k,F)$. The variance is controlled by the growth of covering numbers of word balls, leading to conditions under which depth-induced variance is saturating, polynomial, or exponential. When coupled with exponential or polynomial decay of bias, this yields four canonical regimes (EL, EP, PL, PP) and explicit optimal depths $k^*$, with depth supremacy most pronounced in the EL regime for hierarchical or iterative concepts such as neural ODEs and chain-of-thought models. This framework connects depth-generalization to coarse geometry and dynamical systems, offering a rigorous, architecture-agnostic lens for when deeper models confer statistical advantages and guiding principled depth selection in practice.

Abstract

Why and when is deep better than shallow? We answer this question in a framework that is agnostic to network implementation. We formulate a deep model as an abstract state-transition semigroup acting on a general metric space, and separate the implementation (e.g., ReLU nets, transformers, and chain-of-thought) from the abstract state transition. We prove a bias-variance decomposition in which the variance depends only on the abstract depth-$k$ network and not on the implementation (Theorem 1). We further split the bounds into output and hidden parts to tie the depth dependence of the variance to the metric entropy of the state-transition semigroup (Theorem 2). We then investigate implementation-free conditions under which the variance grow polynomially or logarithmically with depth (Section 4). Combining these with exponential or polynomial bias decay identifies four canonical bias-variance trade-off regimes (EL/EP/PL/PP) and produces explicit optimal depths $k^\ast$. Across regimes, $k^\ast>1$ typically holds, giving a rigorous form of depth supremacy. The lowest generalization error bound is achieved under the EL regime (exp-decay bias + log-growth variance), explaining why and when deep is better, especially for iterative or hierarchical concept classes such as neural ODEs, diffusion/score-matching models, and chain-of-thought reasoning.

Figures (3)

• Figure 1: Example of a neural network (depth $k=4$) in consideration. The input layer is formulated as state space $\mathcal{X}$, the hidden layers as state transition functions $f_i: \mathcal{X} \to \mathcal{X}$, and the output layer as readout function $h: \mathcal{X} \to \mathbb{R}$. The entire network is formulated as a state transition model.
• Figure 2: Framework of machine learning in consideration. The concept class$\mathcal{C}$ is the class of (unseen) data generators called concept, the hypothesis class$\mathcal{H}$ is the class of learning models, the sample space$\mathcal{S}$ is the space of datasets. The learning (algorithm) $A:\mathcal{S}\to\mathcal{H}$ is a mapping that assigns hypotheses to data. Generalization error is the discrepancy between the concept $c \in \mathcal{C}$ and the outcome $h = A(S_n) \in \mathcal{H}$. Complexity (of learning model) refers to an absolute size of hypothesis class $\mathcal{H}$, while expressive power refers to a size of $\mathcal{H}$ relative to concept class $\mathcal{C}$.
• Figure 3: Typical examples of approximation error and estimation error ($\alpha=1.0, \beta=2.0, \gamma=1.0$)

Theorems & Definitions (77)

• Theorem 1: Bias-Variance Decomposition for ERM over Depth-$k$ Networks
• Theorem 2: Hidden-Output Decomposition of Rademacher Complexity for Depth-$k$ Network
• Remark 1
• proof
• Theorem 3: \ref{['thm:rad.decomp.rad.ent']}, restated
• proof
• Theorem 4: \ref{['thm:rad.decomp.rad.ent']}, simpler parallel
• Lemma 1: Packing-Covering
• Lemma 2: Lipschitz Embedding
• Lemma 3: Subadditivity