The Depth Delusion: Why Transformers Should Be Wider, Not Deeper

TL;DR

This work challenges the conventional wisdom that deeper transformers are inherently superior by introducing architecture-conditioned scaling laws. It demonstrates a critical depth threshold where additional layers harm performance, deriving a sublinear D_crit(W)≈W^{0.44} and showing optimal scaling exponents D^*∝C^{0.12}, W^*∝C^{0.34}; wider models accumulate learning signals more persistently (τ(W)∝W^{0.44}). Across 30 architectures from 17M to 7B parameters, the framework achieves strong fit (R^2=0.922) and, at 7B, a 32-layer model outperforms a 64-layer counterpart with similar compute, evidencing the Depth Delusion. These results imply practical guidance to favor width over depth in LLM design and raise considerations for scaling flagship models. The findings have hardware and efficiency implications, suggesting reoriented compute strategies and stabilization techniques to push toward wider, shallower architectures. $D_{ ext{crit}}(W)$, $W^*$, and $D^*$ are the central quantitative insights guiding future model design under compute constraints.

Abstract

Neural scaling laws describe how language model loss decreases with parameters and data, but treat architecture as interchangeable--a billion parameters could arise from a shallow-wide model (10 layers & 8,192 hidden dimension) or a deep-narrow one (80 layers & 2,048 hidden dimension). We propose architecture-conditioned scaling laws decomposing this dependence, finding that optimal depth scales as D* ~ C^0.12 while optimal width scales as W* ~ C^0.34, meaning width should grow 2.8x faster than depth. We discover a critical depth phenomenon: beyond D_crit ~ W^0.44 (sublinear in W), adding layers increases loss despite adding parameters--the Depth Delusion. Empirically, we validate these findings across 30 transformer architectures spanning 17M to 7B parameters, each trained on representative high-compute samples, achieving R^2 = 0.922. Our central finding: at 7B scale, a 64-layer model (6.38B params) underperforms a 32-layer model (6.86B params) by 0.12 nats, despite being significantly deeper. This demonstrates that optimal depth-width tradeoffs persist at the production scale.

Figures (4)

• Figure 1: Primary Evidence. (a) Test loss scales predictably with parameters until depth-width limits are reached. (b) For fixed width (W=512), increasing layers beyond $D_{\mathrm{crit}}=16$ creates a U-shaped penalty. The partial recovery at 32L remains significantly above the 16L optimum.
• Figure 2: Gradient Starvation Mechanism. Exponential decay of gradient signal through layers for different widths. Wider models support more persistence, but all exhibit a 1/e threshold defining $D_{\text{crit}}$.
• Figure 3: Gradient Flow Validation. (a) Gradient magnitude decay across layers for widths 256--1536. Dashed lines show exponential fits. (b) Fitted persistence length $\tau(W)$ vs. width. The power law $\tau \propto W^{0.44}$ ($R^2 = 0.98$) fits well, consistent with our theoretical prediction $\tau \propto \sqrt{W}$ (exponent 0.5). The slight deviation (0.44 vs 0.5) may arise from finite-width corrections.
• Figure 4: Large-Scale Validation. U-curves for 1B, 3B, and 7B models confirming the Depth Delusion at production scale. Optimal depths: 1B at 24L (loss 2.821), 3B at 40L (loss 2.519), 7B at 32L (loss 2.298). The dashed red lines show predicted $D_{\mathrm{crit}}$.

Theorems & Definitions (6)

• Proposition 3.1: Gradient Persistence
• Definition 3.2: Critical Depth
• Corollary 3.5: Optimal Scaling, from Ansatz \ref{['thm:loss']}
• Proposition 3.1
• proof : Sketch of Proof
• Theorem 3.1: Optimal Scaling