Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Does Sparsity Mitigate the Curse of Depth in LLMs

Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta, Max Zimmer, Nico Pelleriti, Thomas Hofmann, Shiwei Liu

Abstract

Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we demonstrate that, sparsity, beyond enabling efficiency, acts as a regulator of variance propagation and thereby improves depth utilization. Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing sparsity in Grouped-Query Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: sparsity improves layer utilization by reducing output variance and promoting functional differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training deptheffective LLMs, yielding a notable 4.6% accuracy improvement on downstream tasks. Our results reveal sparsity, arising naturally from standard design choices, as a key yet previously overlooked mechanism for effective depth scaling in LLMs. Code is available at https://github.com/pUmpKin-Co/SparsityAndCoD.

When Does Sparsity Mitigate the Curse of Depth in LLMs

Abstract

Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we demonstrate that, sparsity, beyond enabling efficiency, acts as a regulator of variance propagation and thereby improves depth utilization. Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing sparsity in Grouped-Query Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: sparsity improves layer utilization by reducing output variance and promoting functional differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training deptheffective LLMs, yielding a notable 4.6% accuracy improvement on downstream tasks. Our results reveal sparsity, arising naturally from standard design choices, as a key yet previously overlooked mechanism for effective depth scaling in LLMs. Code is available at https://github.com/pUmpKin-Co/SparsityAndCoD.
Paper Structure (67 sections, 7 theorems, 110 equations, 29 figures, 8 tables)

This paper contains 67 sections, 7 theorems, 110 equations, 29 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Variance Propagation and Curse of Depth
  3. Experimental Setup
  4. Last-Layer Variance
  5. Causal Score
  6. Permutation Score
  7. Usefulness Score
  8. Main Observation
  9. Sparsity as Variance Regularizer
  10. Theoretical Analysis
  11. Implicit and Explicit Sparsity
  12. Implicit Sparsity
  13. Weight Decay
  14. Sequence Length
  15. Explicit Sparsity
  16. ...and 52 more sections

Key Result

Lemma 1

For a Pre-LN Transformer with $L$ layers using Equations eq:pre_ln_forward, assuming that the input vectors, intermediate vectors, and parameter follow independent zero-mean Gaussian distributions, and that $\sigma^2_{x_\ell}$ grows exponentially, then the partial derivative $\frac{\partial y_L}{\pa We define the intermediate state $x'_\ell \in \mathbb{R}^{d}$ as the post-attention, pre-FFN residu

Figures (29)

  • Figure 1: Progressive performance gains when scaling the 1.2B model to depth $L=32$ via stacking with various "sparsity" modules.
  • Figure 2: (a) Last-layer variance increases with depth under controlled width (note: total parameter count varies across depths). (b) Jacobian Frobenius norm $||J-I||_F$. (d) Layers in deeper models exhibit lower effectiveness (usefulness and causality) and higher redundancy (permutation scores).
  • Figure 3: Jacobian matrices at layers $\{10, 14, 22, 30\}$ for $L \in \{12, 16, 24, 32\}$.
  • Figure 4: (a) Variance decreases with stronger weight decay. (b)-(d) Weight sparsity (fraction of weights $< \epsilon$) increases with weight decay at thresholds $\epsilon \in \{10^{-1}, 10^{-2}, 10^{-4}\}$.
  • Figure 5: (a) Last-layer output variance decreases with longer sequences. (b)-(d) Attention sparsity (fraction of weights below threshold $\epsilon$) increases with sequence length across all thresholds. Results averaged over 3 random seeds.
  • ...and 24 more figures

Theorems & Definitions (13)

  • Lemma 1: gradient converge to identity sun2025curse Theorem 3.3
  • Theorem 1: Sparsity reduces variance propagation in residual-depth
  • proof
  • Theorem 2: Sparsity reduces variance propagation in Transformer
  • proof
  • Theorem 3: Weight decay contracts parameter variance and reduces layer-output variance
  • proof
  • Theorem 4: Sequence length reduces attention-output variance
  • proof
  • Theorem 5: Output variance scaling for Attention / GQA / MQA ($G=H$)
  • ...and 3 more