Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers

William Merrill, Ashish Sabharwal

TL;DR

This work investigates how transformer expressivity changes when depth grows with context length $n$, focusing on universal (looped) transformers. It shows that depth $Θ(\log n)$ suffices to recognize regular languages and solve graph connectivity, tasks believed to lie beyond fixed-depth $\mathsf{TC}^0$, while width must grow superpolynomially and chain-of-thought steps must grow superlogarithmically. The authors provide a concrete division primitive and a range of constructions, along with a bounded-input analysis for fixed-depth regimes, and corroborate theory with curriculum-based experiments on the A5 regular-language task. The results offer practical guidance for depth-aware model design and imply that dynamic depth can be a more efficient inference-time resource than width expansion or chain-of-thought reasoning for long-context sequential tasks.

Abstract

Recent theoretical results show transformers cannot express sequential reasoning problems over long inputs, intuitively because their computational depth is bounded. However, prior work treats the depth as a constant, leaving it unclear to what degree bounded depth may suffice for solving problems over short inputs, or how increasing the transformer's depth affects its expressive power. We address these questions by analyzing transformers whose depth can grow minimally with context length $n$. We show even highly uniform transformers with depth $Θ(\log n)$ can express two important problems: recognizing regular languages, which captures state tracking abilities and was known to be expressible only by an unconventional, non-uniform model of transformers, and graph connectivity, which underlies multi-step reasoning. Notably, both of these problems cannot be expressed by fixed-depth transformers under standard complexity conjectures, demonstrating the expressivity benefit of growing depth. Moreover, our theory quantitatively predicts how depth must grow with input length to express these problems, showing that depth scaling is more efficient than scaling width or chain-of-thought steps. Empirically, our detailed experiments designed to bridge the expressivity vs. learnability gap reveal that our theoretical depth requirements for regular language recognition closely match the practical depth requirements for successfully training transformers. Thus, our results clarify how depth affects a transformer's reasoning capabilities, and provide practical guidance for effective depth selection for sequential reasoning.

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers

TL;DR

This work investigates how transformer expressivity changes when depth grows with context length , focusing on universal (looped) transformers. It shows that depth suffices to recognize regular languages and solve graph connectivity, tasks believed to lie beyond fixed-depth , while width must grow superpolynomially and chain-of-thought steps must grow superlogarithmically. The authors provide a concrete division primitive and a range of constructions, along with a bounded-input analysis for fixed-depth regimes, and corroborate theory with curriculum-based experiments on the A5 regular-language task. The results offer practical guidance for depth-aware model design and imply that dynamic depth can be a more efficient inference-time resource than width expansion or chain-of-thought reasoning for long-context sequential tasks.

Abstract

Recent theoretical results show transformers cannot express sequential reasoning problems over long inputs, intuitively because their computational depth is bounded. However, prior work treats the depth as a constant, leaving it unclear to what degree bounded depth may suffice for solving problems over short inputs, or how increasing the transformer's depth affects its expressive power. We address these questions by analyzing transformers whose depth can grow minimally with context length . We show even highly uniform transformers with depth can express two important problems: recognizing regular languages, which captures state tracking abilities and was known to be expressible only by an unconventional, non-uniform model of transformers, and graph connectivity, which underlies multi-step reasoning. Notably, both of these problems cannot be expressed by fixed-depth transformers under standard complexity conjectures, demonstrating the expressivity benefit of growing depth. Moreover, our theory quantitatively predicts how depth must grow with input length to express these problems, showing that depth scaling is more efficient than scaling width or chain-of-thought steps. Empirically, our detailed experiments designed to bridge the expressivity vs. learnability gap reveal that our theoretical depth requirements for regular language recognition closely match the practical depth requirements for successfully training transformers. Thus, our results clarify how depth affects a transformer's reasoning capabilities, and provide practical guidance for effective depth selection for sequential reasoning.

Paper Structure

This paper contains 34 sections, 22 theorems, 11 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preliminaries: Universal Transformers
  3. Transformer Sublayers.
  4. Positional Encodings.
  5. Memory Management in Universal Transformers
  6. Numerical Datatype and Precision
  7. Fixed Depth Transformers Can Divide Small Integers
  8. Log Depth Enables Recognizing Regular Languages
  9. Fixed Depth and Bounded Length Inputs.
  10. Log Depth Enables Graph Connectivity
  11. Fixed Depth and Bounded Length Inputs.
  12. Comparing Scaling Depth to Scaling Width or Chain of Thought
  13. Wide Transformers with Fixed Depth Remain in $\mathsf{TC}^0$.
  14. Transformers with Logarithmic Chain-of-Thought Steps Remain in $\mathsf{TC}^0$.
  15. Experiments: Learning to Recognize Regular Languages
  16. ...and 19 more sections

Key Result

Lemma 1

Let $a_i, b_i, c_i, m \in \mathbb Z^{\geq 0}$ be such that $a_i = b_i m + c_i$ where $a_i \leq i$ and $c_i < m$. Suppose $\psi(i)$, $\psi(m)$, and $\phi(a_i)$ (or $\psi(a_i)$) are present in the residual stream of a transformer at each token $i$. Then, there exists a block of 7 transformer layers wi

Figures (2)

  • Figure 1: To recognize a regular language over inputs of length $n$, the depth of a universal transformer can grow $\Theta(\log n)$ by \ref{['thm:reg-langs']}. On the other hand, width must grow superpolynomially (\ref{['thm:polyn-width']}), and the number of chain-of-thought steps must be superlogarithmic (\ref{['thm:cot-tc0']}). The precise depth and width coefficients plotted here were obtained experimentally in \ref{['sec:experiments']}.
  • Figure 2: Strong linear fits imply theory/experiment match for modeling the impact of depth (left, $d = 4.8 \log_2 n - 15.8$ with $r^2 = 0.93$) and width (right, $n = 7.2 \log_2 w - 41.7$ with $r^2 = 0.98$) on effective context length for the $A_5$ state tracking task, a canonical hard regular language recognition problem. As predicted by \ref{['thm:reg-langs', 'thm:polyn-width']}, to recognize strings of length $n$, depth only needs to increase minimally $\propto \log n$ while width must increase drastically as $\exp(\Theta(n))$.

Theorems & Definitions (41)

  • Definition 1
  • Definition 2: Self-attention sublayer
  • Definition 3: Feedforward sublayer
  • Definition 4: $p$-Precise Operations
  • Lemma 1: Division
  • proof
  • Theorem 1: Regular Language Recognition
  • Corollary 1.1: Regular Language Recognition, Non-Uniform
  • Corollary 1.2: Depth Scaling for Regular Language
  • Theorem 2: Graph Connectivity
  • ...and 31 more