A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention

TL;DR

This work establishes a provable expressiveness hierarchy separating full attention from hybrid linear-full attention in Transformer models for deep, multi-step reasoning tasks, modeled by L-sequential function composition. Using a hybrid communication framework and indistinguishable decompositions, it proves an unconditional lower bound that limits the gain from interleaving expansive linear attention layers with full-attention layers: an $(L-1,2^{3L^2},\cdots,2^{3L^2})$-hybrid Transformer cannot solve $L$-sequential function composition when $Hdp \le n^{2^{-4L-2}}$, whereas an $(L+1)$-layer full-attention network suffices. The paper also proves a separate lower bound for single-layer sparse attention on the $2$-Sum task, showing a fundamental efficiency gap unless the sparse capacity scales with the block size. Collectively, these results provide a principled theoretical basis for the limited benefits of naive hybrid or sparse attention alternatives on deep reasoning tasks and guide the design of long-context Transformers that preserve essential expressive power.

Abstract

Transformers serve as the foundation of most modern large language models. To mitigate the quadratic complexity of standard full attention, various efficient attention mechanisms, such as linear and hybrid attention, have been developed. A fundamental gap remains: their expressive power relative to full attention lacks a rigorous theoretical characterization. In this work, we theoretically characterize the performance differences among these attention mechanisms. Our theory applies to all linear attention variants that can be formulated as a recurrence, including Mamba, DeltaNet, etc. Specifically, we establish an expressiveness hierarchy: for the sequential function composition-a multi-step reasoning task that must occur within a model's forward pass, an ($L+1$)-layer full attention network is sufficient, whereas any hybrid network interleaving $L-1$ layers of full attention with a substantially larger number ($2^{3L^2}$) of linear attention layers cannot solve it. This result demonstrates a clear separation in expressive power between the two types of attention. Our work provides the first provable separation between hybrid attention and standard full attention, offering a theoretical perspective for understanding the fundamental capabilities and limitations of different attention mechanisms.

Theorems & Definitions (47)

• Theorem 1.1
• Theorem 1.2
• Definition 2.1: Recurrent neural network (RNN)
• Lemma 2.2: Linear attention as RNN
• Definition 2.3: Hybrid Transformer architecture
• Definition 2.4
• Definition 2.5: $L$-sequential function composition
• Definition 2.6: The 2-Sum task
• Definition 4.1: Indistinguishable decomposition
• Lemma 4.2: Lemma \ref{['lemma:id-suffices']}