When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective

TL;DR

The paper tackles the problem of statistical efficiency in sequence-to-sequence learning by introducing the $q$STR model, where each output position depends on a dynamic, sparse subset of input tokens. It shows that a single-layer Transformer with at least $q$ attention heads can learn $q$STR with sample complexity nearly independent of the sequence length $N$, while FFNs require linear dependence on $N$ and RNNs incur $N^{\Omega(1)}$ samples for general $q$STR. The results establish a formal hierarchy: Transformers outperform FFNs and RNNs in the presence of dynamic sparsity, with RNNs capable of learning a simple subset of $q$STR but not the full model, and FFNs facing the strongest lower bounds. The analysis combines approximation/construction arguments for Transformers with minimax lower bounds for FFNs and RNNs, highlighting the impact of attention-based retrieval on length-generalization and sample efficiency. This work provides a principled statistical rationale for when Transformers offer tangible advantages in long-context, sparsity-structured tasks, with implications for architecture design and theoretical understanding of length generalization.

Abstract

Theoretical efforts to prove advantages of Transformers in comparison with classical architectures such as feedforward and recurrent neural networks have mostly focused on representational power. In this work, we take an alternative perspective and prove that even with infinite compute, feedforward and recurrent networks may suffer from larger sample complexity compared to Transformers, as the latter can adapt to a form of dynamic sparsity. Specifically, we consider a sequence-to-sequence data generating model on sequences of length $N$, in which the output at each position depends only on $q$ relevant tokens with $q \ll N$, and the positions of these tokens are described in the input prompt. We prove that a single-layer Transformer can learn this model if and only if its number of attention heads is at least $q$, in which case it achieves a sample complexity almost independent of $N$, while recurrent networks require $N^{Ω(1)}$ samples on the same problem. If we simplify this model, recurrent networks may achieve a complexity almost independent of $N$, while feedforward networks still require $N$ samples. Consequently, our proposed sparse retrieval model illustrates a natural hierarchy in sample complexity across these architectures.

Figures (2)

• Figure 1: Number of samples required to reach a certain test MSE loss threshold while training with online AdamW. We consider (\ref{['fig:n_vs_N']}) the $1\mathrm{STR}$ model with loss threshold $0.7$ and (\ref{['fig:n_vs_N_simple']}) the simple-$1\mathrm{STR}$ model with loss threshold $0.02$, averaged over 5 experiments. We use a linear link function, standard Gaussian input, $d=10$ and $d_e = \lfloor 5 \log(N)\rfloor$. Positional encodings are sampled uniformly from the unit hypercube. This observation is consistent with Theorem \ref{['thm:main']}.
• Figure 2: The trained attention weights $\boldsymbol{W}_Q^\top\boldsymbol{W}_K$ match our theoretical construction, see Equation \ref{['eq:W_QK_constr']}. We use the $1\mathrm{STR}$ setup of Figure \ref{['fig:exp_n_vs_N']} with $N=100$.

Theorems & Definitions (21)

• Theorem 1: Informal
• Definition 2: $q$-Sparse Token Regression
• Proposition 3
• Theorem 4
• Definition 5
• Theorem 6
• Theorem 7
• Proposition 8
• Theorem 9
• Theorem 10