On the "Induction Bias" in Sequence Models

TL;DR

A large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes finds that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs.

Abstract

Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growing body of literature has shown this limitation primarily through failures in out-of-distribution (OOD) generalization, such as length extrapolation. In this work, we shift attention to the in-distribution implications of these limitations. We conduct a large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes. We find that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs. Furthermore, we analyze the extent to which learned state-tracking mechanisms are shared across different sequence lengths. We show that transformers exhibit negligible or even detrimental weight sharing across lengths, indicating that they learn length-specific solutions in isolation. In contrast, recurrent models exhibit effective amortized learning by sharing weights across lengths, allowing data from one sequence length to improve performance on others. Together, these results demonstrate that state tracking remains a fundamental challenge for transformers, even when training and evaluation distributions match.

Figures (7)

• Figure 1: Example of the three task formats for the addition modulo $5$ task applied to the sequence $2\, 1\, 0\, 3\, 4$.
• Figure 2: Minimal dataset size for the uniform length distribution with $m=2$ (parity). RNNs favor ACoT, whereas transformers favor CoT.
• Figure 3: $N^*$ for the outcome supervision format with a uniform length distribution and $m=2$ (parity). In the absence of intermediate supervision, single‑layer RNNs significantly outperform the 6‑layer transformer.
• Figure 4: Sample complexity (log scale) for transformers trained with CoT and RNNs with ACoT on the parity task. RNNs exhibit the expected improvement in sample efficiency with increasing sequence length, while transformers fail to leverage the additional supervision.
• Figure 5: Sample complexity (log scale) in the Outcome Supervision format for the uniform and short-to-long length setting, with $m=2$ (parity). Recurrent models require fewer training samples under the short-to-long setting, indicating that shorter sequences provides a stronger learning signal.