Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Characterizing the Expressivity of Fixed-Precision Transformer Language Models

Jiaoda Li, Ryan Cotterell

TL;DR

This work provides a precise characterization of the expressivity of fixed-precision transformers with soft attention, strict masking, and no positional encodings, proving equivalence to the past-fragment of linear temporal logic $\text{LTL}[\mathrel{P}]$ and to $\text{PFO}^2[<]$ (via PODFAs and $\mathcal{R}$-trivial monoids). It further shows that transformer language models share this expressivity, and it offers a constructive bridge between logical formalisms and transformer architectures. Empirical evaluations on a range of formal languages demonstrate perfect generalization for tasks within the characterized capacity and consistent failure on languages beyond it, aligning theory with practice. The results illuminate the limitations of fixed-precision transformers and provide a principled framework for understanding when such models can reliably generalize across sequence lengths.

Abstract

Transformer-based language models (LMs) have achieved widespread empirical success, but their theoretical expressive power remains only partially understood. In this work, we analyze a restricted idealization of fixed-precision transformers with strict future masking, soft attention, and no positional encodings. We establish that this class of models is exactly as expressive as a specific fragment of linear temporal logic that contains only a single temporal operator: the past operator. We further connect this fragment to established classes in formal language theory, automata theory, and algebra, yielding a unified framework for understanding transformer expressivity under this idealization. Finally, we present empirical results that align closely with our theory: transformers trained on languages within their characterized expressive capacity generalize reliably across sequence lengths, while they consistently fail to generalize on languages beyond it.

Characterizing the Expressivity of Fixed-Precision Transformer Language Models

TL;DR

This work provides a precise characterization of the expressivity of fixed-precision transformers with soft attention, strict masking, and no positional encodings, proving equivalence to the past-fragment of linear temporal logic and to (via PODFAs and -trivial monoids). It further shows that transformer language models share this expressivity, and it offers a constructive bridge between logical formalisms and transformer architectures. Empirical evaluations on a range of formal languages demonstrate perfect generalization for tasks within the characterized capacity and consistent failure on languages beyond it, aligning theory with practice. The results illuminate the limitations of fixed-precision transformers and provide a principled framework for understanding when such models can reliably generalize across sequence lengths.

Abstract

Transformer-based language models (LMs) have achieved widespread empirical success, but their theoretical expressive power remains only partially understood. In this work, we analyze a restricted idealization of fixed-precision transformers with strict future masking, soft attention, and no positional encodings. We establish that this class of models is exactly as expressive as a specific fragment of linear temporal logic that contains only a single temporal operator: the past operator. We further connect this fragment to established classes in formal language theory, automata theory, and algebra, yielding a unified framework for understanding transformer expressivity under this idealization. Finally, we present empirical results that align closely with our theory: transformers trained on languages within their characterized expressive capacity generalize reliably across sequence lengths, while they consistently fail to generalize on languages beyond it.

Paper Structure

This paper contains 79 sections, 36 theorems, 76 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Strings and Languages
  4. ${\textup{LTL}[{\mathrel{\textup{P}}}]}$
  5. $\IfNoValueTF{-NoValue-}{{\textup{PFO}^2[\mathord<]}}{{\textup{PFO}^2[\mathord<,-NoValue-]}}$
  6. Transformers
  7. The Transformer Architecture
  8. From Transformers to $\IfNoValueTF{-NoValue-}{{\textup{PFO}^2[\mathord<]}}{{\textup{PFO}^2[\mathord<,-NoValue-]}}$
  9. From ${\textup{LTL}[{\mathrel{\textup{P}}}]}$ to Transformers
  10. Base Construction.
  11. Handling Vanishing Weights.
  12. Characterizations of ${\textup{LTL}[{\mathrel{\textup{P}}}]}$
  13. Transformer Language Models
  14. Experiments
  15. Language recognition
  16. ...and 64 more sections

Key Result

Theorem 2.0

$\IfNoValueTF{-NoValue-}{{\textup{PFO}^2[\mathord<]}}{{\textup{PFO}^2[\mathord<,-NoValue-]}}$ and ${\textup{LTL}[{\mathrel{\textup{P}}}]}$ have the same expressive power.

Figures (3)

  • Figure 1: Roadmap of the paper. Red arrows indicate novel results.
  • Figure 2: Visualization of the transformer's representations for different DFA states. $-1$ denotes the rejecting state ${{{ q}}_{R}}$. The filled contours represent the decision boundaries of a linear classifier.
  • Figure 3: DFAs for the given languages. Nodes represent states, and arrows represent transitions. The initial state is indicated by an incoming arrow with no source node, and accepting (final) states are shown with double circles.

Theorems & Definitions (74)

  • Theorem 2.0
  • proof
  • Theorem 3.0
  • proof
  • Theorem 3.0
  • proof
  • Theorem 4.0
  • proof
  • Definition 4.1
  • Definition 4.2
  • ...and 64 more