Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Softmax Transformers are Turing-Complete

Hongjian Jiang, Michael Hahn, Georg Zetzsche, Anthony Widjaja Lin

TL;DR

The paper investigates whether softmax-attention transformers with Chain-of-Thought (CoT) steps can be Turing-complete. It shows that, by embedding CoT within C-RASP counting mechanisms, length-generalizable softmax CoT transformers are capable of simulating Minsky counter machines, establishing Turing-completeness for unary and letter-bounded languages, and extending to arbitrary languages with Relative Positional Encodings (RPEs). It proves that CoT C-RASP without RPEs fails for general languages but becomes fully Turing-complete when equipped with RPEs, supported by a formal construction and empirical demonstrations on arithmetic languages. The authors further demonstrate length-generalizable learnability within the len-gen-huang framework for languages expressible in C-RASP CoT, and provide empirical validation showing that unary representations generalize with NoPE while binary representations require RPEs for robust generalization. Overall, the work bridges formal language theory and transformer architectures, offering theoretical guarantees for trainability and practical insights for arithmetic reasoning in CoT-enabled models, with implications for understanding the limits and capabilities of softmax transformers in reasoning tasks.

Abstract

Hard attention Chain-of-Thought (CoT) transformers are known to be Turing-complete. However, it is an open problem whether softmax attention Chain-of-Thought (CoT) transformers are Turing-complete. In this paper, we prove a stronger result that length-generalizable softmax CoT transformers are Turing-complete. More precisely, our Turing-completeness proof goes via the CoT extension of the Counting RASP (C-RASP), which correspond to softmax CoT transformers that admit length generalization. We prove Turing-completeness for CoT C-RASP with causal masking over a unary alphabet (more generally, for letter-bounded languages). While we show this is not Turing-complete for arbitrary languages, we prove that its extension with relative positional encoding is Turing-complete for arbitrary languages. We empirically validate our theory by training transformers for languages requiring complex (non-linear) arithmetic reasoning.

Softmax Transformers are Turing-Complete

TL;DR

The paper investigates whether softmax-attention transformers with Chain-of-Thought (CoT) steps can be Turing-complete. It shows that, by embedding CoT within C-RASP counting mechanisms, length-generalizable softmax CoT transformers are capable of simulating Minsky counter machines, establishing Turing-completeness for unary and letter-bounded languages, and extending to arbitrary languages with Relative Positional Encodings (RPEs). It proves that CoT C-RASP without RPEs fails for general languages but becomes fully Turing-complete when equipped with RPEs, supported by a formal construction and empirical demonstrations on arithmetic languages. The authors further demonstrate length-generalizable learnability within the len-gen-huang framework for languages expressible in C-RASP CoT, and provide empirical validation showing that unary representations generalize with NoPE while binary representations require RPEs for robust generalization. Overall, the work bridges formal language theory and transformer architectures, offering theoretical guarantees for trainability and practical insights for arithmetic reasoning in CoT-enabled models, with implications for understanding the limits and capabilities of softmax transformers in reasoning tasks.

Abstract

Hard attention Chain-of-Thought (CoT) transformers are known to be Turing-complete. However, it is an open problem whether softmax attention Chain-of-Thought (CoT) transformers are Turing-complete. In this paper, we prove a stronger result that length-generalizable softmax CoT transformers are Turing-complete. More precisely, our Turing-completeness proof goes via the CoT extension of the Counting RASP (C-RASP), which correspond to softmax CoT transformers that admit length generalization. We prove Turing-completeness for CoT C-RASP with causal masking over a unary alphabet (more generally, for letter-bounded languages). While we show this is not Turing-complete for arbitrary languages, we prove that its extension with relative positional encoding is Turing-complete for arbitrary languages. We empirically validate our theory by training transformers for languages requiring complex (non-linear) arithmetic reasoning.

Paper Structure

This paper contains 47 sections, 12 theorems, 32 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Are CoT transformers Turing-complete?
  3. Contributions.
  4. Organization.
  5. Models for Trainable CoT Transformers
  6. Transformers and C-RASP
  7. Softmax Transformers.
  8. C-RASPs.
  9. Relative Positional Encodings.
  10. Discussion of Relative Positional Encodings
  11. Extensions with Chain-of-Thought
  12. CoT C-RASPs.
  13. CoT SMATs.
  14. Learnability with CoT
  15. Unary Case
  16. ...and 32 more sections

Key Result

Proposition 2.1

If a language is accepted by a CoT C-RASP (resp. C-RASP[RPEs]), then it is also accepted by a CoT SMAT (resp. SMAT[RPEs]).

Figures (1)

  • Figure 1: 2-CM with transition labels $\tau_i$.

Theorems & Definitions (23)

  • Proposition 2.1
  • proof : Proof Sketch for Proposition \ref{['prop:cot-crasp-to-cot-smat']}; see \ref{['app:model:proofs']} for full details
  • Definition 2.2
  • Proposition 2.3
  • proof : Proof Sketch for Proposition \ref{['prop:learnability-cot-crasp']}; see \ref{['app:model:proofs']} for full proof
  • Theorem 3.1
  • Proposition 3.2
  • Proposition 3.3
  • Proposition 3.4
  • Lemma 3.5
  • ...and 13 more