Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Turing Machine Simulation with Transformers

Qian Li, Yuyi Wang

TL;DR

This work addresses the efficiency gap in theoretically universal Transformers by proving that constant-bit-size Transformers can simulate $(t(n),s(n))$-bounded TMs with a linear context window and sublinear per-step reasoning. The authors develop a two-stage bridge: first translating multi-tape TMs into synchronous multi-queue TMs with favorable time/space, then simulating those multi-queue machines with constant-bit Transformers using a geometric-offset attention pattern. The main result shows per-step CoT overhead can be reduced to $O(s(n)^c)$ for any $c>0$, achieved by tuning the head-layer product, and the model can produce each token in $O(1)$ time under sparse attention. The construction hinges on a careful encoding of queues as stacks across levels and employs a specialized relative positional encoding that evolves with input length. These findings offer architectural guidance for designing efficient, scalable reasoning in Transformers while preserving universality.

Abstract

Constant bit-size Transformers are known to be Turing complete, but existing constructions require $Ω(s(n))$ chain-of-thought (CoT) steps per simulated Turing machine (TM) step, leading to impractical reasoning lengths. In this paper, we significantly reduce this efficiency gap by proving that any $(t(n),s(n))$-bounded multi-tape TM can be simulated by a constant bit-size Transformer with an optimal $O(s(n))$-long context window and only $O(s(n)^c)$ CoT steps per TM step, where $c>0$ can be made arbitrarily small by letting the Transformers' head-layer product sufficiently large. In addition, our construction shows that sparse attention with fixed geometric offsets suffices for efficient universal computation. Our proof leverages multi-queue TMs as a bridge. The main technical novelty is a more efficient simulation of multi-tape TMs by synchronous multi-queue TMs, improving both time and space complexity under stricter model assumptions.

Efficient Turing Machine Simulation with Transformers

TL;DR

This work addresses the efficiency gap in theoretically universal Transformers by proving that constant-bit-size Transformers can simulate -bounded TMs with a linear context window and sublinear per-step reasoning. The authors develop a two-stage bridge: first translating multi-tape TMs into synchronous multi-queue TMs with favorable time/space, then simulating those multi-queue machines with constant-bit Transformers using a geometric-offset attention pattern. The main result shows per-step CoT overhead can be reduced to for any , achieved by tuning the head-layer product, and the model can produce each token in time under sparse attention. The construction hinges on a careful encoding of queues as stacks across levels and employs a specialized relative positional encoding that evolves with input length. These findings offer architectural guidance for designing efficient, scalable reasoning in Transformers while preserving universality.

Abstract

Constant bit-size Transformers are known to be Turing complete, but existing constructions require chain-of-thought (CoT) steps per simulated Turing machine (TM) step, leading to impractical reasoning lengths. In this paper, we significantly reduce this efficiency gap by proving that any -bounded multi-tape TM can be simulated by a constant bit-size Transformer with an optimal -long context window and only CoT steps per TM step, where can be made arbitrarily small by letting the Transformers' head-layer product sufficiently large. In addition, our construction shows that sparse attention with fixed geometric offsets suffices for efficient universal computation. Our proof leverages multi-queue TMs as a bridge. The main technical novelty is a more efficient simulation of multi-tape TMs by synchronous multi-queue TMs, improving both time and space complexity under stricter model assumptions.

Paper Structure

This paper contains 22 sections, 3 theorems, 17 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our contribution
  3. Step 1: From multi-tape TMs to multi-queue TMs.
  4. Step 2: From multi-queue TMs to Transformers.
  5. Other related works
  6. Formal-language expressivity.
  7. Reasoning efficiency.
  8. Preliminaries
  9. Multi-tape Turing machines
  10. Multi-queue Turing machines
  11. Transformer Architecture
  12. Token embedding layer
  13. Decoder Layers
  14. Output layer
  15. Step I: Efficient simulation of multi-tape TMs by multi-queue TMs
  16. ...and 7 more sections

Key Result

Theorem 1

Any $(t(n),s(n))$ time-space bounded $k$-tape Turing Machine can be simulated by a constant bit-size Transformer with head-layer product $K$ and context window length $s(n)+1$, that takes $O(s(n)^{6k/K})$ CoT steps for each simulated TM step, leading to a total CoT length of $O(t(n)\cdot s(n)^{6k/K}

Figures (2)

  • Figure 1: Left: A queue can be viewed as a tape with the head shifting one cell to the right cyclically at each step. Right: Illustration of the queues to simulate a stack.
  • Figure 2: Illustration of our construction in the proof of Theorem \ref{['thm:qmtr']}.

Theorems & Definitions (13)

  • Theorem 1
  • Theorem 2
  • proof
  • Claim 1
  • proof
  • Remark 1
  • Theorem 3
  • Remark 2
  • proof
  • Claim 2
  • ...and 3 more