Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal computation is intrinsic to language model decoding

Alex Lewandowski, Marlos C. Machado, Dale Schuurmans

TL;DR

The paper investigates whether autoregressive decoding in language models can realize universal computation under the Church-Turing perspective. It develops a formal connection between extended autoregressive decoding and Lag systems, and proves exact simulation of a universal Lag system $L(U_{15,2})$ using either a system prompt on a trained model or an injective codebook on randomly initialized models. The authors validate universality across architectures (including $Llama-4-17B-128E-Instruct$) and show that training primarily improves promptability rather than computational expressiveness. This reframes prompting as a programmable interface to universal computation and suggests language models can serve as a natural-language interface between humans and machines, enabling a third age of computation.

Abstract

Language models now provide an interface to express and often solve general problems in natural language, yet their ultimate computational capabilities remain a major topic of scientific debate. Unlike a formal computer, a language model is trained to autoregressively predict successive elements in human-generated text. We prove that chaining a language model's autoregressive output is sufficient to perform universal computation. That is, a language model can simulate the execution of any algorithm on any input. The challenge of eliciting desired computational behaviour can thus be reframed in terms of programmability: the ease of finding a suitable prompt. Strikingly, we demonstrate that even randomly initialized language models are capable of universal computation before training. This implies that training does not give rise to computational expressiveness -- rather, it improves programmability, enabling a natural language interface for accessing these intrinsic capabilities.

Universal computation is intrinsic to language model decoding

TL;DR

The paper investigates whether autoregressive decoding in language models can realize universal computation under the Church-Turing perspective. It develops a formal connection between extended autoregressive decoding and Lag systems, and proves exact simulation of a universal Lag system using either a system prompt on a trained model or an injective codebook on randomly initialized models. The authors validate universality across architectures (including ) and show that training primarily improves promptability rather than computational expressiveness. This reframes prompting as a programmable interface to universal computation and suggests language models can serve as a natural-language interface between humans and machines, enabling a third age of computation.

Abstract

Language models now provide an interface to express and often solve general problems in natural language, yet their ultimate computational capabilities remain a major topic of scientific debate. Unlike a formal computer, a language model is trained to autoregressively predict successive elements in human-generated text. We prove that chaining a language model's autoregressive output is sufficient to perform universal computation. That is, a language model can simulate the execution of any algorithm on any input. The challenge of eliciting desired computational behaviour can thus be reframed in terms of programmability: the ease of finding a suitable prompt. Strikingly, we demonstrate that even randomly initialized language models are capable of universal computation before training. This implies that training does not give rise to computational expressiveness -- rather, it improves programmability, enabling a natural language interface for accessing these intrinsic capabilities.
Paper Structure (13 sections, 6 theorems, 1 equation, 4 figures)

This paper contains 13 sections, 6 theorems, 1 equation, 4 figures.

Table of Contents

  1. Introduction
  2. Autoregression and Computation
  3. Extending Autoregressive Decoding to Universality
  4. Demonstrating Computational Universality With a Proof-of-Simulation
  5. Proof-of-Simulation via a System Prompt
  6. Proof-of-Simulation via an Injective Codebook
  7. Discussion and Implications
  8. Additional Related Work
  9. Extended Autoregressive Decoding as Universal Lag Systems
  10. The Proof-of-Simulation Procedure
  11. Designing a System Prompt for Simulating a Universal Lag System
  12. Learning an Injective Codebook for Simulating a Universal Lag System
  13. Proofs

Key Result

Theorem 1

For a given language model $M:\Phi^*\rightarrow\Phi^*$, assume there exists an injective mapping $E:\Sigma\rightarrow\Phi^*$ such that there exists a reverse mapping $D:\Phi^*\rightarrow\Sigma$ satisfying $D(E(\sigma))=\sigma$ for all $\sigma\in\Sigma$. Assume furthermore there exists a system promp

Figures (4)

  • Figure 1: The three ages of computation: human, formal, and language model computation. A general-purpose computational system involves three fundamental components: directive, processor, and memory. Given a directive, the system manipulates memory using its processor to produce an answer to a question. (a) Human computation: A person uses paper as memory, manipulating it by hand according to the directive of a book. Before the advent of modern digital computers, humans had to manually process and manage computational operations specified by instructions written in natural language. (b) Formal computation: A classical computer uses random access memory, manipulating it using a central processing unit according to the directive of a written program. During the second age, spanning the 80 years since the development of modern digital computers, people have had to undergo a multi-year training process to learn how to specify precise step-by-step operations for a formal machine. (c) Language model computation: A language model uses a text sequence as memory, manipulating it via autoregressive decoding according to the directive of a system prompt. Now, in the emerging third age, language models are increasingly able to produce answers given only informal natural language specifications, without needing explicit step-by-step instructions on how to produce the answer.
  • Figure 2: Autoregressive decoding with a context window of size $N$ can be extended to read and write strings of arbitrary length.Standard autoregressive decoding uses a sliding context window to append its output to the end of its context. However, if the input is longer than the size of the context window, then the beginning is ignored. Generalized autoregressive decoding uses a sliding context window to read from the beginning of the input string and append to the end of the operational string, allowing the entire string to be eventually read regardless of the size of the context window. Extended autoregressive decoding introduces the option to produce more than one output token per step, enabling the generation of arbitrary-length strings.
  • Figure 3: A language model under autoregressive decoding is computationally universal if it can simulate the universal Lag system. Computational universality is achieved when the language model can execute each of the universal Lag system's production rules. For a one-output rule, the language model must map a two-symbol input to an output symbol $t_{1}$, and then the special halt symbol that stops generation. For a two-output rule, the language model must also produce a second output $t_{2}$ before producing the special halt symbol; this case is not shown in the figure.
  • Figure 4: Randomly initialized language models achieve universal computation above an architecture-specific minimal size. We measure log time-to-universality: the log-normalized number of training iterations to learn a codebook that drives a randomly initialized language model to correctly execute all production rules. Lower values indicate faster convergence, and a value of $0$ indicates that the iteration budget was reached without finding a valid codebook. The faded lines show individual runs; dark lines show the average.

Theorems & Definitions (9)

  • Theorem 1
  • Corollary 2
  • Corollary 3
  • Theorem 1
  • proof
  • Corollary 2
  • proof
  • Corollary 3
  • proof