Ask, and it shall be given: On the Turing completeness of prompting
Ruizhong Qiu, Zhe Xu, Wenxuan Bao, Hanghang Tong
TL;DR
This work establishes a formal theory for LLM prompting by proving that prompting is Turing-complete with a finite-size Transformer, meaning any computable function can be realized by an appropriate prompt. It introduces 2-PTMs to provide a tractable, finite-alphabet model that a Decoder-only Transformer can execute via chain-of-thought traces, enabling explicit constructions for prompts, input encoding, and state recording. The authors prove near-optimal complexity bounds: CoT steps scale as O(t(n)) for TIME2(t(n)) and O(t(n) log t(n)) for TIME(t(n)) with a single transformer, and precision scales as O(log(n+t(n))). This work provides a theoretical foundation for prompt engineering, showing that a single finite transformer can efficiently simulate a broad class of computations, which has implications for the universality and design of prompting strategies in practice.
Abstract
Since the success of GPT, large language models (LLMs) have been revolutionizing machine learning and have initiated the so-called LLM prompting paradigm. In the era of LLMs, people train a single general-purpose LLM and provide the LLM with different prompts to perform different tasks. However, such empirical success largely lacks theoretical understanding. Here, we present the first theoretical study on the LLM prompting paradigm to the best of our knowledge. In this work, we show that prompting is in fact Turing-complete: there exists a finite-size Transformer such that for any computable function, there exists a corresponding prompt following which the Transformer computes the function. Furthermore, we show that even though we use only a single finite-size Transformer, it can still achieve nearly the same complexity bounds as that of the class of all unbounded-size Transformers. Overall, our result reveals that prompting can enable a single finite-size Transformer to be efficiently universal, which establishes a theoretical underpinning for prompt engineering in practice.