Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks

TL;DR

This work investigates how transformer networks can perform robust symbol processing by isolating purely semantics-free symbolic tasks and embedding them into a programmable framework. The authors introduce the Transformer Production Framework (TPF), which combines a high-level Production System Language (PSL) with a sequence of compilers (PSL → QKVL → DAT) to realize fully interpretable symbolic computation inside transformers, culminating in a Turing-complete paradigm. Through the Temptatic Generation Task (TGT) and the Swap case study, they demonstrate that a discrete-attention-only transformer (DAT) can execute production-system-like algorithms, achieving precise symbol manipulation and near-universal computability. They also discuss mechanistic interpretability hypotheses, compare to RASP-based approaches, and propose pathways to unify semantics-free and semantics-permeated processing in enhanced transformer architectures. The results provide a principled bridge between symbolic AI concepts and neural computation, outlining concrete methods and tooling (ParGen, a simulator) to study and extend symbol processing in LMs.

Abstract

Large Language Models (LLMs) have demonstrated impressive abilities in symbol processing through in-context learning (ICL). This success flies in the face of decades of critiques asserting that artificial neural networks cannot master abstract symbol manipulation. We seek to understand the mechanisms that can enable robust symbol processing in transformer networks, illuminating both the unanticipated success, and the significant limitations, of transformers in symbol processing. Borrowing insights from symbolic AI and cognitive science on the power of Production System architectures, we develop a high-level Production System Language, PSL, that allows us to write symbolic programs to do complex, abstract symbol processing, and create compilers that precisely implement PSL programs in transformer networks which are, by construction, 100% mechanistically interpretable. The work is driven by study of a purely abstract (semantics-free) symbolic task that we develop, Templatic Generation (TGT). Although developed through study of TGT, PSL is, we demonstrate, highly general: it is Turing Universal. The new type of transformer architecture that we compile from PSL programs suggests a number of paths for enhancing transformers' capabilities at symbol processing. We note, however, that the work we report addresses computability, and not learnability, by transformer networks. Note: The first section provides an extended synopsis of the entire paper.