Context-Free Recognition with Transformers
Selim Jerad, Anej Svete, Sophie Hao, Ryan Cotterell, William Merrill
TL;DR
The paper addresses whether transformers can provably recognize context-free languages and develops a framework in which looping depth and padding budgets govern recognizability. It proves CFLs can be recognized by looped AHAT transformers using $O(\log n)$ loops and $O(n^6)$ padding, with substantially reduced padding ($O(n^3)$) for unambiguous CFLs, and $O(n^2)$ for unambiguous linear CFLs; it also provides a parallel, item-based algorithm to realize CFL recognition. Empirical results on languages like BFVP validate the theoretical predictions that looping helps when the target language requires logarithmic depth, while results on other languages reveal nuanced behavior depending on depth and structure. The findings illuminate the trade-offs between depth, padding, and ambiguity, offering insights into how transformers could process hierarchical syntax in practice and guiding future research on tighter bounds and learnability for syntactic tasks.
Abstract
Transformers excel on tasks that process well-formed inputs according to some grammar, such as natural language and code. However, it remains unclear how they can process grammatical syntax. In fact, under standard complexity conjectures, standard transformers cannot recognize context-free languages (CFLs), a canonical formalism to describe syntax, or even regular languages, a subclass of CFLs (Merrill et al., 2022). Merrill & Sabharwal (2024) show that $\mathcal{O}(\log n)$ looping layers (w.r.t. input length $n$) allows transformers to recognize regular languages, but the question of context-free recognition remained open. In this work, we show that looped transformers with $\mathcal{O}(\log n)$ looping layers and $\mathcal{O}(n^6)$ padding tokens can recognize all CFLs. However, training and inference with $\mathcal{O}(n^6)$ padding tokens is potentially impractical. Fortunately, we show that, for natural subclasses such as unambiguous CFLs, the recognition problem on transformers becomes more tractable, requiring $\mathcal{O}(n^3)$ padding. We empirically validate our results and show that looping helps on a language that provably requires logarithmic depth. Overall, our results shed light on the intricacy of CFL recognition by transformers: While general recognition may require an intractable amount of padding, natural constraints such as unambiguity yield efficient recognition algorithms.