Training Neural Networks as Recognizers of Formal Languages
Alexandra Butoi, Ghazal Khalighinejad, Anej Svete, Josef Valvoda, Ryan Cotterell, Brian DuSell
TL;DR
The paper tackles the gap between formal language recognizers and neural networks by training networks as binary recognizers of strings and evaluating them on languages from the Chomsky hierarchy. It proposes a general data-generation and training framework, including efficient length-bounded sampling for regular languages via a binning semiring, and compares RNNs, LSTMs, and transformers across 18 languages. Key findings show RNNs and LSTMs typically outperform transformers on formal languages, with recognition performance often bounded to regular languages and influenced by auxiliary losses in a language- and architecture-dependent manner. The authors release FLaRe (Formal Language Recognition) as a benchmark, providing datasets and code to enable theoretically grounded empirical testing of language recognizers in future work.
Abstract
Characterizing the computational power of neural network architectures in terms of formal language theory remains a crucial line of research, as it describes lower and upper bounds on the reasoning capabilities of modern AI. However, when empirically testing these bounds, existing work often leaves a discrepancy between experiments and the formal claims they are meant to support. The problem is that formal language theory pertains specifically to recognizers: machines that receive a string as input and classify whether it belongs to a language. On the other hand, it is common instead to evaluate language models on proxy tasks, e.g., language modeling or sequence-to-sequence transduction, that are similar in only an informal sense to the underlying theory. We correct this mismatch by training and evaluating neural networks directly as binary classifiers of strings, using a general method that can be applied to a wide variety of languages. As part of this, we extend an algorithm recently proposed by Snæbjarnarson et al. (2025) for efficient length-controlled sampling of strings from regular languages. We provide results on a variety of languages across the Chomsky hierarchy for three neural architectures: a simple RNN, an LSTM, and a causally-masked transformer. We find that the RNN and LSTM often outperform the transformer, and that auxiliary training objectives such as language modeling can help, although no single objective uniformly improves performance across languages and architectures. Our contributions will facilitate theoretically sound empirical testing of language recognition claims in future work. We have released our datasets as a benchmark called FLaRe (Formal Language Recognition), along with our code.