On the Linguistic Capacity of Real-Time Counter Automata
William Merrill
TL;DR
<3-5 sentence high-level summary> This paper investigates real-time counter automata as formal grammars to shed light on the linguistic capabilities of memory-based NLP models. It analyzes multiple counter-machine variants, proving that general, incremental, stateless, and threshold counter machines all recognize the same class of languages (CL), while the simplified variant (SCL) is strictly weaker. The authors establish broad closure properties for counter languages and demonstrate a critical limitation: counter machines can verify syntactic well-formedness but cannot evaluate the full compositional semantics of boolean expressions. They also show a restricted subclass is semilinear, connecting counter languages to Parikh-based structural properties and offering insight into the potential limits of LSTM-like architectures for deep hierarchical syntax.
Abstract
Counter machines have achieved a newfound relevance to the field of natural language processing (NLP): recent work suggests some strong-performing recurrent neural networks utilize their memory as counters. Thus, one potential way to understand the success of these networks is to revisit the theory of counter computation. Therefore, we study the abilities of real-time counter machines as formal grammars, focusing on formal properties that are relevant for NLP models. We first show that several variants of the counter machine converge to express the same class of formal languages. We also prove that counter languages are closed under complement, union, intersection, and many other common set operations. Next, we show that counter machines cannot evaluate boolean expressions, even though they can weakly validate their syntax. This has implications for the interpretability and evaluation of neural network systems: successfully matching syntactic patterns does not guarantee that counter memory accurately encodes compositional semantics. Finally, we consider whether counter languages are semilinear. This work makes general contributions to the theory of formal languages that are of potential interest for understanding recurrent neural networks.