Beyond Memorization: Testing LLM Reasoning on Unseen Theory of Computation Tasks

TL;DR

This work interrogates whether large language models exhibit genuine symbolic reasoning or rely on memorization when constructing deterministic finite automata ($DFA$) from regular languages. It introduces a principled DFA benchmark with a knowledge-check set, a seen construction set, and an unseen construction set generated via manual constraint composition and Arden’s theorem, enabling controlled tests of compositional generalization. Across frontier models and prompting strategies, results show perfect knowledge accuracy but substantial drops on unseen tasks, revealing a gap between syntactic plausibility and semantic correctness. A three-stage hint protocol can correct shallow errors but does not reliably resolve globally inconsistent automata, underscoring fundamental limits of current LLM-based symbolic reasoning and motivating future hybrid or training-based improvements.

Abstract

Large language models (LLMs) have demonstrated strong performance on formal language tasks, yet whether this reflects genuine symbolic reasoning or pattern matching on familiar constructions remains unclear. We introduce a benchmark for deterministic finite automata (DFA) construction from regular languages, comprising factual knowledge questions, seen construction problems from public sources, and two types of unseen problems: hand-crafted instances with multiple interacting constraints and systematically generated problems via Arden's theorem. Models achieve perfect accuracy on factual questions and 84-90% on seen tasks. However, accuracy drops sharply on unseen problems (by 30-64%), with failures stemming from systematic misinterpretation of language constraints, incorrect handling of Kleene-star semantics, and a failure to preserve global consistency. We evaluate a three-stage hint protocol that enables correction of shallow errors but does not reliably resolve globally inconsistent or structurally flawed automata. Our analysis across multiple prompting strategies (direct, Chain-of-Thought, Tree-of-Thought) reveals that errors persist regardless of prompting approach, exposing a fundamental gap between LLMs' ability to generate syntactically plausible DFAs and their capacity for semantically correct formal reasoning.

Figures (9)

• Figure 1: (a): Minimum DFA for language $L_1$; (b) & (c): Comparison of correct minimal DFA for unseen language $L_2$ (see b) and representative incorrect output (see c). Notation: $\bigcirc$ = state, $\triangleright$ = start state, $\circledcirc$ = accepting state. Transition $q_0 \xrightarrow{a} q_1$ indicates that on input $a$ from state $q_0$, the DFA transitions to state $q_1$ (Please zoom for better readability).
• Figure 2: (a): Random NFA over $\{a,b\}$. (b): Minimal DFA recognizing the derived language $L_3$.
• Figure 5: DFA generated via direct construction for $L = b(a \mid b)^{*}ab$.
• Figure 6: Derivative-based DFA generated for $L = b(a \mid b)^{*}ab$.
• Figure 7: DFA produced prior to minimization for $L = b(a \mid b)^{*}ab$.