PLSemanticsBench: Large Language Models As Programming Language Interpreters

TL;DR

PLSemanticsBench investigates whether large language models can act as interpreters for an imperative language by executing programs according to formal SOS and K-semantics. The authors construct three dataset splits—Human-Written, LLM-Translated, and Fuzzer-Generated—and evaluate 11 state-of-the-art LLMs on three tasks: final-state prediction, semantic-rule prediction, and execution-trace prediction, under standard and two nonstandard semantics. They find that models are generally capable under standard semantics but degrade when faced with semantic mutations, with reasoning models performing better on coarse tasks but struggling on fine-grained ones; execution-trace prediction remains particularly challenging. The work underscores both the promise of LLMs for rapid language prototyping and the need for explicit semantic understanding, and it contributes a new benchmark and codebase to foster progress in semantics-aware AI for programming languages.

Abstract

As large language models (LLMs) excel at code reasoning, a natural question arises: can an LLM execute programs (i.e., act as an interpreter) purely based on a programming language's formal semantics? If so, it will enable rapid prototyping of new programming languages and language features. We study this question using the imperative language IMP (a subset of C), formalized via small-step operational semantics (SOS) and rewriting-based operational semantics (K-semantics). We introduce three evaluation sets-Human-Written, LLM-Translated, and Fuzzer- Generated-whose difficulty is controlled by code-complexity metrics spanning the size, control-flow, and data-flow axes. Given a program and its semantics formalized with SOS/K-semantics, models are evaluated on three tasks ranging from coarse to fine: (1) final-state prediction, (2) semantic rule prediction, and (3) execution trace prediction. To distinguish pretraining memorization from semantic competence, we define two nonstandard semantics obtained through systematic mutations of the standard rules. Across strong code/reasoning LLMs, performance drops under nonstandard semantics despite high performance under the standard one. We further find that (i) there are patterns to different model failures, (ii) most reasoning models perform exceptionally well on coarse grained tasks involving reasoning about highly complex programs often containing nested loop depths beyond five, and surprisingly, (iii) providing formal semantics helps on simple programs but often hurts on more complex ones. Overall, the results show a promise that LLMs could serve as programming language interpreters, but points to the lack of their robust semantics understanding. We release the benchmark and the supporting code at https://github.com/EngineeringSoftware/PLSemanticsBench.

Figures (9)

• Figure 1: The PLSemanticsBench construction workflow and the proposed three tasks. Each program is written in IMP with syntax specified in EBNF, and its standard PL semantics defined using both SOS and K-semantics (). The standard semantics can be systematically transformed into one of two nonstandard semantics, KeywordSwap and KeywordObf (). The standard IMP programs and their semantics will be transformed accordingly. The transformed K-semantics is then used to build a traditional interpreter with the K-framework (), which generates an output trace () for each transformed IMP program, serving as ground truth (GT) for the tasks. The transformed IMP program, its K-semantics () are used to construct prompts for the tasks. Tasks (- ) span from coarse-grained evaluation (PredState) to fine-grained evaluation (PredRule, PredTrace). An almost identical flow can also be achieved using the SOS and EBNF syntax by just replacing the K-framework interpreter with our custom built ANTLR4-based interpreter to evaluate the models on the tasks using SOS instead of K-semantics.
• Figure 2: The Small-step operational semantics (SOS) and rewriting-based operational semantics (K-semantics) for formalizing the semantics of a subset of the IMP programming language.
• Figure 3: Complete syntax of IMP used in our experiments in EBNF.
• Figure 4: An example of re-writing a C++ program into an IMP program in the Human-Written dataset.
• Figure 5: An example IMP program (fuzz_100.imp) from the Fuzzer-Generated dataset. Its code-complexity metric profile is: control-flow complexity ($\Omega_\text{CC}$= 62, $\Omega_\text{If}$= 5, $\Omega_\text{Loop}$= 6, $\hat{\Omega}_\text{If}$= 3, $\hat{\Omega}_\text{Loop}$= 5), data-flow complexity ($\Omega_\text{DD}$= 2603, $\hat{\Omega}_\text{Assign}$= 86), and program-size complexity ($\Omega_\text{Loc}$= 492, $\Omega_\text{Vol}$= 37140, $\Omega_\text{Voc}$= 91, $\hat{\Omega}_\text{Trace}$= 249). The Gemini-2.5-pro model successfully predicted the final program-state of this program in the PredState task.