L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution

TL;DR

This work introduces L0-Bench, a synthetic benchmark that probes level-0 reasoning by requiring language models to generate precise, step-by-step execution traces for simple Python programs. By controlling program generation, inputs, and traces, the framework isolates procedural correctness from broader reasoning, enabling scalable evaluation across 20 models and multiple test-time scales. Key findings show that larger models and reasoning-enhanced configurations improve trace accuracy, but all models degrade as trace length grows; test-time scaling via more demonstrations, larger voting pools, and longer chain-of-thought can yield meaningful gains yet incur higher costs. The results point to substantial room for improvement in level-0 reasoning and offer directions for developing more reliable, trace-consistent reasoning systems through targeted scaling and data-generation strategies.

Abstract

Complex reasoning tasks often rely on the ability to consistently and accurately apply simple rules across incremental steps, a foundational capability which we term "level-0" reasoning. To systematically evaluate this capability, we introduce L0-Bench, a language model benchmark for testing procedural correctness -- the ability to generate correct reasoning processes, complementing existing benchmarks that primarily focus on outcome correctness. Given synthetic Python functions with simple operations, L0-Bench grades models on their ability to generate step-by-step, error-free execution traces. The synthetic nature of L0-Bench enables systematic and scalable generation of test programs along various axes (e.g., number of trace steps). We evaluate a diverse array of recent closed-source and open-weight models on a baseline test set. All models exhibit degradation as the number of target trace steps increases, while larger models and reasoning-enhanced models better maintain correctness over multiple steps. Additionally, we use L0-Bench to explore test-time scaling along three dimensions: input context length, number of solutions for majority voting, and inference steps. Our results suggest substantial room to improve "level-0" reasoning and potential directions to build more reliable reasoning systems.

Figures (6)

• Figure 1: An illustrative example in $L0$-Bench. The prompt includes a program, multiple few-shot execution demonstrations, and a test input. Programs are generated using a constrained generative grammar that defines a simplified Python subset (see Appendix \ref{['sec:appendix_grammar']}). Inputs are randomly sampled and verified by executing the Python function to ensure error-free evaluation.
• Figure 2: Model performance generally declines as the number of trace steps increases. Open-weight models in the figure are instruct version.
• Figure 3: We increase the number of voters up to 1024 to study majority voting at scale. Each voter either permutes the same set of four demonstrations or randomly selects four distinct few-shot demonstrations from a pool of 64. Access to a larger, more diverse demonstration pool improves majority-voted performance and pass@K. The significant gap between pass@K and majority-voted performance indicates large room for models to reach their full potential.
• Figure 4: (Left) The difficulty of default tasks in $L0$-Bench can be increased by : (Task variation #1) using irrelevant (program, input, trace) as few-shot examples, or (Task variation #2) removing the program, forcing the model to infer the algorithm purely from step-by-step demonstrations. We use the simplest setup to test the 0-level reasoning. (Right) Even the strongest closed-source model Claude-3.7-Sonnet shows performance degradation with the modification of task setup, especially when extended thinking is disabled.
• Figure 5: Template for for the default setup of $L0$-Bench. The input programs, few-shot demonstrations, and test input arguments are filled on the fly during evaluation.