Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models
Changshu Liu, Yang Chen, Reyhaneh Jabbarvand
TL;DR
CES addresses the gap in reliably evaluating code execution reasoning in LLMs by unifying intermediate state and final output predictions into a flow-sensitive prompt and introducing coherency and consistency diagnostics. It formalizes simulation divergence, three coherency rules, and a prime-path–based consistency spectrum (weak/strong) to scrutinize how models reason about program execution across diverse tests. The empirical study across 16 LLMs on HumanEval shows substantial but imperfect coherent execution and widespread inconsistency in path-sensitive reasoning, with many outputs being suspiciously correct without faithful reasoning. CES also benchmarks its relation to bug-related tasks, revealing that many apparent successes stem from pattern matching or data leakage and that CES can systematically vet suspicious outcomes. Overall, CES provides a rigorous, execution-centric framework for vetting the reliability and generalizability of code reasoning in LLMs and informs the next generation of execution-aware Code LLMs.
Abstract
This paper proposes CES, a task to evaluate the abilities of LLMs in simulating program execution and using that reasoning in programming tasks. Besides measuring the correctness of variable predictions during execution simulation, CES introduces the notion of coherence to determine whether the simulation complies with commonsense execution logic, even if the predicted values along the simulations are incorrect. This enables CES to rule out suspiciously correct output predictions due to reasoning shortcuts, hallucinations, or potential data leakage. CES also introduces a novel metric to measure reasoning consistency across tests with the same or different prime path coverage in a spectrum: strong, weak, and random. Evaluating 16 LLMs (including three reasoning LLMs) using CES indicates 81.42% coherent execution simulation on HumanEval, 46.92% and 53.08% of which result in correct and incorrect output predictions. Frontier LLMs such as GPT-4 and DeepSeek-R1 have the most incoherent execution reasoning, mostly due to natural language shortcuts. Despite relatively coherent execution simulation, LLMs' reasoning performance across different tests is inconsistent, mostly random (48.87%) or weak (45.37%), potentially explaining their weakness in programming tasks that require path-sensitive program analysis to succeed. We also compare CES with bug prediction/localization/repair, which intuitively requires control- and data-flow awareness. We observe that LLMs barely incorporate execution reasoning into their analysis for bug-related tasks, and their success is primarily due to inherent abilities in pattern matching or natural language shortcuts, if not data leakage. Without reasoning, there is a threat to the generalizability of LLMs in dealing with unseen bugs or patterns in different contexts. CES can be used to vet the suspicious success of LLMs in these tasks systematically.