A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models

TL;DR

This paper tackles the challenge of interpreting code execution reasoning beyond mere input/output predictions. It introduces ExeRScope, a tool that combines static and dynamic program analysis to extract properties such as program constructs, complexity, dynamic behaviors, and variable types, linking them to LLM reasoning outcomes. Key contributions include a taxonomy of program constructs, multiple complexity metrics including CFG-based cyclomatic complexity, dynamic property extraction per input, and a comprehensive visualization/reporting pipeline. Across six LLMs and four benchmarks, the results show that recursive/nested constructs, higher complexity, longer loops, and non-primitive types hinder reasoning, underscoring the need for more representative benchmarks and execution-aware improvements.

Abstract

Code Executing Reasoning is becoming a new non-functional metric that assesses the ability of large language models (LLMs) in programming tasks. State-of-the-art frameworks (CodeMind or REval) and benchmarks (CruxEval) usually focus on LLM's prediction of a given code's input/output or intermediate variable states/values on limited programs. However, there is no tool for more in-depth analysis of the results. Without such a tool, the observations about LLM's code execution reasoning cannot be generalized to more datasets, preventing the research community and practitioners from devising the next generation of LLMs with better code execution reasoning abilities. This paper introduces ExeRScope, a series of tools and heuristics to analyze the result of code execution reasoning frameworks to understand better the impact of code properties in the studied benchmarks on the code execution reasoning. With such tooling, analysis can be generalized to code with similar properties without the urgent need to design more benchmarks, which is a cumbersome effort.

Figures (5)

• Figure 1: Overview of ExeRScope.
• Figure 2: Impact of different programming constructs on code execution reasoning of LLMs. We abbreviate the tags with B (Basic), F (For loop), I (If statement), NI (Nested Ifs), NL (Nested Loops), T (Try), and W (While loop).
• Figure 3: Impact of cyclomatic complexity on execution reasoning of LLMs. The $\rho$ symbol denotes the Spearman's Rank Order Correlation (ROC) coefficient. The blue dots demonstrate the percentage of correct variable prediction.
• Figure 4: Impact of Loop Length on execution reasoning of LLMs. The $\rho$ symbol denotes the Spearman's Rank Order Correlation (ROC) coefficient. The blue dots demonstrate the percentage of correct variable prediction.
• Figure 5: Performance of LLMs on predicting different variable types and values of variables. TM and VM stand for Type Match and Value Match, respectively.