Large Language Model Critics for Execution-Free Evaluation of Code Changes

TL;DR

This paper tackles the challenge of evaluating code edits produced by agentic workflows without executing the code, by introducing test-centric LLM critics that use a gold test patch as a reference. Micro-evaluation predicts per-test outcomes with high accuracy ($91.6\%$) using context-enhanced patches, and macro-evaluation aggregates these signals to predict build success with a macro $F1$ of $82.1\%$, outperforming several baselines. The approach is shown to align with actual progress across multiple agentic workflows and LLMs, offering a robust, execution-free proxy for patch quality and comparability of workflows. An open-source library is released to enable broader adoption and benchmarking in various SE tasks and languages.

Abstract

Large language models (LLMs) offer a promising way forward for automating software engineering tasks, such as bug fixes, feature additions, etc., via multi-step LLM-based agentic workflows. However, existing metrics for evaluating such workflows, mainly build status and occasionally log analysis, are too sparse and limited in providing the information needed to assess the quality of changes made. In this work, we designed LLM-based critics to derive well-structured and rigorous intermediate/step-level, execution-free evaluation proxies for repo-level code changes. Importantly, we assume access to the gold test patch for the problem (i.e., reference-aware) to assess both semantics and executability of generated patches. With the gold test patch as a reference, we predict executability of all editing locations with an F1 score of 91.6%, aggregating which, we can predict the build status in 84.8% of the instances in SWE-bench. In particular, such an execution-focused LLM critic outperforms other reference-free and reference-aware LLM critics by 38.9% to 72.5%. Moreover, we demonstrate the usefulness of such a reference-aware framework in comparing patches generated by different agentic workflows. Finally, we open-source the library developed for this project, which allows further usage for either other agentic workflows or other benchmarks. The source code is available at https://github.com/amazon-science/code-agent-eval.

Figures (4)

• Figure 1: A snapshot of SWE-bench leaderboard for different agentic workflows, as of July 2024. DBLP:conf/iclr/JimenezYWYPPN24 Instance resolution rate provides low discrimination between the different models nor is not informative over the failures or partial progress rate.
• Figure 2: Heatmap graph of predicted and effective test pass rate for all instances of SWE benchmark. Experiment setting and results are further described in \ref{['sec:intrinsic-micro']}
• Figure 3: ( left) Distribution of verbalized confidence scores for both correct and incorrect isolated, test-aware LLM critic predictions. ( right) Plot of these confidence scores against test character length, as a proxy to test complexity. Here, the highlighted region indicates the confidence threshold.
• Figure 4: ( left) Task instance distribution by LLM-predicted test pass rates. ( right) Density plot of the Spearman's ranking coefficient between LLM-predicted and actual task progress rankings.