Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs

TL;DR

The paper investigates whether augmenting code-language-model prompts with program execution traces improves automatic program repair (APR). Through experiments with GPT-3.5 and GPT-4 on Refactory, RunBugRun, and HumanEval-Java, it shows that traces alone do not consistently outperform trace-free prompts; trace complexity (length and number of variable changes) can negatively impact results. The authors explore modified trace representations, finding that LL M-optimized traces deliver the most robust gains across CFA and CPA, while collated traces and simple routing strategies offer limited benefits. Finetuning a smaller model generally underperforms compared to prompting-based approaches, and probing studies reveal that LLMs have nontrivial limitations in generating or aligning traces. Overall, traces can complement LLM reasoning in APR when carefully formatted, but their effectiveness depends on prompt design and model capabilities, with practical implications for enabling trace-aware code repair systems.

Abstract

Large Language Models (LLMs) show promising performance on various programming tasks, including Automatic Program Repair (APR). However, most approaches to LLM-based APR are limited to the static analysis of the programs, while disregarding their runtime behavior. Inspired by knowledge-augmented NLP, in this work, we aim to remedy this potential blind spot by augmenting standard APR prompts with program execution traces. We evaluate our approach using the GPT family of models on three popular APR datasets. Our findings suggest that simply incorporating execution traces into the prompt provides a limited performance improvement over trace-free baselines, in only 2 out of 6 tested dataset / model configurations. We further find that the effectiveness of execution traces for APR diminishes as their complexity increases. We explore several strategies for leveraging traces in prompts and demonstrate that LLM-optimized prompts help outperform trace-free prompts more consistently. Additionally, we show trace-based prompting to be superior to finetuning a smaller LLM on a small-scale dataset; and conduct probing studies reinforcing the notion that execution traces can complement the reasoning abilities of the LLMs.

Figures (7)

• Figure 1: Example buggy program, a failing test case and its execution trace. While the failure message simply indicates the output is wrong, the execution trace provides a detailed explanation how it was produced.
• Figure 2: Distributions of trace lengths and variable changes across correct vs incorrect program fixes generated by GPT-4. Analysis for GPT-3.5, showing a similar trend, is given in the appendix.
• Figure 3: The Flow of Conditional Selection of Traces
• Figure 4: CPA and CFA of Prompting-Based Techniques vs. Model Fine-Tuned for APR Tasks.
• Figure 5: Distributions of trace lengths and variable changes across correct vs incorrect program fixes generated by GPT-3.5