Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Fact Selection Problem in LLM-Based Program Repair

Nikhil Parasaram, Huijie Yan, Boyu Yang, Zineb Flahy, Abriele Qudsi, Damian Ziaber, Earl Barr, Sergey Mechtaev

TL;DR

The paper tackles the problem of selecting which bug-related facts to include in prompts for LLM-based automated program repair, showing that both the type and the number of facts impact success in a non-monotonic manner. It introduces seven bug-related facts (including angelic values) and conducts a large-scale study on 314 BugsInPy bugs, using over 19K prompts with GPT-3.5-Turbo-0125 and llama3-70B. It proves there is no universally optimal fact set and presents Maniple, a bug-tailored random-forest model that selects fact subsets to maximize prompt effectiveness; on a testing set of 157 bugs, Maniple repairs 88 bugs, about 17% more than the best generic configuration. The work highlights the importance of tailored prompt information in LLM-based program repair and provides datasets, code, and a practical method to guide future research in fact-selection strategies.

Abstract

Recent research has shown that incorporating bug-related facts, such as stack traces and GitHub issues, into prompts enhances the bug-fixing capabilities of large language models (LLMs). Considering the ever-increasing context window of these models, a critical question arises: what and how many facts should be included in prompts to maximise the chance of correctly fixing bugs? To answer this question, we conducted a large-scale study, employing over 19K prompts featuring various combinations of seven diverse facts to rectify 314 bugs from open-source Python projects within the BugsInPy benchmark. Our findings revealed that each fact, ranging from simple syntactic details like code context to semantic information previously unexplored in the context of LLMs such as angelic values, is beneficial. Specifically, each fact aids in fixing some bugs that would remain unresolved or only be fixed with a low success rate without it. Importantly, we discovered that the effectiveness of program repair prompts is non-monotonic over the number of used facts; using too many facts leads to subpar outcomes. These insights led us to define the fact selection problem: determining the optimal set of facts for inclusion in a prompt to maximise LLM's performance on a given task instance. We found that there is no one-size-fits-all set of facts for bug repair. Therefore, we developed a basic statistical model, named Maniple, which selects facts specific to a given bug to include in the prompt. This model significantly surpasses the performance of the best generic fact set. To underscore the significance of the fact selection problem, we benchmarked Maniple against the state-of-the-art zero-shot, non-conversational LLM-based bug repair methods. On our testing dataset of 157 bugs, Maniple repairs 88 bugs, 17% above the best configuration.

The Fact Selection Problem in LLM-Based Program Repair

TL;DR

The paper tackles the problem of selecting which bug-related facts to include in prompts for LLM-based automated program repair, showing that both the type and the number of facts impact success in a non-monotonic manner. It introduces seven bug-related facts (including angelic values) and conducts a large-scale study on 314 BugsInPy bugs, using over 19K prompts with GPT-3.5-Turbo-0125 and llama3-70B. It proves there is no universally optimal fact set and presents Maniple, a bug-tailored random-forest model that selects fact subsets to maximize prompt effectiveness; on a testing set of 157 bugs, Maniple repairs 88 bugs, about 17% more than the best generic configuration. The work highlights the importance of tailored prompt information in LLM-based program repair and provides datasets, code, and a practical method to guide future research in fact-selection strategies.

Abstract

Recent research has shown that incorporating bug-related facts, such as stack traces and GitHub issues, into prompts enhances the bug-fixing capabilities of large language models (LLMs). Considering the ever-increasing context window of these models, a critical question arises: what and how many facts should be included in prompts to maximise the chance of correctly fixing bugs? To answer this question, we conducted a large-scale study, employing over 19K prompts featuring various combinations of seven diverse facts to rectify 314 bugs from open-source Python projects within the BugsInPy benchmark. Our findings revealed that each fact, ranging from simple syntactic details like code context to semantic information previously unexplored in the context of LLMs such as angelic values, is beneficial. Specifically, each fact aids in fixing some bugs that would remain unresolved or only be fixed with a low success rate without it. Importantly, we discovered that the effectiveness of program repair prompts is non-monotonic over the number of used facts; using too many facts leads to subpar outcomes. These insights led us to define the fact selection problem: determining the optimal set of facts for inclusion in a prompt to maximise LLM's performance on a given task instance. We found that there is no one-size-fits-all set of facts for bug repair. Therefore, we developed a basic statistical model, named Maniple, which selects facts specific to a given bug to include in the prompt. This model significantly surpasses the performance of the best generic fact set. To underscore the significance of the fact selection problem, we benchmarked Maniple against the state-of-the-art zero-shot, non-conversational LLM-based bug repair methods. On our testing dataset of 157 bugs, Maniple repairs 88 bugs, 17% above the best configuration.
Paper Structure (22 sections, 8 equations, 7 figures, 5 tables)

This paper contains 22 sections, 8 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Motivating example
  3. Study Design
  4. Experimental Setup
  5. Bug-Related Facts
  6. Prompt Design
  7. The Fact Selection Problem
  8. Fact Utility
  9. Impact of Fact Set Size on Prompt Performance
  10. Non-Existence of a Universally Optimal Fact Set
  11. Effect of Fact Order on Performance
  12. Selecting Facts with Maniple
  13. Feature Selection
  14. Maniple: A Random Forest for Fact Set Selection
  15. Comparing Maniple with SOTA LLM-Based APR
  16. ...and 7 more sections

Figures (7)

  • Figure 1: A simplified APR prompt incorporating various facts for fixing pandas:128. ... shows information omitted for brevity. ... shows a part of the GitHub issue that is essential to correctly fix the bug. Too much information in the prompt "distracts" the LLM from the relevant part of the issue description, significantly reducing pass@1 and the correctness rate.
  • Figure 2: Comparison of pass@1 (the vertical axis) for around 10K prompts incorporating subsets of the seven considered facts (the horizontal axis) computed over 15 responses for repairing 157 Python bugs within the BugsInPy benchmark. Zero facts corresponds to the prompt containing only the buggy function without any additional information about the bug. The graph plots the average pass@1 score (dashed lines) and the maximum pass@1 score (solid lines) across all bugs. This graph clearly shows the non-monotonic nature of APR prompts over facts for GPT-3.5 and Llama3-70B.
  • Figure 3: This work uses seven facts (dark rectangles) across three categories for constructing program repair prompts. Each fact is composed of related pieces of information . Each prompt contains the buggy function to be repaired; the facts can be included based on the employed fact selection strategy.
  • Figure 4: This upset diagram compares the 5 fact sets that fix the most bugs in BGP157Ply1 with using no facts (encoded as $0000000$) including the buggy function and chain-of-thought instructions. The fact sets shown on the left collectively fix 107 bugs, 18 more than the single best fact set, which fixes 89 bugs. The diagram is constructed by generating prompts from each fact set, sending them to GPT-3.5 for $n=15$ responses, and identifying the set of bugs with at least one passing response.
  • Figure 5: Distribution of mean pass@k evaluated on BGP32. The violet histogram reports the mean performance under permutation; the yellow histogram reports mean by subset. The wider range in the yellow distribution suggests that fact subsets have a stronger impact on performance than fact order.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Definition 4.1: Fact Selection Problem