The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models

TL;DR

This work investigates a practical, budget-aware approach to automatic program repair (APR) using instruction-tuned LLMs, balancing multi-output generation with iterative refinement while capping patches per bug at 10. It demonstrates that very small fine-tuning datasets can yield substantial improvements, while larger datasets may cause diminishing returns due to overfitting, and that base models typically benefit more from iterative feedback than finely-tuned ones. Across benchmarks like HumanEval-Java and Defects4J, iterative strategies significantly aid repair, especially for complex bugs, but optimal strategies depend on model tuning and problem difficulty. The study provides actionable guidance for deploying resource-efficient APR pipelines and highlights the value of combining iterative refinement with limited patch generation, supported by a replication package for reproducibility.

Abstract

Automatic program repair (APR) aims to reduce the manual efforts required to identify and fix errors in source code. Before the rise of LLM-based agents, a common strategy was to increase the number of generated patches, sometimes to the thousands, to achieve better repair results on benchmarks. More recently, self-iterative capabilities enabled LLMs to refine patches over multiple rounds guided by feedback. However, literature often focuses on many iterations and disregards different numbers of outputs. We investigate an APR pipeline that balances these two approaches, the generation of multiple outputs and multiple rounds of iteration, while imposing a limit of 10 total patches per bug. We apply three SOTA instruction-tuned LLMs - DeepSeekCoder-Instruct, Codellama-Instruct, Llama3.1-Instruct - to the APR task. We further fine-tune each model on an APR dataset with three sizes (1K, 30K, 65K) and two techniques (Full Fine-Tuning and LoRA), allowing us to assess their repair capabilities on two APR benchmarks: HumanEval-Java and Defects4J. Our results show that by using only a fraction (<1%) of the fine-tuning dataset, we can achieve improvements of up to 78% in the number of plausible patches generated, challenging prior studies that reported limited gains using Full Fine-Tuning. However, we find that exceeding certain thresholds leads to diminishing outcomes, likely due to overfitting. Moreover, we show that base models greatly benefit from creating patches in an iterative fashion rather than generating them all at once. In addition, the benefit of iterative strategies becomes more pronounced in complex benchmarks. Even fine-tuned models, while benefiting less from iterations, still gain advantages, particularly on complex benchmarks. The research underscores the need for balanced APR strategies that combine multi-output generation and iterative refinement.

Figures (4)

• Figure 1: High-level illustration of different APR strategies: (Strategy X) generating a batch of patches all at once; (Strategy Y) iterating over patches in two rounds; (Strategy Z) iterating over one patch over four rounds.
• Figure 2: Position of the first plausible patches found for 10 outputs. Results are shown as proportion for all unique plausible patches found summarized over the 6 fine-tuning configurations for each model. The base model is shown separately.
• Figure 3: Venn diagram of problems with plausible patches generated by the variants of Llama3.1 comparing the least and most iterative strategies applied for each benchmark.
• Figure 4: Impact of different generation strategies on the number of unique problems with at least one plausible patch. The heatmaps show the difference in the number of plausible patches with regard to the default strategy (Strategy A).