Enhancing Automated Program Repair with Solution Design
Jiuang Zhao, Donghao Yang, Li Zhang, Xiaoli Lian, Zitian Yang, Fang Liu
TL;DR
DRCodePilot introduces a design-rationale–driven, feedback-based framework for automated program repair (APR) that injects real developer reasoning from Jira issue logs into GPT-4–based patch generation. By combining design-rationale acquisition (DRMiner), defective-segment localization, reference patch generation (CodeT5P-220M), identifier recommendations, and iterative refinement, it achieves up to 4.7x more full-match patches than GPT-4 and notable CodeBLEU gains on a 938-patch benchmark from Flink and Solr. Ablation and cross-model analyses demonstrate that high-quality DRs, project-specific references, and identifier feedback substantially improve patch quality, and that the approach generalizes to other LLMs with appropriate DR integration. The work highlights a practical, human-in-the-loop path to more reliable APR and suggests future directions in enhanced DR extraction and retrieval-augmented generation to tackle more complex repairs.
Abstract
Automatic Program Repair (APR) endeavors to autonomously rectify issues within specific projects, which generally encompasses three categories of tasks: bug resolution, new feature development, and feature enhancement. Despite extensive research proposing various methodologies, their efficacy in addressing real issues remains unsatisfactory. It's worth noting that, typically, engineers have design rationales (DR) on solution-planed solutions and a set of underlying reasons-before they start patching code. In open-source projects, these DRs are frequently captured in issue logs through project management tools like Jira. This raises a compelling question: How can we leverage DR scattered across the issue logs to efficiently enhance APR? To investigate this premise, we introduce DRCodePilot, an approach designed to augment GPT-4-Turbo's APR capabilities by incorporating DR into the prompt instruction. Furthermore, given GPT-4's constraints in fully grasping the broader project context and occasional shortcomings in generating precise identifiers, we have devised a feedback-based self-reflective framework, in which we prompt GPT-4 to reconsider and refine its outputs by referencing a provided patch and suggested identifiers. We have established a benchmark comprising 938 issue-patch pairs sourced from two open-source repositories hosted on GitHub and Jira. Our experimental results are impressive: DRCodePilot achieves a full-match ratio that is a remarkable 4.7x higher than when GPT-4 is utilized directly. Additionally, the CodeBLEU scores also exhibit promising enhancements. Moreover, our findings reveal that the standalone application of DR can yield promising increase in the full-match ratio across CodeLlama, GPT-3.5, and GPT-4 within our benchmark suite. We believe that our DRCodePilot initiative heralds a novel human-in-the-loop avenue for advancing the field of APR.