PatchGuru: Patch Oracle Inference from Natural Language Artifacts with Large Language Models
Thanh Le-Cong, Bach Le, Toby Murray, Michael Pradel, Cristian Cadar
TL;DR
PatchGuru tackles the lack of executable specifications for patches by automatically inferring patch oracles from NL artifacts associated with PRs using LLMs and dynamic analysis. It frames the patch intent as a cross version delta between pre and post patch behaviors and validates it via comparison programs that execute pre and post implementations side by side. The approach yields 24 real inconsistencies across 400 PRs, including 12 previously unknown bugs, with a precision of 0.62 and higher mutation scores than a strong baseline, Testora. It operates with a practical cost of roughly 8.9 minutes and USD 0.07 per PR, making it suitable for CI workflows and as a complement to code reviews and regression tests. Overall, PatchGuru demonstrates that executable patch documentation derived from NL artifacts can meaningfully improve patch validation and developer efficiency in real-world open source projects.
Abstract
As software systems evolve, patches may unintentionally alter program behavior. Validating patches against their intended semantics is difficult due to incomplete regression tests and informal, non-executable natural language (NL) descriptions of patch intent. We present PatchGuru, the first automated technique that infers executable patch specifications from real-world pull requests (PRs). Given a PR, PatchGuru uses large language models (LLMs) to extract developer intent from NL artifacts and synthesizes patch oracles: under-approximate yet practical specifications expressed as runtime assertions in comparison programs that integrate pre- and post-patch versions. Patch oracles focus on patch-relevant behaviors, enable automated validation, and support cross-version properties. PatchGuru iteratively refines inferred oracles by comparing pre- and post-patch behaviors, identifies violations, filters inconsistencies via self-review, and generates bug reports. We evaluate PatchGuru on 400 recent PRs from four widely used open-source Python projects. PatchGuru reports 39 warnings with a precision of 0.62, yielding 24 confirmed true positives, including 12 previously unknown bugs, 11 of which were subsequently fixed by developers. Compared to the state-of-the-art technique Testora, PatchGuru detects 17 more bugs (24 vs. 7) while improving precision from 0.32 to 0.62. PatchGuru incurs an average cost of 8.9 minutes and USD 0.07 per PR. These results suggest that PatchGuru complements code review and regression testing by providing executable documentation and automated validation of patch intent.