PatchZero: Zero-Shot Automatic Patch Correctness Assessment

TL;DR

The paper tackles patch overfitting in Automated Program Repair by proposing LLM4PatchCorrect, a zero-shot patch correctness assessment framework that uses in-context learning with an open-source code LLM (Starcoder-7B). It retrieves semantically similar labeled patches via a contrastive embedding model and enriches inputs with bug-descriptions, execution traces, failing tests, and test coverage to guide the model’s judgment. Across cross-tool experiments on Defects4J, LLM4PatchCorrect achieves an average accuracy of $84.4\%$, F1 of $86.5\%$, and AUC of $80.4\%$, outperforming strong static and dynamic baselines with significant gains. The approach also performs well on the Bears benchmark and demonstrates practical utility as a patch-filter to reduce manual verification, with robust behavior across multiple LLMs and configurations. This work suggests that zero-shot, context-rich LLM-based APCA can effectively transfer knowledge from existing labeled data to unseen APR tools, enabling scalable and efficient patch correctness assessment.

Abstract

Automated Program Repair (APR) techniques have shown more and more promising results in fixing real-world bugs. Despite the effectiveness, APR techniques still face an overfitting problem: a generated patch can be incorrect although it passes all tests. It is time-consuming to manually evaluate the correctness of generated patches that can pass all tests. To address this problem, many approaches have been proposed to automatically assess the correctness of patches generated by APR techniques. These approaches are mainly evaluated within the cross-validation setting. However, for patches generated by a new or unseen APR tool, users are implicitly required to manually label a significant portion of these patches in the cross-validation setting before inferring the remaining patches. To mitigate the issue, in this study, we propose \toolname, the patch correctness assessment by adopting a large language model for code. Specifically, for patches generated by a new or unseen APR tool, \toolname does not need labeled patches of this new or unseen APR tool for training but directly queries the large language model for code to get predictions on the correctness labels without training. In this way, \toolname can reduce the manual labeling effort when building a model to automatically assess the correctness of generated patches of new APR tools. \toolname prioritizes labeled patches from existing APR tools that exhibit semantic similarity to those generated by new APR tools, enhancing the accuracy achieved by \toolname for patches from new APR tools. Our experimental results showed that \toolname can achieve an accuracy of 84.4% and an F1-score of 86.5% on average although no labeled patch of the new or unseen APR tool is available. In addition, our proposed technique outperformed the prior state-of-the-art by a large margin.

Figures (10)

• Figure 1: An abstracted example of how to use the large pre-trained models. An unlabeled test patch of a new APR tool is concatenated by several labeled patches of existing APR tools to form the input to the pre-trained model.
• Figure 2: Overall Framework of LLM4PatchCorrect.
• Figure 3: The process of obtaining similar patches from the training set.
• Figure 4: Generating the embeddings of positive samples via the Dropout operation in SIMCSE. It adopts two different dropout operations which results in two different embeddings.
• Figure 5: Training CL-based patch embedding model with the embeddings of the constructed triplet $\langle p, p^{+}, p^{-}\rangle$.