APPT: Boosting Automated Patch Correctness Prediction via Fine-tuning Pre-trained Models

TL;DR

APPT tackles the persistent overfitting problem in automated program repair by integrating a pre-trained encoder with end-to-end fine-tuning, a bidirectional LSTM stack, and a neural classifier to assess patch correctness. The approach tokenizes buggy and patched code, encodes them with a pre-trained model, fuses representations, and models their interactions to predict whether a patch is correct or overfitting. Comprehensive experiments on 1,183 Defects4J patches and 49,694 real-world patches show APPT outperforms state-of-the-art baselines (notably CACHE) across accuracy, precision, recall, F1, and AUC, with especially large gains on the small dataset and near-perfect performance on the large dataset; CodeBERT and GraphCodeBERT further improve results. The study demonstrates the value of fine-tuning pre-trained models for patch correctness reasoning and suggests strong generalizability and practical impact for APCA, including cross-project effectiveness and guidance for integrating sequence-based code representations with pre-trained knowledge.

Abstract

Automated program repair (APR) aims to fix software bugs automatically without human debugging efforts and plays a crucial role in software development and maintenance. Despite promising, APR is still challenged by a long-standing overfitting problem (i.e., the generated patch is plausible but overfitting). Various techniques have thus been proposed to address the overfitting problem. Recently, researchers have employed BERT to extract code features, which are then used to train a classifier for patch correctness prediction. However, BERT is restricted to feature extraction for classifier training without benefiting from the training process, potentially generating sub-optimal vector representations for patched code snippets. In this paper, we propose APPT, a pre-trained model-based automated patch correctness assessment technique by both pre-training and fine-tuning. APPT adopts a pre-trained model as the encoder stack, followed by an LSTM stack and a deep learning classifier. More importantly, the pre-trained model is fine-tuned in conjunction with other components as a whole pipeline to fully adapt it specifically for reasoning about patch correctness. We conduct an extensive experiment on 1,183 Defects4J patches and the experimental results show that APPT achieves prediction accuracy of 79.7% and recall of 83.2%, outperforming CACHE by 4.3% and 6.7%. Our additional investigation on 49,694 real-world patches shows that APPT achieves the optimum performance compared with existing representation learning techniques. We further investigate the impact of each component and find that they all positively contribute to APPT, e.g., the fine-tuning process and the LSTM stack increase F1-score by 10.22% and 4.11%, respectively. We also prove that adopting advanced pre-trained models can further provide substantial advancement, highlighting the generalizability of APPT.

Figures (3)

• Figure 1: Overview of APR
• Figure 2: Overview of APPT
• Figure 3: APR-generated and developer patches for Math-53