Pre-trained Model-based Actionable Warning Identification: A Feasibility Study

TL;DR

The paper addresses the challenge of actionable warning identification (AWI) in static analysis by evaluating pre-trained transformer models (PTMs) on a large SpotBugs warning dataset. It demonstrates that PTMs surpass the current state-of-the-art ML-based AWI approaches in AUC, with CodeBERT achieving up to $70.77\%$ and improvements up to $21.12\%$ over DL baselines. The study systematically analyzes the impact of data preprocessing, training (pre-training and fine-tuning), and prediction (within- vs cross-project) on AWI performance, and explains reasons for underperformance such as limited warning context and lack of diverse training data. Practical guidelines for enhancing PTM-based AWI are offered, including refined warning contexts and broader, more diverse training resources, and the authors release their dataset and scripts for replication and future work.

Abstract

Actionable Warning Identification (AWI) plays a pivotal role in improving the usability of static code analyzers. Currently, Machine Learning (ML)-based AWI approaches, which mainly learn an AWI classifier from labeled warnings, are notably common. However, these approaches still face the problem of restricted performance due to the direct reliance on a limited number of labeled warnings to develop a classifier. Very recently, Pre-Trained Models (PTMs), which have been trained through billions of text/code tokens and demonstrated substantial success applications on various code-related tasks, could potentially circumvent the above problem. Nevertheless, the performance of PTMs on AWI has not been systematically investigated, leaving a gap in understanding their pros and cons. In this paper, we are the first to explore the feasibility of applying various PTMs for AWI. By conducting the extensive evaluation on 10K+ SpotBugs warnings from 10 large-scale and open-source projects, we observe that all studied PTMs are consistently 9.85%~21.12% better than the state-of-the-art ML-based AWI approaches. Besides, we investigate the impact of three primary aspects (i.e., data preprocessing, model training, and model prediction) in the typical PTM-based AWI workflow. Further, we identify the reasons for current PTMs' underperformance on AWI. Based on our findings, we provide several practical guidelines to enhance PTM-based AWI in future work.

Figures (7)

• Figure 1: Overview of our study.
• Figure 2: AUC of DL-based and PTM-based AWI approaches.
• Figure 3: AUC of PTMs on AWI in the data preprocessing.
• Figure 4: AUC of PTMs on AWI with/without pre-training.
• Figure 5: AUC trend of PTMs on AWI with different fine-tuning corpora.