Security Bug Report Prediction Within and Across Projects: A Comparative Study of BERT and Random Forest

TL;DR

This work rigorously compares transformer-based BERT and a strong non-deep baseline RF for predicting security bug reports, across both within-project and cross-project settings using five public datasets. It reveals a nuanced picture: RF generally wins within-project, yet when training data is augmented with external BRs/NSBRs, BERT’s performance improves markedly, and in cross-project scenarios BERT achieves substantially higher $G$-measures. The study also demonstrates that including mixed security and non-security bug reports can cripple RF while boosting BERT, and that cross-project transfer with BERT benefits from combining data across multiple projects. A replication package is released to support future research and benchmarking, highlighting the practical impact of data augmentation and cross-project strategies for SBR detection.

Abstract

Early detection of security bug reports (SBRs) is crucial for preventing vulnerabilities and ensuring system reliability. While machine learning models have been developed for SBR prediction, their predictive performance still has room for improvement. In this study, we conduct a comprehensive comparison between BERT and Random Forest (RF), a competitive baseline for predicting SBRs. The results show that RF outperforms BERT with a 34% higher average G-measure for within-project predictions. Adding only SBRs from various projects improves both models' average performance. However, including both security and nonsecurity bug reports significantly reduces RF's average performance to 46%, while boosts BERT to its best average performance of 66%, surpassing RF. In cross-project SBR prediction, BERT achieves a remarkable 62% G-measure, which is substantially higher than RF.

Figures (1)

• Figure 1: Sequential Dataset Partitioning for WPP