Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models for Security Bug Report Prediction

Farnaz Soltaniani, Shoaib Razzaq, Mohammad Ghafari

TL;DR

The paper benchmarks prompt-based proprietary LLMs and small fine-tuned LLMs for security bug report (SBR) prediction across five public datasets, revealing a clear recall-precision trade-off. Gemini (prompted) achieves high recall and G-measure but incurs high false positives, latency, and cost, whereas GPT is more conservative with faster inference. Among fine-tuned models, DistilBERT delivers the strongest average performance (G-measure ≈ 0.51) with higher precision and substantially faster inference than proprietary LLMs, though at the cost of lower recall. The results highlight practical trade-offs between coverage and accuracy, emphasize the importance of data privacy considerations, and suggest directions for combining models or further tuning to improve SBR detection. Overall, the study demonstrates that LLM-based approaches have potential for SBR prediction but require careful selection of method and deployment strategy depending on operational constraints.

Abstract

Early detection of security bug reports (SBRs) is critical for timely vulnerability mitigation. We present an evaluation of prompt-based engineering and fine-tuning approaches for predicting SBRs using Large Language Models (LLMs). Our findings reveal a distinct trade-off between the two approaches. Prompted proprietary models demonstrate the highest sensitivity to SBRs, achieving a G-measure of 77% and a recall of 74% on average across all the datasets, albeit at the cost of a higher false-positive rate, resulting in an average precision of only 22%. Fine-tuned models, by contrast, exhibit the opposite behavior, attaining a lower overall G-measure of 51% but substantially higher precision of 75% at the cost of reduced recall of 36%. Though a one-time investment in building fine-tuned models is necessary, the inference on the largest dataset is up to 50 times faster than that of proprietary models. These findings suggest that further investigations to harness the power of LLMs for SBR prediction are necessary.

Evaluating Large Language Models for Security Bug Report Prediction

TL;DR

The paper benchmarks prompt-based proprietary LLMs and small fine-tuned LLMs for security bug report (SBR) prediction across five public datasets, revealing a clear recall-precision trade-off. Gemini (prompted) achieves high recall and G-measure but incurs high false positives, latency, and cost, whereas GPT is more conservative with faster inference. Among fine-tuned models, DistilBERT delivers the strongest average performance (G-measure ≈ 0.51) with higher precision and substantially faster inference than proprietary LLMs, though at the cost of lower recall. The results highlight practical trade-offs between coverage and accuracy, emphasize the importance of data privacy considerations, and suggest directions for combining models or further tuning to improve SBR detection. Overall, the study demonstrates that LLM-based approaches have potential for SBR prediction but require careful selection of method and deployment strategy depending on operational constraints.

Abstract

Early detection of security bug reports (SBRs) is critical for timely vulnerability mitigation. We present an evaluation of prompt-based engineering and fine-tuning approaches for predicting SBRs using Large Language Models (LLMs). Our findings reveal a distinct trade-off between the two approaches. Prompted proprietary models demonstrate the highest sensitivity to SBRs, achieving a G-measure of 77% and a recall of 74% on average across all the datasets, albeit at the cost of a higher false-positive rate, resulting in an average precision of only 22%. Fine-tuned models, by contrast, exhibit the opposite behavior, attaining a lower overall G-measure of 51% but substantially higher precision of 75% at the cost of reduced recall of 36%. Though a one-time investment in building fine-tuned models is necessary, the inference on the largest dataset is up to 50 times faster than that of proprietary models. These findings suggest that further investigations to harness the power of LLMs for SBR prediction are necessary.
Paper Structure (21 sections, 1 equation, 2 figures, 6 tables)

This paper contains 21 sections, 1 equation, 2 figures, 6 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Methodology
  4. Dataset
  5. Train and Test Splits
  6. Proprietary LLMs
  7. Fine-Tuned LLMs
  8. BERT
  9. DistilBERT
  10. DistilGPT2
  11. Qwen2.5-Coder
  12. Performance Metrics
  13. RESULTS
  14. Discussion
  15. Zero-shot and Few-shot Prompting.
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Sequential Dataset Partitioning.
  • Figure 2: Prediction consistency scores between GPT and Gemini across five datasets.