Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-shot learning for security bug report identification

Muhammad Laiq

TL;DR

The paper addresses the challenge of identifying security-related bug reports when labeled data are scarce. It adopts SetFit, a few-shot learning framework that combines sentence-transformer embeddings with contrastive learning and parameter-efficient fine-tuning, and compares it against traditional baselines on four public datasets. Results show that SetFit achieves up to an AUC of 0.865 and generally outperforms baselines across all datasets, demonstrating strong performance with limited data. This suggests SetFit as a practical, data-efficient approach for security bug report triage, enabling faster vulnerability identification with reduced annotation effort.

Abstract

Security bug reports require prompt identification to minimize the window of vulnerability in software systems. Traditional machine learning (ML) techniques for classifying bug reports to identify security bug reports rely heavily on large amounts of labeled data. However, datasets for security bug reports are often scarce in practice, leading to poor model performance and limited applicability in real-world settings. In this study, we propose a few-shot learning-based technique to effectively identify security bug reports using limited labeled data. We employ SetFit, a state-of-the-art few-shot learning framework that combines sentence transformers with contrastive learning and parameter-efficient fine-tuning. The model is trained on a small labeled dataset of bug reports and is evaluated on its ability to classify these reports as either security-related or non-security-related. Our approach achieves an AUC of 0.865, at best, outperforming traditional ML techniques (baselines) for all of the evaluated datasets. This highlights the potential of SetFit to effectively identify security bug reports. SetFit-based few-shot learning offers a promising alternative to traditional ML techniques to identify security bug reports. The approach enables efficient model development with minimal annotation effort, making it highly suitable for scenarios where labeled data is scarce.

Few-shot learning for security bug report identification

TL;DR

The paper addresses the challenge of identifying security-related bug reports when labeled data are scarce. It adopts SetFit, a few-shot learning framework that combines sentence-transformer embeddings with contrastive learning and parameter-efficient fine-tuning, and compares it against traditional baselines on four public datasets. Results show that SetFit achieves up to an AUC of 0.865 and generally outperforms baselines across all datasets, demonstrating strong performance with limited data. This suggests SetFit as a practical, data-efficient approach for security bug report triage, enabling faster vulnerability identification with reduced annotation effort.

Abstract

Security bug reports require prompt identification to minimize the window of vulnerability in software systems. Traditional machine learning (ML) techniques for classifying bug reports to identify security bug reports rely heavily on large amounts of labeled data. However, datasets for security bug reports are often scarce in practice, leading to poor model performance and limited applicability in real-world settings. In this study, we propose a few-shot learning-based technique to effectively identify security bug reports using limited labeled data. We employ SetFit, a state-of-the-art few-shot learning framework that combines sentence transformers with contrastive learning and parameter-efficient fine-tuning. The model is trained on a small labeled dataset of bug reports and is evaluated on its ability to classify these reports as either security-related or non-security-related. Our approach achieves an AUC of 0.865, at best, outperforming traditional ML techniques (baselines) for all of the evaluated datasets. This highlights the potential of SetFit to effectively identify security bug reports. SetFit-based few-shot learning offers a promising alternative to traditional ML techniques to identify security bug reports. The approach enables efficient model development with minimal annotation effort, making it highly suitable for scenarios where labeled data is scarce.
Paper Structure (17 sections, 6 equations, 1 figure, 2 tables)

This paper contains 17 sections, 6 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Research Method
  3. Selection of techniques
  4. Selection of datasets
  5. Evaluation approach
  6. Performance evaluation metrics
  7. Data preprocessing
  8. Implementation details
  9. Results and analysis
  10. Discussion
  11. Threats to validity
  12. Generalizability of results:
  13. Validity of the evaluation approach:
  14. Validity of the evaluation metrics:
  15. Reliability of training data:
  16. ...and 2 more sections

Figures (1)

  • Figure 1: AUC, MCC, F-score, Recall, and Precision of the evaluated techniques (Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF), and SetFit) for identifying security bug reports.