Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning

Xuhui Dou, Hayretdin Bahsi, Alejandro Guerra-Manzanares

TL;DR

Investigation of continual fine-tuning of a decoder-style language model on a CVE-linked dataset spanning 2018-2024, organised into bi-monthly windows shows that selective replay with class balancing offers a practical accuracy-efficiency trade-off for LLM-based temporal vulnerability detection under continuous temporal drift.

Abstract

Recent work applies Large Language Models (LLMs) to source-code vulnerability detection, but most evaluations still rely on random train-test splits that ignore time and overestimate real-world performance. In practice, detectors are deployed on evolving code bases and must recognise future vulnerabilities under temporal distribution shift. This paper investigates continual fine-tuning of a decoder-style language model (microsoft/phi-2 with LoRA) on a CVE-linked dataset spanning 2018-2024, organised into bi-monthly windows. We evaluate eight continual learning strategies, including window-only and cumulative training, replay-based baselines and regularisation-based variants. We propose Hybrid Class-Aware Selective Replay (Hybrid-CASR), a confidence-aware replay method for binary vulnerability classification that prioritises uncertain samples while maintaining a balanced ratio of VULNERABLE and FIXED functions in the replay buffer. On bi-monthly forward evaluation Hybrid-CASR achieves a Macro-F1 of 0.667, improving on the window-only baseline (0.651) by 0.016 with statistically significant gains ($p = 0.026$) and stronger backward retention (IBR@1 of 0.741). Hybrid-CASR also reduces training time per window by about 17 percent compared to the baseline, whereas cumulative training delivers only a minor F1 increase (0.661) at a 15.9-fold computational cost. Overall, the results show that selective replay with class balancing offers a practical accuracy-efficiency trade-off for LLM-based temporal vulnerability detection under continuous temporal drift.

Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning

TL;DR

Investigation of continual fine-tuning of a decoder-style language model on a CVE-linked dataset spanning 2018-2024, organised into bi-monthly windows shows that selective replay with class balancing offers a practical accuracy-efficiency trade-off for LLM-based temporal vulnerability detection under continuous temporal drift.

Abstract

Recent work applies Large Language Models (LLMs) to source-code vulnerability detection, but most evaluations still rely on random train-test splits that ignore time and overestimate real-world performance. In practice, detectors are deployed on evolving code bases and must recognise future vulnerabilities under temporal distribution shift. This paper investigates continual fine-tuning of a decoder-style language model (microsoft/phi-2 with LoRA) on a CVE-linked dataset spanning 2018-2024, organised into bi-monthly windows. We evaluate eight continual learning strategies, including window-only and cumulative training, replay-based baselines and regularisation-based variants. We propose Hybrid Class-Aware Selective Replay (Hybrid-CASR), a confidence-aware replay method for binary vulnerability classification that prioritises uncertain samples while maintaining a balanced ratio of VULNERABLE and FIXED functions in the replay buffer. On bi-monthly forward evaluation Hybrid-CASR achieves a Macro-F1 of 0.667, improving on the window-only baseline (0.651) by 0.016 with statistically significant gains () and stronger backward retention (IBR@1 of 0.741). Hybrid-CASR also reduces training time per window by about 17 percent compared to the baseline, whereas cumulative training delivers only a minor F1 increase (0.661) at a 15.9-fold computational cost. Overall, the results show that selective replay with class balancing offers a practical accuracy-efficiency trade-off for LLM-based temporal vulnerability detection under continuous temporal drift.
Paper Structure (29 sections, 4 figures, 7 tables)

This paper contains 29 sections, 4 figures, 7 tables.

Table of Contents

  1. INTRODUCTION
  2. METHODOLOGY
  3. Experimental Setup
  4. Model Architecture and Configuration
  5. Dataset Construction and Temporal Design
  6. Data Sources and Quality Control
  7. Function-level Instance Generation
  8. Temporal Window Granularity
  9. Experimental Protocol
  10. Forward Temporal Evaluation Framework
  11. Backward Retention Assessment
  12. Continual Learning Strategies
  13. Evaluation Metrics and Statistical Analysis
  14. Evaluation Metrics
  15. Temporal Aggregation and Statistical Testing
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Distribution of vulnerability disclosures across bi-monthly windows (2018--2024), illustrating volume growth and temporal variability.
  • Figure 2: Forward one-step F1 scores across different temporal granularities (1, 2, 3, 6, 12 months), showing variability and trends over chronological evaluation windows.
  • Figure 3: Forward F1 comparison across all methods using bi-monthly windows, highlighting differences in performance level and stability over time.
  • Figure 4: Plasticity--stability trade-off, showing the relationship between forward learning capability and performance consistency. Methods in the upper-right region achieve the best balance.