Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets

Partha Chakraborty, Krishna Kanth Arumugam, Mahmoud Alfadel, Meiyappan Nagappan, Shane McIntosh

TL;DR

The paper addresses the mismatch between reported performance of deep-learning vulnerability detectors and their effectiveness in real-world codebases. It introduces Real-Vul, a realistic dataset with full project code and label-consistency, and rigorously evaluates four state-of-the-art models, plus augmentation strategies. The results reveal substantial drops in precision and F1 in realistic settings, with large false-positive rates and variability across vulnerability types and severities; dead-code augmentation and training-data realism partially mitigate these issues. The work underscores the need for new modeling and evaluation paradigms to bridge the gap between academic benchmarks and practical deployment of vulnerability detectors.

Abstract

The impact of software vulnerabilities on everyday software systems is significant. Despite deep learning models being proposed for vulnerability detection, their reliability is questionable. Prior evaluations show high recall/F1 scores of up to 99%, but these models underperform in practical scenarios, particularly when assessed on entire codebases rather than just the fixing commit. This paper introduces Real-Vul, a comprehensive dataset representing real-world scenarios for evaluating vulnerability detection models. Evaluating DeepWukong, LineVul, ReVeal, and IVDetect shows a significant drop in performance, with precision decreasing by up to 95 percentage points and F1 scores by up to 91 points. Furthermore, Model performance fluctuates based on vulnerability characteristics, with better F1 scores for information leaks or code injection than for path resolution or predictable return values. The results highlight a significant performance gap that needs addressing before deploying deep learning-based vulnerability detection in practical settings. Overfitting is identified as a key issue, and an augmentation technique is proposed, potentially improving performance by up to 30%. Contributions include a dataset creation approach for better model evaluation, Real-Vul dataset, and empirical evidence of deep learning models struggling in real-world settings.

Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets

TL;DR

The paper addresses the mismatch between reported performance of deep-learning vulnerability detectors and their effectiveness in real-world codebases. It introduces Real-Vul, a realistic dataset with full project code and label-consistency, and rigorously evaluates four state-of-the-art models, plus augmentation strategies. The results reveal substantial drops in precision and F1 in realistic settings, with large false-positive rates and variability across vulnerability types and severities; dead-code augmentation and training-data realism partially mitigate these issues. The work underscores the need for new modeling and evaluation paradigms to bridge the gap between academic benchmarks and practical deployment of vulnerability detectors.

Abstract

The impact of software vulnerabilities on everyday software systems is significant. Despite deep learning models being proposed for vulnerability detection, their reliability is questionable. Prior evaluations show high recall/F1 scores of up to 99%, but these models underperform in practical scenarios, particularly when assessed on entire codebases rather than just the fixing commit. This paper introduces Real-Vul, a comprehensive dataset representing real-world scenarios for evaluating vulnerability detection models. Evaluating DeepWukong, LineVul, ReVeal, and IVDetect shows a significant drop in performance, with precision decreasing by up to 95 percentage points and F1 scores by up to 91 points. Furthermore, Model performance fluctuates based on vulnerability characteristics, with better F1 scores for information leaks or code injection than for path resolution or predictable return values. The results highlight a significant performance gap that needs addressing before deploying deep learning-based vulnerability detection in practical settings. Overfitting is identified as a key issue, and an augmentation technique is proposed, potentially improving performance by up to 30%. Contributions include a dataset creation approach for better model evaluation, Real-Vul dataset, and empirical evidence of deep learning models struggling in real-world settings.
Paper Structure (14 sections, 4 figures, 8 tables)

This paper contains 14 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Limitations of Existing Datasets
  3. Real-Vul Dataset
  4. Study Design
  5. Employed Models
  6. Evaluation Metrics.
  7. Research Questions
  8. Preliminary Analysis
  9. Results
  10. Discussion
  11. Related Work
  12. Threats to Validity
  13. Implication
  14. Conclusion

Figures (4)

  • Figure 1: Example of label inconsistency.
  • Figure 2: Scatter plots showing the class separation between vulnerable and uncertain samples in the original datasets of the approaches. • denotes vulnerable samples and • denotes uncertain samples.
  • Figure 3: Scatter plots showing the class separation between the vulnerable and uncertain samples. • denotes vulnerable samples and • denotes uncertain samples.
  • Figure 5: Performance of the models per SFP cluster.