R+R: Security Vulnerability Dataset Quality Is Critical

TL;DR

The paper tackles a critical bottleneck in vulnerability repair research: the reliability of the datasets used to train and evaluate models. By systematically identifying and removing duplicates, inconsistent labels, and incomplete samples in the VulRepair dataset, the authors show that previous performance claims were overstated due to data quality issues. They perform replication across several models, assess the impact of data cleaning, and explore transfer learning with a deduplicated bug-fix corpus, demonstrating that high-quality pre-training data can yield substantial performance gains. The work underscores the need for rigorous data curation and transparent reporting to derive credible conclusions about vulnerability repair capabilities and to guide future dataset construction, benchmark establishment, and model development.

Abstract

Large Language Models (LLMs) are of great interest in vulnerability detection and repair. The effectiveness of these models hinges on the quality of the datasets used for both training and evaluation. Our investigation reveals that a number of studies featured in prominent software engineering conferences have employed datasets that are plagued by high duplication rates, questionable label accuracy, and incomplete samples. Using these datasets for experimentation will yield incorrect results that are significantly different from actual expected behavior. For example, the state-of-the-art VulRepair Model, which is reported to have 44% accuracy, on average yielded 9% accuracy when test-set duplicates were removed from its training set and 13% accuracy when training-set duplicates were removed from its test set. In an effort to tackle these data quality concerns, we have retrained models from several papers without duplicates and conducted an accuracy assessment of labels for the top ten most hazardous Common Weakness Enumerations (CWEs). Our findings indicate that 56% of the samples had incorrect labels and 44% comprised incomplete samples--only 31% were both accurate and complete. Finally, we employ transfer learning using a large deduplicated bugfix corpus to show that these models can exhibit better performance if given larger amounts of high-quality pre-training data, leading us to conclude that while previous studies have over-estimated performance due to poor dataset quality, this does not demonstrate that better performance is not possible.

Figures (1)

• Figure 1: %PP as a Function of Beam Size for RQ5 (Transfer Learning)