Automated Code-centric Software Vulnerability Assessment: How Far Are We? An Empirical Study in C/C++

TL;DR

This study tackles the problem of function-level SV assessment in C/C++ using CVSS metrics, addressing a gap where most work targets SV detection rather than assessment. It empirically compares multiple ML and DL approaches, using a 9,993-record dataset derived from Big-Vul and CVSSv2 outputs, and evaluates both standard multi-class models and multi-task DL variants. Key findings show that a carefully tuned ML baseline (LGBM with BoST features) is competitive with DL, while multi-task DL consistently improves performance (8-22% MCC gains), with multi-task CodeBERT delivering the best overall results and notable efficiency gains. The work provides practical guidance on selecting data-driven approaches for function-level SV assessment, highlights efficiency considerations, and releases data, code, and models to support reproducible future research.

Abstract

Background: The C and C++ languages hold significant importance in Software Engineering research because of their widespread use in practice. Numerous studies have utilized Machine Learning (ML) and Deep Learning (DL) techniques to detect software vulnerabilities (SVs) in the source code written in these languages. However, the application of these techniques in function-level SV assessment has been largely unexplored. SV assessment is increasingly crucial as it provides detailed information on the exploitability, impacts, and severity of security defects, thereby aiding in their prioritization and remediation. Aims: We conduct the first empirical study to investigate and compare the performance of ML and DL models, many of which have been used for SV detection, for function-level SV assessment in C/C++. Method: Using 9,993 vulnerable C/C++ functions, we evaluated the performance of six multi-class ML models and five multi-class DL models for the SV assessment at the function level based on the Common Vulnerability Scoring System (CVSS). We further explore multi-task learning, which can leverage common vulnerable code to predict all SV assessment outputs simultaneously in a single model, and compare the effectiveness and efficiency of this model type with those of the original multi-class models. Results: We show that ML has matching or even better performance compared to the multi-class DL models for function-level SV assessment with significantly less training time. Employing multi-task learning allows the DL models to perform significantly better, with an average of 8-22% increase in Matthews Correlation Coefficient (MCC). Conclusions: We distill the practices of using data-driven techniques for function-level SV assessment in C/C++, including the use of multi-task DL to balance efficiency and effectiveness. This can establish a strong foundation for future work in this area.

Figures (5)

• Figure 1: An example of a C++ vulnerable function in Git, extracted from the fixing commit 34fa79a of the SV [CVE-2016-2315]. Note: line 5 and 6 are vulnerable. Additions and removals are colored in green and red, respectively.
• Figure 2: Methodology used to answer the research questions.
• Figure 3: The CVSS metrics distributions in the dataset. Note: Access Complex., Auth., and Adj. Net. refer to Access Complexity, Authentication, and Adjacent Network, respectively.
• Figure 4: Average performance (MCC) of ML combinations (classifiers and features) for function-level SV assessment.
• Figure 5: Training time of CodeBERT, multi-task CodeBERT and LGBM + BoST (in seconds). Note: CodeBERT multi-task does not have training time for each task.