Learning-based Models for Vulnerability Detection: An Extensive Study

TL;DR

This paper conducts a comprehensive, multi-dimensional evaluation of learning-based vulnerability detection, contrasting sequence-based and graph-based approaches on MegaVul and including ChatGPT. It finds sequence-based models generally outperform graph-based ones, while ChatGPT remains underpowered for vulnerability detection tasks. Interpretability evidence shows models concentrate on Function Calls and Field Expressions, with instability observed across semantically equivalent inputs. The work also clarifies ease of use and economic considerations, offering practical guidelines for deploying VD models safely and cost-effectively in real-world settings.

Abstract

Though many deep learning-based models have made great progress in vulnerability detection, we have no good understanding of these models, which limits the further advancement of model capability, understanding of the mechanism of model detection, and efficiency and safety of practical application of models. In this paper, we extensively and comprehensively investigate two types of state-of-the-art learning-based approaches (sequence-based and graph-based) by conducting experiments on a recently built large-scale dataset. We investigate seven research questions from five dimensions, namely model capabilities, model interpretation, model stability, ease of use of model, and model economy. We experimentally demonstrate the priority of sequence-based models and the limited abilities of both LLM (ChatGPT) and graph-based models. We explore the types of vulnerability that learning-based models skilled in and reveal the instability of the models though the input is subtlely semantical-equivalently changed. We empirically explain what the models have learned. We summarize the pre-processing as well as requirements for easily using the models. Finally, we initially induce the vital information for economically and safely practical usage of these models.

Figures (5)

• Figure 1: Performance on Vulnerability Type
• Figure 2: Vulnerability detection prompt templates used in our study
• Figure 3: Number of occurrences for each statement type in the Top 10 most probable vulnerability lines. (diagonal shadow indicates sequence-base models)
• Figure 4: LineVul interpretation result(CVE-2022-47519CVE-2022-47519)
• Figure 5: Visualization of the separation between vulnerable (denoted by +) and non-vulnerable (denoted by $\bigcirc$).