Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-based Vulnerability Detection at Project Scale: An Empirical Study

Fengjie Li, Jiajun Jiang, Dongchi Chen, Yingfei Xiong

TL;DR

The paper tackles the problem of scaling vulnerability detection to large project contexts by empirically comparing five LLM-based detectors with two traditional static analyzers across 222 known vulnerabilities and 24 open-source projects. It reveals that LLM-based approaches exhibit low recall on curated benchmarks yet uncover more unique vulnerabilities than traditional tools, while real-world deployment suffers from massive false positives and substantial token/time costs. The authors provide a taxonomy of false-positive root causes, quantify overheads, and demonstrate that current LLM-based detectors are not yet robust or scalable enough for routine project-level use. The study offers concrete implications for designing more practical, hybrid vulnerability detection workflows and highlights where future research should focus to close gaps in dataflow reasoning, source–sink inference, and efficiency. All experimental artifacts are released to support reproducibility and further research.

Abstract

In this paper, we present the first comprehensive empirical study of specialized LLM-based detectors and compare them with traditional static analyzers at the project scale. Specifically, our study evaluates five latest and representative LLM-based methods and two traditional tools using: 1) an in-house benchmark of 222 known real-world vulnerabilities (C/C++ and Java) to assess detection capability, and 2) 24 active open-source projects, where we manually inspected 385 warnings to assess their practical usability and underlying root causes of failures. Our evaluation yields three key findings: First, while LLM-based detectors exhibit low recall on the in-house benchmark, they still uncover more unique vulnerabilities than traditional tools. Second, in open-source projects, both LLM-based and traditional tools generate substantial warnings but suffer from very high false discovery rates, hindering practical use. Our manual analysis further reveals shallow interprocedural reasoning and misidentified source/sink pairs as primary failure causes, with LLM-based tools exhibiting additional unique failures. Finally, LLM-based methods incurs substantial computational costs-hundreds of thousands to hundreds of millions of tokens and multi-hour to multi-day runtimes. Overall, our findings underscore critical limitations in the robustness, reliability, and scalability of current LLM-based detectors. We ultimately summarize a set of implications for future research toward more effective and practical project-scale vulnerability detection.

LLM-based Vulnerability Detection at Project Scale: An Empirical Study

TL;DR

The paper tackles the problem of scaling vulnerability detection to large project contexts by empirically comparing five LLM-based detectors with two traditional static analyzers across 222 known vulnerabilities and 24 open-source projects. It reveals that LLM-based approaches exhibit low recall on curated benchmarks yet uncover more unique vulnerabilities than traditional tools, while real-world deployment suffers from massive false positives and substantial token/time costs. The authors provide a taxonomy of false-positive root causes, quantify overheads, and demonstrate that current LLM-based detectors are not yet robust or scalable enough for routine project-level use. The study offers concrete implications for designing more practical, hybrid vulnerability detection workflows and highlights where future research should focus to close gaps in dataflow reasoning, source–sink inference, and efficiency. All experimental artifacts are released to support reproducibility and further research.

Abstract

In this paper, we present the first comprehensive empirical study of specialized LLM-based detectors and compare them with traditional static analyzers at the project scale. Specifically, our study evaluates five latest and representative LLM-based methods and two traditional tools using: 1) an in-house benchmark of 222 known real-world vulnerabilities (C/C++ and Java) to assess detection capability, and 2) 24 active open-source projects, where we manually inspected 385 warnings to assess their practical usability and underlying root causes of failures. Our evaluation yields three key findings: First, while LLM-based detectors exhibit low recall on the in-house benchmark, they still uncover more unique vulnerabilities than traditional tools. Second, in open-source projects, both LLM-based and traditional tools generate substantial warnings but suffer from very high false discovery rates, hindering practical use. Our manual analysis further reveals shallow interprocedural reasoning and misidentified source/sink pairs as primary failure causes, with LLM-based tools exhibiting additional unique failures. Finally, LLM-based methods incurs substantial computational costs-hundreds of thousands to hundreds of millions of tokens and multi-hour to multi-day runtimes. Overall, our findings underscore critical limitations in the robustness, reliability, and scalability of current LLM-based detectors. We ultimately summarize a set of implications for future research toward more effective and practical project-scale vulnerability detection.
Paper Structure (31 sections, 2 equations, 13 figures, 11 tables)

This paper contains 31 sections, 2 equations, 13 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Large Language Models
  4. Static Vulnerability Detection
  5. Existing Studies on LLM-based Vulnerability Detection
  6. Study Design
  7. Research Questions
  8. Evaluated Methods
  9. LLM-based Methods
  10. Traditional Static Methods
  11. Datasets
  12. Evaluation Metrics
  13. Experimental Setup
  14. Result Analysis
  15. RQ1: Effectiveness in Detecting In-house Vulnerabilities
  16. ...and 16 more sections

Figures (13)

  • Figure 1: Reasons for undetected C/C++ vulnerabilities.
  • Figure 2: Illustrating example of why these tools fail to detect the vulnerability.
  • Figure 3: Unique vulnerabilities detected by each tool
  • Figure 4: Taxonomy of FP reasons that are related to the cases detected by the selected methods.
  • Figure 5: Distribution of false positive reasons introduced by evaluated methods.
  • ...and 8 more figures