SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection

TL;DR

SecVulEval presents a statement-level, context-rich benchmark for C/C++ vulnerability detection, addressing gaps in prior datasets by providing granular labels, CVE/CWE metadata, and interprocedural context. Leveraging a four-stage data-collection pipeline and a GPT-4.1-derived context annotation workflow, the authors assemble 25,440 function samples from 5,867 CVEs (1999–2024) across 707 projects and 145 CWE types, with a multi-agent LLM evaluation spanning open-source and proprietary models. Across two experiments—vulnerability detection and context identification—the results reveal that current LLMs struggle to accurately localize vulnerable statements and to extract essential context, with Claude-3.7-Sonnet achieving the best statement-level F1 of 23.83% and relatively better context extraction. The work highlights the need for richer, context-aware benchmarks to push toward reliable real-world vulnerability localization and provides a valuable resource and methodology for future research and tool development in secure software engineering.

Abstract

Large Language Models (LLMs) have shown promise in software engineering tasks, but evaluating their effectiveness in vulnerability detection is challenging due to the lack of high-quality datasets. Most existing datasets are limited to function-level labels, ignoring finer-grained vulnerability patterns and crucial contextual information. Also, poor data quality such as mislabeling, inconsistent annotations, and duplicates can lead to inflated performance and weak generalization. Moreover, by including only the functions, these datasets miss broader program context, like data/control dependencies and interprocedural interactions, that are essential for accurately understanding real-world security flaws. Without this context, detection models are evaluated under unrealistic assumptions. To address these limitations, this paper introduces SecVulEval, a benchmark designed to support fine-grained evaluation of LLMs and other detection methods with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model's ability to localize vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for vulnerability detection benchmarks in realistic scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated the SOTA LLMs with a multi-agent-based approach. The evaluation on our dataset shows that the models are still far from accurately predicting vulnerable statements in a given function. The best-performing Claude-3.7-Sonnet model achieves 23.83% F1-score for detecting vulnerable statements with correct reasoning. Finally, we analyze the LLM outputs and provide insights into their behavior in vulnerability detection for C/C++.

Figures (6)

• Figure 1: Data collection pipeline of SecVulEval
• Figure 2: Top 10 projects in SecVulEval.
• Figure 3: Number of vulnerable functions in Top-20 CWE types in SecVulEval. The x-axis represents the list of CWE IDs, while the y-axis indicates the number of corresponding samples.
• Figure 4: The multi-agent vulnerability detection pipeline. Each agent, except the first one, is driven by an LLM.
• Figure 4: Accuracy of essential context identification. No models identified any Environment-level context.