Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection

TL;DR

This work introduces a long-context, file-level benchmark for multi-vulnerability detection across C, C++, Python, and JavaScript by synthetically injecting controlled CWE densities (1, 3, 5, 9) into 40,000 code files sourced from CodeParrot. An Oracle LLM maps feasible injections, enabling a principled evaluation of 5 state-of-the-art models under zero-shot prompting with metrics including Precision, Recall, F1, MAE, and a strict ExactFile score. Results show substantial performance erosion as vulnerability density increases, with notable under-counting and higher MAE, even for strong models like Llama-3.3-70B, and distinct failure modes across compiled versus interpreted languages. The study highlights fundamental limits of current LLMs for exhaustive security audits and outlines future directions, such as hybrid static analysis, retrieval-augmented long-context reasoning, and broader, more realistic vulnerability coverage to drive progress in trustworthy automated vulnerability detection.

Abstract

Large Language Models (LLMs) have demonstrated significant potential in automated software security, particularly in vulnerability detection. However, existing benchmarks primarily focus on isolated, single-vulnerability samples or function-level classification, failing to reflect the complexity of real-world software where multiple interacting vulnerabilities often coexist within large files. Recent studies indicate that LLMs suffer from "count bias" and "selection bias" in multi-label tasks, yet this has not been rigorously quantified in the domain of code security. In this work, we introduce a comprehensive benchmark for Multi-Vulnerability Detection across four major languages: C, C++, Python, and JavaScript. We construct a dataset of 40,000 files by systematically injecting controlled counts of vulnerabilities (1, 3, 5, and 9) into long-context code samples (7.5k-10k tokens) sourced from CodeParrot. We evaluate five state-of-the-art LLMs, including GPT-4o-mini, Llama-3.3-70B, and the Qwen-2.5 series. Our results reveal a sharp degradation in performance as vulnerability density increases. While Llama-3.3-70B achieves near-perfect F1 scores (approximately 0.97) on single-vulnerability C tasks, performance drops by up to 40% in high-density settings. Notably, Python and JavaScript show distinct failure modes compared to C/C++, with models exhibiting severe "under-counting" (Recall dropping to less than 0.30) in complex Python files.

Figures (1)

• Figure 1: Evaluation of LLMs on Code Vulnerability Detection Across Varying Density and Language Types.