VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation

TL;DR

VADER introduces a human-evaluated benchmark to assess LLMs on end-to-end vulnerability handling across four dimensions: assessment, detection, explanation, and remediation. It pairs 174 real-world vulnerability cases with expert ground truth, a four-stage evaluation protocol, and broad multi-language coverage to test deep reasoning, patch generation, and test-plan validation. Across six state-of-the-art models, results show moderate success (top ~54.7%), with remediation quality highly correlated to correct classification and test plans, underscoring the need for more capable vulnerability-aware reasoning. The dataset, evaluation tools, and results are publicly released to propel reproducible progress in secure coding with LLMs, spanning multi-file and cross-language contexts that resemble real-world software development environments.

Abstract

Ensuring that large language models (LLMs) can effectively assess, detect, explain, and remediate software vulnerabilities is critical for building robust and secure software systems. We introduce VADER, a human-evaluated benchmark designed explicitly to assess LLM performance across four key vulnerability-handling dimensions: assessment, detection, explanation, and remediation. VADER comprises 174 real-world software vulnerabilities, each carefully curated from GitHub repositories and annotated by security experts. For each vulnerability case, models are tasked with identifying the flaw, classifying it using Common Weakness Enumeration (CWE), explaining its underlying cause, proposing a patch, and formulating a test plan. Using a one-shot prompting strategy, we benchmark six state-of-the-art LLMs (Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, GPT-4.5, Grok 3 Beta, and o3) on VADER, and human security experts evaluated each response according to a rigorous scoring rubric emphasizing remediation (quality of the code fix, 50%), explanation (20%), and classification and test plan (30%) according to a standardized rubric. Our results show that current state-of-the-art LLMs achieve only moderate success on VADER - OpenAI's o3 attained 54.7% accuracy overall, with others in the 49-54% range, indicating ample room for improvement. Notably, remediation quality is strongly correlated (Pearson r > 0.97) with accurate classification and test plans, suggesting that models that effectively categorize vulnerabilities also tend to fix them well. VADER's comprehensive dataset, detailed evaluation rubrics, scoring tools, and visualized results with confidence intervals are publicly released, providing the community with an interpretable, reproducible benchmark to advance vulnerability-aware LLMs. All code and data are available at: https://github.com/AfterQuery/vader

Figures (11)

• Figure 2: Distribution of severity levels across VADER cases.
• Figure 3: Distribution of Cases by Number of Files per Case
• Figure 4: Curation and evaluation pipeline for VADER.
• Figure 5: Distribution of programming languages across VADER cases.
• Figure 6: Language co-occurrence matrix showing the frequency of language pairs across cases.