SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks

TL;DR

SEC-bench introduces an automated, multi-agent framework for benchmarking LLM agents on real-world software security tasks, specifically PoC generation and vulnerability patching. By leveraging a Preprocessor-Verifier-Evaluator pipeline and sanitizer-based ground truth, it constructs reproducible CVE-derived datasets with high-quality PoCs and patches. Across extensive experiments with multiple agent scaffolds and models, state-of-the-art code agents exhibit substantial gaps — PoC success up to 18% and patch success up to 34% — highlighting the need for advanced reasoning, debugging, and program-analysis capabilities in security-focused AI agents. The work demonstrates the practicality of automated, scalable, and objective security benchmarks and outlines clear directions for improving agent autonomy and reliability in real-world security engineering.

Abstract

Rigorous security-focused evaluation of large language model (LLM) agents is imperative for establishing trust in their safe deployment throughout the software development lifecycle. However, existing benchmarks largely rely on synthetic challenges or simplified vulnerability datasets that fail to capture the complexity and ambiguity encountered by security engineers in practice. We introduce SEC-bench, the first fully automated benchmarking framework for evaluating LLM agents on authentic security engineering tasks. SEC-bench employs a novel multi-agent scaffold that automatically constructs code repositories with harnesses, reproduces vulnerabilities in isolated environments, and generates gold patches for reliable evaluation. Our framework automatically creates high-quality software vulnerability datasets with reproducible artifacts at a cost of only $0.87 per instance. Using SEC-bench, we implement two critical software security tasks to rigorously evaluate LLM agents' capabilities: proof-of-concept (PoC) generation and vulnerability patching. A comprehensive evaluation of state-of-the-art LLM code agents reveals significant performance gaps, achieving at most 18.0% success in PoC generation and 34.0% in vulnerability patching on our complete dataset. These results highlight the crucial steps needed toward developing LLM agents that are more practical, intelligent, and autonomous for security engineering.

Figures (11)

• Figure 1: Overview of SEC-bench.
• Figure 2: Failure types in vulnerability patching. NP (No Patch): the agent fails to generate any patch; IF (Improper Format): the generated patch has an incorrect format; CE (Compilation Error): the patch causes the repository to fail compilation; SV (Still Vulnerable): the patch compiles but does not successfully remediate the security vulnerability when tested.
• Figure 3: Distribution of CVSS scores (upper figure) and CWE types (lower figure) for CVE instances in SEC-bench.
• Figure 4: A prompt for generating a Proof of Concept (PoC) that reproduces a specific sanitizer error. The task provides only the sanitizer error message in the original bug description in the [0.5]bug_description field. The goal is to craft a PoC that reliably triggers the identical sanitizer error.
• Figure 5: A prompt for generating a patch for each CVE instance. The task provides the original bug description in the [0.5]bug_description field. The goal is to craft a patch that fixes the vulnerability preventing the crash points specified in the [0.5]bug_description.