CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

TL;DR

CVE-Factory addresses the critical challenge of generating executable, expert-quality vulnerability tasks from sparse CVE metadata to enable scalable evaluation and training of code-security agents. It introduces a six-stage, decoupled yet tightly coordinated multi-agent pipeline controlled by a central Orchestrator, achieving expert-level reproduction with high solution correctness ($\$95\%\$) and environment fidelity ($\$96\%\$) and demonstrating robust applicability on real-world CVEs ($66.2\%$ verified success). The framework yields LiveCVEBench—a continuously updated, multi-language benchmark—and a large-scale training corpus that enables significant improvements in medical-scale agentic vulnerability repair (e.g., $6.8\times$ on LiveCVEBench for Qwen3-32B). The approach advances practical security testing for AI/LLM-driven code agents by delivering high-fidelity tasks, scalable reproducibility, and transferable training data, with open-source resources enabling broad adoption. Overall, CVE-Factory establishes a scalable foundation for evaluating and training agentic code-security workflows in real-world, multi-language contexts.

Abstract

Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95\% solution correctness and 96\% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2\% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3\% to 35.8\% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5\% to 31.3\%). We open-source CVE-Factory, LiveCVEBench, Abacus-cve (fine-tuned model), training dataset, and leaderboard. All resources are available at https://github.com/livecvebench/CVE-Factory .

Figures (6)

• Figure 1: Comparison between raw CVE metadata and a comprehensive agentic task. (Top) Sparse CVE metadata consisting of vulnerability descriptions, classifications, and reference URLs. (Bottom) An agentic task including natural-language instructions, an interactive environment, and verification tests.
• Figure 2: Overview of CVE-Factory. CVE Metadata are processed through six stages: three decoupling stages (Stages 1–3) generate task components independently, and three coupling stages (Stages 4–6) progressively verify and align them. A central Orchestrator manages the workflow, activating specialized agents and executing verification scripts per stage. Agents communicate with Orchestrator via continue, error, or pause signals, where pause triggers the feedback mechanism to route revisions to original file creators.
• Figure 3: Distribution of CVE reproduction outcomes by programming language (top) and CWE type (bottom). Colors indicate verified success (green), false positives failing manual verification (orange), and CVE-Factory reported failures (gray).
• Figure 4: Model performance on CVEs before versus after each model's release date.
• Figure 5: Execution time distribution per agent across 2,000 CVE reproductions.