PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities

TL;DR

PatchEval offers a large-scale, multilingual benchmark for evaluating LLM-driven vulnerability repair across Python, Go, and JavaScript, built from 1,000 CVEs and 65 CWEs with 230 PoCs in docker sandboxes. By measuring patch generation with known locations and end-to-end localization, the study reveals that even the strongest models achieve only about 23% success on real-world vulnerabilities, highlighting the critical roles of accurate localization and dynamic patch validation. The framework provides dual validation (static similarity and dynamic PoCs/unit tests) and open artifacts to enable reproducible, community-driven progress in AVR. The findings demonstrate strong complementarity among models and agents, suggesting that integrated, domain-specific tooling and collaboration will be essential to advance practical vulnerability repair.

Abstract

Software vulnerabilities are increasing at an alarming rate. However, manual patching is both time-consuming and resource-intensive, while existing automated vulnerability repair (AVR) techniques remain limited in effectiveness. Recent advances in large language models (LLMs) have opened a new paradigm for AVR, demonstrating remarkable progress. To examine the capability of LLMs in AVR, several vulnerability benchmarks have been proposed recently. However, they still suffer from key limitations of outdated vulnerabilities, limited language coverage, unreliable patch validation, and insufficient reproducibility. To overcome these challenges, we introduce PATCHEVAL, a multilingual benchmark for Go, JavaScript, and Python, languages for which existing benchmarks remain unexplored. PATCHEVAL curates a dataset of 1,000 vulnerabilities drawn from CVEs reported between 2015 and 2025, covering 65 distinct CWEs. A subset of 230 CVEs is further equipped with runtime sandbox environments, enabling patch verification through both security tests and functionality tests. To provide a systematic comparison of LLM-based vulnerability repair, we evaluate a series of state-of-the-art LLMs and agents, presenting an in-depth analysis that empirically yields key insights to guide future research in AVR.

Figures (11)

• Figure 1: Overview of PatchEval
• Figure 2: Programming Language Selection Criterial
• Figure 3:
• Figure 4: LLM-generated Patches with Compilation Errors
• Figure 5: Cumulative Number of Vulnerabilities Solved by LLMs Across Different Rounds of Feedback