Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?

Bin Liu, Yanjie Zhao, Guoai Xu, Haoyu Wang

TL;DR

The paper systematically evaluates 20 state-of-the-art LLM agents on automated web vulnerability reproduction, constructing a benchmark of 80 real-world CVEs across 7 vulnerability types and 6 web technologies. The study identifies three top-performing agents (OpenHands, SWE-agent, CAI) and conducts an in-depth analysis of their end-to-end reproduction capabilities, revealing that agents can handle simple library-based flaws but struggle with complex service-based scenarios requiring multi-component environments and authentication. A key finding is the large gap between PoC execution and actual vulnerability triggering, with robust performance heavily dependent on workflow orchestration and tool usage rather than mere capability breadth. The authors advocate for advances in environmental adaptation and autonomous problem-solving, and propose that browser-based, MCP-aligned approaches with browser automation could bridge the gap, highlighting a path toward practical, autonomous vulnerability reproduction in real-world settings.

Abstract

Large language model (LLM) agents have demonstrated remarkable capabilities in software engineering and cybersecurity tasks, including code generation, vulnerability discovery, and automated testing. One critical but underexplored application is automated web vulnerability reproduction, which transforms vulnerability reports into working exploits. Although recent advances suggest promising potential, challenges remain in applying LLM agents to real-world web vulnerability reproduction scenarios. In this paper, we present the first comprehensive evaluation of state-of-the-art LLM agents for automated web vulnerability reproduction. We systematically assess 20 agents from software engineering, cybersecurity, and general domains across 16 dimensions, including technical capabilities, environment adaptability, and user experience factors, on 3 representative web vulnerabilities. Based on the results, we select three top-performing agents (OpenHands, SWE-agent, and CAI) for in-depth evaluation on our benchmark dataset of 80 real-world CVEs spanning 7 vulnerability types and 6 web technologies. Our results reveal that while LLM agents achieve reasonable success on simple library-based vulnerabilities, they consistently fail on complex service-based vulnerabilities requiring multi-component environments. Complex environment configurations and authentication barriers create a gap where agents can execute exploit code but fail to trigger actual vulnerabilities. We observe high sensitivity to input guidance, with performance degrading by over 33% under incomplete authentication information. Our findings highlight the significant gap between current LLM agent capabilities and the demands of reliable automated vulnerability reproduction, emphasizing the need for advances in environmental adaptation and autonomous problem-solving capabilities.

LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?

TL;DR

The paper systematically evaluates 20 state-of-the-art LLM agents on automated web vulnerability reproduction, constructing a benchmark of 80 real-world CVEs across 7 vulnerability types and 6 web technologies. The study identifies three top-performing agents (OpenHands, SWE-agent, CAI) and conducts an in-depth analysis of their end-to-end reproduction capabilities, revealing that agents can handle simple library-based flaws but struggle with complex service-based scenarios requiring multi-component environments and authentication. A key finding is the large gap between PoC execution and actual vulnerability triggering, with robust performance heavily dependent on workflow orchestration and tool usage rather than mere capability breadth. The authors advocate for advances in environmental adaptation and autonomous problem-solving, and propose that browser-based, MCP-aligned approaches with browser automation could bridge the gap, highlighting a path toward practical, autonomous vulnerability reproduction in real-world settings.

Abstract

Large language model (LLM) agents have demonstrated remarkable capabilities in software engineering and cybersecurity tasks, including code generation, vulnerability discovery, and automated testing. One critical but underexplored application is automated web vulnerability reproduction, which transforms vulnerability reports into working exploits. Although recent advances suggest promising potential, challenges remain in applying LLM agents to real-world web vulnerability reproduction scenarios. In this paper, we present the first comprehensive evaluation of state-of-the-art LLM agents for automated web vulnerability reproduction. We systematically assess 20 agents from software engineering, cybersecurity, and general domains across 16 dimensions, including technical capabilities, environment adaptability, and user experience factors, on 3 representative web vulnerabilities. Based on the results, we select three top-performing agents (OpenHands, SWE-agent, and CAI) for in-depth evaluation on our benchmark dataset of 80 real-world CVEs spanning 7 vulnerability types and 6 web technologies. Our results reveal that while LLM agents achieve reasonable success on simple library-based vulnerabilities, they consistently fail on complex service-based vulnerabilities requiring multi-component environments. Complex environment configurations and authentication barriers create a gap where agents can execute exploit code but fail to trigger actual vulnerabilities. We observe high sensitivity to input guidance, with performance degrading by over 33% under incomplete authentication information. Our findings highlight the significant gap between current LLM agent capabilities and the demands of reliable automated vulnerability reproduction, emphasizing the need for advances in environmental adaptation and autonomous problem-solving capabilities.
Paper Structure (40 sections, 4 figures, 8 tables)

This paper contains 40 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Agents
  4. Core Mechanisms.
  5. Representative Agents and Frameworks
  6. Web Vulnerability Reproduction
  7. Motivation
  8. Empirical Study Design
  9. Data Preparation
  10. Agent Selection
  11. Task Specification
  12. Experimental Setup
  13. Studied Dataset and Agents
  14. Metrics
  15. Success Rate Metrics.
  16. ...and 25 more sections

Figures (4)

  • Figure 1: Systematic workflow for evaluating LLM agents in web vulnerability reproduction tasks.
  • Figure 2: Prompt template and example agent response.
  • Figure 3: Comparison of vulnerability reproduction success rates between different web technologies.
  • Figure 4: Performance comparison between agents using Claude-Sonnet-4 with and without reasoning across different vulnerability reproduction stages.