PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing

TL;DR

PenForge tackles the challenge of scalable, adaptive penetration testing in zero-day web applications by constructing expert agents on the fly guided by automated reconnaissance. It introduces a Meta-Planner that orchestrates a two-phase workflow to gather target context and iteratively instantiate attack-type specialists via an AutoGPT-based executor. The approach achieves a 30.0% exploit success rate on CVE-Bench in the zero-day setting, tripling prior state-of-the-art performance. The results highlight the value of contextualized agent construction for improving exploitation effectiveness and suggest directions for richer knowledge retrieval and more comprehensive benchmarks.

Abstract

Penetration testing is essential for identifying vulnerabilities in web applications before real adversaries can exploit them. Recent work has explored automating this process with Large Language Model (LLM)-powered agents, but existing approaches either rely on a single generic agent that struggles in complex scenarios or narrowly specialized agents that cannot adapt to diverse vulnerability types. We therefore introduce PenForge, a framework that dynamically constructs expert agents during testing rather than relying on those prepared beforehand. By integrating automated reconnaissance of potential attack surfaces with agents instantiated on the fly for context-aware exploitation, PenForge achieves a 30.0% exploit success rate (12/40) on CVE-Bench in the particularly challenging zero-day setting, which is a 3 times improvement over the state-of-the-art. Our analysis also identifies three opportunities for future work: (1) supplying richer tool-usage knowledge to improve exploitation effectiveness; (2) extending benchmarks to include more vulnerabilities and attack types; and (3) fostering developer trust by incorporating explainable mechanisms and human review. As an emerging result with substantial potential impact, PenForge embodies the early-stage yet paradigm-shifting idea of on-the-fly agent construction, marking its promise as a step toward scalable and effective LLM-driven penetration testing.

Figures (2)

• Figure 1: Overview of PenForge's two-phase workflow: (1) Target Reconnaissance to gather context and rank attack types; (2) Sequential Attack Attempts where expert agents are constructed and executed guided by Phase 1, repeating until success.
• Figure 2: Zero-day exploitation success of PenForge: (a) success rate; (b) distribution of successful exploit types.