What Makes a Good LLM Agent for Real-world Penetration Testing?

TL;DR

This work addresses the inconsistent performance of LLM-based penetration testing agents by distinguishing two persistent failure classes: Type A (capability gaps) and Type B (complexity barriers). The authors diagnose a root cause for Type B failures as the lack of real-time task difficulty estimation and introduce PentestGPT v2 with a Tool and Skill Layer to fix Type A failures and a Task Difficulty Assessment (TDA) integrated into an Evidence-Guided Attack Tree Search (EGATS) to address Type B. The system augments planning with four difficulty signals, a memory subsystem for state persistence, and a structured search to prune unproductive branches, achieving significant gains across benchmarks (XBOW: 91% peak; GOAD: 4/5 hosts; PentestGPT Benchmark: 12/13 machines) and performing competitively in real-world competition settings. The results demonstrate that difficulty-aware planning provides end-to-end benefits that model scaling alone cannot deliver, and the work provides open-source artifacts and evaluation methodologies to advance reproducibility in this domain.

Abstract

LLM-based agents show promise for automating penetration testing, yet reported performance varies widely across systems and benchmarks. We analyze 28 LLM-based penetration testing systems and evaluate five representative implementations across three benchmarks of increasing complexity. Our analysis reveals two distinct failure modes: Type A failures stem from capability gaps (missing tools, inadequate prompts) that engineering readily addresses, while Type B failures persist regardless of tooling due to planning and state management limitations. We show that Type B failures share a root cause that is largely invariant to the underlying LLM: agents lack real-time task difficulty estimation. As a result, agents misallocate effort, over-commit to low-value branches, and exhaust context before completing attack chains. Based on this insight, we present Excalibur, a penetration testing agent that couples strong tooling with difficulty-aware planning. A Tool and Skill Layer eliminates Type A failures through typed interfaces and retrieval-augmented knowledge. A Task Difficulty Assessment (TDA) mechanism addresses Type B failures by estimating tractability through four measurable dimensions (horizon estimation, evidence confidence, context load, and historical success) and uses these estimates to guide exploration-exploitation decisions within an Evidence-Guided Attack Tree Search (EGATS) framework. Excalibur achieves up to 91% task completion on CTF benchmarks with frontier models (39 to 49% relative improvement over baselines) and compromises 4 of 5 hosts on the GOAD Active Directory environment versus 2 by prior systems. These results show that difficulty-aware planning yields consistent end-to-end gains across models and addresses a limitation that model scaling alone does not eliminate.

Figures (4)

• Figure 1: Failure type distribution by the task depth, measured as the number of distinct exploitation steps required for task completion.
• Figure 2: PentestGPT v2 architecture. The TDA-EGATS Planner addresses Type B failures through difficulty-aware tree search with Upper Confidence Bound (UCB) selection, TDI-guided mode switching, and evidence-based pruning. The Tool & Skill Layer addresses Type A failures through typed tool interfaces and RAG-augmented knowledge. The Memory Subsystem maintains structured state and enables selective context injection based on tree position.
• Figure 3: Ablation study across benchmarks (GPT-5.2 thinking). Performance is normalized to percentage scale.
• Figure 4: HTB Falafel exploration comparison. (a) PentestGPT commits to password brute-force after extracting hashes and stalls after 47 attempts. (b) PentestGPT v2's TDI-guided exploration discovers the type juggling bypass when hash cracking fails, then navigates the privilege escalation chain.