VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework

TL;DR

VulnBot addresses the inefficiencies of manual and prior LLM-assisted penetration testing by introducing a tri-phase, autonomous, multi-agent framework guided by a Penetration Task Graph. It decomposes tasks into reconnaissance, scanning, and exploitation, with role specialization, inter-agent summarization, and a memory-driven planning loop to maintain context and adapt plans. Experimental results on AUTOPENBENCH and real-world AI-Pentest-Benchmark targets show VulnBot outperforming baselines and achieving end-to-end autonomy, especially when augmented with Retrieval Augmented Generation. The work highlights the importance of structured task planning, modular cooperation, and memory-based reasoning for scalable, autonomous cybersecurity testing, while also identifying limitations in processing non-textual data and real-world deployment challenges.

Abstract

Penetration testing is a vital practice for identifying and mitigating vulnerabilities in cybersecurity systems, but its manual execution is labor-intensive and time-consuming. Existing large language model (LLM)-assisted or automated penetration testing approaches often suffer from inefficiencies, such as a lack of contextual understanding and excessive, unstructured data generation. This paper presents VulnBot, an automated penetration testing framework that leverages LLMs to simulate the collaborative workflow of human penetration testing teams through a multi-agent system. To address the inefficiencies and reliance on manual intervention in traditional penetration testing methods, VulnBot decomposes complex tasks into three specialized phases: reconnaissance, scanning, and exploitation. These phases are guided by a penetration task graph (PTG) to ensure logical task execution. Key design features include role specialization, penetration path planning, inter-agent communication, and generative penetration behavior. Experimental results demonstrate that VulnBot outperforms baseline models such as GPT-4 and Llama3 in automated penetration testing tasks, particularly showcasing its potential in fully autonomous testing on real-world machines.

Figures (7)

• Figure 1: The workflow comparison of three approaches to automated penetration testing: (a) LLM Assistant-Guided Pentest Agent, which requires assistance due to inefficiency; (b) Conventional Automated Pentest Agent, which struggles with information overload and context loss; and (c) Collaborative Multi-Agent system, which employs a phased and modular approach, enhancing the overall efficiency and autonomy of the penetration testing process through multi-agent coordination.
• Figure 2: Overview of VulnBot.
• Figure 3: The process of generating Penetration Task Graph (PTG). The green circle represents the current task being executed, while the dark circle indicates that the task has been successfully completed.
• Figure 4: The failure counts of VulnBot and baseline models across the Reconnaissance, Scanning, and Exploitation phases.
• Figure 5: Ablation study of VulnBot on AUTOPENBENCH. This figure demonstrates the impact of removing key components—role specialization, the PTG, and the Summarizer—on model performance.