RedTeamLLM: an Agentic AI framework for offensive security
Brian Challita, Pierre Parrend
TL;DR
The paper addresses the dual-use risk and potential of agentic AI in offensive cybersecurity. It proposes RedTeamLLM, a seven-component architecture combining ADaPT Enhanced, Plan Corrector, Memory Management, and a ReAct-based three-step pipeline to automate pentest tasks under a strict security model. Evaluation on five VULNHUB-derived use cases shows that incorporating reasoning significantly reduces required tool calls and enhances task completion compared with competitive baselines like PentestGPT, demonstrating strong automation and generality. The work highlights reasoning as a central lever in agentic AI performance and outlines future directions in recursive planning, memory architectures, and safety controls to guide responsible deployment.
Abstract
From automated intrusion testing to discovery of zero-day attacks before software launch, agentic AI calls for great promises in security engineering. This strong capability is bound with a similar threat: the security and research community must build up its models before the approach is leveraged by malicious actors for cybercrime. We therefore propose and evaluate RedTeamLLM, an integrated architecture with a comprehensive security model for automatization of pentest tasks. RedTeamLLM follows three key steps: summarizing, reasoning and act, which embed its operational capacity. This novel framework addresses four open challenges: plan correction, memory management, context window constraint, and generality vs. specialization. Evaluation is performed through the automated resolution of a range of entry-level, but not trivial, CTF challenges. The contribution of the reasoning capability of our agentic AI framework is specifically evaluated.