Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning

He Kong, Die Hu, Jingguo Ge, Liangxiong Li, Hui Li, Tong Li

TL;DR

This work tackles the challenge of automating penetration testing with large language models by introducing Pentest-R1, a two-stage reinforcement learning framework. It offline-learns from a large, real-world collection of expert walkthroughs to build foundational attack logic, then online-fine-tunes in an interactive CTF environment to develop robust, adaptive strategies using GRPO and LoRA. The approach achieves state-of-the-art performance among open-source models on Cybench and AutoPenBench benchmarks, closely approaching proprietary models, and ablations confirm that both training stages are necessary for peak performance. The results suggest that end-to-end online RL can bridge static knowledge and dynamic real-world tasks in cybersecurity, with practical implications for scalable, autonomous pentesting and future multimodal extensions.

Abstract

Automating penetration testing is crucial for enhancing cybersecurity, yet current Large Language Models (LLMs) face significant limitations in this domain, including poor error handling, inefficient reasoning, and an inability to perform complex end-to-end tasks autonomously. To address these challenges, we introduce Pentest-R1, a novel framework designed to optimize LLM reasoning capabilities for this task through a two-stage reinforcement learning pipeline. We first construct a dataset of over 500 real-world, multi-step walkthroughs, which Pentest-R1 leverages for offline reinforcement learning (RL) to instill foundational attack logic. Subsequently, the LLM is fine-tuned via online RL in an interactive Capture The Flag (CTF) environment, where it learns directly from environmental feedback to develop robust error self-correction and adaptive strategies. Our extensive experiments on the Cybench and AutoPenBench benchmarks demonstrate the framework's effectiveness. On AutoPenBench, Pentest-R1 achieves a 24.2\% success rate, surpassing most state-of-the-art models and ranking second only to Gemini 2.5 Flash. On Cybench, it attains a 15.0\% success rate in unguided tasks, establishing a new state-of-the-art for open-source LLMs and matching the performance of top proprietary models. Ablation studies confirm that the synergy of both training stages is critical to its success.

Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning

TL;DR

This work tackles the challenge of automating penetration testing with large language models by introducing Pentest-R1, a two-stage reinforcement learning framework. It offline-learns from a large, real-world collection of expert walkthroughs to build foundational attack logic, then online-fine-tunes in an interactive CTF environment to develop robust, adaptive strategies using GRPO and LoRA. The approach achieves state-of-the-art performance among open-source models on Cybench and AutoPenBench benchmarks, closely approaching proprietary models, and ablations confirm that both training stages are necessary for peak performance. The results suggest that end-to-end online RL can bridge static knowledge and dynamic real-world tasks in cybersecurity, with practical implications for scalable, autonomous pentesting and future multimodal extensions.

Abstract

Automating penetration testing is crucial for enhancing cybersecurity, yet current Large Language Models (LLMs) face significant limitations in this domain, including poor error handling, inefficient reasoning, and an inability to perform complex end-to-end tasks autonomously. To address these challenges, we introduce Pentest-R1, a novel framework designed to optimize LLM reasoning capabilities for this task through a two-stage reinforcement learning pipeline. We first construct a dataset of over 500 real-world, multi-step walkthroughs, which Pentest-R1 leverages for offline reinforcement learning (RL) to instill foundational attack logic. Subsequently, the LLM is fine-tuned via online RL in an interactive Capture The Flag (CTF) environment, where it learns directly from environmental feedback to develop robust error self-correction and adaptive strategies. Our extensive experiments on the Cybench and AutoPenBench benchmarks demonstrate the framework's effectiveness. On AutoPenBench, Pentest-R1 achieves a 24.2\% success rate, surpassing most state-of-the-art models and ranking second only to Gemini 2.5 Flash. On Cybench, it attains a 15.0\% success rate in unguided tasks, establishing a new state-of-the-art for open-source LLMs and matching the performance of top proprietary models. Ablation studies confirm that the synergy of both training stages is critical to its success.

Paper Structure

This paper contains 20 sections, 4 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Penetration Testing
  4. Large Language Models
  5. Motivation
  6. Methodology
  7. Overview
  8. Stage 1: Offline Reinforcement Learning from Walkthroughs
  9. Reward Function.
  10. Stage 2: Online Reinforcement Learning in Interactive Environments
  11. Experiments
  12. Experimental Setup
  13. Datasets.
  14. Baselines.
  15. Evaluation Metrics.
  16. ...and 5 more sections

Figures (4)

  • Figure 1: An illustration of the core challenges in LLM-driven penetration testing. Current LLMs often exhibit ineffective reasoning and poor error recovery, leading to failed attack chains. This is exacerbated by a lack of realistic, multi-step training data and RL paradigms ill-suited for long-horizon, interactive tasks.
  • Figure 2: The framework architecture of Pentest-R1.
  • Figure 3: Token consumption analysis. "Thinking Token" refers to tokens for Chain-of-Thought reasoning.
  • Figure 4: Total token consumption on Cybench.