Mind the Gap: Towards Generalizable Autonomous Penetration Testing via Domain Randomization and Meta-Reinforcement Learning

TL;DR

This work tackles the generalization and realism gaps in autonomous pentesting by introducing GAP, a Real-to-Sim-to-Real framework that combines end to end policy learning in unknown real like environments with synthetic simulations generated via LLM powered domain randomization and meta reinforcement learning. By training policies in diverse synthetic environments and employing MAML style meta learning, GAP enables zero shot policy transfer in similar environments and rapid few shot adaptation in dissimilar ones, outperforming baselines on Vulhub based vulnerability scenarios. The experimental results demonstrate that GAP supports end to end learning in realistic settings, bridges the generalization gap for unseen but similar targets, and accelerates adaptation when faced with novel vulnerability configurations, thereby significantly enhancing practical applicability of autonomous pentesting agents. Overall GAP presents a feasible pathway to robust, generalizable autonomous pentesting and offers a framework that can be extended to other safety critical cyber testing tasks.

Abstract

With increasing numbers of vulnerabilities exposed on the internet, autonomous penetration testing (pentesting) has emerged as a promising research area. Reinforcement learning (RL) is a natural fit for studying this topic. However, two key challenges limit the applicability of RL-based autonomous pentesting in real-world scenarios: (a) training environment dilemma -- training agents in simulated environments is sample-efficient while ensuring their realism remains challenging; (b) poor generalization ability -- agents' policies often perform poorly when transferred to unseen scenarios, with even slight changes potentially causing significant generalization gap. To this end, we propose GAP, a generalizable autonomous pentesting framework that aims to realizes efficient policy training in realistic environments and train generalizable agents capable of drawing inferences about other cases from one instance. GAP introduces a Real-to-Sim-to-Real pipeline that (a) enables end-to-end policy learning in unknown real environments while constructing realistic simulations; (b) improves agents' generalization ability by leveraging domain randomization and meta-RL learning.Specially, we are among the first to apply domain randomization in autonomous pentesting and propose a large language model-powered domain randomization method for synthetic environment generation. We further apply meta-RL to improve agents' generalization ability in unseen environments by leveraging synthetic environments. The combination of two methods effectively bridges the generalization gap and improves agents' policy adaptation performance.Experiments are conducted on various vulnerable virtual machines, with results showing that GAP can enable policy learning in various realistic environments, achieve zero-shot policy transfer in similar environments, and realize rapid policy adaptation in dissimilar environments.

Figures (11)

• Figure 1: Gap between the training and testing environments. (a) Shows the reality gap between training and testing environments. The training environment is a virtual machine with the CVE-2018-7600 vulnerability, set up using Vulhub. We utilized the Shodan engine to search real-world hosts that may possess the same vulnerability, which is assumed to be a testing environment. Even though both training and testing environments have the same vulnerability, differences in host configurations create a gap, thereby affecting the agent's transfer learning performance. (b) Shows the learning curve of April in the training environment and its zero-shot performance in the testing environment. The averages and 95% confidence intervals (shaded areas) across 3 random seeds are shown.
• Figure 2: Overview of GAP.
• Figure 3: The process of policy learning and simulated environment construction.
• Figure 4: Prompt pattern for synthetic environment generation.
• Figure 5: The workflow and example of synthetic environment generation using LLM. In this example, we construct the simulation by using the CVE-2018-7600 vulnerable host from Vulhub as the original training environment. Then, we employ GLM-4 glm2024chatglm for domain randomization.