Adversarial Reinforcement Learning for Offensive and Defensive Agents in a Simulated Zero-Sum Network Environment

TL;DR

The paper addresses autonomous attacker-defender dynamics in network security using adversarial reinforcement learning. It introduces a compact OpenAI Gym–like multi-agent zero-sum environment that models brute-force exploitation and reactive defenses, including background traffic, IP rotation, honeypots, and rate-limiting. The contributions include detailed environment design, systematic ablations across defense strategies and learning dynamics, and practical guidance for training stability in adversarial settings. Findings show that defender observability and adaptive defense strategies substantially raise the barrier to attack, while reward shaping and training schedules are critical for stable convergence; the work provides a foundation for studying autonomous cyber defense, attacker-defender co-evolution, and transfer to real-world scenarios.

Abstract

This paper presents a controlled study of adversarial reinforcement learning in network security through a custom OpenAI Gym environment that models brute-force attacks and reactive defenses on multi-port services. The environment captures realistic security trade-offs including background traffic noise, progressive exploitation mechanics, IP-based evasion tactics, honeypot traps, and multi-level rate-limiting defenses. Competing attacker and defender agents are trained using Deep Q-Networks (DQN) within a zero-sum reward framework, where successful exploits yield large terminal rewards while incremental actions incur small costs. Through systematic evaluation across multiple configurations (varying trap detection probabilities, exploitation difficulty thresholds, and training regimens), the results demonstrate that defender observability and trap effectiveness create substantial barriers to successful attacks. The experiments reveal that reward shaping and careful training scheduling are critical for learning stability in this adversarial setting. The defender consistently maintains strategic advantage across 50,000+ training episodes, with performance gains amplifying when exposed to complex defensive strategies including adaptive IP blocking and port-specific controls. Complete implementation details, reproducible hyperparameter configurations, and architectural guidelines are provided to support future research in adversarial RL for cybersecurity. The zero-sum formulation and realistic operational constraints make this environment suitable for studying autonomous defense systems, attacker-defender co-evolution, and transfer learning to real-world network security scenarios.

Figures (5)

• Figure 1: Reinforcement Learning agent workflow in an environment
• Figure 2: Flowchart of attacker and defender action spaces.
• Figure 3: Early training dynamics with 50-episode moving average rewards. The defender achieves rapid positive escalation while attacker rewards remain deeply negative, illustrating strong divergence in the initial learning phase.
• Figure 4: Learning Performance of Attacker and Defender Agents during initial training
• Figure 5: Learning Performance of Attacker and Defender Agents after adapting them to complex strategies