Unveiling the Black Box: A Multi-Layer Framework for Explaining Reinforcement Learning-Based Cyber Agents

TL;DR

This paper tackles the opacity of reinforcement-learning agents used to model autonomous cyberattacks by introducing a unified, multi-layer explainability framework. The approach combines strategic, MDP-level insights derived from modeling attacker behaviour as a POMDP with tactical, policy-level analyses that track temporal Q-values and surface high-impact transitions via Prioritised Experience Replay. Through CyberBattleSim experiments across increasingly complex environments, the authors demonstrate how exploration-exploitation dynamics, phase transitions, and evolving action preferences reveal the intent and learning progression of adversarial agents. The framework is designed to be agent- and environment-agnostic, enabling practical use for red-team planning, policy debugging, and defense design, while laying groundwork for real-time and multi-agent extensions. This work advances transparent, trustworthy RL in cybersecurity by converting black-box learning into interpretable behavioral intelligence.

Abstract

Reinforcement Learning (RL) agents are increasingly used to simulate sophisticated cyberattacks, but their decision-making processes remain opaque, hindering trust, debugging, and defensive preparedness. In high-stakes cybersecurity contexts, explainability is essential for understanding how adversarial strategies are formed and evolve over time. In this paper, we propose a unified, multi-layer explainability framework for RL-based attacker agents that reveals both strategic (MDP-level) and tactical (policy-level) reasoning. At the MDP level, we model cyberattacks as a Partially Observable Markov Decision Processes (POMDPs) to expose exploration-exploitation dynamics and phase-aware behavioural shifts. At the policy level, we analyse the temporal evolution of Q-values and use Prioritised Experience Replay (PER) to surface critical learning transitions and evolving action preferences. Evaluated across CyberBattleSim environments of increasing complexity, our framework offers interpretable insights into agent behaviour at scale. Unlike previous explainable RL methods, which are often post-hoc, domain-specific, or limited in depth, our approach is both agent- and environment-agnostic, supporting use cases ranging from red-team simulation to RL policy debugging. By transforming black-box learning into actionable behavioural intelligence, our framework enables both defenders and developers to better anticipate, analyse, and respond to autonomous cyber threats.

Figures (7)

• Figure 1: Cumulative Rewards Comparison of Attacker Policies Across Environments (The shaded region in the figures represents the standard deviation of cumulative rewards across training steps for each agent).
• Figure 2: Impact of Exploration Strategies on Cumulative Rewards Across Environments.
• Figure 3: Impact of Exploration Strategies on Node Discovery Rate Across Environments.
• Figure 4: Cumulative Rewards Comparison in Early vs. Late Phase Attack Across Environments.
• Figure 5: Emergence of Action Preferences via State-Aggregated Q-Values Across Episodes.