A System for Interactive Examination of Learned Security Policies

TL;DR

The paper addresses the gap between aggregate performance metrics and the actual behavior of learned security policies by introducing an interactive policy-examination system. It couples an emulation environment (for realistic statistics) with a simulation system (for policy learning via reinforcement learning, specifically PPO) to train policies represented by a neural network $\pi_{\theta}(a|h)$ that maps history $h$ and infrastructure metrics $h$ to action probabilities. The key contributions are the end-to-end architecture that enables interactive inspection of POMDP episodes, the implementation details (Docker-based emulation, PPO training, React/Flask frontend, PostgreSQL), and a demonstrated intrusion-prevention use case revealing policy structure and edge-case behavior. This approach enhances safe deployment of learned security policies by exposing how policies react in edge cases and under adversarial scenarios, bridging the gap between training-time metrics and real-world policy behavior.

Abstract

We present a system for interactive examination of learned security policies. It allows a user to traverse episodes of Markov decision processes in a controlled manner and to track the actions triggered by security policies. Similar to a software debugger, a user can continue or or halt an episode at any time step and inspect parameters and probability distributions of interest. The system enables insight into the structure of a given policy and in the behavior of a policy in edge cases. We demonstrate the system with a network intrusion use case. We examine the evolution of an IT infrastructure's state and the actions prescribed by security policies while an attack occurs. The policies for the demonstration have been obtained through a reinforcement learning approach that includes a simulation system where policies are incrementally learned and an emulation system that produces statistics that drive the simulation runs.

Figures (4)

• Figure 1: The IT infrastructure and the actors in the intrusion prevention use case: an attacker, a client population, and a defender.
• Figure 2: Our framework for learning and evaluating security policies.
• Figure 3: Architecture of the system for inspecting and examining learned security policies.
• Figure 4: The demonstration view of the system which allows the user to interactively step through a POMDP episode.