Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intrusion Prevention through Optimal Stopping

Kim Hammar, Rolf Stadler

TL;DR

This work reframes intrusion prevention as a finite-horizon, multi-stop optimal stopping problem under partial observability, revealing that optimal defender policies have a threshold structure over a belief state. It introduces T-SPSA, a threshold-focused reinforcement-learning algorithm that learns a vector of stopping thresholds $\alpha^{*}_l$ (via a smooth policy) to maximize total reward, and demonstrates its advantages over state-of-the-art baselines. The authors implement a two-system workflow—an emulation system that captures the target infrastructure and an adjacent simulation system for policy learning—to reliably learn and validate defender policies with realistic statistics $\hat{f}_{XYZ|s}$. Empirical results show fast convergence and strong performance, with learned policies closely approaching an optimal benchmark and significantly surpassing baselines across attacker types. This framework enables practical deployment of automated intrusion prevention policies in real IT infrastructures by leveraging threshold-based control and validated emulation-based evaluation.

Abstract

We study automated intrusion prevention using reinforcement learning. Following a novel approach, we formulate the problem of intrusion prevention as an (optimal) multiple stopping problem. This formulation gives us insight into the structure of optimal policies, which we show to have threshold properties. For most practical cases, it is not feasible to obtain an optimal defender policy using dynamic programming. We therefore develop a reinforcement learning approach to approximate an optimal threshold policy. We introduce T-SPSA, an efficient reinforcement learning algorithm that learns threshold policies through stochastic approximation. We show that T-SPSA outperforms state-of-the-art algorithms for our use case. Our overall method for learning and validating policies includes two systems: a simulation system where defender policies are incrementally learned and an emulation system where statistics are produced that drive simulation runs and where learned policies are evaluated. We show that this approach can produce effective defender policies for a practical IT infrastructure.

Intrusion Prevention through Optimal Stopping

TL;DR

This work reframes intrusion prevention as a finite-horizon, multi-stop optimal stopping problem under partial observability, revealing that optimal defender policies have a threshold structure over a belief state. It introduces T-SPSA, a threshold-focused reinforcement-learning algorithm that learns a vector of stopping thresholds (via a smooth policy) to maximize total reward, and demonstrates its advantages over state-of-the-art baselines. The authors implement a two-system workflow—an emulation system that captures the target infrastructure and an adjacent simulation system for policy learning—to reliably learn and validate defender policies with realistic statistics . Empirical results show fast convergence and strong performance, with learned policies closely approaching an optimal benchmark and significantly surpassing baselines across attacker types. This framework enables practical deployment of automated intrusion prevention policies in real IT infrastructures by leveraging threshold-based control and validated emulation-based evaluation.

Abstract

We study automated intrusion prevention using reinforcement learning. Following a novel approach, we formulate the problem of intrusion prevention as an (optimal) multiple stopping problem. This formulation gives us insight into the structure of optimal policies, which we show to have threshold properties. For most practical cases, it is not feasible to obtain an optimal defender policy using dynamic programming. We therefore develop a reinforcement learning approach to approximate an optimal threshold policy. We introduce T-SPSA, an efficient reinforcement learning algorithm that learns threshold policies through stochastic approximation. We show that T-SPSA outperforms state-of-the-art algorithms for our use case. Our overall method for learning and validating policies includes two systems: a simulation system where defender policies are incrementally learned and an emulation system where statistics are produced that drive simulation runs and where learned policies are evaluated. We show that this approach can produce effective defender policies for a practical IT infrastructure.

Paper Structure

This paper contains 30 sections, 4 theorems, 16 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. The Intrusion Prevention Use Case
  3. Theoretical Background
  4. Markov Decision Processes
  5. The Reinforcement Learning Problem
  6. The Markovian Optimal Stopping Problem
  7. Formalizing The Intrusion Prevention Use Case and Our Reinforcement Learning Approach
  8. A POMDP Model of the Intrusion Prevention Use Case
  9. Actions $\mathcal{A}$
  10. States $\mathcal{S}$ and Initial State Distribution $\rho_1$
  11. Observations $\mathcal{O}$
  12. Transition Probabilities $\mathcal{P}^{a_t}_{s_t,s_{t+1},l_t}$
  13. Observation Function $\mathcal{Z}(o_{t+1},s_{t+1},a_t)$
  14. Reward Function $\mathcal{R}_{s_t,l_t}^{a_t}$
  15. Time Horizon $T_{\emptyset}$
  16. ...and 15 more sections

Key Result

Theorem 1

Given the POMDP in Section sec:pomdp_model, let $L$ denote the number of stop actions, $f_{XYZ|s}$ the conditional distribution of the observations, $b(1)$ the belief state, $\mathscr{S}_{l}$ the stopping set, and $\mathscr{C}_{l}$ the continuation set. The following holds:

Figures (9)

  • Figure 1: The IT infrastructure and the actors in the use case.
  • Figure 2: Our approach for finding and evaluating intrusion prevention policies.
  • Figure 3: Optimal multiple stopping formulation of intrusion prevention; the horizontal axis represents time; $T$ is the time horizon; the episode length is $T-1$; the dashed line shows the intrusion start time; the optimal policy is to prevent the attacker at the time of intrusion.
  • Figure 4: The cumulative distribution function (CDF) of the intrusion start time $I_t$.
  • Figure 5: State transition diagram of the POMDP: each circle represents a state; an arrow represents a state transition; a label indicates the event that triggers the state transition; an episode starts in state $s_1=0$ with $l_1=L$.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Lemma 1
  • Lemma 2
  • Lemma 3