Learning Near-Optimal Intrusion Responses Against Dynamic Attackers

TL;DR

This work treats intrusion response as an optimal stopping game between a defender and a dynamic attacker, revealing threshold structures in optimal strategies. It introduces Threshold Fictitious Self-Play (T-FP), a gradient-based, threshold-parametrized learning procedure that efficiently converges to near-Nash defender policies against adaptive adversaries. The authors validate the framework with both a simulation system and a hardware-like emulation platform, showing that learned strategies achieve near-equilibrium performance and robustness against a changing attacker. The combination of a formal stopping-game model, a scalable learning algorithm, and emulation-informed evaluation provides a practical pathway to deploy near-optimal intrusion responses in real IT infrastructures.

Abstract

We study automated intrusion response and formulate the interaction between an attacker and a defender as an optimal stopping game where attack and defense strategies evolve through reinforcement learning and self-play. The game-theoretic modeling enables us to find defender strategies that are effective against a dynamic attacker, i.e. an attacker that adapts its strategy in response to the defender strategy. Further, the optimal stopping formulation allows us to prove that optimal strategies have threshold properties. To obtain near-optimal defender strategies, we develop Threshold Fictitious Self-Play (T-FP), a fictitious self-play algorithm that learns Nash equilibria through stochastic approximation. We show that T-FP outperforms a state-of-the-art algorithm for our use case. The experimental part of this investigation includes two systems: a simulation system where defender strategies are incrementally learned and an emulation system where statistics are collected that drive simulation runs and where learned strategies are evaluated. We argue that this approach can produce effective defender strategies for a practical IT infrastructure.

Figures (14)

• Figure 1: The IT infrastructure and the actors in the intrusion response use case.
• Figure 2: Our framework for finding and evaluating intrusion response strategies hammar_stadler_tnsm.
• Figure 3: State transition diagram of a game episode: each disk represents a state; an arrow represents a state transition; a label indicates the conditions for the state transition (w.p means "with probability"); a game episode starts in state $s_1=0$ with $l=L$ and ends in state $s_T=\emptyset$.
• Figure 4: Stopping times of the defender and the attacker in a game episode; the bottom horizontal axis represents time; the black circles on the middle axis and the upper axis represent time-steps of the defender's stop actions and the attacker's stop actions, respectively; $\tau_{i,j}$ denotes the $j$th stopping time of player $i$; the cross shows the time the intrusion is stopped; an intrusion starts when the attacker takes the first stop action (at time $\tau_{\mathrm{A},1}$); an episode ends either when the attacker is stopped (as a consequence of defender actions) or when the attacker terminates its intrusion.
• Figure 5: Illustration of Theorem \ref{['thm:best_responses']}; the upper part shows $L$ thresholds $\tilde{\alpha}_{1} \geq \tilde{\alpha}_{2}, \hdots, \geq \tilde{\alpha}_{L} \in [0,1]$ that define a best response strategy $\tilde{\pi}_{\mathrm{D}} \in \mathscr{B}_{\mathrm{D}}(\pi_{\mathrm{A}})$ for the defender (\ref{['eq:prop_br_defender']}); the lower part shows $2L$ thresholds $\tilde{\beta}_{0,1}, \tilde{\beta}_{1,1}, \hdots, \tilde{\beta}_{0,L}, \tilde{\beta}_{1,L} \in [0,1]$ that define a best response strategy $\tilde{\pi}_{\mathrm{A}} \in \mathscr{B}_{\mathrm{A}}(\pi_{\mathrm{D}})$ for the attacker (\ref{['eq:prop_br_attacker_1']})--(\ref{['eq:prop_br_attacker_2']}).

Theorems & Definitions (7)

• Theorem 1
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5
• Lemma 6