Towards Type Agnostic Cyber Defense Agents

TL;DR

This work formulates cyber defense as a partially observable stochastic Bayesian game and extends the YAWNING-TITAN framework to train defenders against multiple attacker types via self-play. Using PPO-based agents and a HiPPO-inspired hierarchical policy, the authors compare single-type, alternating, and hierarchical multi-type training against two attacker archetypes: $Ransomware$ and $APT$. Key findings show that multi-type training yields more robust, transferable defender policies, with the hierarchical defender achieving the best average performance across attacker types, though win rates remain modest and highly sensitive to detection parameters. These results suggest that attacker-type-agnostic defense agents are feasible and scalable, motivating future work on online mixtures of experts and richer simulation environments for real-world deployment.

Abstract

With computing now ubiquitous across government, industry, and education, cybersecurity has become a critical component for every organization on the planet. Due to this ubiquity of computing, cyber threats have continued to grow year over year, leading to labor shortages and a skills gap in cybersecurity. As a result, many cybersecurity product vendors and security organizations have looked to artificial intelligence to shore up their defenses. This work considers how to characterize attackers and defenders in one approach to the automation of cyber defense -- the application of reinforcement learning. Specifically, we characterize the types of attackers and defenders in the sense of Bayesian games and, using reinforcement learning, derive empirical findings about how to best train agents that defend against multiple types of attackers.

Figures (6)

• Figure 1: The Diamond Model of intrusion analysis: Adversary, Victim, Capability, and Infrastructure. Edges represent the relationships between features that can be used to analyze and discover malicious activity.
• Figure 2: Training curves for all four training settings. Note that curves are smoothed using a best fit line with order 5.
• Figure 3: Mean evaluation reward for each defender against ransomware and APT-type attackers
• Figure 4: Interquartile mean evaluation reward for each defender against ransomware and APT-type attackers. Note that all rewards in this graph are negative; less negative values are better.
• Figure 5: Evaluation win rate for each defender against ransomware and APT-type attackers