Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Rewards in Reinforcement Learning for Cyber Defence

Elizabeth Bates, Chris Hicks, Vasilios Mavroudis

TL;DR

This work tackles the problem that dense, human-engineered rewards used in autonomous cyber defence (ACD) DRL agents can bias learning toward suboptimal, high-risk policies. It introduces a ground truth scoring framework (Score_GT) and applies reliability and risk metrics to compare sparse versus dense rewards across Yawning Titan and miniCAGE, using PPO and DQN. The findings show sparse rewards (SP, SPN) deliver better policy quality (lower compromise counts), higher training reliability (lower dispersion across runs), and reduced risk in worst-case outcomes, even as network size grows and attacker timing varies. The approach provides a robust, reward-agnostic evaluation method with practical implications for designing safer, more effective cyber defence agents in complex network environments.

Abstract

Recent years have seen an explosion of interest in autonomous cyber defence agents trained to defend computer networks using deep reinforcement learning. These agents are typically trained in cyber gym environments using dense, highly engineered reward functions which combine many penalties and incentives for a range of (un)desirable states and costly actions. Dense rewards help alleviate the challenge of exploring complex environments but risk biasing agents towards suboptimal and potentially riskier solutions, a critical issue in complex cyber environments. We thoroughly evaluate the impact of reward function structure on learning and policy behavioural characteristics using a variety of sparse and dense reward functions, two well-established cyber gyms, a range of network sizes, and both policy gradient and value-based RL algorithms. Our evaluation is enabled by a novel ground truth evaluation approach which allows directly comparing between different reward functions, illuminating the nuanced inter-relationships between rewards, action space and the risks of suboptimal policies in cyber environments. Our results show that sparse rewards, provided they are goal aligned and can be encountered frequently, uniquely offer both enhanced training reliability and more effective cyber defence agents with lower-risk policies. Surprisingly, sparse rewards can also yield policies that are better aligned with cyber defender goals and make sparing use of costly defensive actions without explicit reward-based numerical penalties.

Beyond Rewards in Reinforcement Learning for Cyber Defence

TL;DR

This work tackles the problem that dense, human-engineered rewards used in autonomous cyber defence (ACD) DRL agents can bias learning toward suboptimal, high-risk policies. It introduces a ground truth scoring framework (Score_GT) and applies reliability and risk metrics to compare sparse versus dense rewards across Yawning Titan and miniCAGE, using PPO and DQN. The findings show sparse rewards (SP, SPN) deliver better policy quality (lower compromise counts), higher training reliability (lower dispersion across runs), and reduced risk in worst-case outcomes, even as network size grows and attacker timing varies. The approach provides a robust, reward-agnostic evaluation method with practical implications for designing safer, more effective cyber defence agents in complex network environments.

Abstract

Recent years have seen an explosion of interest in autonomous cyber defence agents trained to defend computer networks using deep reinforcement learning. These agents are typically trained in cyber gym environments using dense, highly engineered reward functions which combine many penalties and incentives for a range of (un)desirable states and costly actions. Dense rewards help alleviate the challenge of exploring complex environments but risk biasing agents towards suboptimal and potentially riskier solutions, a critical issue in complex cyber environments. We thoroughly evaluate the impact of reward function structure on learning and policy behavioural characteristics using a variety of sparse and dense reward functions, two well-established cyber gyms, a range of network sizes, and both policy gradient and value-based RL algorithms. Our evaluation is enabled by a novel ground truth evaluation approach which allows directly comparing between different reward functions, illuminating the nuanced inter-relationships between rewards, action space and the risks of suboptimal policies in cyber environments. Our results show that sparse rewards, provided they are goal aligned and can be encountered frequently, uniquely offer both enhanced training reliability and more effective cyber defence agents with lower-risk policies. Surprisingly, sparse rewards can also yield policies that are better aligned with cyber defender goals and make sparing use of costly defensive actions without explicit reward-based numerical penalties.
Paper Structure (32 sections, 6 equations, 4 figures, 23 tables)

This paper contains 32 sections, 6 equations, 4 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Autonomous Cyber Defence
  4. Reliability and Risk in RL
  5. Dispersion variability across time (DT)
  6. Dispersion variability across runs (DR)
  7. Conditional Value at Risk (CVaR)
  8. Risk across Fixed-Policy Rollouts (RF)
  9. Methodology
  10. Yawning Titan Cyber Gym
  11. Cyber Autonomy Gym for Experimentation
  12. Ground Truth
  13. Ground Truth Score ($\mathrm{Score}_\mathrm{GT}$)
  14. Evaluating reliability across different rewards
  15. Experiments
  16. ...and 17 more sections

Figures (4)

  • Figure 1: One step of the YT network environment illustrating an intra-step node compromise that is concealed by standard cyber gym evaluations.
  • Figure 2: Training curves for the 50 node network size, red then blue agent order in the extended action space for reward functions: sparse positive (SP), sparse negative (SN), sparse positive negative (SPN), dense negative (DN) and complex dense negative (CDN).
  • Figure 3: Training curves for the 50 node network size, blue then red agent order in the extended action space for reward functions: sparse positive (SP), sparse negative (SN), sparse positive negative (SPN), dense negative (DN) and complex dense negative (CDN).
  • Figure 4: Training curves for the 50 node network size, random agent order in the extended action space for reward functions: sparse positive (SP), sparse negative (SN), sparse positive negative (SPN), dense negative (DN) and complex dense negative (CDN).