ACDZero: Graph-Embedding-Based Tree Search for Mastering Automated Cyber Defense

TL;DR

ACDZero tackles sample efficiency in automated cyber defense by fusing graph-embedding representations with MuZero-like latent-space planning and MCTS to guide policy improvement. Framing CC4 as a graph-based Dec-POMDP, it trains a GNN-based actor via distillation from search targets, enabling fast, strategically informed decisions across diverse network topologies. The approach delivers a 29.2% performance gain over a graph-based baseline, faster convergence, and reduced variance, demonstrating robust multi-step defense planning in complex, partially observable networks. Practically, ACDZero offers a scalable framework that combines deliberate search with fast reactive execution for autonomous cyber defense.

Abstract

Automated cyber defense (ACD) seeks to protect computer networks with minimal or no human intervention, reacting to intrusions by taking corrective actions such as isolating hosts, resetting services, deploying decoys, or updating access controls. However, existing approaches for ACD, such as deep reinforcement learning (RL), often face difficult exploration in complex networks with large decision/state spaces and thus require an expensive amount of samples. Inspired by the need to learn sample-efficient defense policies, we frame ACD in CAGE Challenge 4 (CAGE-4 / CC4) as a context-based partially observable Markov decision problem and propose a planning-centric defense policy based on Monte Carlo Tree Search (MCTS). It explicitly models the exploration-exploitation tradeoff in ACD and uses statistical sampling to guide exploration and decision making. We make novel use of graph neural networks (GNNs) to embed observations from the network as attributed graphs, to enable permutation-invariant reasoning over hosts and their relationships. To make our solution practical in complex search spaces, we guide MCTS with learned graph embeddings and priors over graph-edit actions, combining model-free generalization and policy distillation with look-ahead planning. We evaluate the resulting agent on CC4 scenarios involving diverse network structures and adversary behaviors, and show that our search-guided, graph-embedding-based planning improves defense reward and robustness relative to state-of-the-art RL baselines.

Figures (3)

• Figure 1: Overview of the ACDZero framework applied to the CAGE Challenge 4 environment. CAGE-4 simulates a high-fidelity enterprise network where autonomous defenders must protect critical assets against adaptive adversaries in a partially observable, multi-agent setting. To address the brittleness of standard reactive policies that degrade under topological changes, our approach treats defense not only as policy learning but as online decision-time planning. The system first transforms local observations into attributed graphs ($G_t$) to enable permutation-invariant reasoning over hosts and their relationships. Uniquely, ACDZero utilizes a Monte Carlo Tree Search (MCTS) module (bottom right) to perform look-ahead planning within a learned latent space, simulating alternative futures to discover optimal strategies. These high-quality search policies serve as distillation targets for the GNN-based Actor (top right), allowing the agent to internalize strategic foresight while maintaining the inference speed required for real-time autonomous cyber defense.
• Figure 2: ACDZero framework. Graph observations are encoded into latent states. MCTS performs tree search with learned dynamics $g$ to generate improved policy and value estimates $(p, v)$, which are distilled via prediction head $f$ into the actor $\pi_\theta$ for action selection.
• Figure 3: Training performance on CAGE Challenge 4. ACDZero converges faster ($\sim$30k episodes) and to better performance ($-150$) than the GCN baseline ($-193.68$ at $\sim$40k episodes). Tabular methods plateau early at suboptimal performance. Shaded regions indicate standard deviation.