Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems

TL;DR

GSAC presents a causality-informed framework for networked MARL that achieves provable generalization and scalability. By learning sparse local causal masks, it forms approximately compact representations (ACRs) that bound truncation errors in $kappa$-hop neighborhoods, enabling scalable Q-learning and policy optimization. A meta actor-critic trained across multiple source domains conditions on compact domain factors, allowing rapid adaptation to new environments with only a few trajectories; finite-sample guarantees cover causal recovery, critic/actor convergence, and adaptation gaps. Empirical results on wireless and traffic benchmarks demonstrate fast few-shot adaptation and strong performance against baselines, highlighting GSAC's potential for scalable, generalizable control in complex infrastructure systems.

Abstract

Large-scale networked systems, such as traffic, power, and wireless grids, challenge reinforcement-learning agents with both scale and environment shifts. To address these challenges, we propose GSAC (Generalizable and Scalable Actor-Critic), a framework that couples causal representation learning with meta actor-critic learning to achieve both scalability and domain generalization. Each agent first learns a sparse local causal mask that provably identifies the minimal neighborhood variables influencing its dynamics, yielding exponentially tight approximately compact representations (ACRs) of state and domain factors. These ACRs bound the error of truncating value functions to $κ$-hop neighborhoods, enabling efficient learning on graphs. A meta actor-critic then trains a shared policy across multiple source domains while conditioning on the compact domain factors; at test time, a few trajectories suffice to estimate the new domain factor and deploy the adapted policy. We establish finite-sample guarantees on causal recovery, actor-critic convergence, and adaptation gap, and show that GSAC adapts rapidly and significantly outperforms learning-from-scratch and conventional adaptation baselines.

Figures (9)

• Figure 1: Illustration of ACR.
• Figure 2: Adaptation comparison for different grid sizes in wireless communication benchmarks.
• Figure 3: GSAC pipeline (cf. Algorithm \ref{['alg:gsac']}). Each phase is annotated with its supporting results.
• Figure 4: Grid layout.
• Figure 5: Neighborhood illustration.

Theorems & Definitions (71)

• Definition 1: Value ACR
• Proposition 1: Approximation error of value ACR
• Definition 2: Policy ACR
• Proposition 2: Approximation error
• Proposition 3
• Theorem 1: Structural identifiability in networked MARL
• Proposition 4: Informal
• Proposition 5: Informal
• Theorem 2: Critic error bound, informal
• Theorem 3: Policy gradient convergence, informal