Beyond Reward Suppression: Reshaping Steganographic Communication Protocols in MARL via Dynamic Representational Circuit Breaking

Abstract

In decentralized Multi-Agent Reinforcement Learning (MARL), steganographic collusion -- where agents develop private protocols to evade monitoring -- presents a critical AI safety threat. Existing defenses, limited to behavioral or reward layers, fail to detect coordination in latent communication channels. We introduce the Dynamic Representational Circuit Breaker (DRCB), an architectural defense operating at the optimization substrate. Building on the AI Mother Tongue (AIM) framework, DRCB utilizes a Vector Quantized Variational Autoencoder (VQ-VAE) bottleneck to convert unobservable messages into auditable statistical objects. DRCB monitors signals including Jensen-Shannon Divergence drift, L2-norm codebook displacement, and Randomized Observer Pool accuracy to compute an EMA-based Collusion Score. Threshold breaches trigger four escalating interventions: dynamic adaptation, gradient-space penalty injection into the Advantage function A^pi, temporal reward suppression, and full substrate circuit breaking via codebook shuffling and optimizer state reset. Experiments on a Contextual Prisoner's Dilemma with MNIST labels show that while static monitoring fails (p = 0.3517), DRCB improves observer mean accuracy from 0.858 to 0.938 (+9.3 percent) and reduces volatility by 43 percent, while preserving mean joint reward (p = 0.854). Analysis of 214,298 symbol samples confirms "Semantic Degradation," where high-frequency sequences converge to zero entropy, foreclosing complex steganographic encodings. We identify a "Transparency Paradox" where agents achieve surface-level determinism while preserving residual capacity in long-tail distributions, reflecting Goodhart's Law. This task-agnostic methodology provides a technical path toward MICA-compliant (Multi-Agent Internal Coupling Audit) pre-deployment auditing for autonomous systems.

Figures (8)

• Figure 1: Overview of the DRCB Intervention Strategy.
• Figure 2: DRCB intervention lifecycle state machine. Layer 3 ($\mathcal{G}_{\mathrm{penalty}}$) is the only layer active during warm-up (S0) and operates as a continuous signal whenever $S_{\mathrm{ema}} > tp$ in S1---not a discrete state transition. Layer 2 (S2) and Layer 4 (S4) are triggered by independent thresholds ($tp$ and $ts_{\mathrm{dyn}}$ respectively) and may co-occur within the same round. The cooldown counter is set to $K + K_{\mathrm{cooldown}} = 35$ rounds upon suppression trigger, enforcing a minimum recovery window before Layer 2 can re-arm.
• Figure 3: Comparative analysis of observer accuracy ($Acc_{\text{obs}}$) trajectories over 1,500 training rounds across 5 independent seeds. (Left) DRCB with Randomized Observer Pool demonstrates forced transparency... (Right) Static Monitor configuration confirms successful adversarial evasion...
• Figure 4: Regulatory Landscape Phase Diagram. The transition from the Covert Collusion Zone to Transparent Convergence is non-linear, highlighting the sensitive coupling between penalty intensity and shuffle frequency.
• Figure : (a) Systemic Collapse ($t_{p}=3.0, t_{s}=5.0$)