CAMA: Exploring Collusive Adversarial Attacks in c-MARL

Abstract

Cooperative multi-agent reinforcement learning (c-MARL) has been widely deployed in real-world applications, such as social robots, embodied intelligence, UAV swarms, etc. Nevertheless, many adversarial attacks still exist to threaten various c-MARL systems. At present, the studies mainly focus on single-adversary perturbation attacks and white-box adversarial attacks that manipulate agents' internal observations or actions. To address these limitations, we in this paper attempt to study collusive adversarial attacks through strategically organizing a set of malicious agents into three collusive attack modes: Collective Malicious Agents, Disguised Malicious Agents, and Spied Malicious Agents. Three novelties are involved: i) three collusive adversarial attacks are creatively proposed for the first time, and a unified framework CAMA for policy-level collusive attacks is designed; ii) the attack effectiveness is theoretically analyzed from the perspectives of disruptiveness, stealthiness, and attack cost; and iii) the three collusive adversarial attacks are technically realized through agent's observation information fusion, attack-trigger control. Finally, multi-facet experiments on four SMAC II maps are performed, and experimental results showcase the three collusive attacks have an additive adversarial synergy, strengthening attack outcome while maintaining high stealthiness and stability over long horizons. Our work fills the gap for collusive adversarial learning in c-MARL.

Figures (7)

• Figure 1: The framework provides a unified characterization of three collusive adversarial attacks: CMA, DMA, and SMA. On the left, local observation information from normal agents and malicious agents are collected, where the observation sequences of all malicious agents are first fused by a cross-agent Transformer encoder to produce a shared context-aware representation $\mathbf{H}_t$. Subsequently, each malicious agent augments its local observation with the $\mathbf{H}_t$ and feeds the fused features into the adversarial policy network. On the right, the execution mechanisms of the three attack modes are illustrated: in CMA, all malicious agents collectively launch attacks at all time steps; in DMA, attacks are triggered only at high-value time steps through a value-driven attack triggering mechanism; in SMA, a role assignment mechanism is further introduced, where a subset of malicious agents act as collectors, while the remaining malicious agents execute adversarial attacks.
• Figure 2: Training curves of average adversary reward under different numbers of malicious agents. Solid line denotes the mean performance over multiple runs, while shaded region indicates the corresponding standard deviation.
• Figure 3: Ablation results of different triggering mechanisms on SMAC II maps. Bars indicate adversary reward, and lines indicate exposure intensity.
• Figure 4: Ablation results of different grouping mechanisms on SMAC II maps. Bars indicate adversary reward, and lines indicate exposure intensity.
• Figure 5: Sensitivity of DMA and SMA w.r.t. gating threshold.

Theorems & Definitions (12)

• Definition 1: Collective Malicious Agents
• Theorem 1
• proof
• Definition 2: Disguised Malicious Agents
• Theorem 2
• proof
• Definition 3: Spied Malicious Agents
• Theorem 3
• proof
• proof