SAJA: A State-Action Joint Attack Framework on Multi-Agent Deep Reinforcement Learning

TL;DR

This work addresses the vulnerability of multi-agent deep reinforcement learning (MADRL) to adversarial perturbations by introducing SAJA, a state-action joint attack that optimizes perturbations in two phases using both actor and critic guidance. A theoretical framework bounds the attack and motivates a practical two-stage optimization with a Heuristic Loss Function that blends Q-value degradation with action-distance regularization. Empirical results in the Multi-Agent Particle Environment show SAJA outperforming state-only and action-only attacks, including against defended models (PAAD, ATLA, M3DDPG), while maintaining stealthiness through distributed perturbations. The findings highlight a significant security risk in MADRL systems and point to the need for joint defense strategies that counter these dual perturbations in real-world cooperative-competitive tasks. The work also opens avenues for extending SAJA to black-box settings and more complex domains such as StarCraft II.”

Abstract

Multi-Agent Deep Reinforcement Learning (MADRL) has shown potential for cooperative and competitive tasks such as autonomous driving and strategic gaming. However, models trained by MADRL are vulnerable to adversarial perturbations on states and actions. Therefore, it is essential to investigate the robustness of MADRL models from an attack perspective. Existing studies focus on either state-only attacks or action-only attacks, but do not consider how to effectively joint them. Simply combining state and action perturbations such as randomly perturbing states and actions does not exploit their potential synergistic effects. In this paper, we propose the State-Action Joint Attack (SAJA) framework that has a good synergistic effects. SAJA consists of two important phases: (1) In the state attack phase, a multi-step gradient ascent method utilizes both the actor network and the critic network to compute an adversarial state, and (2) in the action attack phase, based on the perturbed state, a second gradient ascent uses the critic network to craft the final adversarial action. Additionally, a heuristic regularizer measuring the distance between the perturbed actions and the original clean ones is added into the loss function to enhance the effectiveness of the critic's guidance. We evaluate SAJA in the Multi-Agent Particle Environment (MPE), demonstrating that (1) it outperforms and is more stealthy than state-only or action-only attacks, and (2) existing state or action defense methods cannot defend its attacks.

Figures (3)

• Figure 1: Impact of Perturbation Budget Allocation on SAJA. The x-axis represents the state perturbation budget $\epsilon_s$. Each line corresponds to a fixed total budget $s$, as indicated in the legend. The y-axis represents the average episodic reward, with each point calculated as the mean of 5 independent runs (500 episodes each) for each $(\epsilon_s,\epsilon_a)$ setting.
• Figure 2: Distribution of the average Euclidean distance under 6 attack methods. The x-axis represents the average L2 distance $||a" - a^{(0)}||_2 / m$ between the original and final actions of victim agents in an episode. The y-axis shows the frequency of these distances within bins of width 0.01. For clarity, distances greater than 0.40 are omitted due to their low frequency.
• Figure 3: (Left) Learning curves for the vanilla models. (Right) Learning curves for the defended models. Training curves for all benchmark models used in the experiments. The solid lines represent the average team reward over 10 evaluation episodes, and the shaded areas represent the standard deviation.Training curves for all benchmark models used in the experiments.

Theorems & Definitions (8)

• Definition 1: State–Action–Adversarial Markov Game
• Definition 2: Perturbation Sets
• Proposition 1: Heuristic Transition
• Theorem 1: Critic Accuracy
• Definition 3: Multi-Agent Action Difference
• Theorem 2: Bounded Reward Degradation
• proof
• proof