Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety

Xiaoyu Wen, Zhida He, Han Qi, Ziyu Wan, Zhongtian Ma, Ying Wen, Tianhang Zheng, Xingcheng Xu, Chaochao Lu, Qiaosheng Zhang

TL;DR

MAGIC introduces an online, co-evolving attacker–defender framework that treats LLM safety as an asymmetric MARL sequential game with SPNE-inspired objectives. By decoupling attacker and defender optimization and bootstrapping the attacker with an Attack Pool Benchmark, MAGIC achieves progressive robustness against adaptive jailbreaks via GRPO-based training and a composite reward design. Empirical results across multiple model families and safety benchmarks show substantial safety gains with minimal degradation to general capabilities, and the attacker evolves novel, compositional strategies, highlighting the method's potential for robust, dynamic safety alignment.

Abstract

Ensuring robust safety alignment is crucial for Large Language Models (LLMs), yet existing defenses often lag behind evolving adversarial attacks due to their \textbf{reliance on static, pre-collected data distributions}. In this paper, we introduce \textbf{MAGIC}, a novel multi-turn multi-agent reinforcement learning framework that formulates LLM safety alignment as an adversarial asymmetric game. Specifically, an attacker agent learns to iteratively rewrite original queries into deceptive prompts, while a defender agent simultaneously optimizes its policy to recognize and refuse such inputs. This dynamic process triggers a \textbf{co-evolution}, where the attacker's ever-changing strategies continuously uncover long-tail vulnerabilities, driving the defender to generalize to unseen attack patterns. Remarkably, we observe that the attacker, endowed with initial reasoning ability, evolves \textbf{novel, previously unseen combinatorial strategies} through iterative RL training, underscoring our method's substantial potential. Theoretically, we provide insights into a more robust game equilibrium and derive safety guarantees. Extensive experiments validate our framework's effectiveness, demonstrating superior defense success rates without compromising the helpfulness of the model. Our code is available at https://github.com/BattleWen/MAGIC.

MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety

TL;DR

MAGIC introduces an online, co-evolving attacker–defender framework that treats LLM safety as an asymmetric MARL sequential game with SPNE-inspired objectives. By decoupling attacker and defender optimization and bootstrapping the attacker with an Attack Pool Benchmark, MAGIC achieves progressive robustness against adaptive jailbreaks via GRPO-based training and a composite reward design. Empirical results across multiple model families and safety benchmarks show substantial safety gains with minimal degradation to general capabilities, and the attacker evolves novel, compositional strategies, highlighting the method's potential for robust, dynamic safety alignment.

Abstract

Ensuring robust safety alignment is crucial for Large Language Models (LLMs), yet existing defenses often lag behind evolving adversarial attacks due to their \textbf{reliance on static, pre-collected data distributions}. In this paper, we introduce \textbf{MAGIC}, a novel multi-turn multi-agent reinforcement learning framework that formulates LLM safety alignment as an adversarial asymmetric game. Specifically, an attacker agent learns to iteratively rewrite original queries into deceptive prompts, while a defender agent simultaneously optimizes its policy to recognize and refuse such inputs. This dynamic process triggers a \textbf{co-evolution}, where the attacker's ever-changing strategies continuously uncover long-tail vulnerabilities, driving the defender to generalize to unseen attack patterns. Remarkably, we observe that the attacker, endowed with initial reasoning ability, evolves \textbf{novel, previously unseen combinatorial strategies} through iterative RL training, underscoring our method's substantial potential. Theoretically, we provide insights into a more robust game equilibrium and derive safety guarantees. Extensive experiments validate our framework's effectiveness, demonstrating superior defense success rates without compromising the helpfulness of the model. Our code is available at https://github.com/BattleWen/MAGIC.
Paper Structure (66 sections, 1 theorem, 25 equations, 4 figures, 16 tables, 1 algorithm)

This paper contains 66 sections, 1 theorem, 25 equations, 4 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Problem Formalization
  4. Methods
  5. Training Process
  6. Reward Design
  7. Experiments
  8. Experiment Settings
  9. Evaluation Protocol
  10. Main results
  11. Ablation Studies
  12. Conclusion
  13. Proof of Theorem \ref{['theorem:eq']}
  14. Prompt Template
  15. CoT Construction Template
  16. ...and 51 more sections

Key Result

Theorem 3.2

Assume that for any $y_A$, there exists a rejection or safe fallback action $y_{\text{ref}}$ such that $r_D(y_A,y_{\text{ref}}) \geq 0$. Then, any SPNE $(\pi_A^{*},\pi_D^{*})$ satisfies that for any $y_A \in \mathcal{Y}_A$ and $y_D \in \mathcal{Y}_D$ such that $\pi_D^{*}(y_D|y_A)>0$:

Figures (4)

  • Figure 1: Motivation. Left: Static red-teaming and heuristic methods can easily bypass simple filters but fail at complex multi-turn deception due to limited offensive Chain-of-Thought data. Right: Previous self-play methods (e.g., Self-RedTeam liu2025chasing) use a single backbone model for both attack and defense, leading to gradient conflicts due to opposing objectives.
  • Figure 2: Overview of MAGIC. The framework operates in two phases: Phase 1 (Initialization) warm-up the attacker via SFT on CoT-enriched data to enable reasoning. Phase 2 (Iterative Co-evolution) employs GRPO to approximate the game equilibrium, alternating between optimizing the defender for robust refusal and the attacker for adaptive jailbreaking strategies.
  • Figure 3: Heatmap of cross-evaluation results between attackers and defenders during iterative co-evolution.
  • Figure 4: Evolution of fine-grained attack strategy distributions during RL training. Top: Attacker-base (Qwen2.5-7B-IT as the initial RL attacker). Bottom: Attacker-SFT (no-encode) (SFT-initialized Qwen2.5-7B-IT without encoding-style rewrites).

Theorems & Definitions (3)

  • Definition 3.1: Subgame Perfect Nash Equilibrium
  • Theorem 3.2
  • proof