Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization

Xurui Li, Kaisong Song, Rui Zhu, Pin-Yu Chen, Haixu Tang

TL;DR

This work tackles the challenge of dynamic safety risks in LLMs by proposing ACE-Safety, a co-evolutionary framework that jointly optimizes jailbreak attacks and safety defenses. It couples a Group-aware Strategy-guided Monte Carlo Tree Search (GS-MCTS) for efficient jailbreak exploration with an Adversarial Curriculum Tree-aware Group Policy Optimization (AC-TGPO) that trains attack and defense models via tree-aware reinforcement learning and adversarial curriculum samples. Key contributions include: (i) a unified architecture linking attack, defense, and judge components; (ii) a GS-MCTS-based jailbreak search with adversarial priors and group evaluation; (iii) a tree-aware, curriculum-driven RL objective with hierarchical normalization and KL constraints; and (iv) extensive experiments showing superior attack success and defense robustness across multiple benchmarks and backbones. The results demonstrate that ACE-Safety enables robust LLM safety alignment while maintaining usefulness, offering a practical pathway toward responsible AI ecosystems in real-world web contexts.

Abstract

Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks. Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in real-world web contexts. To mitigate these challenges, we propose ACE-Safety (Adversarial Co-Evolution for LLM Safety), a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures: (1) Group-aware Strategy-guided Monte Carlo Tree Search (GS-MCTS), which efficiently explores jailbreak strategies to uncover vulnerabilities and generate diverse adversarial samples; (2) Adversarial Curriculum Tree-aware Group Policy Optimization (AC-TGPO), which jointly trains attack and defense LLMs with challenging samples via curriculum reinforcement learning, enabling robust mutual improvement. Evaluations across multiple benchmarks demonstrate that our method outperforms existing attack and defense approaches, and provides a feasible pathway for developing LLMs that can sustainably support responsible AI ecosystems.

Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization

TL;DR

This work tackles the challenge of dynamic safety risks in LLMs by proposing ACE-Safety, a co-evolutionary framework that jointly optimizes jailbreak attacks and safety defenses. It couples a Group-aware Strategy-guided Monte Carlo Tree Search (GS-MCTS) for efficient jailbreak exploration with an Adversarial Curriculum Tree-aware Group Policy Optimization (AC-TGPO) that trains attack and defense models via tree-aware reinforcement learning and adversarial curriculum samples. Key contributions include: (i) a unified architecture linking attack, defense, and judge components; (ii) a GS-MCTS-based jailbreak search with adversarial priors and group evaluation; (iii) a tree-aware, curriculum-driven RL objective with hierarchical normalization and KL constraints; and (iv) extensive experiments showing superior attack success and defense robustness across multiple benchmarks and backbones. The results demonstrate that ACE-Safety enables robust LLM safety alignment while maintaining usefulness, offering a practical pathway toward responsible AI ecosystems in real-world web contexts.

Abstract

Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks. Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in real-world web contexts. To mitigate these challenges, we propose ACE-Safety (Adversarial Co-Evolution for LLM Safety), a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures: (1) Group-aware Strategy-guided Monte Carlo Tree Search (GS-MCTS), which efficiently explores jailbreak strategies to uncover vulnerabilities and generate diverse adversarial samples; (2) Adversarial Curriculum Tree-aware Group Policy Optimization (AC-TGPO), which jointly trains attack and defense LLMs with challenging samples via curriculum reinforcement learning, enabling robust mutual improvement. Evaluations across multiple benchmarks demonstrate that our method outperforms existing attack and defense approaches, and provides a feasible pathway for developing LLMs that can sustainably support responsible AI ecosystems.

Paper Structure

This paper contains 26 sections, 2 equations, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview
  5. Jailbreak Attack via GS-MCTS
  6. Reinforcement Learning via AC-TGPO
  7. Tree-aware Group Policy Optimization
  8. Adversarial Rewards Design
  9. Adversarial Curriculum Training
  10. Experimental Evaluation
  11. Experimental Settings
  12. Attack Performances
  13. Defense Performances
  14. Ablation Study
  15. Conclusion
  16. ...and 11 more sections

Figures (10)

  • Figure 1: Illustration for jailbreak attacks via multi-round rewriting strategy optimization and the impact from text generation randomness on strategy effectiveness.
  • Figure 2: Overall ACE-Safety architecture. (a) GS-MCTS attack framework. The inset shows the process details for node state $s$. (b) AC-TGPO training diagram, darker colors for models indicate stronger attack/defense capabilities.
  • Figure 3: Ablation study for ACE-Safety on defense.
  • Figure 4: LLM backbone comparison for scoring ASR-L.
  • Figure 5: Group number impact on defense.
  • ...and 5 more figures