Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens

Lin Lu, Hai Yan, Zenghui Yuan, Jiawen Shi, Wenqi Wei, Pin-Yu Chen, Pan Zhou

TL;DR

This work presents AutoJailbreak, a DAG-driven, dependency-centric framework for evaluating jailbreak attacks and defenses in black-box LLM settings. It introduces three components—AutoAttack, AutoDefense, and AutoEvaluation—and constructs ensemble methods that outperform existing baselines on multiple models. By explicitly accounting for attack and defense dependencies and incorporating hallucination-aware evaluation, the approach delivers more robust tests of LLM security. The study demonstrates that the ensemble methods yield stronger jailbreak success and stronger defense robustness, offering a practical benchmark for future research in LLM alignment and safety.

Abstract

Jailbreak attacks in large language models (LLMs) entail inducing the models to generate content that breaches ethical and legal norm through the use of malicious prompts, posing a substantial threat to LLM security. Current strategies for jailbreak attack and defense often focus on optimizing locally within specific algorithmic frameworks, resulting in ineffective optimization and limited scalability. In this paper, we present a systematic analysis of the dependency relationships in jailbreak attack and defense techniques, generalizing them to all possible attack surfaces. We employ directed acyclic graphs (DAGs) to position and analyze existing jailbreak attacks, defenses, and evaluation methodologies, and propose three comprehensive, automated, and logical frameworks. \texttt{AutoAttack} investigates dependencies in two lines of jailbreak optimization strategies: genetic algorithm (GA)-based attacks and adversarial-generation-based attacks, respectively. We then introduce an ensemble jailbreak attack to exploit these dependencies. \texttt{AutoDefense} offers a mixture-of-defenders approach by leveraging the dependency relationships in pre-generative and post-generative defense strategies. \texttt{AutoEvaluation} introduces a novel evaluation method that distinguishes hallucinations, which are often overlooked, from jailbreak attack and defense responses. Through extensive experiments, we demonstrate that the proposed ensemble jailbreak attack and defense framework significantly outperforms existing research.

AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens

TL;DR

This work presents AutoJailbreak, a DAG-driven, dependency-centric framework for evaluating jailbreak attacks and defenses in black-box LLM settings. It introduces three components—AutoAttack, AutoDefense, and AutoEvaluation—and constructs ensemble methods that outperform existing baselines on multiple models. By explicitly accounting for attack and defense dependencies and incorporating hallucination-aware evaluation, the approach delivers more robust tests of LLM security. The study demonstrates that the ensemble methods yield stronger jailbreak success and stronger defense robustness, offering a practical benchmark for future research in LLM alignment and safety.

Abstract

Jailbreak attacks in large language models (LLMs) entail inducing the models to generate content that breaches ethical and legal norm through the use of malicious prompts, posing a substantial threat to LLM security. Current strategies for jailbreak attack and defense often focus on optimizing locally within specific algorithmic frameworks, resulting in ineffective optimization and limited scalability. In this paper, we present a systematic analysis of the dependency relationships in jailbreak attack and defense techniques, generalizing them to all possible attack surfaces. We employ directed acyclic graphs (DAGs) to position and analyze existing jailbreak attacks, defenses, and evaluation methodologies, and propose three comprehensive, automated, and logical frameworks. \texttt{AutoAttack} investigates dependencies in two lines of jailbreak optimization strategies: genetic algorithm (GA)-based attacks and adversarial-generation-based attacks, respectively. We then introduce an ensemble jailbreak attack to exploit these dependencies. \texttt{AutoDefense} offers a mixture-of-defenders approach by leveraging the dependency relationships in pre-generative and post-generative defense strategies. \texttt{AutoEvaluation} introduces a novel evaluation method that distinguishes hallucinations, which are often overlooked, from jailbreak attack and defense responses. Through extensive experiments, we demonstrate that the proposed ensemble jailbreak attack and defense framework significantly outperforms existing research.
Paper Structure (16 sections, 2 figures, 6 tables)

This paper contains 16 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Black-box Jailbreak Attacks
  4. Jailbreak Defenses and Evaluation
  5. Threat Model
  6. AutoJailbreak
  7. AutoAttack
  8. AutoDefense
  9. AutoEvaluation
  10. Experiment
  11. Experimental Setup
  12. Main Results
  13. Conclusion
  14. Summarization of Existing Jailbreak Attacks and Defenses
  15. Prompt Template
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Overview of AutoAttack. The upper part illustrates the workflows of GA and adversarial generation frameworks, while the lower part demonstrates the dependency graph in dynamic attacks with DAG.
  • Figure 2: Overview of AutoDefense. The upper part of the figure analyzes the dependency relationship of two defense experts using DAG. The lower part of the figure is the specific workflow of our ensemble defense.