Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models

Kai Hu, Abhinav Aggarwal, Mehran Khodabandeh, David Zhang, Eric Hsin, Li Chen, Ankit Jain, Matt Fredrikson, Akash Bharadwaj

TL;DR

Jailbreak-Zero introduces a policy-based red-teaming framework that shifts safety evaluation of large language models from fixed unsafe examples to abstract safety policies, enabling Pareto-optimal trade-offs among coverage, diversity, and fidelity. It combines a zero-shot attack LLM with a fine-tuning stage using preference data (DPO) to produce human-readable, effective adversarial prompts while maintaining scalability. The method achieves superior attack success rates against both open-source and proprietary models, including GPT-4o and Claude 3.5, and maintains effectiveness under iterative safety alignment. By providing controllable mechanisms to balance risk discovery with realism, Jailbreak-Zero offers a practical, scalable tool for identifying and mitigating safety vulnerabilities in evolving LLM deployments.

Abstract

This paper introduces Jailbreak-Zero, a novel red teaming methodology that shifts the paradigm of Large Language Model (LLM) safety evaluation from a constrained example-based approach to a more expansive and effective policy-based framework. By leveraging an attack LLM to generate a high volume of diverse adversarial prompts and then fine-tuning this attack model with a preference dataset, Jailbreak-Zero achieves Pareto optimality across the crucial objectives of policy coverage, attack strategy diversity, and prompt fidelity to real user inputs. The empirical evidence demonstrates the superiority of this method, showcasing significantly higher attack success rates against both open-source and proprietary models like GPT-40 and Claude 3.5 when compared to existing state-of-the-art techniques. Crucially, Jailbreak-Zero accomplishes this while producing human-readable and effective adversarial prompts with minimal need for human intervention, thereby presenting a more scalable and comprehensive solution for identifying and mitigating the safety vulnerabilities of LLMs.

Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models

TL;DR

Jailbreak-Zero introduces a policy-based red-teaming framework that shifts safety evaluation of large language models from fixed unsafe examples to abstract safety policies, enabling Pareto-optimal trade-offs among coverage, diversity, and fidelity. It combines a zero-shot attack LLM with a fine-tuning stage using preference data (DPO) to produce human-readable, effective adversarial prompts while maintaining scalability. The method achieves superior attack success rates against both open-source and proprietary models, including GPT-4o and Claude 3.5, and maintains effectiveness under iterative safety alignment. By providing controllable mechanisms to balance risk discovery with realism, Jailbreak-Zero offers a practical, scalable tool for identifying and mitigating safety vulnerabilities in evolving LLM deployments.

Abstract

This paper introduces Jailbreak-Zero, a novel red teaming methodology that shifts the paradigm of Large Language Model (LLM) safety evaluation from a constrained example-based approach to a more expansive and effective policy-based framework. By leveraging an attack LLM to generate a high volume of diverse adversarial prompts and then fine-tuning this attack model with a preference dataset, Jailbreak-Zero achieves Pareto optimality across the crucial objectives of policy coverage, attack strategy diversity, and prompt fidelity to real user inputs. The empirical evidence demonstrates the superiority of this method, showcasing significantly higher attack success rates against both open-source and proprietary models like GPT-40 and Claude 3.5 when compared to existing state-of-the-art techniques. Crucially, Jailbreak-Zero accomplishes this while producing human-readable and effective adversarial prompts with minimal need for human intervention, thereby presenting a more scalable and comprehensive solution for identifying and mitigating the safety vulnerabilities of LLMs.
Paper Structure (31 sections, 8 equations, 5 figures, 24 tables)

This paper contains 31 sections, 8 equations, 5 figures, 24 tables.

Table of Contents

  1. Introduction
  2. Policy-Based Red Teaming with Pareto Optimality
  3. Example-Based Evaluation
  4. Policy-Based Evaluation
  5. 1. Coverage
  6. 2: Diversity
  7. 3. Fidelity
  8. Red Teaming with Controllable Pareto Optimality
  9. Jailbreak-Zero: Simple Red Teaming Method with No Human Strategies
  10. Jailbreak-Zero: The Zero-shot Variant
  11. Jailbreak-Zero: The Fine-tuning Variant
  12. Experiments
  13. Performance of Jailbreak-Zero for Example-based Evaluation
  14. Performance of Jailbreak-Zero for Policy-based Evaluation
  15. Zero-Shot Variant Performance
  16. ...and 16 more sections

Figures (5)

  • Figure 1: The pipeline of our base method for Jailbreak-Zero
  • Figure 2: ASR performance after fine-tuning the attack LLM with varying DPO dataset sizes. Top row: ASR on 4 training policies (out of 9 total; 5 additional results in Appendix). Bottom row: ASR on 4 novel policies unseen during fine-tuning.
  • Figure 3: The zero-short ASR performance for 9 covered policies (ASR $> 10\%$ for Gemma 3 (27B)). We will use these policies for the fine-tuning method.
  • Figure 4: The zero-short ASR performance for 5 uncovered policies (ASR < 10% for Gemma 3 (27B)). We use these policies to simulate the performance for novel policies for the fine-tuning method.
  • Figure 5: Ablation Study on the Clustering Hyper-parameter