Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs

Xueluan Gong, Mingzhe Li, Yilin Zhang, Fengyuan Ran, Chen Chen, Yanjiao Chen, Qian Wang, Kwok-Yan Lam

TL;DR

PAPILLON introduces a black-box fuzz-testing jailbreaking framework that starts with an empty seed pool and uses LLM-powered mutations plus a two-level judge to automatically generate concise, coherent jailbreak prompts. It demonstrates superior attack effectiveness across seven LLMs (including GPT-3.5 Turbo, GPT-4, and Gemini-Pro) with high ASR and low query costs, while showing transferability and resilience to state-of-the-art defenses. The work also provides human-in-the-loop validation and ethical considerations, outlining defense implications and future directions for strengthening safety and alignment in LLM deployments.

Abstract

Large Language Models (LLMs) have excelled in various tasks but are still vulnerable to jailbreaking attacks, where attackers create jailbreak prompts to mislead the model to produce harmful or offensive content. Current jailbreak methods either rely heavily on manually crafted templates, which pose challenges in scalability and adaptability, or struggle to generate semantically coherent prompts, making them easy to detect. Additionally, most existing approaches involve lengthy prompts, leading to higher query costs. In this paper, to remedy these challenges, we introduce a novel jailbreaking attack framework called PAPILLON, which is an automated, black-box jailbreaking attack framework that adapts the black-box fuzz testing approach with a series of customized designs. Instead of relying on manually crafted templates,PAPILLON starts with an empty seed pool, removing the need to search for any related jailbreaking templates. We also develop three novel question-dependent mutation strategies using an LLM helper to generate prompts that maintain semantic coherence while significantly reducing their length. Additionally, we implement a two-level judge module to accurately detect genuine successful jailbreaks. We evaluated PAPILLON on 7 representative LLMs and compared it with 5 state-of-the-art jailbreaking attack strategies. For proprietary LLM APIs, such as GPT-3.5 turbo, GPT-4, and Gemini-Pro, PAPILLONs achieves attack success rates of over 90%, 80%, and 74%, respectively, exceeding existing baselines by more than 60\%. Additionally, PAPILLON can maintain high semantic coherence while significantly reducing the length of jailbreak prompts. When targeting GPT-4, PAPILLON can achieve over 78% attack success rate even with 100 tokens. Moreover, PAPILLON demonstrates transferability and is robust to state-of-the-art defenses. Code: https://github.com/aaFrostnova/Papillon

PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs

TL;DR

PAPILLON introduces a black-box fuzz-testing jailbreaking framework that starts with an empty seed pool and uses LLM-powered mutations plus a two-level judge to automatically generate concise, coherent jailbreak prompts. It demonstrates superior attack effectiveness across seven LLMs (including GPT-3.5 Turbo, GPT-4, and Gemini-Pro) with high ASR and low query costs, while showing transferability and resilience to state-of-the-art defenses. The work also provides human-in-the-loop validation and ethical considerations, outlining defense implications and future directions for strengthening safety and alignment in LLM deployments.

Abstract

Large Language Models (LLMs) have excelled in various tasks but are still vulnerable to jailbreaking attacks, where attackers create jailbreak prompts to mislead the model to produce harmful or offensive content. Current jailbreak methods either rely heavily on manually crafted templates, which pose challenges in scalability and adaptability, or struggle to generate semantically coherent prompts, making them easy to detect. Additionally, most existing approaches involve lengthy prompts, leading to higher query costs. In this paper, to remedy these challenges, we introduce a novel jailbreaking attack framework called PAPILLON, which is an automated, black-box jailbreaking attack framework that adapts the black-box fuzz testing approach with a series of customized designs. Instead of relying on manually crafted templates,PAPILLON starts with an empty seed pool, removing the need to search for any related jailbreaking templates. We also develop three novel question-dependent mutation strategies using an LLM helper to generate prompts that maintain semantic coherence while significantly reducing their length. Additionally, we implement a two-level judge module to accurately detect genuine successful jailbreaks. We evaluated PAPILLON on 7 representative LLMs and compared it with 5 state-of-the-art jailbreaking attack strategies. For proprietary LLM APIs, such as GPT-3.5 turbo, GPT-4, and Gemini-Pro, PAPILLONs achieves attack success rates of over 90%, 80%, and 74%, respectively, exceeding existing baselines by more than 60\%. Additionally, PAPILLON can maintain high semantic coherence while significantly reducing the length of jailbreak prompts. When targeting GPT-4, PAPILLON can achieve over 78% attack success rate even with 100 tokens. Moreover, PAPILLON demonstrates transferability and is robust to state-of-the-art defenses. Code: https://github.com/aaFrostnova/Papillon
Paper Structure (26 sections, 8 equations, 2 figures, 17 tables, 1 algorithm)

This paper contains 26 sections, 8 equations, 2 figures, 17 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Model
  4. Jailbreaking Attacks
  5. Jailbreaking Defenses
  6. Fuzz Testing
  7. Threat Model
  8. Methodology of Papillon
  9. Framework of Papillon
  10. Template Mutation and Optimization
  11. Two-level Judge Module
  12. Experiments
  13. Experimental Setup
  14. Comparison with Baselines
  15. Ablation Studies
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Examples of the jailbreaking attacks. The texts are derived from our experimental results.
  • Figure 2: Workflow of Papillon. Papillon features a fuzz testing-enabled jailbreaking attack with two main phases: template mutation/optimization and a two-level judge module. Papillon begins with an empty seed pool and operates in two phases: pre-jailbreak and final-jailbreak. In the pre-jailbreak phase, Role-play and Contextualization mutations are used to generate initial jailbreak attempts and populate the empty seed pool. The final-jailbreak phase builds on this by adding Expand mutation to the process. A two-level judge module evaluates LLM responses to identify successful jailbreaks.