FlipAttack: Jailbreak LLMs via Flipping

TL;DR

FlipAttack addresses jailbreak risk in black-box LLMs by exploiting autoregressive left-to-right processing and disguising prompts with left-side noise across four flipping modes. It introduces a two-module pipeline—attack disguise and flipping guidance—that enables single-query jailbreak across 8 LLMs, achieving near-98% success on GPT-4Turbo/4o and high guardrail bypass rates. The study provides extensive empirical evidence, including ablations and analyses of why the method works, and demonstrates that simple defenses (SPD, perplexity filters) are ineffective. Together, these results highlight persistent vulnerabilities in safety-aligned LLMs and underscore the need for stronger defense and red-teaming strategies.

Abstract

This paper proposes a simple yet effective jailbreak attack named FlipAttack against black-box LLMs. First, from the autoregressive nature, we reveal that LLMs tend to understand the text from left to right and find that they struggle to comprehend the text when noise is added to the left side. Motivated by these insights, we propose to disguise the harmful prompt by constructing left-side noise merely based on the prompt itself, then generalize this idea to 4 flipping modes. Second, we verify the strong ability of LLMs to perform the text-flipping task, and then develop 4 variants to guide LLMs to denoise, understand, and execute harmful behaviors accurately. These designs keep FlipAttack universal, stealthy, and simple, allowing it to jailbreak black-box LLMs within only 1 query. Experiments on 8 LLMs demonstrate the superiority of FlipAttack. Remarkably, it achieves $\sim$98\% attack success rate on GPT-4o, and $\sim$98\% bypass rate against 5 guardrail models on average. The codes are available at GitHub\footnote{https://github.com/yueliu1999/FlipAttack}.

Figures (31)

• Figure 1: The attack success rate of our proposed FlipAttack, the runner-up black-box attack ReNeLLM, and the best white-box attack AutoDAN on 8 LLMs for 7 categories of harm contents.
• Figure 2: Overview of FlipAttack. First, the attack disguise module (upper part) disguises the harmful prompt by constructing left-side noise based on the prompt itself and generalizes it to four flipping modes. Then, based on four guidance units, the flipping guidance module (lower part) manipulates LLMs to denoise, understand, and execute the harmful behavior in the disguised prompt.
• Figure 3: Token cost & attack performance of 16 attack methods. A larger bubble indicates higher token costs. FlipAttack, ReNeLLM, AutoDAN denotes the best black-box attack, the runner-up black-box attack, and the best white-box attack in terms of attack performance, respectively.
• Figure 4: Ablation studies of flip modes on 8 LLMs. Variants are Flip Word Order (I), Flip Characters in Word (II), Flip Characters in Sentence (III), and Fool Model Mode (IV). The performance is tested based on Vanilla (A), and shaded regions show the performance improvement of adding CoT.
• Figure 5: Ablation studies of modules in FlipAttack on 8 LLMs. Variants are Vanilla (A), Vanilla+CoT (B), Vanilla+CoT+LangGPT (C), Vanilla+CoT+LangGPT+Few-shot (D).