Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V. Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, Abhradeep Thakurta, Kai Yuanqing Xiao, Andreas Terzis, Florian Tramèr

TL;DR

This work argues that evaluating LLM defenses against jailbreaks and prompt injections requires strong adaptive attackers, not static prompts or weak optimizers. It introduces a general adaptive attack framework and demonstrates its effectiveness against 12 diverse defenses, including prompting, training-time, detector-based, and secret-knowledge defenses, often achieving attack success rates above 90%. The study emphasizes the critical role of human red-teaming and cautions against overreliance on static benchmarks or automated evaluators, showing that many defenses believed robust are easily bypassed. The authors advocate security-style rigor in robustness assessment and call for open, accessible evaluation tools to improve defense reliability in practice.

Abstract

How should we evaluate the robustness of language model defenses? Current defenses against jailbreaks and prompt injections (which aim to prevent an attacker from eliciting harmful knowledge or remotely triggering malicious actions, respectively) are typically evaluated either against a static set of harmful attack strings, or against computationally weak optimization methods that were not designed with the defense in mind. We argue that this evaluation process is flawed. Instead, we should evaluate defenses against adaptive attackers who explicitly modify their attack strategy to counter a defense's design while spending considerable resources to optimize their objective. By systematically tuning and scaling general optimization techniques-gradient descent, reinforcement learning, random search, and human-guided exploration-we bypass 12 recent defenses (based on a diverse set of techniques) with attack success rate above 90% for most; importantly, the majority of defenses originally reported near-zero attack success rates. We believe that future defense work must consider stronger attacks, such as the ones we describe, in order to make reliable and convincing claims of robustness.

The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

TL;DR

This work argues that evaluating LLM defenses against jailbreaks and prompt injections requires strong adaptive attackers, not static prompts or weak optimizers. It introduces a general adaptive attack framework and demonstrates its effectiveness against 12 diverse defenses, including prompting, training-time, detector-based, and secret-knowledge defenses, often achieving attack success rates above 90%. The study emphasizes the critical role of human red-teaming and cautions against overreliance on static benchmarks or automated evaluators, showing that many defenses believed robust are easily bypassed. The authors advocate security-style rigor in robustness assessment and call for open, accessible evaluation tools to improve defense reliability in practice.

Abstract

How should we evaluate the robustness of language model defenses? Current defenses against jailbreaks and prompt injections (which aim to prevent an attacker from eliciting harmful knowledge or remotely triggering malicious actions, respectively) are typically evaluated either against a static set of harmful attack strings, or against computationally weak optimization methods that were not designed with the defense in mind. We argue that this evaluation process is flawed. Instead, we should evaluate defenses against adaptive attackers who explicitly modify their attack strategy to counter a defense's design while spending considerable resources to optimize their objective. By systematically tuning and scaling general optimization techniques-gradient descent, reinforcement learning, random search, and human-guided exploration-we bypass 12 recent defenses (based on a diverse set of techniques) with attack success rate above 90% for most; importantly, the majority of defenses originally reported near-zero attack success rates. We believe that future defense work must consider stronger attacks, such as the ones we describe, in order to make reliable and convincing claims of robustness.

Paper Structure

This paper contains 47 sections, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. A Brief History of Adversarial ML Evaluations
  3. Problem Statement, Threat Model, and Attacker's Capabilities
  4. General Attack Methods
  5. Experiments
  6. Prompting Defenses
  7. Training Against Existing Attacks
  8. Filtering Model Defenses
  9. Secret–Knowledge Defenses
  10. Lesson and Discussion
  11. Conclusion
  12. General Adaptive Attack Framework
  13. Gradient-Based Methods
  14. Reinforcement Learning Methods
  15. Search-Based Methods
  16. ...and 32 more sections

Figures (6)

  • Figure 1: Attack success rate of our adaptive attacks compared to the weaker or static attacks considered in the original paper evaluation. None of the 12 defenses across four common techniques is robust to strong adaptive attacks. On the rightmost bars, human red-teaming succeeds on all of the scenarios while the static attack succeeds on none.
  • Figure 2: A diagram of our generalized adaptive attacks against LLMs.
  • Figure 3: Competitors are provided with a challenge interface in which they 1) receive a complete set of instructions for how to interact with the environment 2) can test prompts and watch the agent's actions/outputs in real time 3) can see the user task and attacker (injection) task 4) can see whether successfully completed the challenge. After clicking the Try Again button, an input textbox will be shown where the Chat currently is.
  • Figure 4: Score of the RL-based attack against Data Sentinel. We also include the trigger at the first step and the last step of the RL update.
  • Figure 5: Reward Hacking example
  • ...and 1 more figures