Jailbreaking in the Haystack

TL;DR

This work addresses safety risks arising from expanding context windows in transformer-based LLMs by introducing the NINJA jailbreak, which embeds a harmful goal inside a long, benign, thematically related context. The key insight is that goal positioning within the context—especially placing the dangerous instruction at the beginning—can dramatically increase attack success while the context remains unobtrusive, revealing a structural vulnerability in long-context processing. The authors formulate a compute-aware scaling law, showing that under a fixed compute budget, longer contexts can be more effective than increasing attack attempts, and demonstrate this across multiple models (e.g., LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-v0.3, Gemini 2.0 Flash) with ASR improvements up to near 60%. The findings imply urgent need for defenses that consider context structure and positional biases, not just content filtering, to safeguard future long-context and agentic systems.

Abstract

Recent advances in long-context language models (LMs) have enabled million-token inputs, expanding their capabilities across complex tasks like computer-use agents. Yet, the safety implications of these extended contexts remain unclear. To bridge this gap, we introduce NINJA (short for Needle-in-haystack jailbreak attack), a method that jailbreaks aligned LMs by appending benign, model-generated content to harmful user goals. Critical to our method is the observation that the position of harmful goals play an important role in safety. Experiments on standard safety benchmark, HarmBench, show that NINJA significantly increases attack success rates across state-of-the-art open and proprietary models, including LLaMA, Qwen, Mistral, and Gemini. Unlike prior jailbreaking methods, our approach is low-resource, transferable, and less detectable. Moreover, we show that NINJA is compute-optimal -- under a fixed compute budget, increasing context length can outperform increasing the number of trials in best-of-N jailbreak. These findings reveal that even benign long contexts -- when crafted with careful goal positioning -- introduce fundamental vulnerabilities in modern LMs.

Figures (6)

• Figure 1: An overview of the Ninja attack, (A) In the standard baseline case, a direct harmful goal with no added context is correctly identified and refused by the target model. (B) In the successful Ninja attack, the same goal is placed at the beginning of a long, benign 'haystack,' causing the model to bypass its safety filters. (C) This highlights the core principle of goal positioning that enables the attack: effectiveness is maximized when the harmful goal is at the start of the context window.
• Figure 2: Attack Success Rate (ASR) of the Ninja attack The y-axis shows the Attack Success Rate (ASR), our primary metric for a successful jailbreak, where the model provides an actionable response to the harmful goal. This is a stricter metric than the Non-Refusal Rate (see Section \ref{['sec:metric']}), which only measures the absence of a direct safety refusal.
• Figure 3: Goal positioning reveals capability-safety trade-offs in long-context jailbreaks. Left: Llama-3.1 shows monotonic decrease in both ASR and non-refusal rate as goal moves from beginning to end. Right: Qwen2.5 exhibits a "needle-in-haystack" effect with lowest performance when goal is in the middle (0.25-0.5), where capability limitations inadvertently provide safety benefits. For both models, placing the goal at the beginning maximizes ASR while maintaining high acceptance rates.
• Figure 4: Effect of goal positioning on ASR in BrowserART (OpenDevin GPT-4o Agent) Across direct prompting, GCG, human-written, and prefix-based attacks, placing the goal at the beginning consistently yields higher ASR than placing it at the end.
• Figure 5: The effect of context relevance on the Ninja attack's success.(a) Attack Success Rate (ASR) comparison between thematically relevant and irrelevant contexts across different context lengths. (b) Example showing how relevant context shares keywords with the harmful goal, while irrelevant context has no semantic connection.