Mitigating Many-Shot Jailbreaking

TL;DR

This work interrogates many-shot jailbreaking (MSJ), a vulnerability arising from long context windows that enables jailbreaks via in-context demonstrations. It analyzes three mitigation paths—input sanitization, vector-based activation steering, and safety-focused fine-tuning—and finds that combining adversarial fine-tuning with input sanitization most effectively reduces MSJ risk while preserving in-context learning and conversational abilities. The authors provide a thorough evaluation framework combining NLL-based scaling, human judgments, and ability-preservation tests (OR-Bench, MT-Bench, and parity tasks). They show that the combined approach raises the baseline difficulty of jailbreaking and flattens the $NLL$-vs-shots slope, suggesting practical deployment guidance for safety post-training. Overall, the study argues for integrating this mitigation into model safety pipelines to improve robustness against MSJ in both closed and open-weight deployments.

Abstract

Many-shot jailbreaking (MSJ) is an adversarial technique that exploits the long context windows of modern LLMs to circumvent model safety training by including in the prompt many examples of a "fake" assistant responding inappropriately before the final request. With enough examples, the model's in-context learning abilities override its safety training, and it responds as if it were the "fake" assistant. In this work, we probe the effectiveness of different fine-tuning and input sanitization approaches on mitigating MSJ attacks, alone and in combination. We find incremental mitigation effectiveness for each, and show that the combined techniques significantly reduce the effectiveness of MSJ attacks, while retaining model performance in benign in-context learning and conversational tasks. We suggest that our approach could meaningfully ameliorate this vulnerability if incorporated into model safety post-training.

Figures (9)

• Figure 1: A: Illustration of the MSJ concept; four "shots" are shown in this example. B: Effectiveness of MSJ attacks from three different data sources on Llama-3.1-8b-Instruct, demonstrating that the model increases its probability of outputing the jailbroken response as the number of shots increases.
• Figure 2: Prompt formatting examples: user text is highlighted in blue; assistant in orange. The normal conversation on the left uses Llama3's standard tags, while in the MSJ attack on the right the attacker has replaced them with "(Assistant)" and "< h>" tags to circumvent input sanitization.
• Figure 3: Mitigation of MSJ attacks across four datasets. Y-axis is log of average per-token NLL of inappropriate response. "Sanit.": attacks that use random role tokens to circumvent input sanitization.
• Figure 4: Appropriateness of model generations to maximum-length MSJ attacks. "Sanit": attacks that use random role tokens to circumvent input sanitization.
• Figure 5: A: OR-Bench results showing preserved (or enhanced) abilities of the fine-tuned model. B: Preserved ICL abilities in the parity judgment task.