JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift

TL;DR

This paper tackles the problem of jailbreak attacks on safety-aligned LLMs under distribution shift by introducing JailbreaksOverTime, a timestamped benchmark spanning 10 months of real user interactions with both benign prompts and jailbreak attempts. It proposes a two-pronged defense: a Continuous Detector that uses self-training to adapt to drift without ongoing human labeling, and an Active Monitoring pipeline that unsupervisedly identifies novel, unseen jailbreaks by testing how templates interact with harmful payloads. Empirical results show the Continuous Detector achieving ~99.7% true positive rate at 0.1% false positive rate, matching a fully supervised model, while Active Monitoring more effectively detects unseen attacks and can surface new threats for human review. Together, these approaches form a resilient framework for maintaining jailbreak safety in real-world AI deployments, with dataset, methods, and code available to researchers and practitioners.

Abstract

Safety and security remain critical concerns in AI deployment. Despite safety training through reinforcement learning with human feedback (RLHF) [ 32], language models remain vulnerable to jailbreak attacks that bypass safety guardrails. Universal jailbreaks - prefixes that can circumvent alignment for any payload - are particularly concerning. We show empirically that jailbreak detection systems face distribution shift, with detectors trained at one point in time performing poorly against newer exploits. To study this problem, we release JailbreaksOverTime, a comprehensive dataset of timestamped real user interactions containing both benign requests and jailbreak attempts collected over 10 months. We propose a two-pronged method for defenders to detect new jailbreaks and continuously update their detectors. First, we show how to use continuous learning to detect jailbreaks and adapt rapidly to new emerging jailbreaks. While detectors trained at a single point in time eventually fail due to drift, we find that universal jailbreaks evolve slowly enough for self-training to be effective. Retraining our detection model weekly using its own labels - with no new human labels - reduces the false negative rate from 4% to 0.3% at a false positive rate of 0.1%. Second, we introduce an unsupervised active monitoring approach to identify novel jailbreaks. Rather than classifying inputs directly, we recognize jailbreaks by their behavior, specifically, their ability to trigger models to respond to known-harmful prompts. This approach has a higher false negative rate (4.1%) than supervised methods, but it successfully identified some out-of-distribution attacks that were missed by the continuous learning approach.

Figures (10)

• Figure 1: Our proposed two-pronged jailbreak detection: (1) Continuous Detector for in-distribution samples that adapt slowly under distribution shift over time, and (2) Active Monitoring, an unsupervised method for detecting out-of-distribution unseen jailbreaks.
• Figure 2: False negative rate and false positive rate of a fixed detector, trained on the first month of data, evaluated on the rest of JailbreaksOverTime. We observe an increase in false negatives over time. Metrics are averaged over 28-day windows.
• Figure 3: Proportion of daily sampled jailbreaks which correspond to five representative prompts from JailbreakChat. We smooth proportions over a 14-day interval for visualization. Over time, different jailbreaks become more popular and grow to dominate the daily samples.
• Figure 4: JailbreakHub data is poorly suited for training or evaluating detectors. Out of 2,674 de-duplicated prompts, we find 866 are standalone jailbreak prompts and 72 datapoints are jailbreaks with a payloads.
• Figure 5: Confidence scores of the binary classifier over time showing drift in jailbreak patterns. We compute the worst confidence score each day (lowest among all jailbreaks, highest among all benign samples), then average these daily values over a 28-day window.