In-Context Experience Replay Facilitates Safety Red-Teaming of Text-to-Image Diffusion Models

TL;DR

This work tackles the challenge of evaluating safety mechanisms in deployed text-to-image diffusion models by proposing ICER, an in-context experience replay framework. ICER uses a large language model as a surrogate to generate jailbreaking prompts, guided by a bandit-based exemplar selection drawn from a growing playbook of past red-teaming successes, enabling efficient black-box probing without internal model access. Across nudity and violence prompts and multiple safety mechanisms, ICER outperforms existing prompt-attacks and variant strategies, producing fluent, semantically aligned prompts that reveal vulnerabilities. The study also uncovers that knowledge transfer from historical jailbreaks facilitates the discovery of new weaknesses and discusses implications for defense and the risk of misuse when attackers leverage this transferability.

Abstract

Text-to-image (T2I) models have shown remarkable progress, but their potential to generate harmful content remains a critical concern in the ML community. While various safety mechanisms have been developed, the field lacks systematic tools for evaluating their effectiveness against real-world misuse scenarios. In this work, we propose ICER, a novel red-teaming framework that leverages Large Language Models (LLMs) and a bandit optimization-based algorithm to generate interpretable and semantic meaningful problematic prompts by learning from past successful red-teaming attempts. Our ICER efficiently probes safety mechanisms across different T2I models without requiring internal access or additional training, making it broadly applicable to deployed systems. Through extensive experiments, we demonstrate that ICER significantly outperforms existing prompt attack methods in identifying model vulnerabilities while maintaining high semantic similarity with intended content. By uncovering that successful jailbreaking instances can systematically facilitate the discovery of new vulnerabilities, our work provides crucial insights for developing more robust safety mechanisms in T2I systems.

Figures (11)

• Figure 2: An overview of our ICER framework. Our framework leverages past experiences to guide future red-teaming attempts through three interconnected components: (1) Surrogate Model: an LLM-based module that generates interpretable adversarial prompts by utilizing system instructions and in-context exemplars sampled from prior successful attempts; (2) Prior Sampling: a Thomson Sampling-based strategy that maintains and samples from a database of past experiences, balancing exploration and exploitation; and (3) Evaluation: a two-stage assessment process that validates semantic consistency with the original intent and measures red-teaming effectiveness. Posterior are stored back in the prior database, enabling continuous learning and adaptation.
• Figure 3: Qualitative comparison of jailbreaking prompts from different red-teaming methods and their generated images across safe T2I models. Original I2P prompts and their generated "safe" images are shown in the first column. Ours (TS) refers to our Thompson Sampling setting. The n-gram perplexity scores ($\times 10^3$) are provided as ppl, where lower values suggest more fluent prompts.
• Figure 4: FR under different textual similarity constraints, showing the achieved FR (y-axis) as the cosine similarity threshold between input prompt and jailbreaking prompt pairs (x-axis) decreases.
• Figure 5: $n$-shot attack ablation results
• Figure 6: Effect of varying the number of exemplars $k$ on ICER's FR across different safe T2I models.