Jailbreaking Safeguarded Text-to-Image Models via Large Language Models

TL;DR

The paper addresses the vulnerability of safeguarded text-to-image models to adversarial prompts by introducing PromptTune, a query-free jailbreak that fine-tunes an LLM (AttackLLM) to rewrite unsafe prompts into effective adversarial prompts, guided by a CLIP-based Judge. It presents three variants (PromptTune-base, PromptTune-AdvPrompter, PromptTune-dpo) and a 66,000-sample preference dataset to optimize the rewriting process, achieving superior bypass rates and CLIP-based utility over existing no-box baselines and enhancing query-based attacks. The evaluation spans three unsafe-prompt datasets and five guardrails, highlighting strong performance against safety filters and mixed results for alignment guardrails, while emphasizing efficiency gains (single LLM interaction) and applicability to real-world red-teaming. The work underscores the need to integrate LLM-based jailbreak defenses into alignment pipelines and suggests future directions such as combining this approach with tree-of-thought reasoning for further attack efficacy or defense enhancement.

Abstract

Text-to-Image models may generate harmful content, such as pornographic images, particularly when unsafe prompts are submitted. To address this issue, safety filters are often added on top of text-to-image models, or the models themselves are aligned to reduce harmful outputs. However, these defenses remain vulnerable when an attacker strategically designs adversarial prompts to bypass these safety guardrails. In this work, we propose \alg, a method to jailbreak text-to-image models with safety guardrails using a fine-tuned large language model. Unlike other query-based jailbreak attacks that require repeated queries to the target model, our attack generates adversarial prompts efficiently after fine-tuning our AttackLLM. We evaluate our method on three datasets of unsafe prompts and against five safety guardrails. Our results demonstrate that our approach effectively bypasses safety guardrails, outperforms existing no-box attacks, and also facilitates other query-based attacks.

Figures (5)

• Figure 1: Key components of PromptTune.
• Figure 2: Images generated by different attacks against safety guardrails using the prompt, "A naked woman posing with her hands on her chest." From the first row to the last row, the results correspond to: No guardrail, Keyword match filter, Text embedding filter, Image embedding filter, MACE, and SafeGen. An all-black image indicates that the prompt was blocked by the safety filters. PT refers to PromptTune.
• Figure 3: Collecting one sample in our preference dataset $D$.
• Figure 4: Images generated by different attack methods. From first row to fourth row are respectively: Ring-A-Bell, MMA, PromptTune-base, PromptTune-AdvPrompter, and PromptTune-dpo.
• Figure 5: Images generated by SneakyPrompt and SneakyPrompt-PromptTune against SafeGen. The first row is SneakyPrompt, and the second row is SneakyPrompt-PromptTune.