AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts

TL;DR

The paper addresses the challenge of evaluating safety in text-to-image models by proposing AutoPrompT, a black-box red-teaming framework that uses LLM-driven adversarial suffixes to convert benign prompts into unsafe outputs. It combines an alternating optimization–finetuning loop with a dual-evasion strategy that uses a perplexity constraint and banned-token penalties to generate human-readable, filter-resistant prompts that transfer to unseen models. Key contributions include the formulation of a token-wise adversarial suffix optimization objective, the integration of a replay-buffer–based fine-tuning regime, and extensive experiments showing superior red-teaming performance and zero-shot transferability across multiple safe T2I models and online platforms. The approach enables scalable safety evaluations and highlights the need for robust, open safety benchmarks for rapid deployment of safer AIGC technologies.

Abstract

Despite rapid advancements in text-to-image (T2I) models, their safety mechanisms are vulnerable to adversarial prompts, which maliciously generate unsafe images. Current red-teaming methods for proactively assessing such vulnerabilities usually require white-box access to T2I models, and rely on inefficient per-prompt optimization, as well as inevitably generate semantically meaningless prompts easily blocked by filters. In this paper, we propose APT (AutoPrompT), a black-box framework that leverages large language models (LLMs) to automatically generate human-readable adversarial suffixes for benign prompts. We first introduce an alternating optimization-finetuning pipeline between adversarial suffix optimization and fine-tuning the LLM utilizing the optimized suffix. Furthermore, we integrates a dual-evasion strategy in optimization phase, enabling the bypass of both perplexity-based filter and blacklist word filter: (1) we constrain the LLM generating human-readable prompts through an auxiliary LLM perplexity scoring, which starkly contrasts with prior token-level gibberish, and (2) we also introduce banned-token penalties to suppress the explicit generation of banned-tokens in blacklist. Extensive experiments demonstrate the excellent red-teaming performance of our human-readable, filter-resistant adversarial prompts, as well as superior zero-shot transferability which enables instant adaptation to unseen prompts and exposes critical vulnerabilities even in commercial APIs (e.g., Leonardo.Ai.).

Figures (6)

• Figure 1: The overall framework of the proposed AutoPrompT(APT) method. We propose an alternating optimization-finetuning strategy to train a suffix generator. During the optimization phase, we employ a stochastic beam search algorithm to iteratively optimize adversarial suffixes token-by-token for given benign prompts, storing optimized suffixes in a replay buffer. We then sample high-priority suffixes from the replay buffer as finetuning targets for the suffix generator. Additionally, we introduce a dual-evasion strategy during optimization—combining perplexity constraints and banned-token penalties—to bypass both perplexity-based filter and blacklist word filter. During the inference phase, the trained suffix generator can automatically generate adversarial suffixes for unseen prompts.
• Figure 1: Visualizations against SLD-MAX on "shocking".
• Figure 2: Average blocking rate for each method across four safe T2I models
• Figure 2: More qualitative evaluation.
• Figure 3: Visualizations of different red-teaming methods. Previous methods generate undesirable blacklist words. Instead, our method generates human-readable adversarial prompts with lowest (best) PPL, inducing inappropriate contents across diverse safe T2I models.