T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model

TL;DR

T2I-RiskyPrompt introduces a hierarchical 6-by-14 risk taxonomy and a six-stage pipeline to curate 6,432 high-quality risky prompts with detailed risk reasons, addressing gaps in prior safety datasets. A reason-driven risky image detector aligns MLLMs with granular human annotations to achieve robust risk identification, showing 91.8% accuracy with a 3B model. The benchmark enables comprehensive evaluation across eight T2I models, nine defense methods, five safety filters, and five jailbreaking attacks, yielding nine insights into safety strengths and vulnerabilities. The work highlights persistent safety risks in high-capability T2I models and underscores the need for category-specific defenses and principled evaluation to advance safe generative AI. The dataset and code are available at https://github.com/datar001/T2I-RiskyPrompt, facilitating future research in risk-aware generation and mitigation.

Abstract

Using risky text prompts, such as pornography and violent prompts, to test the safety of text-to-image (T2I) models is a critical task. However, existing risky prompt datasets are limited in three key areas: 1) limited risky categories, 2) coarse-grained annotation, and 3) low effectiveness. To address these limitations, we introduce T2I-RiskyPrompt, a comprehensive benchmark designed for evaluating safety-related tasks in T2I models. Specifically, we first develop a hierarchical risk taxonomy, which consists of 6 primary categories and 14 fine-grained subcategories. Building upon this taxonomy, we construct a pipeline to collect and annotate risky prompts. Finally, we obtain 6,432 effective risky prompts, where each prompt is annotated with both hierarchical category labels and detailed risk reasons. Moreover, to facilitate the evaluation, we propose a reason-driven risky image detection method that explicitly aligns the MLLM with safety annotations. Based on T2I-RiskyPrompt, we conduct a comprehensive evaluation of eight T2I models, nine defense methods, five safety filters, and five attack strategies, offering nine key insights into the strengths and limitations of T2I model safety. Finally, we discuss potential applications of T2I-RiskyPrompt across various research fields. The dataset and code are provided in https://github.com/datar001/T2I-RiskyPrompt.

Figures (14)

• Figure 1: The hierarchical risk taxonomy of T2I-RiskyPrompt and six representative examples.
• Figure 1: The prompt template for GPT-4o to generate cartoon-themed prompts based on given images.
• Figure 2: T2I-RiskyPrompt includes a hierarchical risk taxonomy and a six-stage pipeline for data collection and annotation.
• Figure 2: The prompt template for GPT-4o to generate additional cartoon-themed prompts based on existing prompts.
• Figure 3: Evaluation of Jailbreaking attacks on SD1.4 with various safety filters. The black line indicates the baseline risk ratio without attack. Values closer to the center indicate lower risk ratio, while values near the outer edge indicate higher risk ratio. Category mapping is: C1-Explicit_Pornography, C2-Borderline_Pornography, C3-Weapons&Conflicts, C4-Bloody_Content, C5-Drug_Crimes, C6-Illegal_Trade, C7-Theft&Robbery, C8-Other_Illegal_Content, C9-LOGO, C10-Cartoon_Characters, C11-Terrifying_Content, C12-Other_Disturbing_Content, C13-Political_Figures, and C14-Political_Metaphor.