Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Prompt Optimizer for Safe Text-to-Image Generation

Zongyu Wu, Hongcheng Gao, Yueze Wang, Xiang Zhang, Suhang Wang

TL;DR

This work tackles unsafe prompt-to-image generation by introducing POSI, a universal black-box prompt optimizer that rewrites toxic prompts into safe, semantically faithful versions. It builds a toxic-clean prompt dataset via GPT-3.5 Turbo, performs supervised fine-tuning on a language model, and further refines prompts with PPO using a reward that couples toxicity reduction and text alignment, without accessing T2I internals. Experimental results across Stable Diffusion variants show substantial reductions in unsafe outputs and competitive text fidelity, with good transferability to newer SD versions and other models like DALL-E 3 and Midjourney when combined with existing safety methods. The approach offers a practical, plug-in defense for real-world T2I systems, balancing safety with user-expressed intent and enabling broader deployment of safe T2I technologies.

Abstract

Text-to-Image (T2I) models have shown great performance in generating images based on textual prompts. However, these models are vulnerable to unsafe input to generate unsafe content like sexual, harassment and illegal-activity images. Existing studies based on image checker, model fine-tuning and embedding blocking are impractical in real-world applications. Hence, we propose the first universal prompt optimizer for safe T2I (POSI) generation in black-box scenario. We first construct a dataset consisting of toxic-clean prompt pairs by GPT-3.5 Turbo. To guide the optimizer to have the ability of converting toxic prompt to clean prompt while preserving semantic information, we design a novel reward function measuring toxicity and text alignment of generated images and train the optimizer through Proximal Policy Optimization. Experiments show that our approach can effectively reduce the likelihood of various T2I models in generating inappropriate images, with no significant impact on text alignment. It is also flexible to be combined with methods to achieve better performance. Our code is available at https://github.com/wu-zongyu/POSI.

Universal Prompt Optimizer for Safe Text-to-Image Generation

TL;DR

This work tackles unsafe prompt-to-image generation by introducing POSI, a universal black-box prompt optimizer that rewrites toxic prompts into safe, semantically faithful versions. It builds a toxic-clean prompt dataset via GPT-3.5 Turbo, performs supervised fine-tuning on a language model, and further refines prompts with PPO using a reward that couples toxicity reduction and text alignment, without accessing T2I internals. Experimental results across Stable Diffusion variants show substantial reductions in unsafe outputs and competitive text fidelity, with good transferability to newer SD versions and other models like DALL-E 3 and Midjourney when combined with existing safety methods. The approach offers a practical, plug-in defense for real-world T2I systems, balancing safety with user-expressed intent and enabling broader deployment of safe T2I technologies.

Abstract

Text-to-Image (T2I) models have shown great performance in generating images based on textual prompts. However, these models are vulnerable to unsafe input to generate unsafe content like sexual, harassment and illegal-activity images. Existing studies based on image checker, model fine-tuning and embedding blocking are impractical in real-world applications. Hence, we propose the first universal prompt optimizer for safe T2I (POSI) generation in black-box scenario. We first construct a dataset consisting of toxic-clean prompt pairs by GPT-3.5 Turbo. To guide the optimizer to have the ability of converting toxic prompt to clean prompt while preserving semantic information, we design a novel reward function measuring toxicity and text alignment of generated images and train the optimizer through Proximal Policy Optimization. Experiments show that our approach can effectively reduce the likelihood of various T2I models in generating inappropriate images, with no significant impact on text alignment. It is also flexible to be combined with methods to achieve better performance. Our code is available at https://github.com/wu-zongyu/POSI.
Paper Structure (27 sections, 5 equations, 6 figures, 8 tables)

This paper contains 27 sections, 5 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. T2I Generation.
  4. T2I DMs with Safety Mechanisms.
  5. Prompt Engineering.
  6. Proposed Framework
  7. Toxic-Clean Prompt Construction
  8. Supervised Fine-tuning
  9. Reward Score
  10. Proximal Policy Optimization
  11. Experiments
  12. Datasets
  13. Baseline Methods
  14. Settings
  15. Evaluation Metric
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Comparison between the original sample and the optimized sample. The original image is blurred manually for display purposes.
  • Figure 2: An overview of the proposed POSI. The first step is using GPT to preprocess toxic prompts to produce a dataset composed of toxic-clean dataset pairs. The second step is to do SFT on the language model based on the dataset produced in the first step. The third step is to employ PPO on the language model based on the designed reward to further enhance the model.
  • Figure 3: Text alignment of different methods on I2P for eval
  • Figure 4: Examples of various methods to prevent the generation of inappropriate content on SD v1.4. Some images were manually blurred after being generated for display purposes. The prompts and their corresponding seeds and guidance scales are in Appendix \ref{['cs1.4']}.
  • Figure 5: Case study on Midjourney. The upper half of the images were generated based on the original prompts. The lower half of the images were generated based on the corresponding modified prompts.
  • ...and 1 more figures