SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models

TL;DR

SafeGen is presented, a framework to mitigate sexual content generation by text-to-image models in a text-agnostic manner that outperforms eight state-of-the-art baseline methods and achieves 99.4% sexual content removal performance.

Abstract

Text-to-image (T2I) models, such as Stable Diffusion, have exhibited remarkable performance in generating high-quality images from text descriptions in recent years. However, text-to-image models may be tricked into generating not-safe-for-work (NSFW) content, particularly in sexually explicit scenarios. Existing countermeasures mostly focus on filtering inappropriate inputs and outputs, or suppressing improper text embeddings, which can block sexually explicit content (e.g., naked) but may still be vulnerable to adversarial prompts -- inputs that appear innocent but are ill-intended. In this paper, we present SafeGen, a framework to mitigate sexual content generation by text-to-image models in a text-agnostic manner. The key idea is to eliminate explicit visual representations from the model regardless of the text input. In this way, the text-to-image model is resistant to adversarial prompts since such unsafe visual representations are obstructed from within. Extensive experiments conducted on four datasets and large-scale user studies demonstrate SafeGen's effectiveness in mitigating sexually explicit content generation while preserving the high-fidelity of benign images. SafeGen outperforms eight state-of-the-art baseline methods and achieves 99.4% sexual content removal performance. Furthermore, our constructed benchmark of adversarial prompts provides a basis for future development and evaluation of anti-NSFW-generation methods.

Figures (17)

• Figure 1: Despite defending against the generation of sexually explicit images prompted by naive cues, prior methods can be bypassed or compromised by adversarial prompts. SafeGen eliminates explicit visual representations that inherently share high similarity within text-to-image (T2I) models, achieving text-agnostic mitigation against adversarial prompts since unsafe visual representations are removed from within.
• Figure 2: Inference workflow of text-to-image Stable Diffusion. The user input is converted into embeddings and projected through cross-attention layers in each denoising step.
• Figure 3: Diagram of a cross-attention layer (in the dashed box) in text-to-image models. Text-based attention matrices $\mathbf{W_K}$ and $\mathbf{W_V}$ transform each token's embedding into $\mathbf{K}$ and $\mathbf{V}$, respectively. Similarly, the matrix $\mathbf{W_Q}$ transforms visual latent to $\mathbf{Q}$.
• Figure 4: Diagram of self-attention. The query, key, and value $\mathbf{Q,K,V}$ vectors are all obtained by the learned attention matrices $\mathbf{W_Q,W_K,W_V}$ transforming the same visual latent.
• Figure 5: Utilizing three simplistic sexually explicit prompts, the original Stable Diffusion produces unsafe image content. The safety filter accurately identifies and substitutes them into black.