Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Safe Synthetic Image Generation On the Web: A Multimodal Robust NSFW Defense and Million Scale Dataset

Muhammad Shahid Muneer, Simon S. Woo

TL;DR

The paper tackles the safety challenges of web-based image generation by addressing weaknesses in unimodal NSFW defenses against multimodal adversarial attacks. It introduces NSFWCorpus, a million-scale dataset of text–image pairs including safe, NSFW, and adversarial variants, and a multimodal defense that uses cross-attention between text and image modalities to detect NSFW content. Experimental results demonstrate reduced attack success rates and improved detection metrics compared with state-of-the-art unimodal defenses, highlighting the practical value of context-aware multimodal moderation. The work provides a scalable baseline for safer web content generation and points to extensions toward multilingual data and additional modalities like video and audio.

Abstract

In the past years, we have witnessed the remarkable success of Text-to-Image (T2I) models and their widespread use on the web. Extensive research in making T2I models produce hyper-realistic images has led to new concerns, such as generating Not-Safe-For-Work (NSFW) web content and polluting the web society. To help prevent misuse of T2I models and create a safer web environment for users features like NSFW filters and post-hoc security checks are used in these models. However, recent work unveiled how these methods can easily fail to prevent misuse. In particular, adversarial attacks on text and image modalities can easily outplay defensive measures. %Exploiting such leads to the growing concern of preventing adversarial attacks on text and image modalities. Moreover, there is currently no robust multimodal NSFW dataset that includes both prompt and image pairs and adversarial examples. This work proposes a million-scale prompt and image dataset generated using open-source diffusion models. Second, we develop a multimodal defense to distinguish safe and NSFW text and images, which is robust against adversarial attacks and directly alleviates current challenges. Our extensive experiments show that our model performs well against existing SOTA NSFW detection methods in terms of accuracy and recall, drastically reducing the Attack Success Rate (ASR) in multimodal adversarial attack scenarios. Code: https://github.com/shahidmuneer/multimodal-nsfw-defense.

Towards Safe Synthetic Image Generation On the Web: A Multimodal Robust NSFW Defense and Million Scale Dataset

TL;DR

The paper tackles the safety challenges of web-based image generation by addressing weaknesses in unimodal NSFW defenses against multimodal adversarial attacks. It introduces NSFWCorpus, a million-scale dataset of text–image pairs including safe, NSFW, and adversarial variants, and a multimodal defense that uses cross-attention between text and image modalities to detect NSFW content. Experimental results demonstrate reduced attack success rates and improved detection metrics compared with state-of-the-art unimodal defenses, highlighting the practical value of context-aware multimodal moderation. The work provides a scalable baseline for safer web content generation and points to extensions toward multilingual data and additional modalities like video and audio.

Abstract

In the past years, we have witnessed the remarkable success of Text-to-Image (T2I) models and their widespread use on the web. Extensive research in making T2I models produce hyper-realistic images has led to new concerns, such as generating Not-Safe-For-Work (NSFW) web content and polluting the web society. To help prevent misuse of T2I models and create a safer web environment for users features like NSFW filters and post-hoc security checks are used in these models. However, recent work unveiled how these methods can easily fail to prevent misuse. In particular, adversarial attacks on text and image modalities can easily outplay defensive measures. %Exploiting such leads to the growing concern of preventing adversarial attacks on text and image modalities. Moreover, there is currently no robust multimodal NSFW dataset that includes both prompt and image pairs and adversarial examples. This work proposes a million-scale prompt and image dataset generated using open-source diffusion models. Second, we develop a multimodal defense to distinguish safe and NSFW text and images, which is robust against adversarial attacks and directly alleviates current challenges. Our extensive experiments show that our model performs well against existing SOTA NSFW detection methods in terms of accuracy and recall, drastically reducing the Attack Success Rate (ASR) in multimodal adversarial attack scenarios. Code: https://github.com/shahidmuneer/multimodal-nsfw-defense.

Paper Structure

This paper contains 10 sections, 11 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Dataset Generation Pipeline
  4. Proposed Multimodal Model
  5. Experiments
  6. Experimental Settings
  7. Benchmark Performance
  8. Ethical Consideration & Social Impact
  9. Limitation & Future Work
  10. Conclusion

Figures (4)

  • Figure 1: a) Existing prompt filters and post-hoc security filters are employed in AI Image generation models, while b) and c) are our proposed implementations for context-dependent multimodal defense in T2I and I2I models respectively.
  • Figure 2: Overall dataset generation pipeline.
  • Figure 3: Illustration of a high-level diagram of our proposed framework to classify safe and NSFW content.
  • Figure 4: Illustration of images depicting racism, nudity, violence, and gender discrimination context included in the generated dataset.