Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Boundaries of Content Moderation in Text-to-Image Generation

Piera Riccio, Georgina Curto, Nuria Oliver

Abstract

This paper analyzes the community safety guidelines of five text-to-image (T2I) generation platforms and audits five T2I models, focusing on prompts related to the representation of humans in areas that might lead to societal stigma. While current research primarily focuses on ensuring safety by restricting the generation of harmful content, our study offers a complementary perspective. We argue that the concept of safety is difficult to define and operationalize, reflected in a discrepancy between the officially published safety guidelines and the actual behavior of the T2I models, and leading at times to over-censorship. Our findings call for more transparency and an inclusive dialogue about the platforms' content moderation practices, bearing in mind their global cultural and social impact.

Exploring the Boundaries of Content Moderation in Text-to-Image Generation

Abstract

This paper analyzes the community safety guidelines of five text-to-image (T2I) generation platforms and audits five T2I models, focusing on prompts related to the representation of humans in areas that might lead to societal stigma. While current research primarily focuses on ensuring safety by restricting the generation of harmful content, our study offers a complementary perspective. We argue that the concept of safety is difficult to define and operationalize, reflected in a discrepancy between the officially published safety guidelines and the actual behavior of the T2I models, and leading at times to over-censorship. Our findings call for more transparency and an inclusive dialogue about the platforms' content moderation practices, bearing in mind their global cultural and social impact.
Paper Structure (17 sections, 2 figures, 4 tables)

This paper contains 17 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Safety Guidelines in T2I Systems
  4. Auditing
  5. 1. Physical Appearance and Personal Traits
  6. 2. Health
  7. 3. Reproduction, Women's Health and Romantic Relationships
  8. 4. Legal and Illegal Activities
  9. 5. Politics and ideologies
  10. 6. Artistic Nudity
  11. Discussion
  12. Opacity of Content Moderation in T2I platforms
  13. Image generation is different from Web search
  14. Social and cultural consequences of prompt and content moderation
  15. Artistic nudity as a special case
  16. ...and 2 more sections

Figures (2)

  • Figure 1: (a) Histogram of the types of prompt and content moderation experienced in the auditing process; (b) Percentage of moderated/censored prompts per T2I model.
  • Figure 2: From left to right: A revisitation of Botticelli's The Birth of Venus by Stable Image Ulta (SIU), A revisitation of Titian’s Venus of Urbino, by Stable Image Core (SIC), A revisitation of Botticelli's The Birth of Venus, by Stable Diffusion 3 (SD3), and A revisitation of Michelangelo's David, by Midjourney (MJ).