Harm Amplification in Text-to-Image Models

TL;DR

This work formalizes harm amplification in text-to-image (T2I) models, defining amplification via $H(i) > H(t) + \tau$ and focusing on sexually explicit and violent harms. It introduces a triad of methodologies—Distribution-Based Thresholds, Bucket Flip, and Image-Text Co-embedding—to quantify amplification across deployment contexts, demonstrated on the Nibbler/adversarial data with varying resource requirements. Empirical results show strong performance for sexually explicit amplification in several methods and reveal gender-related disparities, with explainability techniques (DAAM) and counterfactuals identifying input cues that drive harmful outputs. The paper offers scalable safety measurement tools for responsible T2I deployment, reduces dependence on manual annotation, and points to future work on training-data biases and mitigation strategies such as safe latent diffusion.

Abstract

Text-to-image (T2I) models have emerged as a significant advancement in generative AI; however, there exist safety concerns regarding their potential to produce harmful image outputs even when users input seemingly safe prompts. This phenomenon, where T2I models generate harmful representations that were not explicit in the input prompt, poses a potentially greater risk than adversarial prompts, leaving users unintentionally exposed to harms. Our paper addresses this issue by formalizing a definition for this phenomenon which we term harm amplification. We further contribute to the field by developing a framework of methodologies to quantify harm amplification in which we consider the harm of the model output in the context of user input. We then empirically examine how to apply these different methodologies to simulate real-world deployment scenarios including a quantification of disparate impacts across genders resulting from harm amplification. Together, our work aims to offer researchers tools to comprehensively address safety challenges in T2I systems and contribute to the responsible deployment of generative AI models.

Figures (7)

• Figure 1: Examples of harm amplifictation for images generated with Stable Diffusion 2.1 with the input prompt shown underneath each image for sexually explicit content (A) and violent content (B). A gaussian blur or black box was applied on some images to limit exposure of harms to readers.
• Figure 2: Difference in harm amplification rates across perceived genders in the Nibbler evaluation dataset. Images containing females were significantly more oversexualized than males whereas there was no significant difference in harm amplification of violence. $*p<0.05,\ **p<0.01,\ ***p<0.001$
• Figure 3: Attribution maps tang-etal-2023-daam for notable harm amplification prompts.
• Figure 4: Counterfactual analysis for the prompt "black gay man", altering the sexual orientation term (red) and the race term (blue).
• Figure 5: Method 1: Distribution-based thresholds applied on the measurements dataset. A) Machine annotations for sexually explicit harm scores were obtained for text and images. Text sexually explicit scores were bucketed into 5 buckets. Distributions of the image sexually explicit scores were then derived for images with corresponding text in that bucket. 2 standard deviations above the mean was used as an initial raw threshold where we then fitted a 1 degree polynomial to obtain new fitted thresholds (right most image). B) We then repeated the same procedure for machine annotations for violence.