Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Prompt Modifiers Control Bias? A Comparative Analysis of Text-to-Image Generative Models

Philip Wootaek Shin, Jihyun Janice Ahn, Wenpeng Yin, Jack Sampson, Vijaykrishnan Narayanan

TL;DR

The paper investigates whether prompt modifiers can control bias in text-to-image generation across Stable Diffusion, DALL·E 3, and Adobe Firefly. It analyzes base prompts, modifiers, and sequencing to reveal how biases related to gender, race, geography, and culture are encoded and how they can be (in)mitigated by prompt design. A bias sensitivity taxonomy and a distribution-shift metric are introduced to enable cross-model comparisons and standardize future bias analyses. The findings show that modifiers have uneven effects and that prompt order and model choice significantly influence outcomes, underscoring the need for more robust, ethical approaches to bias in AI-powered imagery.

Abstract

It has been shown that many generative models inherit and amplify societal biases. To date, there is no uniform/systematic agreed standard to control/adjust for these biases. This study examines the presence and manipulation of societal biases in leading text-to-image models: Stable Diffusion, DALL-E 3, and Adobe Firefly. Through a comprehensive analysis combining base prompts with modifiers and their sequencing, we uncover the nuanced ways these AI technologies encode biases across gender, race, geography, and region/culture. Our findings reveal the challenges and potential of prompt engineering in controlling biases, highlighting the critical need for ethical AI development promoting diversity and inclusivity. This work advances AI ethics by not only revealing the nuanced dynamics of bias in text-to-image generation models but also by offering a novel framework for future research in controlling bias. Our contributions-panning comparative analyses, the strategic use of prompt modifiers, the exploration of prompt sequencing effects, and the introduction of a bias sensitivity taxonomy-lay the groundwork for the development of common metrics and standard analyses for evaluating whether and how future AI models exhibit and respond to requests to adjust for inherent biases.

Can Prompt Modifiers Control Bias? A Comparative Analysis of Text-to-Image Generative Models

TL;DR

The paper investigates whether prompt modifiers can control bias in text-to-image generation across Stable Diffusion, DALL·E 3, and Adobe Firefly. It analyzes base prompts, modifiers, and sequencing to reveal how biases related to gender, race, geography, and culture are encoded and how they can be (in)mitigated by prompt design. A bias sensitivity taxonomy and a distribution-shift metric are introduced to enable cross-model comparisons and standardize future bias analyses. The findings show that modifiers have uneven effects and that prompt order and model choice significantly influence outcomes, underscoring the need for more robust, ethical approaches to bias in AI-powered imagery.

Abstract

It has been shown that many generative models inherit and amplify societal biases. To date, there is no uniform/systematic agreed standard to control/adjust for these biases. This study examines the presence and manipulation of societal biases in leading text-to-image models: Stable Diffusion, DALL-E 3, and Adobe Firefly. Through a comprehensive analysis combining base prompts with modifiers and their sequencing, we uncover the nuanced ways these AI technologies encode biases across gender, race, geography, and region/culture. Our findings reveal the challenges and potential of prompt engineering in controlling biases, highlighting the critical need for ethical AI development promoting diversity and inclusivity. This work advances AI ethics by not only revealing the nuanced dynamics of bias in text-to-image generation models but also by offering a novel framework for future research in controlling bias. Our contributions-panning comparative analyses, the strategic use of prompt modifiers, the exploration of prompt sequencing effects, and the introduction of a bias sensitivity taxonomy-lay the groundwork for the development of common metrics and standard analyses for evaluating whether and how future AI models exhibit and respond to requests to adjust for inherent biases.
Paper Structure (17 sections, 6 figures, 7 tables)

This paper contains 17 sections, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Biases in Text-to-Image Model
  4. Biases in Large Language Model
  5. Bias Evaluation
  6. Methodology
  7. Results
  8. Qualitative Characterization & Analysis
  9. Quantitative Analysis
  10. Discussion
  11. Limitations
  12. Opportunities and Insights
  13. Conclusion
  14. Comprehensive List
  15. Image Generated for Different Model
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Example of images in different model. Note that we tried to maintain the percentage of Asian presented by our prompt
  • Figure 2: Example of images Generated by Stable Diffusion(SD), DallE(DE), Firefly(FF) with prompt "Korean Soldier"
  • Figure 3: Example of images Generated by DallE with prompt "An Asian person living in Africa"
  • Figure :
  • Figure :
  • ...and 1 more figures