BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models

TL;DR

BIGbench introduces a unified, four-dimensional bias benchmark for text-to-image models, enabling granular analysis of acquired and protected attributes, manifestation, and visibility. It builds a 47,040-prompt dataset and uses automated, multimodal-LLM evaluations complemented by human alignment checks across multiple models and debiasing methods. The study reveals that while bias in sex is relatively better mitigated, race and age biases persist, and distillation can amplify biases, underscoring complex trade-offs in debiasing. Overall, BIGbench provides a scalable framework for systematic bias evaluation, guiding debiasing research, dataset design, and responsible deployment of AIGC systems, with open-source resources for reproducibility.

Abstract

Text-to-Image (T2I) generative models are becoming increasingly crucial due to their ability to generate high-quality images, but also raise concerns about social biases, particularly in human image generation. Sociological research has established systematic classifications of bias. Yet, existing studies on bias in T2I models largely conflate different types of bias, impeding methodological progress. In this paper, we introduce BIGbench, a unified benchmark for Biases of Image Generation, featuring a carefully designed dataset. Unlike existing benchmarks, BIGbench classifies and evaluates biases across four dimensions to enable a more granular evaluation and deeper analysis. Furthermore, BIGbench applies advanced multi-modal large language models to achieve fully automated and highly accurate evaluations. We apply BIGbench to evaluate eight representative T2I models and three debiasing methods. Our human evaluation results by trained evaluators from different races underscore BIGbench's effectiveness in aligning images and identifying various biases. Moreover, our study also reveals new research directions about biases with insightful analysis of our results. Our work is openly accessible at https://github.com/BIGbench2024/BIGbench2024/.

Figures (9)

• Figure 1: The proportion distribution in BIGbench. The number of explicit prompts outnumbers implicit prompts by nearly 9:1 as nine set protected attributes.
• Figure 2: Overview of our multi-stage pipeline for evaluating T2I models on multi-dimensional social biases. Yellow box denotes generated images; purple box denotes the metadata from alignment; green box represents selected prompts for manifestation factors; orange box denotes attribute bias scores; red box represents the ground truth.
• Figure 3: Comparative analysis of implicit and explicit bias scores across eight T2I models. A) and C) show implicit bias; B) and D) show explicit bias. Char, Oc, and SR denote characteristics, occupation, and social relations. Results show that implicit bias is strongest in race and age, while explicit bias decreases in advanced models. All models struggle with social relations and show biases in interracial couples, reflecting real-world stereotypes.
• Figure 4: Visualized results of bias in prompt "one East Asian husband with one White wife".
• Figure 5: Implicit bias results of debiasing methods.