SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples

TL;DR

This work tackles intersectional biases in vision-language models by introducing SocialCounterfactuals, a scalable pipeline that uses cross-attention-controlled diffusion to generate counterfactual image-text pairs varying across race, gender, and physical attributes for occupations. A rigorous three-stage filtering process (CLIP similarity, NSFW ViT, and CLIP attribute detectability) yields 13,824 counterfactual sets (≈170,832 image-text pairs). Through comprehensive probing across six VLMs, the authors demonstrate significant intersectional skew in retrieval and show that debiasing with synthetic counterfactuals can meaningfully reduce bias while maintaining task performance. The dataset also supports cross-dataset validation and offers a path toward broader debiasing for multimodal systems, with public release of data and code. Overall, SocialCounterfactuals provides a scalable, principled resource to both diagnose and mitigate intersectional biases in vision-language understanding and retrieval tasks.

Abstract

While vision-language models (VLMs) have achieved remarkable performance improvements recently, there is growing evidence that these models also posses harmful biases with respect to social attributes such as gender and race. Prior studies have primarily focused on probing such bias attributes individually while ignoring biases associated with intersections between social attributes. This could be due to the difficulty of collecting an exhaustive set of image-text pairs for various combinations of social attributes. To address this challenge, we employ text-to-image diffusion models to produce counterfactual examples for probing intersectional social biases at scale. Our approach utilizes Stable Diffusion with cross attention control to produce sets of counterfactual image-text pairs that are highly similar in their depiction of a subject (e.g., a given occupation) while differing only in their depiction of intersectional social attributes (e.g., race & gender). Through our over-generate-then-filter methodology, we produce SocialCounterfactuals, a high-quality dataset containing 171k image-text pairs for probing intersectional biases related to gender, race, and physical characteristics. We conduct extensive experiments to demonstrate the usefulness of our generated dataset for probing and mitigating intersectional social biases in state-of-the-art VLMs.

Figures (16)

• Figure 1: Examples of our counterfactual image-text pairs for probing intersectional race-gender bias in VLMs for the "construction worker" occupation. See Section \ref{['app:additional-examples']} in Supplementary Material for additional examples.
• Figure 2: Overview of our methodology for generating SocialCounterfactuals.
• Figure 3: Distribution of $\textrm{MaxSkew}@K$ measured across occupations for (a) Race-Gender, (b) Physical Characteristics-Gender, and (c) Physical Characteristics-Race intersectional biases. Max (min) values are plotted as red (green) circles with corresponding occupation names
• Figure 4: Mean of (marginal) gender $\textrm{MaxSkew}@K$ measured across occupations for different races.
• Figure 5: Proportion of images retrieved $@k=12$ using neutral prompts for the 'Doctor' occupation.