Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos

TL;DR

This work tackles attribution of social bias in vision-language models under visual confounding by introducing a face-only counterfactual paradigm. It builds $FOCUS$, a real-photo dataset of 480 scene-matched face edits across six occupations and ten race–gender groups, and $REFLECT$, a three-task benchmark (2AFC, MCQ, Salary Recommendation) for decision-oriented bias evaluation under strict visual control. Across five state-of-the-art VLMs, the study finds that demographic disparities persist under counterfactual control and that bias magnitude and direction vary with task formulation and scenario, underscoring the need for controlled audits and careful task design in multimodal bias assessment. The framework enables attribution-focused audits while maintaining realism, providing a practical path toward safer deployment of VLMs in socially consequential settings.

Abstract

Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings, raising concerns about social bias driven by demographic cues. A central challenge in measuring such social bias is attribution under visual confounding: real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution. We propose a \textbf{face-only counterfactual evaluation paradigm} that isolates demographic effects while preserving real-image realism. Starting from real photographs, we generate counterfactual variants by editing only facial attributes related to race and gender, keeping all other visual factors fixed. Based on this paradigm, we construct \textbf{FOCUS}, a dataset of 480 scene-matched counterfactual images across six occupations and ten demographic groups, and propose \textbf{REFLECT}, a benchmark comprising three decision-oriented tasks: two-alternative forced choice, multiple-choice socioeconomic inference, and numeric salary recommendation. Experiments on five state-of-the-art VLMs reveal that demographic disparities persist under strict visual control and vary substantially across task formulations. These findings underscore the necessity of controlled, counterfactual audits and highlight task design as a critical factor in evaluating social bias in multimodal models.

Figures (18)

• Figure 1: FOCUS isolate facial demographic cues while keeping background, clothing, pose, and lighting fixed.
• Figure 2: Overview of REFLECT with FOCUS dataset construction. Starting from real photos, we generate scene-matched counterfactuals by editing only facial demographic cues while keeping all other context fixed. Using these controlled images, we evaluate VLMs with three decision-oriented tasks: (1) 2AFC, head-to-head comparisons between paired counterfactuals from the same source photo; (2) MCQ, single-image categorical judgments; and (3) Salary Recommendation, numeric salary outputs conditioned on a portrait and a standardized biography.
• Figure 3: FOCUS example from one source photo. Ten face-only counterfactual variants (5 races $\times$ 2 genders) generated from the same real source photo, illustrating the visual control used in REFLECT.
• Figure 4: 2AFC results for Gemini-2.5-Pro on FOCUS. (a--c) Pairwise win-rate matrices over the 10 race–gender groups for Income, Education, and Perceived Safety; each cell shows the fraction of retained comparisons in which the row group is selected over the column group. Groups are abbreviated by race (A/B/L/ME/W) $\times$ gender (M/F). (d--e) Race win rates within male (d) and female (e) variants, reported separately for each scenario. (f) Gender effect by race, measured as the male win rate in within-race male–female comparisons; the dashed line denotes 50% (no preference). Results for other models are reported in Appendix \ref{['app:2AFC_more_result']}.
• Figure 5: MCQ results on FOCUS. Mean-based percentage gaps $\Delta_g$ relative to reference groups (White for race; Female for gender).