Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models

TL;DR

The paper tackles social bias in contrastive vision–language models by independently varying encoder size, pretraining data scale, and data source while keeping the objective fixed. Through a controlled CLIP/OpenCLIP comparison and bias probes on FairFace and PATA, it shows that data source often dominates bias patterns, with scale effects diverging by model family. It also evaluates three test-time debiasing methods, finding that their effectiveness is model- and data-dependent and that debiasing cannot fully eradicate harm. The work provides a public auditing framework, opensource evaluation tools, and argues for data-centric, scale-stable approaches to building fair open-vocabulary multimodal systems. Overall, it emphasizes that fairness is not an automatic outcome of scaling and that corpus-aware curation and calibration are essential for robust bias mitigation in VLMs.

Abstract

Vision-language models (VLMs) deliver strong zero-shot recognition but frequently inherit social biases from their training data. We systematically disentangle three design factors -- model size, training-data scale, and training-data source -- by comparing CLIP and OpenCLIP, two models that share an identical contrastive objective yet differ in encoder width and in the image-text corpora on which they are pre-trained (400M proprietary pairs vs. 400M/2B LAION). Across balanced face-analysis benchmarks, enlarging the encoder reduces gender skew in CLIP but amplifies both gender and racial skew in OpenCLIP; increasing the LAION corpus from 400M to 2B further increases OpenCLIP bias. At matched model and data budgets, substituting proprietary data with LAION improves gender fairness while increasing racial skew, underscoring data source as the primary driver of bias patterns. We also evaluate three post-hoc, test-time debiasing strategies -- Bias Prompts, Prompt Array, and SANER. Debiasing reduces but does not eliminate harm, and its effectiveness is source- and size-dependent: Bias Prompts most effectively reduce gender skew in CLIP at smaller model sizes, whereas Prompt Array and SANER more reliably reduce racial skew in OpenCLIP; scaling LAION reconfigures which method is most fair. Taken together, these findings challenge the assumption that bigger models or datasets are automatically fairer and foreground training data source as the key determinant of both bias and mitigation efficacy. We release code and evaluation scripts to enable transparent, reproducible auditing of future VLMs.

Figures (10)

• Figure 1: Effect of scaling the encoder from ViT-B/32 to ViT-L/14 at a fixed 400M-image corpus. Bars show the change in Max Skew (higher = more bias) for gender (gold) and race (orange). Solid bars use FairFace; hatched bars use PATA. Negative values denote mitigation. Enlarging the backbone reduces gender skew in CLIP but increases both gender and race skew in OpenCLIP, underscoring that parameter count interacts with the loss function rather than acting as a universal regulariser. Error bars give 95% bootstrap CIs.
• Figure 2: Effect of enlarging the corpus from LAION-400M to LAION-2B while holding the encoder fixed. Deltas are plotted as in Fig. \ref{['fig:mod']}. CLIP’s bias profile is essentially flat (all shifts within ±0.06), whereas OpenCLIP—especially the smaller encoder—shows a doubling of gender skew and a substantial rise in race skew, contradicting the common intuition that “more data dilutes bias.”
• Figure 3: CLIP vs. OpenCLIP at matched encoder and corpus size (400M images). Bars are absolute Max Skew values rather than deltas. OpenCLIP (purple) is consistently more gender-biased; CLIP (teal) is more race-biased once either the encoder or the corpus is scaled. The crossed pattern signals that neither objective is uniformly preferable and that balancing harms may require ensembling or calibration.
• Figure 4: Corpus-level prevalence of harmful top-1 predictions on 10,000 FairFace images. Each cluster shows the share of portraits tagged Criminal (yellow), negative Communion (orange), or negative Agency (red). Negative-communion stereotypes dominate—reaching 62% for CLIP-B/32@400M. Crime mislabelling tops out at 13% (CLIP-L/14@400M), while negative-agency labels peak at 36% (OpenCLIP-L/14@2B). Shorter bars indicate safer behaviour; whiskers give 95% bootstrap confidence intervals.
• Figure 5: CLIP vs. CLIP debiased with Bias Prompts. Panels: (a) Gender–Crime, (b) Gender–Communion, (c) Race–Crime, (d) Race–Communion. Bars show Bias Score (lower is better) across four checkpoints (B/32 FF, L/14 FF, B/32 PT, L/14 PT). Bias Prompts reduce skew in several settings but effects vary by axis and size, motivating validation at the deployment model size.