The Bias of Harmful Label Associations in Vision-Language Models

TL;DR

The paper addresses the risk that vision-language models encode harmful associations in their shared text-vision space, potentially harming fairness. It analyzes CLIP variants and BLIP-2 on the Casual Conversations datasets (CCv1/CCv2) using zero-shot classification over ImageNet classes, with harm defined via a taxonomy and assessed through top-5 predictions and confidence-weighted scores. Key findings show that harmful associations are roughly four to seven times more likely for darker skin tones, larger model sizes increase prediction confidence in harmful labels, and improvements on standard vision benchmarks do not reduce these disparities. The work highlights the need for fairness-aware evaluation and data curation to address harmful-label associations in open-world perception settings, beyond traditional accuracy gains.

Abstract

Despite the remarkable performance of foundation vision-language models, the shared representation space for text and vision can also encode harmful label associations detrimental to fairness. While prior work has uncovered bias in vision-language models' (VLMs) classification performance across geography, work has been limited along the important axis of harmful label associations due to a lack of rich, labeled data. In this work, we investigate harmful label associations in the recently released Casual Conversations datasets containing more than 70,000 videos. We study bias in the frequency of harmful label associations across self-provided labels for age, gender, apparent skin tone, and physical adornments across several leading VLMs. We find that VLMs are $4-7$x more likely to harmfully classify individuals with darker skin tones. We also find scaling transformer encoder model size leads to higher confidence in harmful predictions. Finally, we find improvements on standard vision tasks across VLMs does not address disparities in harmful label associations.

Figures (8)

• Figure 1: Most commonly selected harmful class labels for CLIP ViT-L14 (left) along with sample frames leading to harmful label associations from CCv2 (right).
• Figure 2: CCv2 breakdowns of harmful label associations. Second row shows confidence weighted scores.
• Figure 3: CCv2 physical adornments breakdowns.
• Figure 4: CCv2 physical adornments breakdowns (confidence weighted).
• Figure 5: CCv1 breakdowns of harmful label associations. Second row shows confidence weighted scores.