A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks

TL;DR

The experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance.

Abstract

Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debiasing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper introduces Selective Feature Imputation for Debiasing (SFID), a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs. SFID is versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance. This approach not only enhances the fairness of VLMs applications but also preserves their efficiency and utility across diverse scenarios.

Figures (8)

• Figure 1: Bias in VLMs' various downstream tasks. VLMs tend to prefer certain gender for a subject in image or text, while SFID mitigates the bias issue in VLMs.
• Figure 2: GradCAM visualization for feature indices sorted by their importance in predicting an attribute (e.g., gender). Highly important features (left) focus on attribute-related characteristics such as face in the image, while the least important features (right) are associated with the background. SFID not only identifies the crucial biased features but also imputes these biased features with ambiguous values derived from low-confidence samples.
• Figure 3: Selective Feature Imputation for Debiasing (SFID) utilizes RandomForest to extract feature importance ($j_k$) identifying bias-related features, and low-confidence samples in the validation set which indicate ambiguous representations. During the inference stage, the extracted feature indices and imputing values ($\mu_k$) from low-confidence samples are imputed into the embedding used in the downstream task.
• Figure 4: Comparison of zero-value imputation, zero-centered Gaussian noise, and low confidence samples. Different colors of points indicate different sensitive attributes. Gray points represent zero-centered Gaussian noise, which is out-of-distribution from the original embedding. SFID utilizes the centroid of low confidence samples (red $\mathbf{\times}$), which remain in-distribution of the original samples.
• Figure 5: Feature importances for gender prediction by RandomForest for each frozen representation.