Cross-modality debiasing: using language to mitigate sub-population shifts in imaging

TL;DR

Sub-population shifts threaten robustness in vision-language models like CLIP, especially when fine-tuning erodes distributional robustness. The authors introduce L-DRO, a language-based distributional robustness framework that debiases CLIP image representations via natural language prompts and an entropy–consistency objective, without requiring instance-level labels. They formalize the link between language supervision and subgroup robustness and validate on CelebA and Waterbirds, reporting substantial gains in worst-case accuracy and improved training stability. The work highlights the practical potential of cross-modality debiasing for robust multimodal systems, while acknowledging sensitivity to prompt design and the need for domain-aware validation.

Abstract

Sub-population shift is a specific type of domain shift that highlights changes in data distribution within specific sub-groups or populations between training and testing. Sub-population shift accounts for a significant source of algorithmic bias and calls for distributional robustness. Recent studies found inherent distributional robustness in multi-modality foundation models, such as the vision-language model CLIP, yet this robustness is vulnerable through parameter fine-tuning. In this paper, we propose leveraging the connection of robustness among different modalities and reshaping the distributional robustness of one modality with another. Specifically, in the context of the distributional robustness of CLIP, we propose to leverage natural language inputs to debias the image feature representations, to improve worst-case performance on sub-populations. Our extensive empirical studies show that image representations debiased by natural language can achieve significant performance improvement and reduction of performance instability under sub-population shifts.

Figures (2)

• Figure 1: Training phase of L-DRO on CelebA liu2015faceattributes. Given the training dataset (without instance-wise label information) and concerned sup-populations, L-LDR aims to learn a feature adapter $A_{\theta_{a}}$ to transform image representation from original embedding to a debiased embedding. The goal is to ensure that the debiased embedding does not reveal any information about sub-population membership while minimizing significant changes from the original embedding.
• Figure 2: Under CLIP (ViT-B/32), the average and worst-case accuracy of validation dataset and test dataset over epochs using L-DRO and baseline methods. The left figure shows the performance of CelebA dataset, and the right figure shows the performance of Waterbirds dataset.