Debiasing CLIP: Interpreting and Correcting Bias in Attention Heads
Wei Jie Yeo, Rui Mao, Moloud Abdar, Erik Cambria, Ranjan Satapathy
TL;DR
This work tackles spurious bias in CLIP by introducing Locate-Then-Correct (LTC), a diagnose-then-correct framework that operates at the level of individual Vision Transformer attention heads. By leveraging a linear decomposition of ViT representations, LTC localizes spurious and target states via logit-lens projections and a contrastive mechanism, then mitigates bias through mean ablation of spurious states and knowledge injection using discriminative text-derived vectors. Empirically, LTC yields substantive improvements in worst-group accuracy on background- and gender-bias benchmarks (often exceeding 50% gains) while preserving or enhancing overall performance, and it provides interpretable head-level insights through SHAP and visualization analyses. The approach emphasizes training-free or lightweight interventions with strong interpretability, offering a practical and scalable path to debiasing large multimodal models with minimal parameter updates. Overall, LTC demonstrates that targeted, interpretable head-level corrections can robustly reduce spurious correlations in CLIP and similar models, with potential for extension to other architectures and modalities.
Abstract
Multimodal models like CLIP have gained significant attention due to their remarkable zero-shot performance across various tasks. However, studies have revealed that CLIP can inadvertently learn spurious associations between target variables and confounding factors. To address this, we introduce \textsc{Locate-Then-Correct} (LTC), a contrastive framework that identifies spurious attention heads in Vision Transformers via mechanistic insights and mitigates them through targeted ablation. Furthermore, LTC identifies salient, task-relevant attention heads, enabling the integration of discriminative features through orthogonal projection to improve classification performance. We evaluate LTC on benchmarks with inherent background and gender biases, achieving over a $>50\%$ gain in worst-group accuracy compared to non-training post-hoc baselines. Additionally, we visualize the representation of selected heads and find that the presented interpretation corroborates our contrastive mechanism for identifying both spurious and salient attention heads. Code available at https://github.com/wj210/CLIP_LTC.