When are Lemons Purple? The Concept Association Bias of Vision-Language Models

TL;DR

This work identifies Concept Association Bias (CAB) in vision–language models, where models fill in missing cross-modal concepts leading to incorrect zero-shot predictions in tasks like VQA. It demonstrates CAB with a lemon–eggplant color task, shows its presence across CLIP, BLIP, and BLIP-2 (but not OFA), and links CAB strength to VQA performance; it also shows that stronger cross-modal interaction and autoregressive-only training mitigate CAB. The authors quantify association strength with ConceptNet and examine non-color attributes (e.g., part–whole relations) to argue for a general binding bias. They propose mitigation via deeper modality interaction and fine-tuning, but caution that this does not fully solve the underlying binding problem, suggesting avenues for future research on object-centric representations and cross-modal binding. Overall, the work highlights brittleness in cross-modal alignment and provides guidance for developing more robust vision–language systems.

Abstract

Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval. However, such performance does not realize in tasks that require a finer-grained correspondence between vision and language, such as Visual Question Answering (VQA). As a potential cause of the difficulty of applying these models to VQA and similar tasks, we report an interesting phenomenon of vision-language models, which we call the Concept Association Bias (CAB). We find that models with CAB tend to treat input as a bag of concepts and attempt to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. We demonstrate CAB by showing that CLIP's zero-shot classification performance greatly suffers when there is a strong concept association between an object (e.g. eggplant) and an attribute (e.g. color purple). We also show that the strength of CAB predicts the performance on VQA. We observe that CAB is prevalent in vision-language models trained with contrastive losses, even when autoregressive losses are jointly employed. However, a model that solely relies on autoregressive loss seems to exhibit minimal or no signs of CAB.

Figures (18)

• Figure 1: When we ask CLIP the color of the lemon in the above image, CLIP answers "purple". The text prompt we use is "In this picture, the color of the lemon is [mask]", where CLIP picks one from [red, green, yellow, orange, purple].
• Figure 2: Example images from Natural-Color Dataset (NCD) anwarImageColorizationSurvey2022, modified for our color recognition tasks so that each image contains two different objects.
• Figure 3: Zero-shot performance of CLIP on color recognition tasks using NCD anwarImageColorizationSurvey2022. CLIP achieves almost perfect accuracy when there is a single object in the image, but the accuracy significantly drops with two objects. "Two object*" refer to the case in which we instead measure the accuracy of predicting the color of the object B when it is asked for the color of the object A, where we see 80% zero-shot accuracy. We claim this gap between Two objects and Two objects* is a result of the Concept Association Bias (CAB).
• Figure 4: The concept binding diagram. Two variables per object represent the object name and its attribute (e.g. color), respectively. We suggest that the text prompt and the image are represented as two separate "bags of concepts" in CLIP. When a pair of object-attribute concepts are naturally associated with each other, both concepts can be accounted for by including in the prompt either of the object or the attribute. When only some of the concepts in the image are included in the text, this leaves other concepts in the image unaccounted for.
• Figure 5: Examples from UNCD. Single object (Top) and Two objects per image (Bottom).