Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge

TL;DR

The paper tackles interpreting CLIP's zero-shot image classification by analyzing the mutual knowledge shared between its vision and language encoders through a discrete textual-concept space. It introduces a training-free multimodal concept framework that grounds visual parts to textual descriptors and retrieves textual concepts around the predicted class to compute a mutual-information measure $I(\mathcal{D}_{v}; \mathcal{D}_{\hat{y}})$. Key contributions include a method to quantify mutual information via MI dynamics and AUC, validation across 13 CLIP models, and evidence that larger models and richer pretraining data yield stronger shared knowledge which correlates with zero-shot accuracy, with zero-shot gains up to 3.75%. The work also provides human-friendly explanations and visualizations of mutual concepts, informing model selection and interpretability in vision-language models.

Abstract

Contrastive Language-Image Pretraining (CLIP) performs zero-shot image classification by mapping images and textual class representation into a shared embedding space, then retrieving the class closest to the image. This work provides a new approach for interpreting CLIP models for image classification from the lens of mutual knowledge between the two modalities. Specifically, we ask: what concepts do both vision and language CLIP encoders learn in common that influence the joint embedding space, causing points to be closer or further apart? We answer this question via an approach of textual concept-based explanations, showing their effectiveness, and perform an analysis encompassing a pool of 13 CLIP models varying in architecture, size and pretraining datasets. We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions. Our approach demonstrates an effective and human-friendly way of understanding zero-shot classification decisions with CLIP.

Figures (16)

• Figure 1: CLIP maps visual and textual inputs into a joint embedding space, with an information channel expressed in terms of the Mutual Information (MI) between them (a). We interpret the visual features from the vision encoder with multimodal concepts (b) which represent object parts and their corresponding textual description. From the language encoder, we identify points (shown in grey) around the zero-shot prediction (shown in green) as textual descriptions of the predicted class (c). By considering the textual descriptors corresponding to the visual concepts, and the textual descriptors of the language encoder for the predicted class, the two encoders establish a common space of textual concepts allowing us to identify mutual concepts and analyze their shared knowledge (d).
• Figure 1: CLIP layer similarity analysis for the token features (a) and key features (b).
• Figure 2: A high-level overview of our method for deriving visual concepts at the vision encoder (a), querying each visual concept individually from a textual bank to describe the visual concept in natural text (b), and then deriving textual concepts at the language encoder (c). The outputs of (b) and (c) share a common space of fine-grained textual concepts such that mutual information can be better calculated.
• Figure 2: Optimal Transport (OT) diversifies textual concepts across their visual counterparts
• Figure 3: MI Dynamics curve comparing model families (left) and pretraining datasets (middle). Correlation of AUC with zero-shot classification accuracy is shown right for ViTs and ResNets.