Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP

TL;DR

This work provides a formal theory for how CLIP learns transferable, cross-modal representations and enables zero-shot transfer. It identifies key challenges in aligning image and text features and introduces an $(\alpha,\beta,\gamma)$-completeness framework under which near-optimal training yields separable, aligned representations across modalities; it also contrasts CLIP's contrastive loss with square loss, showing the latter's failure for zero-shot tasks. Building on these insights, the authors propose a one-sided regularization term that increases the learned margin and empirically improves zero-shot transfer and linear probing on benchmark datasets. The combination of theory and experiments yields practical guidance for improving multimodal transfer learning with small batches and temperature-aware training, while outlining limitations and avenues for extending to more modalities.

Abstract

Multi-modal learning has become increasingly popular due to its ability to leverage information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and exhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the huge practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets.

Figures (6)

• Figure 1: Illustration of the Challenges. Left: The feature domains are different and not one-to-one mapping. We need to learn transferrable features while preserving the shared features. Right: The image-text data show in the same batch can have similar shared features since the shared features are sparse (here is "stop sign"). The learned similarities between each image-text pair are very close.
• Figure 2: Illustration of zero-shot transfer learning. With the encoders jointly pre-trained on the image-text dataset, zero-shot transfer is done by issuing prompts according to all the potential labels of the task. With similarity score computed between the image embedding and all prompt embeddings, the label that resulted in highest similarity is the prediction.
• Figure 3: The distribution of the margins with regard to CLIP models trained at different temperature values. Margin is computed within each batch of the data.
• Figure 4: Distribution of the image-caption pairs in MSCOCO, where we count the number of object that appeared in the image but was absent from the captions.
• Figure 5: Examples of the image-text pairs from CC3M. We identify a few missing visual objects in the captions.

Theorems & Definitions (23)

• Remark 3.2
• Theorem 3.3
• Theorem 4.2
• Remark 4.3
• Remark 4.4: Choice of temperature parameter
• Remark 4.5: Batch size
• Corollary 5.1
• Remark 5.2
• Definition 5.3: A Case Study
• Lemma 5.4: Completeness