Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP

Sedigheh Eslami, Gerard de Melo

TL;DR

AlignCLIP is designed to reduce the modality gap in the CLIP embedding space and achieves noticeable enhancements in the cross-modal alignment of the embeddings, and thereby, reduces the modality gap, while improving the performance across several zero-shot and fine-tuning downstream evaluations.

Abstract

Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found to have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we aim at answering three main questions: 1. Does sharing the parameter space between the multi-modal encoders reduce the modality gap? 2. Can the gap be mitigated by pushing apart the uni-modal embeddings via intra-modality separation? 3. How do these gap reduction approaches affect the downstream performance? We design AlignCLIP, in order to answer these questions and through extensive experiments, we show that AlignCLIP achieves noticeable enhancements in the cross-modal alignment of the embeddings, and thereby, reduces the modality gap, while improving the performance across several zero-shot and fine-tuning downstream evaluations.

Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP

TL;DR

AlignCLIP is designed to reduce the modality gap in the CLIP embedding space and achieves noticeable enhancements in the cross-modal alignment of the embeddings, and thereby, reduces the modality gap, while improving the performance across several zero-shot and fine-tuning downstream evaluations.

Abstract

Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found to have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we aim at answering three main questions: 1. Does sharing the parameter space between the multi-modal encoders reduce the modality gap? 2. Can the gap be mitigated by pushing apart the uni-modal embeddings via intra-modality separation? 3. How do these gap reduction approaches affect the downstream performance? We design AlignCLIP, in order to answer these questions and through extensive experiments, we show that AlignCLIP achieves noticeable enhancements in the cross-modal alignment of the embeddings, and thereby, reduces the modality gap, while improving the performance across several zero-shot and fine-tuning downstream evaluations.
Paper Structure (20 sections, 13 equations, 5 figures, 10 tables)

This paper contains 20 sections, 13 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background, Notations and Concepts
  3. AlignCLIP
  4. Sharing the Learnable Parameter Space in CLIP
  5. Intra-Modality Separation
  6. Experiments
  7. Training Dataset and Setup
  8. Cross-Modal Alignment
  9. Classification
  10. Robustness to Natural Distribution Shift
  11. Multi-Modal Retrieval
  12. Ablation Study
  13. Results Analysis
  14. Related Work
  15. Conclusion
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Modality gap and cross-modal alignment. (A) The average pairwise cosine similarity of the encoded image--text pairs is about 0.22, i.e., the average angle is about 78 degrees. (B) Schematic illustration of unaligned (left) versus aligned (right) embedding spaces.
  • Figure 2: Overview of sharing the transformer and projection layer in SharedCLIP.
  • Figure 3: Schematic summary of the Intra-Modality Separation approach in AlignCLIP.
  • Figure 4: Cumulative distribution of pairwise cosine similarities of positive samples in MSCOCO.
  • Figure 5: DOSNES visualization of the multi-modal embeddings using CC3M