Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning

François Role, Sébastien Meyer, Victor Amblard

TL;DR

The paper tackles the modality gap between image and text embeddings in vision-language models, which hampers fully multimodal retrieval. It introduces two post-processing approaches—spectral embedding and Laplacian-regularized optimal transport—to realign cross-modal representations, accompanied by three evaluation frameworks (distance-based, IR-based heterogeneity indices, and distribution-based) to quantify alignment quality. Experiments on COCO-PAIRS and CC-PAIRS with CLIP, SigLIP, and LLM2CLIP demonstrate substantial bias reduction and improved cross-modal retrieval, with spectral methods achieving strong gains using around 100 components. The methods are model-agnostic and practical, offering a path to better multimodal fusion without heavy architectural changes, and the authors provide code to enable broader adoption.

Abstract

Vision-language models (VLMs) allow to embed texts and images in a shared representation space. However, it has been shown that these models are subject to a modality gap phenomenon meaning there exists a clear separation between the embeddings from one modality and another in the embedding space. While this misalignment is detrimental for downstream tasks such as multimodal retrieval, multimodal clustering or zero-shot classification, etc. no generic and practical methods have so far been proposed to assess it precisely and even reduce it. We therefore propose novel measures and effective techniques (spectral- and optimal transport-based methods) to achieve this goal. Extensive experiments conducted on several image-text datasets and models demonstrate their effectiveness and beneficial effects on downstream tasks. Our code is available at the URL provided in the paper's abstract.

Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning

TL;DR

The paper tackles the modality gap between image and text embeddings in vision-language models, which hampers fully multimodal retrieval. It introduces two post-processing approaches—spectral embedding and Laplacian-regularized optimal transport—to realign cross-modal representations, accompanied by three evaluation frameworks (distance-based, IR-based heterogeneity indices, and distribution-based) to quantify alignment quality. Experiments on COCO-PAIRS and CC-PAIRS with CLIP, SigLIP, and LLM2CLIP demonstrate substantial bias reduction and improved cross-modal retrieval, with spectral methods achieving strong gains using around 100 components. The methods are model-agnostic and practical, offering a path to better multimodal fusion without heavy architectural changes, and the authors provide code to enable broader adoption.

Abstract

Vision-language models (VLMs) allow to embed texts and images in a shared representation space. However, it has been shown that these models are subject to a modality gap phenomenon meaning there exists a clear separation between the embeddings from one modality and another in the embedding space. While this misalignment is detrimental for downstream tasks such as multimodal retrieval, multimodal clustering or zero-shot classification, etc. no generic and practical methods have so far been proposed to assess it precisely and even reduce it. We therefore propose novel measures and effective techniques (spectral- and optimal transport-based methods) to achieve this goal. Extensive experiments conducted on several image-text datasets and models demonstrate their effectiveness and beneficial effects on downstream tasks. Our code is available at the URL provided in the paper's abstract.
Paper Structure (19 sections, 8 equations, 3 figures, 6 tables)

This paper contains 19 sections, 8 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Post-processing
  4. Spectral Techniques
  5. Optimal Transport
  6. Evaluation Measures
  7. Distance-based Evaluation Measures
  8. IR-based Heterogeneity Indices
  9. Distribution-based Evaluation Measures
  10. Experiments
  11. Datasets and Models Used
  12. Datasets
  13. Models
  14. Parameters for Optimal Transport
  15. Experimental Results and Discussion
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Definition of the heterogeneity indices: $\textbf{ITR}$ and $\textbf{TIR}$.
  • Figure 2: Definition of the ranking-based measures: $\mathbf{TMR}$ and $\mathbf{IMR}$.
  • Figure 3: Distribution of the cosine distances between the images and their corresponding texts in both cases (original and transformed embeddings). The dashed lines represent the mean distances across all image-text pairs.