Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

TL;DR

This work identifies intra-modal misalignment caused by CLIP's inter-modal training objective and the resulting modality gap between image and text embeddings. It introduces modality inversion techniques, Optimization-based Textual Inversion (OTI) and Optimization-based Visual Inversion (OVI), to transform intra-modal tasks into inter-modal ones, enabling retrieval-based improvements across image-to-image and text-to-text tasks on 15 datasets by exploiting CLIP's inter-modal alignment. The study also analyzes how intra-modal losses during pre-training (e.g., SLIP) and reducing the modality gap impact misalignment, revealing that intra-modal pretraining can mitigate the issue while closed gaps may hurt downstream performance. Overall, the paper provides practical insights into when and why inter-modal comparisons help intra-modal tasks and highlights directions for more efficient mitigation strategies in VLM pre-training and fine-tuning.

Abstract

Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these powerful multi-modal models is highly suboptimal for intra-modal tasks like image-to-image retrieval. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints, leading to what we call intra-modal misalignment. To demonstrate this, we leverage two optimization-based modality inversion techniques that map representations from their input modality to the complementary one without any need for auxiliary data or additional trained adapters. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intra-modal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching a native inter-modal task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or narrowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://github.com/miccunifi/Cross-the-Gap.

Figures (5)

• Figure 1: Motivation and overview. Left: The inter-modal contrastive loss used in pretraining enforces paired images and texts to be at a given distance $r$ (i.e.$r_{dog}\text{ and }r_{cat}$) but does not encourage intra-modal alignment. Consequently, intra-modal similarity scores might not correspond to those of actual images and texts (i.e.$d_1 < d_2$). Right: We show that the common practice of individually exploiting only one encoder is suboptimal and approaching intra-modal tasks (e.g. image-to-image retrieval) inter-modally via modality inversion improves performance.
• Figure 2: (a, b) Loss values and retrieval performance over OTI optimization steps for different numbers of pseudo-tokens $R$. (c) Distribution of pairwise image-image, text-image, and OTI-image cosine similarities. We consider the OTI-inverted features using four pseudo-tokens ($R = 4$) at two distinct optimization steps: the performance peak (step 17) and the final step (step 1000).
• Figure A0: Optimization-based Textual Inversion (OTI)
• Figure A1: Distribution of pairwise dog-dog and dog-cat image similarities. Overlap highlights the intra-modal misalignment issue.
• Figure A2: Captions generated by pre-trained captioning models for an image from the $\mathcal{R}$Oxford dataset.