Closing the Modality Gap Aligns Group-Wise Semantics

TL;DR

This work reframes the modality gap in CLIP-like multimodal learning as a core determinant of group-wise semantic structure rather than a mere byproduct of alignment. It introduces two losses, Align True Pairs ($\mathcal{L}_{\text{ATP}}$) and Centroid Uniformity ($\mathcal{L}_{\text{CU}}$), and combines them with the standard contrastive objective to form $\mathcal{L}_{\text{gap}}$ and $\mathcal{L}_{\text{CL}_{\text{gap}}}$, enabling zero-gap multimodal spaces without architectural changes. Across bimodal and trimodal benchmarks (e.g., CIFAR-10, AV-MNIST, MSCOCO, MSR-VTT), the method consistently improves clustering metrics like V-Measure while preserving retrieval performance, illustrating that closing the gap enhances semantic grouping without sacrificing instance-level accuracy. This suggests a paradigm shift: focusing on gap reduction can yield more coherent multimodal representations with practical benefits for tasks requiring semantic clustering and reasoning. The approach scales to multiple modalities and provides a principled way to balance true-pair alignment with global latent-space coverage.

Abstract

In multimodal learning, CLIP has been recognized as the \textit{de facto} method for learning a shared latent space across multiple modalities, placing similar representations close to each other and moving them away from dissimilar ones. Although CLIP-based losses effectively align modalities at the semantic level, the resulting latent spaces often remain only partially shared, revealing a structural mismatch known as the modality gap. While the necessity of addressing this phenomenon remains debated, particularly given its limited impact on instance-wise tasks (e.g., retrieval), we prove that its influence is instead strongly pronounced in group-level tasks (e.g., clustering). To support this claim, we introduce a novel method designed to consistently reduce this discrepancy in two-modal settings, with a straightforward extension to the general $n$-modal case. Through our extensive evaluation, we demonstrate our novel insight: while reducing the gap provides only marginal or inconsistent improvements in traditional instance-wise tasks, it significantly enhances group-wise tasks. These findings may reshape our understanding of the modality gap, highlighting its key role in improving performance on tasks requiring semantic grouping.

Figures (11)

• Figure 1: Reducing the gap consistently improves clustering metrics, while leaving unaffected retrieval ones. On the contrary, increasing the gap downgrades the V-Measure, bringing no improvements in R@1. In CLIP, the gap results in very poor clustering performance due to the latent space fragmentation.
• Figure 2: AV-MNIST multimodal latent space. The CLIP-based learning creates a fragmented latent space with embeddings clearly clustered by modality and not by multimodal semantics. Our method closes the gap and enhances group-wise semantics, placing embeddings of the same class in the same portion of the space, effectively learning a semantically meaningful multimodal latent space.
• Figure 3: (a) Following liang2022mind, we place six simulated image-text embedding pairs on a 3D sphere, with two mismatched pairs. We artificially move these pairs toward closing or enlarging the gap among them and we track the loss landscape during the simulation. (b) During the same simulation we keep track also of the gradient magnitude received by the six embeddings pairs through our design loss function $\mathcal{L}_{\text{CL}_{\text{gap}}}$. When the gap is closer to zero, the contribution to the loss is just matter of the non matching pairs.
• Figure 4: Most similar vectors from the MSCOCO vector databases with both modalities embedded. CLIP learned space has a gap of 0.47 and, for a text query, all the closest embeddings come from the text modality, meaning that embeddings are clustered according to the modality and not according to the overall multimodal semantics. On the contrary, the proposed method has nearly a zero gap, and most similar samples come from both the image and text modalities, proving that we can effectively build a multimodal latent space that is semantically coherent among the modalities. The same behavior holds for an image query.
• Figure 5: CosTP and gap during training.