Decipher the Modality Gap in Multimodal Contrastive Learning: From Convergent Representations to Pairwise Alignment

TL;DR

This work provides the first theoretical framework for analyzing representational convergence in multimodal contrastive learning and the modality gap. It shows that, in the absence of distributional constraints, MCL representations converge to a paired uniform distribution on the unit sphere, driving the modality gap to zero, but that a dimension-collapse-induced subspace constraint can yield a nonzero gap equal to the minimal angle between two modality-specific hyperplanes. The authors prove that perfect cross-modal alignment cannot be achieved for all pairs under subspace constraints, yet offer two mechanisms—hyperplane rotation and shared space projection (SSP)—that can realize perfect alignment, with SSP demonstrated as a practical post-hoc method in CLIP embeddings on MSCOCO. The results connect gap size to downstream modality alignment and provide a principled path to improve cross-modal consistency while preserving downstream performance, potentially guiding both pretraining and post-hoc alignment of multimodal models. Key insights identify dimension collapse as the root cause of the modality gap and show how strategic geometric transformations can achieve alignment without sacrificing the benefits of MCL.

Abstract

Multimodal contrastive learning (MCL) aims to embed data from different modalities in a shared embedding space. However, empirical evidence shows that representations from different modalities occupy completely separate regions of embedding space, a phenomenon referred to as the modality gap. Moreover, experimental findings on how the size of the modality gap influences downstream performance are inconsistent. These observations raise two key questions: (1) What causes the modality gap? (2) How does it affect downstream tasks? To address these questions, this paper introduces the first theoretical framework for analyzing the convergent optimal representations of MCL and the modality alignment when training is optimized. Specifically, we prove that without any constraint or under the cone constraint, the modality gap converges to zero. Under the subspace constraint (i.e., representations of two modalities fall into two distinct hyperplanes due to dimension collapse), the modality gap converges to the smallest angle between the two hyperplanes. This result identifies \emph{dimension collapse} as the fundamental origin of the modality gap. Furthermore, our theorems demonstrate that paired samples cannot be perfectly aligned under the subspace constraint. The modality gap influences downstream performance by affecting the alignment between sample pairs. We prove that, in this case, perfect alignment between two modalities can still be achieved via two ways: hyperplane rotation and shared space projection.

Figures (8)

• Figure 1: The COR of MCL. Orange and blue dots represent $X$ and $Y$. Starts are centers of $X$ and $Y$ (i.e., $c_x, c_y$). $\Delta_{\theta}$ denotes the size of modality gap. (a): When a model is initialized, $(X, Y)$ lie within two distinct cones. (b): Without any constraint, $(X, Y)$ converge to a paired uniform distribution and $\Delta_{\theta} \rightarrow 0$. (c): Under the cone constraint, $\Delta_{\theta} \rightarrow 0$. (d): $(X, Y)$ collapse into two distinct hyperplanes $\mathbb{A}$ and $\mathbb{B}$, respectively. $X \in \mathbb{S}_X$ (orange circle) $\in \mathbb{A}$ and $Y \in \mathbb{S}_Y$ (blue circle) $\in \mathbb{B}$. $\phi$ is the angle between $\mathbb{A}$ and $\mathbb{B}$. The green line represents the shared space $\mathbb{C} = \mathbb{A} \cap \mathbb{B}$. See \ref{['def:subspace']} for details. (e): Under the subspace constraint, when training is optimized, $c_x, c_y \perp \mathbb{C}$ and $\Delta_{\theta} \rightarrow \phi_{\mathrm{min}}$.
• Figure 2: Distributional constraints. CLIP ViT-B/32 representations of the MSCOCO validation set. (a): Density plot of cosine similarities between representations of image–image (I2I), text–text (T2T), paired image–text (P I2T), and unpaired image–text (NP-I2T). (b): UMAP plot of $X$ and $Y$. (c): Explained variance ratio of singular values of $X-\mu_X$ and $Y-\mu_Y$. (d): Values of principal angles.
• Figure 3: Modality alignment. Notations follow \ref{['fig:opt']}. (a): Condition (A6) ($c_x, c_y \perp \mathbb{C}$) and IMS ($x_i \cdot c_x = y_i \cdot c_y)$ hold. (b): The projections of $(x_i, y_i)_{i \neq c}$ onto $\mathbb{C}$ converge to $p_i$ (green points), i.e., $P_C x_i = P_C y_i = p_i$. (c): When Condition (A6) and (A8) ($P_C x_i = P_C y_i$) hold, $P_B x_i \nparallel y_i$, $P_A y_i \nparallel x_i$. Let $y_j = \frac{P_B x_i}{\|P_B x_i\|}$ (purple dot). Then $x_i \cdot y_j > x_i \cdot y_i$, and $(x_i, y_i)_{i \neq c}$ are not perfectly aligned. (d): Rotating $X$ with the hyperplane $\mathbb{A}$ towards $\mathbb{B}$, $X$ and $Y$ can be aligned perfectly. (c): Project $(x_i, y_i)$ onto $\mathbb{C}$ and re-normalize, then $(x_i^{*}, y_i^{*})$ (yellow dots) are perfectly aligned.
• Figure 4: Results. CLIP ViT-B/32 embeddings of MSCOCO validation set after applying SSP are used. (a): UMAP plot. (b): Density plot of cosine similarities.
• Figure 5: Translation-based method. Notations follow \ref{['fig:opt']}. (a): Condition (A6) ($c_x, c_y \perp \mathbb{C}$) and (A8) ($P_C x_i = P_C y_i$) hold. Orange and blue triangles represent $\mu_x$ and $\mu_y$, respectively. Red arrows indicate the direction and magnitude of the constant translation ($\mu_y - \mu_x$). (b): Translate $X$. (c): $X$ are translated to $X^{*}$. (d): Re-normalize $X^{*}$. Purple arrows indicate the direction and magnitude of the normalization. (e): $X^{*}$ are re-normalized to $X^{**}$. (f): The distribution of $X$ is altered after translation, such that $P_C x_i^{**} \neq P_C y_i^{**}$.

Theorems & Definitions (70)

• Definition 1: Multimodal Contrastive Loss (MCL Loss)
• Definition 2: Modality Gap
• Definition 3: $\mathrm{vMF}$ Distribution
• Theorem 1
• Corollary 1
• Definition 4
• Theorem 2
• Definition 5
• Definition 6
• Theorem 3