Understanding the Emergence of Multimodal Representation Alignment

TL;DR

The paper investigates the emergence of multimodal representation alignment and its relationship to downstream performance across synthetic and real-world datasets. By varying redundancy, uniqueness, and heterogeneity, it shows that alignment is constrained by data properties and does not universally predict performance; model capacity can improve performance even when alignment is weak, and the alignment-performance relationship is highly dataset-dependent. A practical takeaway is to use dataset-aware analysis of alignment to guide explicit cross-modal alignment strategies, rather than assuming that more alignment will always yield better results. The work combines CK A and mutual-KNN alignment measures with synthetic data and real multimodal benchmarks to provide nuanced guidance for practitioners. Code is released to enable further exploration and validation.

Abstract

Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research primarily focused on explicitly aligning these representations through targeted learning objectives and model architectures, a recent line of work has found that independently trained unimodal models of increasing scale and performance can become implicitly aligned with each other. These findings raise fundamental questions regarding the emergence of aligned representations in multimodal learning. Specifically: (1) when and why does alignment emerge implicitly? and (2) is alignment a reliable indicator of performance? Through a comprehensive empirical investigation, we demonstrate that both the emergence of alignment and its relationship with task performance depend on several critical data characteristics. These include, but are not necessarily limited to, the degree of similarity between the modalities and the balance between redundant and unique information they provide for the task. Our findings suggest that alignment may not be universally beneficial; rather, its impact on performance varies depending on the dataset and task. These insights can help practitioners determine whether increasing alignment between modalities is advantageous or, in some cases, detrimental to achieving optimal performance. Code is released at https://github.com/MeganTj/multimodal_alignment.

Figures (29)

• Figure 1: Emergence of multimodal alignment? Triangles and circles correspond to different modalities. While the Platonic Representation Hypothesis huh_platonic_2024 argues that better cross-modal alignment predicts better performance, our findings demonstrate that the relation between alignment and performance is more nuanced and depends on several dataset characteristics including the degree of heterogeneity and interactions between modalities.
• Figure 2: Two principal dimensions of multimodal data. This study empirically evaluates data across two key dimensions: heterogeneity and interactions. Heterogeneity, represented on the x-axis, reflects the similarity between two data modalities, $X_1$ and $X_2$, regardless of the task. Interactions, on the y-axis, indicate the balance between redundant and unique information across modalities that is relevant to task $Y$. We expect the Platonic Representation Hypothesis to hold in cases of redundancy and similar modalities, but when and why alignment emerges implicitly, and whether alignment is a reliable indicator of performance, remain open questions.
• Figure 3: Synthetic data generation and training. We generate synthetic data with varying levels of uniqueness and heterogeneity. The building blocks are the redundant and unique components $[x_r, x_{u_1}, x_{u_2}]$, where $[x_r, x_{u_1}]$ are used in creating $X_1$ and $[x_r, x_{u_2}]$ are for $X_2$. The level of uniqueness is determined by the number of features from $x_r$ that are used to compute the labels $Y$, given that the total number of features used for label computation is held constant. $X_2$ is transformed into a heterogeneous modality using a transformation network $\phi$. In our experiments, we compute alignment between unimodal encoders $E_1$, $E_2$ trained on $X_1$, $X_2$ respectively.
• Figure 4: Alignment vs uniqueness on synthetic datasets. Alignment is computed between unimodal encoders trained on datasets with different levels of informational uniqueness $U$. Each dot is an independent run on a different model size on a given dataset. We see that the maximum level of achievable level of alignment, shown by the red dots, decreases as the level of uniqueness increases. Five different figures shows the different levels of nonlinear transformation we apply to the original data. We report the Spearman correlation $\rho$ between the maximum alignment values and $U$.
• Figure 5: Alignment vs uniqueness on real large-scale vision-language datasets. Alignment is computed between DINOv2 vision models and large language models. Each dot is an independent run on a different model size on a dataset with a given level of uniqueness. The maximum achievable alignment decreases as uniqueness increases.