Principled Multimodal Representation Learning

TL;DR

PMRL tackles the problem of aligning multiple modalities without fixed anchors by linking full multimodal alignment to a rank-1 Gram matrix. It introduces a principled objective that maximizes the leading singular value $\sigma_1$ of the multimodal representation matrix $\mathbf{Z}$ using a softmax over singular values, and augments this with instance-wise regularization over leading eigenvectors to prevent collapse. The approach is evaluated on diverse tasks, including text–video and text–audio retrieval and autism classification, demonstrating consistent gains over anchor-based and volume-based baselines and exhibiting robustness to noise. By reframing alignment as a rank-1, anchor-free problem and providing theoretical insights into Gram matrices and singular values, PMRL offers a scalable, interpretable framework for multimodal representation learning with strong practical impact.

Abstract

Multimodal representation learning seeks to create a unified representation space by integrating diverse data modalities to improve multimodal understanding. Traditional methods often depend on pairwise contrastive learning, which relies on a predefined anchor modality, restricting alignment across all modalities. Recent advances have investigated the simultaneous alignment of multiple modalities, yet several challenges remain, such as limitations imposed by fixed anchor points and instability arising from optimizing the product of singular values. To address the challenges, in this paper, we propose Principled Multimodal Representation Learning (PMRL), a novel framework that achieves simultaneous alignment of multiple modalities without anchor dependency in a more stable manner. Specifically, grounded in the theoretical insight that full alignment corresponds to a rank-1 Gram matrix, PMRL optimizes the dominant singular value of the representation matrix to align modalities along a shared leading direction. We propose a softmax-based loss function that treats singular values as logits to prioritize the largest singular value. Besides, instance-wise contrastive regularization on the leading eigenvectors maintains inter-instance separability and prevents representation collapse. Extensive experiments across diverse tasks demonstrate PMRL's superiority compared to baseline methods. The source code will be publicly available.

Figures (9)

• Figure 1: The illustration of multimodal representations within a hypersphere. The left demonstrates pairwise contrastive learning to align multiple modalities with a predefined anchor (i.e., caption), where modalities are sampled to multiple pairs. The right illustrates our method that aligns all modalities simultaneously with a leading direction.
• Figure 2: The overall framework of PMRL. Different modalities of the instance are encoded into multimodal representations $\mathbf{Z}$. PMRL utilizes SVD to obtain the maximum singular value $\sigma_1$ and maximizes it with the objective $\mathcal{L}^{\mathcal{M}}$. The leading directions (arrows in red) corresponding to $\sigma_1$ from different instances are regularized by $\mathcal{L}^{\mathcal{M}'}$.
• Figure 3: The ablation study across 4 datasets in terms of Recall@1. The instance-wise regularization loss (PMRL w/o reg) and instance matching loss (PMRL w/o IM) are canceled from PMRL and then compared with VAST and GRAM.
• Figure 4: The examples of any modality retrieval. With a unified space, different modalities can retrieve others. PMRL is capable of retrieving from any modality pair with higher accuracy.
• Figure 5: Performance comparison for any modality retrieval across 6 benchmark datasets. PMRL is compared with GRAM in terms of Recall@1. Blue regions highlight where PMRL outperforms GRAM, while gray regions indicate the opposite. Diagonal regions (colored in white) represent self-modal retrieval, which is not meaningful.

Theorems & Definitions (3)

• Theorem 2: Principled learning
• proof
• proof