Gramian Multimodal Representation Learning and Alignment

TL;DR

GRAM introduces a volume-based multimodal alignment measure that operates directly in the high-dimensional embedding space to jointly align 2 to $n$ modalities. By using the Gramian volume of the modality vectors, it generalizes beyond anchor-based cosine similarities and provides a GRAM-based contrastive loss plus a data-anchor matching term. Empirical results show state-of-the-art improvements in video-audio-text retrieval and audio-video classification, with clear evidence that the GRAM space is more disentangled and scalable to additional modalities. The approach also provides a correlated performance metric, $1-\text{GRAM}$, that tracks downstream task success, and the work demonstrates reproducibility through available code and models.

Abstract

Human perception integrates multiple modalities, such as vision, hearing, and language, into a unified understanding of the surrounding reality. While recent multimodal models have achieved significant progress by aligning pairs of modalities via contrastive learning, their solutions are unsuitable when scaling to multiple modalities. These models typically align each modality to a designated anchor without ensuring the alignment of all modalities with each other, leading to suboptimal performance in tasks requiring a joint understanding of multiple modalities. In this paper, we structurally rethink the pairwise conventional approach to multimodal learning and we present the novel Gramian Representation Alignment Measure (GRAM), which overcomes the above-mentioned limitations. GRAM learns and then aligns $n$ modalities directly in the higher-dimensional space in which modality embeddings lie by minimizing the Gramian volume of the $k$-dimensional parallelotope spanned by the modality vectors, ensuring the geometric alignment of all modalities simultaneously. GRAM can replace cosine similarity in any downstream method, holding for 2 to $n$ modalities and providing more meaningful alignment with respect to previous similarity measures. The novel GRAM-based contrastive loss function enhances the alignment of multimodal models in the higher-dimensional embedding space, leading to new state-of-the-art performance in downstream tasks such as video-audio-text retrieval and audio-video classification. The project page, the code, and the pretrained models are available at https://ispamm.github.io/GRAM/.

Figures (6)

• Figure 1: Visualization of the GRAM intuition: on the left, embedding vectors from semantically aligned multimodal data build a parallelotope with a small volume. On the right, where modalities are not aligned with each other, the formed parallelotope has a large volume.
• Figure 2: GRAM-based model architecture. Class tokens from each modality are involved in shaping the $k$-dimensional parallelotope, whose volume indicates the semantic alignment of the modalities. All the tokens are then involved in the multimodal encoder to enhance the predictions. The model is pretrained with the proposed Gramian multimodal contrastive losses $\mathcal{L}_{D2A}$ and $\mathcal{L}_{DAM}$.
• Figure 3: The proposed GRAM similarity is strongly correlated ($\rho=0.923$) with large multimodal models performance in downstream tasks.
• Figure 4: t-SNE visualization on VGGSound of VAST, cosine-based, (left) and GRAM (right) latent spaces. GRAM better models the latent space, resulting in a space more disentangled and highly interpretable. Class clusters are easily recognizable in space, while video (squares) and audio (triangles) modalities that are closer to the classification text (star).
• Figure 5: Multimodal V2T/T2V, V2T/T2V, Gramian Value and training loss of GRAM trained with our loss functions and GRAM trained with only TV-TA loss functions. ActivityNet Dataset, training from scratch.