Generative Modeling of Class Probability for Multi-Modal Representation Learning

TL;DR

CALM tackles modality discrepancies between video and text by aligning class probability distributions to class anchors rendered as prompts. It introduces a cross-modal probabilistic variational autoencoder to model uncertainty and reconstruct intra-modal distributions from inter-modal distributions. Key contributions include anchor-based class prompts, a probabilistic alignment mechanism, and strong out-of-domain generalization demonstrated on four benchmarks. The approach offers a scalable, robust framework for cross-modal alignment that leverages independent class knowledge to enrich joint representations.

Abstract

Multi-modal understanding plays a crucial role in artificial intelligence by enabling models to jointly interpret inputs from different modalities. However, conventional approaches such as contrastive learning often struggle with modality discrepancies, leading to potential misalignments. In this paper, we propose a novel class anchor alignment approach that leverages class probability distributions for multi-modal representation learning. Our method, Class-anchor-ALigned generative Modeling (CALM), encodes class anchors as prompts to generate and align class probability distributions for each modality, enabling more effective alignment. Furthermore, we introduce a cross-modal probabilistic variational autoencoder to model uncertainty in the alignment, enhancing the ability to capture deeper relationships between modalities and data variations. Extensive experiments on four benchmark datasets demonstrate that our approach significantly outperforms state-of-the-art methods, especially in out-of-domain evaluations. This highlights its superior generalization capabilities in multi-modal representation learning.

Figures (3)

• Figure 1: (a) Videos contain subtle semantic information, whereas textual descriptions often have limited expressive capacity. This mismatch leads to an information imbalance and modality discrepancy between video and text, resulting in the collapse of diverse video features to a limited textual representation scope. (b) To address this issue, we propose a class-anchor-aligned generative modeling approach. Our method generates class probability distributions by aligning prompts with inputs from each modality, effectively bridging the modality gap and preserving the diverse semantics of video content.
• Figure 2: An overview of our framework. We employ class labels from an independent dataset, transform them into prompts, and extract their linguistic features to serve as class anchors. We then compute class probability distributions for video and text features by measuring the similarities between their features and the class anchors, effectively representing intra-modal and inter-modal relationships. For modality alignment, we employ a cross-modal probabilistic variational autoencoder that takes the inter-modal probability distribution as input and reconstructs the intra-modal probability distribution to align the modalities in a shared latent space.
• Figure 3: Qualitative video retrieval results on the MSR-VTT dataset. Selected anchors capture distinct semantic cues, either aligning shared content (a) or highlighting complementary information to address modality imbalance (b). Inter-modal and intra-modal relationships serve as supplementary semantic cues, enhancing the semantic alignment between video and text and improving retrieval performance.