PROMISE: Prompt-Attentive Hierarchical Contrastive Learning for Robust Cross-Modal Representation with Missing Modalities

TL;DR

PROMISE tackles the problem of degraded cross-modal representations when modalities are missing by integrating a Prompt Attention mechanism with modality-specific prompts and a hierarchical, dual-level contrastive framework. The Fusion-driven Nexus Contrastive Learning (FNCL) and Cohesion-driven Core Contrastive Learning (CCCL) jointly enforce cross-modal alignment and intra-modal discriminability, respectively, while a frozen encoder backbone provides stable representations. Across benchmarks with varying missing-patterns, PROMISE consistently outperforms state-of-the-art baselines, and ablations confirm the necessity of both FNCL and CCCL for robustness. The approach yields strong semantic consistency between available and reconstructed modalities and demonstrates practical potential for real-world multimodal systems subject to data incompleteness.

Abstract

Multimodal models integrating natural language and visual information have substantially improved generalization of representation models. However, their effectiveness significantly declines in real-world situations where certain modalities are missing or unavailable. This degradation primarily stems from inconsistent representation learning between complete multimodal data and incomplete modality scenarios. Existing approaches typically address missing modalities through relatively simplistic generation methods, yet these approaches fail to adequately preserve cross-modal consistency, leading to suboptimal performance. To overcome this limitation, we propose a novel multimodal framework named PROMISE, a PROMpting-Attentive HIerarchical ContraStive LEarning approach designed explicitly for robust cross-modal representation under conditions of missing modalities. Specifically, PROMISE innovatively incorporates multimodal prompt learning into a hierarchical contrastive learning framework, equipped with a specially designed prompt-attention mechanism. This mechanism dynamically generates robust and consistent representations for scenarios where particular modalities are absent, thereby effectively bridging the representational gap between complete and incomplete data. Extensive experiments conducted on benchmark datasets, along with comprehensive ablation studies, clearly demonstrate the superior performance of PROMISE compared to current state-of-the-art multimodal methods.

Figures (6)

• Figure 1: Architectural overview of the PROMISE framework for robust multimodal representation learning with missing modalities. The framework integrates: (1) a frozen multimodal encoder backbone, (2) a prompt-attention mechanism with modality-specific prompt pools for generating missing representations, (3) fusion-driven nexus contrastive learning (FNCL) for cross-modal alignment, and (4) cohesion-driven core contrastive learning (CCCL) for enhancing discriminative power. This hierarchical design effectively bridges the representation gap between complete and incomplete modality scenarios through dynamic prompt-based generation and multi-level contrastive optimization.
• Figure 2: The proposed Prompt Attention mechanism. It generates representations for a missing modality by feeding a multi-head attention module with a concatenation of available modality features and learnable, modality-specific prompts.
• Figure 3: Quantitative results and the chart on the N24News dataset with different missing rates under different missing-modality scenarios. Each data point on the figure represents that training with are the same 70% missing rate and testing are with $\eta$% missing rate.
• Figure 4: Quantitative results and the chart on the N24News, Hateful Memes, UPMC Food101 datasets with different missing rates. Each data point represents training and testing with the same missing rate $\eta$.
• Figure 5: Parameter sensitivity study on the number and length of prompts, using AUROC as evaluation metric on the Hateful Memes dataset with 70% missing rate in training and testing.