DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities

TL;DR

This work tackles the robustness gap of Vision-Language Transformers when input modalities are missing. It introduces Decoupled Prototype Learning (DPL), a missing-aware prediction head that uses class-wise prototypes conditioned on complete, image-missing, text-missing, or mixed-missing configurations and decomposed into modality-specific components. Two losses, an adaptive ArcFace margin and a Prototype Relational Contrastive Loss, regularize both inter-class and intra-class prototype relations, enabling the model to adapt its decision boundary to the observed modalities without altering the pretrained backbone. Across MM-IMDb, UPMC Food-101, and Hateful Memes, DPL consistently improves performance over state-of-the-art prompt-based methods and FC heads, and remains compatible with existing prompt-tuning frameworks. This approach enhances practical robustness for multimodal systems operating under real-world data missingness scenarios.

Abstract

The performance of Visio-Language Transformers drops sharply when an input modality (e.g., image) is missing, because the model is forced to make predictions using incomplete information. Existing missing-aware prompt methods help reduce this degradation, but they still rely on conventional prediction heads (e.g., a Fully-Connected layer) that compute class scores in the same way regardless of which modality is present or absent. We introduce Decoupled Prototype Learning (DPL), a new prediction head architecture that explicitly adjusts its decision process to the observed input modalities. For each class, DPL selects a set of prototypes specific to the current missing-modality cases (image-missing, text-missing, or mixed-missing). Each prototype is then decomposed into image-specific and text-specific components, enabling the head to make decisions that depend on the information actually present. This adaptive design allows DPL to handle inputs with missing modalities more effectively while remaining fully compatible with existing prompt-based frameworks. Extensive experiments on MM-IMDb, UPMC Food-101, and Hateful Memes demonstrate that DPL outperforms state-of-the-art approaches across all widely used multimodal imag-text datasets and various missing cases.

Figures (5)

• Figure 1: Comparison between the conventional method (mid) and our proposed DPL framework (right) for multimodal classification under missing modalities. When some modalities are unavailable, the conventional method yield corrupted logits, whereas DPL uses missing-aware decoupling to mitigate the adverse impact of missing modalities.
• Figure 2: The workflow of DPL. Class-wise prototypes are decoupled through a missing-aware mechanism. After feature extraction by the frozen pretrained encoder (with or without learnable prompts), the resulting features are matched with their corresponding prototypes to compute logits for predicting class labels. The prototypes are then updated based on the losses $\mathcal{L}_{\text{ArcFace}}$ and $\mathcal{L}_{\text{SupCon}}$.
• Figure 3: F1-Macro comparison of MAP and DCP frameworks with different prediction heads (FC, DePT, DPL) on MM-IMDb under varying missing rates. The proposed DPL consistently outperforms others across all missing-modality scenarios: (a) both missing, (b) image missing, and (c) text missing.
• Figure 4: F1-Macro comparison of prediction heads (FC, DePT, DPL) integrated with the missing-aware DCP framework under three missing-modality settings: (a) both missing, (b) image missing, and (c) text missing. The proposed DPL achieves the highest F1-Macro across all cases, demonstrating superior robustness.
• Figure 5: Test accuracy under a 70% missing rate in a mixed-missing scenario (e.g., either image or text could be missing), showing the influence of $\lambda$, $m^{r_{I}}$ and $m^{r_{T}}$ with fixed $m^{c}$=0.15.