Deep Metric Loss for Multimodal Learning

TL;DR

This work tackles the challenge that different modalities contribute variably across instances in multimodal learning. It introduces MultiModal loss, a proxy-based, modality-aware objective that uses multiple class proxies and soft attention to subgroup instances by unimodal contributions and preserve per-modality outputs. The approach yields improvements on synthetic data and four real multimodal datasets (RAVDESS, OPPORTUNITY, EPIC-KITCHENS, TCGA), with ablations showing the importance of soft attention and normalization. It also demonstrates faster convergence and richer, more reliable modality predictions, enhancing robustness and explainability in multimodal models. The method is applicable to intermediate/late fusion architectures and offers a practical path toward more trustworthy multimodal systems.

Abstract

Multimodal learning often outperforms its unimodal counterparts by exploiting unimodal contributions and cross-modal interactions. However, focusing only on integrating multimodal features into a unified comprehensive representation overlooks the unimodal characteristics. In real data, the contributions of modalities can vary from instance to instance, and they often reinforce or conflict with each other. In this study, we introduce a novel \text{MultiModal} loss paradigm for multimodal learning, which subgroups instances according to their unimodal contributions. \text{MultiModal} loss can prevent inefficient learning caused by overfitting and efficiently optimize multimodal models. On synthetic data, \text{MultiModal} loss demonstrates improved classification performance by subgrouping difficult instances within certain modalities. On four real multimodal datasets, our loss is empirically shown to improve the performance of recent models. Ablation studies verify the effectiveness of our loss. Additionally, we show that our loss generates a reliable prediction score for each modality, which is essential for subgrouping. Our \text{MultiModal} loss is a novel loss function to subgroup instances according to the contribution of modalities in multimodal learning and is applicable to a variety of multimodal models with unimodal decisions. Our code is available at https://github.com/SehwanMoon/MultiModalLoss.

Figures (10)

• Figure 1: Illustration of the proposed MultiModal loss. It generates multiple subgroups based on the classification contributions and difficulty of each modality. Circles with different colors represent instances of different classes. Each bar represents the probability for each class with the same color on three modalities, $m_1$, $m_2$, and $m_3$. Subgroup 1 is largely contributed by $m_2$, subgroup 2 is difficult to classify for all modalities, and subgroup 3 is easy on all modalities. Subgroup 4 is largely contributed by $m_1$ and has a higher probability for the blue class than the red class.
• Figure 1: (A) UMAP visualization of the two-dimensional embeddings of the learned outputs of each modality on TCGA is shown. (B) Examples of subgroups A, B, C and D for the LUSC class are shown, including proxies and instances. The size of the bars represents the relative probabilities of 34 cancer classes for four subgroups.
• Figure 2: Illustration of the differences between SoftTriple and MultiModal loss. SoftTriple minimizes the distances between instances and proxies in the same class based on distance (i.e. similarity). Otherwise, the distance is maximized. MultiModal loss additionally considers and preserves the characteristics of each modality ($m_1$ and $m_2$).
• Figure 2: UMAP visualization of the two-dimensional embeddings of the learned outputs from each modality on TCGA is shown. A, B, and C are the results of using MultiModal with normalization on proxy, MultiModal without Attention and MultiModal with normalization on proxy and Attention moved, respectively.
• Figure 3: Architecture of MultiModal loss. The darkness of the color represents the probability. The similarity $sim$ between the output$$x (as query) and the proxies$$w (as key) is determined, and the proxies are highlighted based on their similarity score. The predicted distribution over different classes is computed with the attended output$$A (sum of existing outputs and highlighted proxies) and the normalized similarities$$S obtained from each class.