AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning

TL;DR

AM^2-EmoJE tackles emotion recognition in conversation with incomplete multimodal data (text, video, audio). It introduces two core ideas: Query Adaptive Fusion to learn modality importance per utterance and a Multimodal Joint Embedding to align cross-modal descriptors and compensate for missing modalities during inference. The method combines an Adaptive Cross-attended Network for fusion, a privacy-preserving Guided NCE loss, and a CLIP-like joint-embedding objective, trained on MELD and IEMOCAP. Empirically, it achieves improvements in weighted-F1 for complete data and substantial gains in missing-modality scenarios, demonstrating robustness for real-world ERC systems.

Abstract

Human emotion can be presented in different modes i.e., audio, video, and text. However, the contribution of each mode in exhibiting each emotion is not uniform. Furthermore, the availability of complete mode-specific details may not always be guaranteed in the test time. In this work, we propose AM^2-EmoJE, a model for Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning model that is grounded on two-fold contributions: First, a query adaptive fusion that can automatically learn the relative importance of its mode-specific representations in a query-specific manner. By this the model aims to prioritize the mode-invariant spatial query details of the emotion patterns, while also retaining its mode-exclusive aspects within the learned multimodal query descriptor. Second the multimodal joint embedding learning module that explicitly addresses various missing modality scenarios in test-time. By this, the model learns to emphasize on the correlated patterns across modalities, which may help align the cross-attended mode-specific descriptors pairwise within a joint-embedding space and thereby compensate for missing modalities during inference. By leveraging the spatio-temporal details at the dialogue level, the proposed AM^2-EmoJE not only demonstrates superior performance compared to the best-performing state-of-the-art multimodal methods, by effectively leveraging body language in place of face expression, it also exhibits an enhanced privacy feature. By reporting around 2-5% improvement in the weighted-F1 score, the proposed multimodal joint embedding module facilitates an impressive performance gain in a variety of missing-modality query scenarios during test time.

Figures (1)

• Figure 1: Proposed AM$^2$-EmoJE framework