SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion Recognition

TL;DR

This work addresses the challenge of incomplete multimodal learning in Conversational Emotion Recognition (MERC) by introducing SDR-GNN, a graph neural network that integrates speaker- and context-driven hyperedges with a spectral-domain reconstruction mechanism. It preserves both high- and low-frequency information through a frequency-aware propagation scheme and multi-frequency aggregation, while reconstructing missing modalities via linear recovery and multi-head attention fusion, guided by the loss $\mathcal{L}=(1-e)\mathcal{L}_{ce}+e\mathcal{L}_{rec}$. Empirical results on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate state-of-the-art performance and robustness to increasing missing rates, with ablations confirming the value of the frequency-aware and hypergraph components. The approach provides a principled framework for recovering incomplete multimodal signals and improves emotion recognition accuracy, offering practical benefits for real-world MERC deployments where data can be partially missing.

Abstract

Multimodal Emotion Recognition in Conversations (MERC) aims to classify utterance emotions using textual, auditory, and visual modal features. Most existing MERC methods assume each utterance has complete modalities, overlooking the common issue of incomplete modalities in real-world scenarios. Recently, graph neural networks (GNNs) have achieved notable results in Incomplete Multimodal Emotion Recognition in Conversations (IMERC). However, traditional GNNs focus on binary relationships between nodes, limiting their ability to capture more complex, higher-order information. Moreover, repeated message passing can cause over-smoothing, reducing their capacity to preserve essential high-frequency details. To address these issues, we propose a Spectral Domain Reconstruction Graph Neural Network (SDR-GNN) for incomplete multimodal learning in conversational emotion recognition. SDR-GNN constructs an utterance semantic interaction graph using a sliding window based on both speaker and context relationships to model emotional dependencies. To capture higher-order and high-frequency information, SDR-GNN utilizes weighted relationship aggregation, ensuring consistent semantic feature extraction across utterances. Additionally, it performs multi-frequency aggregation in the spectral domain, enabling efficient recovery of incomplete modalities by extracting both high- and low-frequency information. Finally, multi-head attention is applied to fuse and optimize features for emotion recognition. Extensive experiments on various real-world datasets demonstrate that our approach is effective in incomplete multimodal learning and outperforms current state-of-the-art methods.

Figures (8)

• Figure 1: A toy example of complete multimodal features and incomplete multimodal features in conversation. Missing modalities pose a considerable challenge to capturing intra- and inter-modal semantic dependencies.
• Figure 2: The overall structure of the framework. First, we encode features of the utterance using a Bi-GRU to obtain the contextual embedding of each node. Then, we apply the SDR-GNN to capture features, jointly considering higher-order and multi-frequency information. Finally, we reconstruct the incomplete features and classify the emotion labels.
• Figure 3: Confusion matrices of the test set on IEMOCAP at varying missing rates. The matrices present the true labels along its rows and the predicted labels across its columns.
• Figure 4: Classification performance comparison between SDR-GNN and Lower bound under different missing rates.
• Figure 5: Reconstruction performance comparison between SDR-GNN and other methods under different missing rates. Lower MSE indicates better imputation performance.