MCIHN: A Hybrid Network Model Based on Multi-path Cross-modal Interaction for Multimodal Emotion Recognition
Haoyang Zhang, Zhou Yang, Ke Sun, Yucai Pang, Guoliang Xu
TL;DR
The paper tackles the challenge of multimodal emotion recognition by addressing inter-modal differences and cross-modal interactions. It proposes MCIHN, a framework that combines per-modality adversarial autoencoders to learn discriminative latent features, a cross-modal gate mechanism to align and relate interacting modalities via an MMD-based objective, and a multi-head attention–based feature fusion module to produce robust joint representations. Empirical results on CH-SIMS and CMU-MOSI show state-of-the-art performance in both regression metrics (MAE, Corr) and classification-like metrics, with ablations confirming the essential roles of CGMM and full three-modality fusion. The approach offers a scalable blueprint for incorporating additional modalities in real-world human–computer interaction systems, highlighting the value of structured cross-modal interaction and adaptive fusion.
Abstract
Multimodal emotion recognition is crucial for future human-computer interaction. However, accurate emotion recognition still faces significant challenges due to differences between different modalities and the difficulty of characterizing unimodal emotional information. To solve these problems, a hybrid network model based on multipath cross-modal interaction (MCIHN) is proposed. First, adversarial autoencoders (AAE) are constructed separately for each modality. The AAE learns discriminative emotion features and reconstructs the features through a decoder to obtain more discriminative information about the emotion classes. Then, the latent codes from the AAE of different modalities are fed into a predefined Cross-modal Gate Mechanism model (CGMM) to reduce the discrepancy between modalities, establish the emotional relationship between interacting modalities, and generate the interaction features between different modalities. Multimodal fusion using the Feature Fusion module (FFM) for better emotion recognition. Experiments were conducted on publicly available SIMS and MOSI datasets, demonstrating that MCIHN achieves superior performance.