A Unified Framework for Emotion Recognition and Sentiment Analysis via Expert-Guided Multimodal Fusion with Large Language Models
Jiaqi Qiao, Xiujuan Xu, Xinran Li, Yu Liu
TL;DR
This work introduces EGMF, a unified framework for simultaneous emotion recognition and sentiment analysis that fuses text, audio, and visual modalities through three functionally specialized experts (fine-grained local, semantic correlation, global context) with hierarchical dynamic gating, and leverages large language models via pseudo token injection and prompt conditioning. The architecture achieves classification for ERC and regression for MSA by generating outputs from an LLM through a LoRA-tuned, parameter-efficient setup, enabling cross-lingual robustness across English and Chinese datasets. Comprehensive experiments on MELD, CHERMA, MOSEI, and SIMS-V2 demonstrate state-of-the-art performance and reveal that cross-lingual gains are particularly strong for Chinese, underscoring the effectiveness of adaptive multimodal fusion. The work also provides detailed ablation analyses to validate the contributions of each expert, the gating mechanism, and modality combinations, establishing a new paradigm for unified multimodal affective computing with practical implications for robust, multilingual emotion understanding systems.
Abstract
Multimodal emotion understanding requires effective integration of text, audio, and visual modalities for both discrete emotion recognition and continuous sentiment analysis. We present EGMF, a unified framework combining expert-guided multimodal fusion with large language models. Our approach features three specialized expert networks--a fine-grained local expert for subtle emotional nuances, a semantic correlation expert for cross-modal relationships, and a global context expert for long-range dependencies--adaptively integrated through hierarchical dynamic gating for context-aware feature selection. Enhanced multimodal representations are integrated with LLMs via pseudo token injection and prompt-based conditioning, enabling a single generative framework to handle both classification and regression through natural language generation. We employ LoRA fine-tuning for computational efficiency. Experiments on bilingual benchmarks (MELD, CHERMA, MOSEI, SIMS-V2) demonstrate consistent improvements over state-of-the-art methods, with superior cross-lingual robustness revealing universal patterns in multimodal emotional expressions across English and Chinese. We will release the source code publicly.