Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis

TL;DR

Omni-Emotion tackles the challenge of multimodal emotion analysis by extending video MLLMs with dedicated audio and facial encoders, and by building high-quality emotion instruction datasets. The approach aligns audio, facial, and visual cues in a unified space and tunes the model through a three-phase instruction regime, yielding state-of-the-art performance in both open-vocabulary emotion recognition and multimodal emotion reasoning. The work contributes substantial: (1) SRE and HRE datasets for rich emotion annotations, (2) architectural integration of Whisper-large-v3 and FaceXFormer into a video MLLM, and (3) comprehensive experiments demonstrating robust gains across DFEW/MAFW and EMER benchmarks. These advances hold practical significance for real-world human-computer interaction, enabling more accurate and explainable emotion understanding in video content.

Abstract

Understanding emotions accurately is essential for fields like human-computer interaction. Due to the complexity of emotions and their multi-modal nature (e.g., emotions are influenced by facial expressions and audio), researchers have turned to using multi-modal models to understand human emotions rather than single-modality. However, current video multi-modal large language models (MLLMs) encounter difficulties in effectively integrating audio and identifying subtle facial micro-expressions. Furthermore, the lack of detailed emotion analysis datasets also limits the development of multimodal emotion analysis. To address these issues, we introduce a self-reviewed dataset and a human-reviewed dataset, comprising 24,137 coarse-grained samples and 3,500 manually annotated samples with detailed emotion annotations, respectively. These datasets allow models to learn from diverse scenarios and better generalize to real-world applications. Moreover, in addition to the audio modeling, we propose to explicitly integrate facial encoding models into the existing advanced Video MLLM, enabling the MLLM to effectively unify audio and the subtle facial cues for emotion understanding. By aligning these features within a unified space and employing instruction tuning in our proposed datasets, our Omni-Emotion achieves state-of-the-art performance in both emotion recognition and reasoning tasks.

Figures (6)

• Figure 1: An example from MAFW liu2022mafw with our self-reviewed annotation shows the integration of multimodal explainable reasons for emotion analysis, involving background information, detailed facial expressions, and audio cues. Besides, the open vocabulary labels provide a more accurate description of the character's emotions in the video.
• Figure 2: The processing pipeline of our proposed datasets consists of three aspects, including the extraction of global visual information, fine-grained facial details, and audio cues. First, we conduct sparse sampling at 1 FPS for each video, using Qwen2-VL to capture global visual information. For detailed facial analysis, a face detector and tracking method are employed to extract face tracklets, followed by dense sampling and analysis using an age/gender estimator and Qwen2-VL. Lastly, we use Whisper-large-v3 to analyze the audio cues. We then employ GPT-3.5 to ensure consistency and accuracy, discarding descriptions that do not align with other clues. Based on these results, we extract explainable reasons, open vocabulary descriptions, and intensity for emotion of each video. Finally, we utilize GPT-3.5 to review the generated data referring to existing coarse-grained classification labels and obtain the self-reviewed emotion (SRE) dataset. We select partial data from SRE dataset and perform manual verification to build the human-reviewed emotion (HRE) dataset.
• Figure 3: Statistics of the GPT-based scores assessing the alignment between emotion reasoning descriptions and ground-truth classification labels on the MAFW liu2022mafw and DFEW jiang2020dfew datasets. The scores range from 0 to 10, with lower scores indicating greater deviation from the ground truth. Samples scoring below 5 are considered misaligned.
• Figure 4: The distribution of sample sizes across several emotion recognition datasets. FERV39K has the largest portion with 10,302 samples, followed by MAFW with 5,006. DFEW and CAER contribute 3,700 and 2,903 samples, respectively. Meanwhile, MER24 and AFEW-VA add 1,050 and 1,176 samples, respectively, providing a diverse distribution for emotion analysis.
• Figure 5: Illustration of our proposed Omni-Emotion MLLM. Our model includes an LLM decoder, a general vision encoder for extracting visual clues, a facial encoder for capturing fine-grained facial information, and an audio encoder for processing auditory input. Our Omni-Emotion MLLM can leverage facial and audio details to better comprehend human emotion.