Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding

TL;DR

Emotion-LLaMAv2 delivers an end-to-end multimodal emotion recognition and reasoning framework by incorporating a Conv-Attention pre-fusion module and a perception-to-cognition curriculum, enabling robust cross-modal emotion understanding. MMEVerse provides a large, standardized benchmark by aggregating twelve emotion datasets into synchronized tri-modal data with unified annotations, supporting scalable instruction tuning and evaluation. The approach yields state-of-the-art performance on both recognition and reasoning benchmarks, with ablation studies confirming the value of explicit pre-fusion, modality-aware alignment, and curriculum training. Together, they offer a scalable foundation for nuanced affective AI capable of interpretable cross-modal emotion reasoning in real-world settings.

Abstract

Understanding human emotions from multimodal signals poses a significant challenge in affective computing and human-robot interaction. While multimodal large language models (MLLMs) have excelled in general vision-language tasks, their capabilities in emotional reasoning remain limited. The field currently suffers from a scarcity of large-scale datasets with high-quality, descriptive emotion annotations and lacks standardized benchmarks for evaluation. Our preliminary framework, Emotion-LLaMA, pioneered instruction-tuned multimodal learning for emotion reasoning but was restricted by explicit face detectors, implicit fusion strategies, and low-quality training data with limited scale. To address these limitations, we present Emotion-LLaMAv2 and the MMEVerse benchmark, establishing an end-to-end pipeline together with a standardized evaluation setting for emotion recognition and reasoning. Emotion-LLaMAv2 introduces three key advances. First, an end-to-end multiview encoder eliminates external face detection and captures nuanced emotional cues via richer spatial and temporal multiview tokens. Second, a Conv Attention pre-fusion module is designed to enable simultaneous local and global multimodal feature interactions external to the LLM backbone. Third, a perception-to-cognition curriculum instruction tuning scheme within the LLaMA2 backbone unifies emotion recognition and free-form emotion reasoning. To support large-scale training and reproducible evaluation, MMEVerse aggregates twelve publicly available emotion datasets, including IEMOCAP, MELD, DFEW, and MAFW, into a unified multimodal instruction format. The data are re-annotated via a multi-agent pipeline involving Qwen2 Audio, Qwen2.5 VL, and GPT 4o, producing 130k training clips and 36k testing clips across 18 evaluation benchmarks.

Figures (7)

• Figure 1: Overall architecture of Emotion-LLaMAv2. The framework processes audio, visual, and textual inputs through a structured pipeline for multimodal emotion recognition and reasoning. Modality-specific encoders extract unimodal representations, which are integrated via a Conv-Attention pre-fusion module to model emotion-aware cross-modal interactions. The fused features are then aligned to the language model embedding space through a modal adapter, enabling a LoRA-tuned large language model to perform joint emotion recognition and multimodal reasoning.
• Figure 2: User Study on Data Quality.
• Figure 3: Ablation of temporal representations, mainly evaluated on MMEVerse-Bench. We analyze the model's performance as a function of: (a) number of audio tokens, (b) temporal sampling density (number of video frames), and (c) spatial granularity of each frame (number of visual tokens). We default to sampling 16 frames per video and 64 frames per audio.
• Figure 4: Ablation of MLLM architecture. Our Emotion-LLaMA-v2 is based on MiniGPT4v2, with LLaMA2 serving as its backbone. Qwen2.5-Omni and MiniCPM-o are natively supported audio, video, and text.
• Figure 5: Visualization of our Emotion-LLaMAv2 on multimodal emotion understanding tasks. The left side displays correct predictions with high confidence across various emotion tasks, while the right side shows error predictions.