Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoning

Zhixian Zhao, Wenjie Tian, Xiaohai Tian, Jun Zhang, Lei Xie

TL;DR

This work targets robust multimodal emotion reasoning by addressing uni-modal dominance in Multimodal LLMs. It introduces SABER, a 600k-clip dataset with a six-dimension annotation schema, and SABER-LLM, a two-stage framework combining Structured Evidence Decomposition $P(Y|X,I)=P(E_v|X,I)\cdot P(E_a|E_v,X,I)\cdot P(R|E_v,E_a,X,I)$ and Consistency-Aware Preference Alignment via CA-DPO to ground reasoning in explicit evidence. Empirical results on EMER, EmoBench-M, and SABER-Test show state-of-the-art open-source performance and robustness approaching closed-source models like Gemini-2.5-Pro, including strong results under cross-modal conflicts. The approach advances fine-grained perception in affective reasoning and provides a scalable pipeline for high-fidelity audiovisual grounding in social-emotional analytics.

Abstract

Multimodal emotion analysis is shifting from static classification to generative reasoning. Beyond simple label prediction, robust affective reasoning must synthesize fine-grained signals such as facial micro-expressions and prosodic which shifts to decode the latent causality within complex social contexts. However, current Multimodal Large Language Models (MLLMs) face significant limitations in fine-grained perception, primarily due to data scarcity and insufficient cross-modal fusion. As a result, these models often exhibit unimodal dominance which leads to hallucinations in complex multimodal interactions, particularly when visual and acoustic cues are subtle, ambiguous, or even contradictory (e.g., in sarcastic scenery). To address this, we introduce SABER-LLM, a framework designed for robust multimodal reasoning. First, we construct SABER, a large-scale emotion reasoning dataset comprising 600K video clips, annotated with a novel six-dimensional schema that jointly captures audiovisual cues and causal logic. Second, we propose the structured evidence decomposition paradigm, which enforces a "perceive-then-reason" separation between evidence extraction and reasoning to alleviate unimodal dominance. The ability to perceive complex scenes is further reinforced by consistency-aware direct preference optimization, which explicitly encourages alignment among modalities under ambiguous or conflicting perceptual conditions. Experiments on EMER, EmoBench-M, and SABER-Test demonstrate that SABER-LLM significantly outperforms open-source baselines and achieves robustness competitive with closed-source models in decoding complex emotional dynamics. The dataset and model are available at https://github.com/zxzhao0/SABER-LLM.

Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoning

TL;DR

This work targets robust multimodal emotion reasoning by addressing uni-modal dominance in Multimodal LLMs. It introduces SABER, a 600k-clip dataset with a six-dimension annotation schema, and SABER-LLM, a two-stage framework combining Structured Evidence Decomposition and Consistency-Aware Preference Alignment via CA-DPO to ground reasoning in explicit evidence. Empirical results on EMER, EmoBench-M, and SABER-Test show state-of-the-art open-source performance and robustness approaching closed-source models like Gemini-2.5-Pro, including strong results under cross-modal conflicts. The approach advances fine-grained perception in affective reasoning and provides a scalable pipeline for high-fidelity audiovisual grounding in social-emotional analytics.

Abstract

Multimodal emotion analysis is shifting from static classification to generative reasoning. Beyond simple label prediction, robust affective reasoning must synthesize fine-grained signals such as facial micro-expressions and prosodic which shifts to decode the latent causality within complex social contexts. However, current Multimodal Large Language Models (MLLMs) face significant limitations in fine-grained perception, primarily due to data scarcity and insufficient cross-modal fusion. As a result, these models often exhibit unimodal dominance which leads to hallucinations in complex multimodal interactions, particularly when visual and acoustic cues are subtle, ambiguous, or even contradictory (e.g., in sarcastic scenery). To address this, we introduce SABER-LLM, a framework designed for robust multimodal reasoning. First, we construct SABER, a large-scale emotion reasoning dataset comprising 600K video clips, annotated with a novel six-dimensional schema that jointly captures audiovisual cues and causal logic. Second, we propose the structured evidence decomposition paradigm, which enforces a "perceive-then-reason" separation between evidence extraction and reasoning to alleviate unimodal dominance. The ability to perceive complex scenes is further reinforced by consistency-aware direct preference optimization, which explicitly encourages alignment among modalities under ambiguous or conflicting perceptual conditions. Experiments on EMER, EmoBench-M, and SABER-Test demonstrate that SABER-LLM significantly outperforms open-source baselines and achieves robustness competitive with closed-source models in decoding complex emotional dynamics. The dataset and model are available at https://github.com/zxzhao0/SABER-LLM.
Paper Structure (17 sections, 3 equations, 3 figures, 5 tables)

This paper contains 17 sections, 3 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Multimodal Emotion Reasoning
  5. SABER Dataset
  6. Data Construction Pipeline
  7. Quality Assurance and Diagnostic Benchmark
  8. Method
  9. Structured Evidence Decomposition
  10. Consistency-Aware Preference Alignment
  11. Experiments
  12. Experimental Setup
  13. Comparison with State-of-the-Art Methods
  14. Robustness against Modal Inconsistency
  15. Ablation Study
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Fragmented vs. Holistic Emotion Reasoning. Top: Conventional methods suffer from uni-modal dominance, failing to reconcile contradictory cues (e.g., a smile masking a sharp tone) and misinterpreting the intent. Bottom: Holistic reasoning integrates fine-grained evidence to correctly decipher complex social dynamics (e.g., "Playful Teasing")
  • Figure 2: (a) Data Pipeline: A scalable construction process featuring six-dimensional fine-grained annotation and automated hallucination filtering. (b) Training Paradigm: The Structured Evidence Decomposition (SED) stage (Stage 1) enforces sequential "perceive-then-reason" grounding, followed by CA-DPO (Stage 2) to align reasoning with human preference in modality-conflicting scenarios.
  • Figure 3: Qualitative comparison on a 'Feigned Sincerity' case. While the baseline is misled by the visual smile into predicting "Concern," SABER-LLM correctly identifies the mismatch between the "unsmiling eyes" and the "probing tone," successfully deducing the underlying manipulative intent.