Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AcoustEmo: Open-Vocabulary Emotion Reasoning via Utterance-Aware Acoustic Q-Former

Liyun Zhang, Xuanmeng Sha, Shuqiong Wu, Fengkai Liu

Abstract

Multimodal Large Language Models (MLLMs) excel in Open-Vocabulary (OV) emotion recognition but often neglect fine-grained acoustic modeling. Existing methods typically use global audio encoders, failing to capture subtle, local temporal dynamics like micro-prosody and intonation shifts within individual utterances. To address this, we propose AcoustEmo, a time-sensitive MLLM featuring a novel Utterance-Aware Acoustic Q-Former. Our approach utilizes a timestamp-synchronized sliding window to dynamically extract segment-level audio tokens instead of coarse global representations. This enables the model to explicitly trace the temporal evolution of subtle acoustic clues and capture deep contextual dependencies in dialogues. Experiments on the Explainable Multimodal Emotion Recognition (EMER) task show that AcoustEmo significantly enhances complex emotion reasoning, outperforming baselines while maintaining robust contextual accuracy.

AcoustEmo: Open-Vocabulary Emotion Reasoning via Utterance-Aware Acoustic Q-Former

Abstract

Multimodal Large Language Models (MLLMs) excel in Open-Vocabulary (OV) emotion recognition but often neglect fine-grained acoustic modeling. Existing methods typically use global audio encoders, failing to capture subtle, local temporal dynamics like micro-prosody and intonation shifts within individual utterances. To address this, we propose AcoustEmo, a time-sensitive MLLM featuring a novel Utterance-Aware Acoustic Q-Former. Our approach utilizes a timestamp-synchronized sliding window to dynamically extract segment-level audio tokens instead of coarse global representations. This enables the model to explicitly trace the temporal evolution of subtle acoustic clues and capture deep contextual dependencies in dialogues. Experiments on the Explainable Multimodal Emotion Recognition (EMER) task show that AcoustEmo significantly enhances complex emotion reasoning, outperforming baselines while maintaining robust contextual accuracy.
Paper Structure (17 sections, 4 equations, 2 figures, 2 tables)

This paper contains 17 sections, 4 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Emotion Recognition in MLLMs
  5. Methodology
  6. Overall Architecture
  7. Timestamp-Synchronized Sliding Window
  8. Utterance-Aware Acoustic Q-Former
  9. Multimodal Alignment and LLM Reasoning
  10. Experiments
  11. Experimental Setup
  12. Baseline Models
  13. Quantitative Results
  14. Ablation Study
  15. Qualitative Analysis
  16. ...and 2 more sections

Figures (2)

  • Figure 1: The overall architecture of AcoustEmo. The multimodal pathways are strictly decoupled before late fusion. Crucially, the Utterance-Aware Acoustic Q-Former (center right) extracts local acoustic dynamics bounded by text timestamps, bypassing the limitations of purely global audio aggregation.
  • Figure 2: Qualitative comparison between the global audio encoder and our AcoustEmo. The baseline averages out the transient micro-prosodic shift (voice tremor) at the end of the utterance, resulting in a neutral prediction. In contrast, AcoustEmo explicitly isolates this local acoustic anomaly via the utterance-aware sliding window, accurately deducing the underlying anxious state.