AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization

TL;DR

This work addresses the fragility of multimodal emotion understanding in LLMs, identifying spurious audiovisual cue associations and text-prior-driven hallucinations as core bottlenecks. It introduces EmoReAlM, a 4,000-question MCQA benchmark (4 tasks) built from 2,649 videos to assess emotion reasoning, modality agreement, and stress-test hallucinations, with rigorous automatic data creation and human verification. To mitigate the issues, the authors propose AVEm-DPO, a multimodal direct preference optimization method that (i) aligns outputs with the correct audiovisual input via prompt-based modality preferences, (ii) debiases text priors with a dedicated penalty, and (iii) leverages a large automatically generated preference dataset. Across EmoReAlM and existing emotion datasets (DFEW, RAVDESS, EMER), AVEm-DPO delivers 6–19% relative gains in zero-shot scenarios and demonstrates improved grounding, reduced spurious cues, and robustness to adversarial modality inputs, highlighting its potential for principled, interpretable social-AI emotion reasoning.

Abstract

Emotion understanding is essential for building socially intelligent agents. Although recent multimodal large language models have shown strong performance on this task, two key challenges remain - spurious associations between emotions and irrelevant audiovisual cues, and hallucinations of audiovisual cues driven by text priors in the language model backbone. To quantify and understand these issues, we introduce EmoReAlM, a benchmark designed to evaluate MLLMs for cue-emotion associations, hallucinations and modality agreement. We then propose AVEm-DPO, a preference optimization technique that aligns model responses with both audiovisual inputs and emotion-centric queries. Specifically, we construct preferences over responses exhibiting spurious associations or hallucinations, and audiovisual input pairs guided by textual prompts. We also include a regularization term that penalizes reliance on text priors, thereby mitigating modality-specific cue hallucinations. Experimental results on DFEW, RAVDESS and EMER demonstrate that our method significantly improves the performance of the reference baseline models with 6-19% of relative performance gains in zero-shot settings. By providing both a rigorous benchmark and a robust optimization framework, this work enables principled evaluation and improvement of MLLMs for emotion understanding and social AI. Code, models and benchmark will be released at https://avere-iclr.github.io.

Figures (37)

• Figure 1: Existing MLLMs (i) include spurious associations between AV cues and emotions -- reasoning errors (blue highlight) and (ii) hallucinate AV cues to explain emotions -- perception errors (red highlight). AV: audiovisual.
• Figure 2: EmoReAlM Tasks. In addition to basic emotion reasoning, we include tasks for Modality Agreement and Emotion Reasoning - Stress Test to test spurious cue-emotion associations and cue hallucinations. Red text is a hallucinated cue, blue text is an emotion-irrelevant cue and green text is a cue relevant for emotion understanding. Correct choices are underlined.
• Figure 3: EmoReAlM Creation Pipeline. We first disentangle the audiovisual information by separate captioning and verify the cues with text-based emotion prediction to find emotion-relevant cues. Finally, GPT-4o is used to generate MCQA samples that are later verified manually.
• Figure 4: Preference pairs in AVEm-DPO. (Top) Fine-grained preference over modality input based on current prompt. (Bottom) Each chosen response $y_w$ has two rejected responses -- $y_l^{vr}$ relevant to the video but with spurious emotion association and $y_l^{er}$ irrelevant to the video (hallucinated) but related to the emotion.
• Figure 5: Effect of AVEm-DPO on -- (Left two plots) the distribution of attention over video and audio tokens taken as a percentage over the total attention over all multimodal tokens for audio and visual reasoning tasks in EmoReAlM; (Right two plots) the log-likelihood distribution shift of the correct answer for visual reasoning tasks on corrupting the audio input $a_{ori}$ with adversary $a_{adv}$.