EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models

TL;DR

EmotionHallucer introduces the first benchmark dedicated to emotion hallucinations in multimodal LLMs, decomposing evaluation into emotion psychology knowledge and real-world multimodal perception. It employs an adversarial QA framework with basic and hallucinated pairs to rigorously assess detection capabilities across 38 models, revealing pervasive emotion hallucinations and a gap between knowledge-grounded and multimodal perception reasoning. The study finds closed-source models generally outperform open-source ones in detection and that emotion psychology knowledge tasks are easier than multimodal perception tasks, while introducing PEP-MEK, a plug-and-play framework that improves detection by about 9.9% on average. Together, these contributions illuminate current limits in emotion-aware, multimodal reasoning and offer a practical mitigation strategy to bolster robustness in future MLLMs.

Abstract

Emotion understanding is a critical yet challenging task. Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities in this area. However, MLLMs often suffer from hallucinations, generating irrelevant or nonsensical content. To the best of our knowledge, despite the importance of this issue, there has been no dedicated effort to evaluate emotion-related hallucinations in MLLMs. In this work, we introduce EmotionHallucer, the first benchmark for detecting and analyzing emotion hallucinations in MLLMs. Unlike humans, whose emotion understanding stems from the interplay of biology and social learning, MLLMs rely solely on data-driven learning and lack innate emotional instincts. Fortunately, emotion psychology provides a solid foundation of knowledge about human emotions. Building on this, we assess emotion hallucinations from two dimensions: emotion psychology knowledge and real-world multimodal perception. To support robust evaluation, we utilize an adversarial binary question-answer (QA) framework, which employs carefully crafted basic and hallucinated pairs to assess the emotion hallucination tendencies of MLLMs. By evaluating 38 LLMs and MLLMs on EmotionHallucer, we reveal that: i) most current models exhibit substantial issues with emotion hallucinations; ii) closed-source models outperform open-source ones in detecting emotion hallucinations, and reasoning capability provides additional advantages; iii) existing models perform better in emotion psychology knowledge than in multimodal emotion perception. As a byproduct, these findings inspire us to propose the PEP-MEK framework, which yields an average improvement of 9.90% in emotion hallucination detection across selected models. Resources will be available at https://github.com/xxtars/EmotionHallucer.

Figures (24)

• Figure 1: Emotion understanding differences and the EmotionHallucer. (a) The distinction between how humans and MLLMs understand emotions. Based on the component process model scherer2009dynamic and dynamic systems approach lewis2005bridging, human emotion understanding involves dynamic interactions among cognitive appraisals, physiological changes, feelings, and behaviors. In contrast, MLLMs rely on data-driven learning from external behavioral cues, which limits their ability to accurately infer the underlying emotional states. (b) EmotionHallucer is organized along two main dimensions, Emotion Knowledge and Multimodality Perception, and includes seven subcategories across four modalities.
• Figure 2: Example tasks in EmotionHallucer. Each pairs consists of a basic question, used to test the basic ability of MLLMs, and a hallucinated question, containing hallucinated content to evaluate the models' ability to detect hallucination. Emotion Knowledge Hallucination targets emotion psychology knowledge scherer2009dynamiclewis2005bridgingshiota2017emotion, whereas Multimodality Perception Hallucination centers on real-world emotion understanding deng2023soulzhang2018adaptivelivingstone2018ryersonlian2023merzadeh2019socialsocialiq2github. More details could be seen in Appendix\ref{['app:examples']}.
• Figure 3: Word cloud of EmotionHallucer.
• Figure 4: Dataset statistics of EmotionHallucer.
• Figure 5: Unimodal performance of partial selected models. T, I, A, V/S, and V/L stand for Text, Image, Audio, Short Video, and Long Video, respectively.Additional models and implementation details are provided in the Appendix\ref{['app:experiment']}.