Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMs

Sewon Kim, Jiwon Kim, Seungwoo Shin, Hyejin Chung, Daeun Moon, Yejin Kwon, Hyunsoo Yoon

TL;DR

The paper identifies Affective Hallucination as a distinct safety risk where LLMs simulate emotional presence in vulnerable conversations. It formalizes the risk along three dimensions, builds AHaBench to diagnose and AHaPairs to train via Direct Preference Optimization (DPO), and demonstrates that emotion-focused alignment can significantly reduce affective hallucination without harming factual or reasoning performance. Across multiple model families, DPO with AHaPairs achieves near-zero AHa rates and shows strong agreement with human judgments (r ≈ 0.85), while maintaining standard benchmarks. The work provides practical resources and a framework for developing emotionally safe, factually reliable LLMs, and reframes emotional alignment as responsible boundary-setting rather than unconditional empathy.

Abstract

Large Language Models (LLMs) are increasingly engaged in emotionally vulnerable conversations that extend beyond information seeking to moments of personal distress. As they adopt affective tones and simulate empathy, they risk creating the illusion of genuine relational connection. We term this phenomenon Affective Hallucination, referring to emotionally immersive responses that evoke false social presence despite the model's lack of affective capacity. To address this, we introduce AHaBench, a benchmark of 500 mental-health-related prompts with expert-informed reference responses, evaluated along three dimensions: Emotional Enmeshment, Illusion of Presence, and Fostering Overdependence. We further release AHaPairs, a 5K-instance preference dataset enabling Direct Preference Optimization (DPO) for alignment with emotionally responsible behavior. DPO fine-tuning substantially reduces affective hallucination without compromising reasoning performance, and the Pearson correlation coefficients between GPT-4o and human judgments is also strong (r=0.85) indicating that human evaluations confirm AHaBench as an effective diagnostic tool. This work establishes affective hallucination as a distinct safety concern and provides resources for developing LLMs that are both factually reliable and psychologically safe. AHaBench and AHaPairs are accessible via https://huggingface.co/datasets/o0oMiNGo0o/AHaBench, and code for fine-tuning and evaluation are in https://github.com/0oOMiNGOo0/AHaBench. Warning: This paper contains examples of mental health-related language that may be emotionally distressing.

Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMs

TL;DR

The paper identifies Affective Hallucination as a distinct safety risk where LLMs simulate emotional presence in vulnerable conversations. It formalizes the risk along three dimensions, builds AHaBench to diagnose and AHaPairs to train via Direct Preference Optimization (DPO), and demonstrates that emotion-focused alignment can significantly reduce affective hallucination without harming factual or reasoning performance. Across multiple model families, DPO with AHaPairs achieves near-zero AHa rates and shows strong agreement with human judgments (r ≈ 0.85), while maintaining standard benchmarks. The work provides practical resources and a framework for developing emotionally safe, factually reliable LLMs, and reframes emotional alignment as responsible boundary-setting rather than unconditional empathy.

Abstract

Large Language Models (LLMs) are increasingly engaged in emotionally vulnerable conversations that extend beyond information seeking to moments of personal distress. As they adopt affective tones and simulate empathy, they risk creating the illusion of genuine relational connection. We term this phenomenon Affective Hallucination, referring to emotionally immersive responses that evoke false social presence despite the model's lack of affective capacity. To address this, we introduce AHaBench, a benchmark of 500 mental-health-related prompts with expert-informed reference responses, evaluated along three dimensions: Emotional Enmeshment, Illusion of Presence, and Fostering Overdependence. We further release AHaPairs, a 5K-instance preference dataset enabling Direct Preference Optimization (DPO) for alignment with emotionally responsible behavior. DPO fine-tuning substantially reduces affective hallucination without compromising reasoning performance, and the Pearson correlation coefficients between GPT-4o and human judgments is also strong (r=0.85) indicating that human evaluations confirm AHaBench as an effective diagnostic tool. This work establishes affective hallucination as a distinct safety concern and provides resources for developing LLMs that are both factually reliable and psychologically safe. AHaBench and AHaPairs are accessible via https://huggingface.co/datasets/o0oMiNGo0o/AHaBench, and code for fine-tuning and evaluation are in https://github.com/0oOMiNGOo0/AHaBench. Warning: This paper contains examples of mental health-related language that may be emotionally distressing.

Paper Structure

This paper contains 50 sections, 1 equation, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Cognitive Safety
  4. Relational and Emotional Safety
  5. Affective Hallucination
  6. Definition
  7. Data Preprocessing
  8. Response Generation
  9. Evaluation Metric
  10. Preference Annotation
  11. Direct Preference Optimization
  12. Experiments
  13. Experimental Setup
  14. Results
  15. Fine-tuning Methods Comparison
  16. ...and 35 more sections

Figures (2)

  • Figure 1: Overview of the AHaBench pipeline. User prompts are adapted from the posts with added emotional dependency cues. Reference responses demonstrate emotionally safe engagement, while the highlighted LLM response shows affective hallucination by simulating relational closeness (e.g., “I'm here for you”). The evaluation framework systematically assesses such risks.
  • Figure 2: AHaPairs construction pipeline. A user query and system prompt are used to generate responses from multiple LLMs. Responses are evaluated on Harmlessness, Helpfulness, and Neutrality using GPT-4o, and pairwise preferences are derived for DPO training.