Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Talk, Listen, Connect: How Humans and AI Evaluate Empathy in Responses to Emotionally Charged Narratives

Mahnaz Roshanaei, Rezvaneh Rezapour, Magy Seif El-Nasr

TL;DR

This study investigates how AI-generated responses and humans perceive and evoke empathy when reacting to emotionally charged personal narratives. By collecting human empathy ratings via MTurk and generating parallel AI responses with GPT-4o—varying prompts through persona attributes and targeted fine-tuning—the authors quantify alignment using multivariate statistics and distributional comparisons. Key findings show that GPT-4o tends to overestimate empathy, particularly cognitive empathy, but alignment with human judgments improves substantially when fine-tuning includes story content and reader-level attributes, especially perceived experience similarity. The work highlights important design and ethical considerations for empathetic AI, including the need to balance persuasive empathic responses with authenticity, avoid over-reliance, and ensure equitable empathy across diverse users and contexts.

Abstract

Social interactions promote well-being, yet barriers like geographic distance, time limitations, and mental health conditions can limit face-to-face interactions. Emotionally responsive AI systems, such as chatbots, offer new opportunities for social and emotional support, but raise critical questions about how empathy is perceived and experienced in human-AI interactions. This study examines how empathy is evaluated in AI-generated versus human responses. Using personal narratives, we explored how persona attributes (e.g., gender, empathic traits, shared experiences) and story qualities affect empathy ratings. We compared responses from standard and fine-tuned AI models with human judgments. Results show that while humans are highly sensitive to emotional vividness and shared experience, AI-responses are less influenced by these cues, often lack nuance in empathic expression. These findings highlight challenges in designing emotionally intelligent systems that respond meaningfully across diverse users and contexts, and informs the design of ethically aware tools to support social connection and well-being.

Talk, Listen, Connect: How Humans and AI Evaluate Empathy in Responses to Emotionally Charged Narratives

TL;DR

This study investigates how AI-generated responses and humans perceive and evoke empathy when reacting to emotionally charged personal narratives. By collecting human empathy ratings via MTurk and generating parallel AI responses with GPT-4o—varying prompts through persona attributes and targeted fine-tuning—the authors quantify alignment using multivariate statistics and distributional comparisons. Key findings show that GPT-4o tends to overestimate empathy, particularly cognitive empathy, but alignment with human judgments improves substantially when fine-tuning includes story content and reader-level attributes, especially perceived experience similarity. The work highlights important design and ethical considerations for empathetic AI, including the need to balance persuasive empathic responses with authenticity, avoid over-reliance, and ensure equitable empathy across diverse users and contexts.

Abstract

Social interactions promote well-being, yet barriers like geographic distance, time limitations, and mental health conditions can limit face-to-face interactions. Emotionally responsive AI systems, such as chatbots, offer new opportunities for social and emotional support, but raise critical questions about how empathy is perceived and experienced in human-AI interactions. This study examines how empathy is evaluated in AI-generated versus human responses. Using personal narratives, we explored how persona attributes (e.g., gender, empathic traits, shared experiences) and story qualities affect empathy ratings. We compared responses from standard and fine-tuned AI models with human judgments. Results show that while humans are highly sensitive to emotional vividness and shared experience, AI-responses are less influenced by these cues, often lack nuance in empathic expression. These findings highlight challenges in designing emotionally intelligent systems that respond meaningfully across diverse users and contexts, and informs the design of ethically aware tools to support social connection and well-being.
Paper Structure (26 sections, 7 figures, 8 tables)

This paper contains 26 sections, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Empathy in the Psychological Literature
  3. Empathy in the Era of AI
  4. Method
  5. Data
  6. Human-Generated Data
  7. LLM-Generated Data
  8. LLM-Generated Data with Persona
  9. Fine-tuning Experiment
  10. Analytical Strategy
  11. Results
  12. RQ1: Empathy Alignment in Human and GPT-4o Vanilla
  13. RQ2: Empathy Alignment in Human and GPT-4o Vanilla with Persona
  14. RQ3: Empathy Alignment in Human and GPT-4o Using Fine-Tuned Models
  15. RQ4: Empathy Evocation in Human and GPT-4o
  16. ...and 11 more sections

Figures (7)

  • Figure 1: An example of the human-generated story used in the surveys
  • Figure 4: Empathy Alignment in Human, compared to GPT-4o Vanilla and Fine-Tuned Models Pearson Correlation (left) and Wasserstein Distance (right)
  • Figure : (a) Empathy
  • Figure : (a) Empathy
  • Figure S.1: Prompt used for generating reponses using GPT-4o (base and fine-tuned).
  • ...and 2 more figures