Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmplifAI: a Fine-grained Dataset for Japanese Empathetic Medical Dialogues in 28 Emotion Labels

Wan Jou She, Lis Kanashiro Pereira, Fei Cheng, Sakiko Yahata, Panote Siriaraya, Eiji Aramaki

TL;DR

EmplifAI introduces a Japanese empathetic dialogue dataset tailored to chronic medical condition management, grounded in 28 fine-grained emotions adapted from GoEmotions. The dataset comprises 280 situation anchors and 4,125 two-turn dialogues, created via crowdsourcing with expert review and filtered for empathetic toxicity. The authors validate the emotion taxonomy with reverse-engineering using multiple LLMs and demonstrate that fine-tuning a Japanese LLM on EmplifAI improves fluency and emotion-specific empathy, while also exploring LLM-as-a-Judge for evaluation and comparing it to human judgments. The work highlights practical implications for patient-facing AI in low-resource languages and discusses safety, harms, and the limits of LLM-based evaluation in empathetic contexts.

Abstract

This paper introduces EmplifAI, a Japanese empathetic dialogue dataset designed to support patients coping with chronic medical conditions. They often experience a wide range of positive and negative emotions (e.g., hope and despair) that shift across different stages of disease management. EmplifAI addresses this complexity by providing situation-based dialogues grounded in 28 fine-grained emotion categories, adapted and validated from the GoEmotions taxonomy. The dataset includes 280 medically contextualized situations and 4125 two-turn dialogues, collected through crowdsourcing and expert review. To evaluate emotional alignment in empathetic dialogues, we assessed model predictions on situation--dialogue pairs using BERTScore across multiple large language models (LLMs), achieving F1 scores of 0.83. Fine-tuning a baseline Japanese LLM (LLM-jp-3.1-13b-instruct4) with EmplifAI resulted in notable improvements in fluency, general empathy, and emotion-specific empathy. Furthermore, we compared the scores assigned by LLM-as-a-Judge and human raters on dialogues generated by multiple LLMs to validate our evaluation pipeline and discuss the insights and potential risks derived from the correlation analysis.

EmplifAI: a Fine-grained Dataset for Japanese Empathetic Medical Dialogues in 28 Emotion Labels

TL;DR

EmplifAI introduces a Japanese empathetic dialogue dataset tailored to chronic medical condition management, grounded in 28 fine-grained emotions adapted from GoEmotions. The dataset comprises 280 situation anchors and 4,125 two-turn dialogues, created via crowdsourcing with expert review and filtered for empathetic toxicity. The authors validate the emotion taxonomy with reverse-engineering using multiple LLMs and demonstrate that fine-tuning a Japanese LLM on EmplifAI improves fluency and emotion-specific empathy, while also exploring LLM-as-a-Judge for evaluation and comparing it to human judgments. The work highlights practical implications for patient-facing AI in low-resource languages and discusses safety, harms, and the limits of LLM-based evaluation in empathetic contexts.

Abstract

This paper introduces EmplifAI, a Japanese empathetic dialogue dataset designed to support patients coping with chronic medical conditions. They often experience a wide range of positive and negative emotions (e.g., hope and despair) that shift across different stages of disease management. EmplifAI addresses this complexity by providing situation-based dialogues grounded in 28 fine-grained emotion categories, adapted and validated from the GoEmotions taxonomy. The dataset includes 280 medically contextualized situations and 4125 two-turn dialogues, collected through crowdsourcing and expert review. To evaluate emotional alignment in empathetic dialogues, we assessed model predictions on situation--dialogue pairs using BERTScore across multiple large language models (LLMs), achieving F1 scores of 0.83. Fine-tuning a baseline Japanese LLM (LLM-jp-3.1-13b-instruct4) with EmplifAI resulted in notable improvements in fluency, general empathy, and emotion-specific empathy. Furthermore, we compared the scores assigned by LLM-as-a-Judge and human raters on dialogues generated by multiple LLMs to validate our evaluation pipeline and discuss the insights and potential risks derived from the correlation analysis.
Paper Structure (39 sections, 1 figure, 7 tables)

This paper contains 39 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Three major limitations in existing empathy datasets
  3. Related Work
  4. English Empathy Datasets
  5. Japanese Empathy Datasets
  6. Emotion taxonomy
  7. Building the EmplifAI Dataset
  8. Emotion Taxonomy Translation
  9. Dialogue Formatting
  10. Task Set-up and Data Collection System Development
  11. Manually Review & Filtering Harmful Conversations
  12. EmplifAI Dataset Statistics
  13. Emotion Taxonomy Validity Evaluation
  14. Evaluation Models and Metrics Selection
  15. Emotion Prediction Results and Findings
  16. ...and 24 more sections

Figures (1)

  • Figure 1: Samples of the conversation shown to the workers in the data collection system