Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empathy Applicability Modeling for General Health Queries

Shan Randhawa, Agha Ali Raza, Kentaro Toyama, Julie Hui, Mustafa Naseem

TL;DR

The paper introduces the Empathy Applicability Framework (EAF) to proactively identify when and what type of clinical empathy is warranted for general health queries, addressing the gap in anticipatory empathy modeling. It frames empathy along two dimensions—Emotional Reactions and Interpretations—labeled as Applicable or Not Applicable based on clinical, contextual, and linguistic cues, enabling pre-response guidance for empathy in asynchronous care. A novel benchmark of 1,300 queries annotated by humans and GPT-4o demonstrates that EAF yields reliable and learnable signals, with RoBERTa-based classifiers achieving strong performance and GPT-aligned reasoning across dimensions. The work reveals meaningful human-GPT alignment while uncovering systematic challenges—subjectivity in inferred distress, clinical-severity ambiguity, and contextual hardship—that motivate multi-annotator and culturally sensitive modeling. Overall, EAF provides a practical, theory-grounded step toward anticipatory empathy in NLP for clinical and general-health contexts, supporting empathetic communication with clinician supervision and safeguards.

Abstract

LLMs are increasingly being integrated into clinical workflows, yet they often lack clinical empathy, an essential aspect of effective doctor-patient communication. Existing NLP frameworks focus on reactively labeling empathy in doctors' responses but offer limited support for anticipatory modeling of empathy needs, especially in general health queries. We introduce the Empathy Applicability Framework (EAF), a theory-driven approach that classifies patient queries in terms of the applicability of emotional reactions and interpretations, based on clinical, contextual, and linguistic cues. We release a benchmark of real patient queries, dual-annotated by Humans and GPT-4o. In the subset with human consensus, we also observe substantial human-GPT alignment. To validate EAF, we train classifiers on human-labeled and GPT-only annotations to predict empathy applicability, achieving strong performance and outperforming the heuristic and zero-shot LLM baselines. Error analysis highlights persistent challenges: implicit distress, clinical-severity ambiguity, and contextual hardship, underscoring the need for multi-annotator modeling, clinician-in-the-loop calibration, and culturally diverse annotation. EAF provides a framework for identifying empathy needs before response generation, establishes a benchmark for anticipatory empathy modeling, and enables supporting empathetic communication in asynchronous healthcare.

Empathy Applicability Modeling for General Health Queries

TL;DR

The paper introduces the Empathy Applicability Framework (EAF) to proactively identify when and what type of clinical empathy is warranted for general health queries, addressing the gap in anticipatory empathy modeling. It frames empathy along two dimensions—Emotional Reactions and Interpretations—labeled as Applicable or Not Applicable based on clinical, contextual, and linguistic cues, enabling pre-response guidance for empathy in asynchronous care. A novel benchmark of 1,300 queries annotated by humans and GPT-4o demonstrates that EAF yields reliable and learnable signals, with RoBERTa-based classifiers achieving strong performance and GPT-aligned reasoning across dimensions. The work reveals meaningful human-GPT alignment while uncovering systematic challenges—subjectivity in inferred distress, clinical-severity ambiguity, and contextual hardship—that motivate multi-annotator and culturally sensitive modeling. Overall, EAF provides a practical, theory-grounded step toward anticipatory empathy in NLP for clinical and general-health contexts, supporting empathetic communication with clinician supervision and safeguards.

Abstract

LLMs are increasingly being integrated into clinical workflows, yet they often lack clinical empathy, an essential aspect of effective doctor-patient communication. Existing NLP frameworks focus on reactively labeling empathy in doctors' responses but offer limited support for anticipatory modeling of empathy needs, especially in general health queries. We introduce the Empathy Applicability Framework (EAF), a theory-driven approach that classifies patient queries in terms of the applicability of emotional reactions and interpretations, based on clinical, contextual, and linguistic cues. We release a benchmark of real patient queries, dual-annotated by Humans and GPT-4o. In the subset with human consensus, we also observe substantial human-GPT alignment. To validate EAF, we train classifiers on human-labeled and GPT-only annotations to predict empathy applicability, achieving strong performance and outperforming the heuristic and zero-shot LLM baselines. Error analysis highlights persistent challenges: implicit distress, clinical-severity ambiguity, and contextual hardship, underscoring the need for multi-annotator modeling, clinician-in-the-loop calibration, and culturally diverse annotation. EAF provides a framework for identifying empathy needs before response generation, establishes a benchmark for anticipatory empathy modeling, and enables supporting empathetic communication in asynchronous healthcare.
Paper Structure (54 sections, 9 figures, 7 tables)

This paper contains 54 sections, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Empathy Applicability Framework and Theoretical Grounding
  3. Methods
  4. Data Source
  5. Annotation Task
  6. Annotator Recruitment, Training and Calibration
  7. GPT Annotations
  8. Modeling Task and Approach
  9. Evaluation Setup and Experiments
  10. Annotator Agreement
  11. Conceptual Alignment
  12. Divergence Bar and Qualitative Analysis
  13. Model Evaluation
  14. Model Training and Training Sets
  15. Results
  16. ...and 39 more sections

Figures (9)

  • Figure 1: UpSet plots comparing GPT and human rationales for (a) IA and (b) EA subcategories. For each query, each human annotator selects one best-fit subcategory for their rationale; thus the human set is either a single-dot combination (both humans chose the same subcategory) or a two-dot combination (humans chose different subcategories). Horizontal bars show how often each subcategory appears in the human annotation set across all queries. Each vertical bar shows the frequency of a unique human-combination and is split by GPT agreement: Full (GPT’s subcategory set covers the entire human set), Partial (GPT matches only one of the two human subcategories), and No match (GPT matches neither human subcategory).
  • Figure 2: Three-way divergence for every subcategory. Orange = Annotator Spread in Humans (One Applicable, other not); Blue = LLM‑Adds Empathy Dimension (GPT Applicable, Humans Not); Green = LLM‑Omits Empathy Dimension (GPT Not, Humans Applicable).
  • Figure 3: Screenshot of the annotation spreadsheet provided to annotators. The header shows the instructions and links to the framework document.
  • Figure 4: Empathy Dimension Applicability Model Architecture
  • Figure 5: Dataset overview panel: base rates, EA--IA coupling, and query length distributions. Panels (a--b) report the binary label base rates for Emotional Reactions (EA) and Interpretations (IA) from Human Annotator 1 (HA1), Human Annotator 2 (HA2), and GPT. Bars are shown as stacked proportions of Applicable vs. Not Applicable. Panels (c--d) show EA$\times$IA co-occurrence as 2$\times$2 heatmaps for (c) Human consensus (only items where HA1 and HA2 agree on both EA and IA) and (d) Majority consensus (majority vote over HA1, HA2, and GPT), with each cell annotated by the percentage of items in that consensus subset; Panels (e--f) summarize query length: (e) token-count histogram (simple word tokenization) and (f) character-count histogram; vertical reference lines mark the mean (solid red) and median (dashed black). Together, the figure summarizes label prevalence, the empirical coupling between EA and IA decisions, and the distribution of textual input lengths in patient queries.
  • ...and 4 more figures