Turn-Level Empathy Prediction Using Psychological Indicators

TL;DR

The paper addresses turn-level empathy detection in empathetic dialogues by decomposing empathy into six psychology-inspired indicators and enriching utterances with GPT-4o-derived scores and explanations before training a DeBERTa classifier. The results show that psychological-indicator enrichment yields improvements in the Pearson correlation $r$ and $F_1$ over utterance-only baselines, with DeBERTa achieving $r\approx0.68$ and $F_1\approx0.35$. The study also analyzes per-indicator correlations, finding strong links for Emotional Language ($r\approx0.481$) and Sympathy/Compassion ($r\approx0.437$), while Extroversion is negatively related and Openness shows minimal impact. While GPT-4o provides interpretable diagnostics, its predictive performance lags behind the supervised model, highlighting the value of combining theory-driven features with fine-tuned architectures for reliable, interpretable empathy detection in conversational AI.

Abstract

For the WASSA 2024 Empathy and Personality Prediction Shared Task, we propose a novel turn-level empathy detection method that decomposes empathy into six psychological indicators: Emotional Language, Perspective-Taking, Sympathy and Compassion, Extroversion, Openness, and Agreeableness. A pipeline of text enrichment using a Large Language Model (LLM) followed by DeBERTA fine-tuning demonstrates a significant improvement in the Pearson Correlation Coefficient and F1 scores for empathy detection, highlighting the effectiveness of our approach. Our system officially ranked 7th at the CONV-turn track.

Figures (1)

• Figure 1: Prompt given to GPT-4o.