Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EMPA: Evaluating Persona-Aligned Empathy as a Process

Shiya Zhang, Yuhan Zhan, Ruixi Su, Ruihan Sun, Ziyi Song, Zhaohan Chen, Xiaofan Zhang

TL;DR

EMPA is introduced, a process-oriented framework that evaluates persona-aligned support as sustained intervention rather than isolated replies and distills real interactions into controllable, psychologically grounded scenarios, couples them with an open-ended multi-agent sandbox that exposes strategic adaptation and failure modes.

Abstract

Evaluating persona-aligned empathy in LLM-based dialogue agents remains challenging. User states are latent, feedback is sparse and difficult to verify in situ, and seemingly supportive turns can still accumulate into trajectories that drift from persona-specific needs. We introduce EMPA, a process-oriented framework that evaluates persona-aligned support as sustained intervention rather than isolated replies. EMPA distills real interactions into controllable, psychologically grounded scenarios, couples them with an open-ended multi-agent sandbox that exposes strategic adaptation and failure modes, and scores trajectories in a latent psychological space by directional alignment, cumulative impact, and stability. The resulting signals and metrics support reproducible comparison and optimization of long-horizon empathic behavior, and they extend to other agent settings shaped by latent dynamics and weak, hard-to-verify feedback.

EMPA: Evaluating Persona-Aligned Empathy as a Process

TL;DR

EMPA is introduced, a process-oriented framework that evaluates persona-aligned support as sustained intervention rather than isolated replies and distills real interactions into controllable, psychologically grounded scenarios, couples them with an open-ended multi-agent sandbox that exposes strategic adaptation and failure modes.

Abstract

Evaluating persona-aligned empathy in LLM-based dialogue agents remains challenging. User states are latent, feedback is sparse and difficult to verify in situ, and seemingly supportive turns can still accumulate into trajectories that drift from persona-specific needs. We introduce EMPA, a process-oriented framework that evaluates persona-aligned support as sustained intervention rather than isolated replies. EMPA distills real interactions into controllable, psychologically grounded scenarios, couples them with an open-ended multi-agent sandbox that exposes strategic adaptation and failure modes, and scores trajectories in a latent psychological space by directional alignment, cumulative impact, and stability. The resulting signals and metrics support reproducible comparison and optimization of long-horizon empathic behavior, and they extend to other agent settings shaped by latent dynamics and weak, hard-to-verify feedback.
Paper Structure (47 sections, 19 equations, 16 figures, 6 tables, 2 algorithms)

This paper contains 47 sections, 19 equations, 16 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. System Overview
  5. The Data Pipeline: Real-to-Sim Scenario Generation
  6. The Multi-Agent Simulation Environment
  7. Empathy Potential Model (EPM): A Psychodynamic Vector Formalism
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Ablation Study
  12. Summary
  13. Results
  14. Dataset
  15. Overall Performance
  16. ...and 32 more sections

Figures (16)

  • Figure 1: Implicit Assumptions in Agent Evaluation and Their Breakdown in Empathy-Oriented Psychological Support
  • Figure 2: shows a real EMPA sandbox interaction, revealing a failure of scalar empathy evaluation: high magnitude scores without directional alignment lead to ineffective support, encouraging verbose but misaligned responses.
  • Figure 3: Overview of EMPA. Real affective interaction data are distilled into psychologically consistent user profiles and crisis scenarios. The evaluated model then engages in unscripted, multi-turn interaction with user agents endowed with persona and long-term memory. Empathic behavior is finally quantified from the resulting interaction trajectories using EPM.
  • Figure 4: These dimensions provide an operational basis for analyzing LLM behavior in human–AI interaction and user experience.
  • Figure 5: Sensitivity and robustness under controlled perturbations
  • ...and 11 more figures