EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World

TL;DR

This work tackles the gap between generic empathy and genuine personalization in emotional support conversations by introducing EmoHarbor, a user-centric evaluation framework. It grounds evaluation in a User-as-a-Judge paradigm using a Chain-of-Agent architecture (User Thinker, User Talker, User Evaluator) to simulate a user's internal world across 100 profiles. The benchmark evaluates 20 LLMs over ten dimensions spanning affective understanding, adaptation, and safety, revealing that while models excel at empathetic language, they struggle to tailor responses to individual user contexts over multi-turn interactions. EmoHarbor offers a reproducible, scalable path for developing and assessing truly user-aware emotional support systems, while detailing limitations and ethical considerations including dataset release and participant safeguards.

Abstract

Current evaluation paradigms for emotional support conversations tend to reward generic empathetic responses, yet they fail to assess whether the support is genuinely personalized to users' unique psychological profiles and contextual needs. We introduce EmoHarbor, an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. EmoHarbor employs a Chain-of-Agent architecture that decomposes users' internal processes into three specialized roles, enabling agents to interact with supporters and complete assessments in a manner similar to human users. We instantiate this benchmark using 100 real-world user profiles that cover a diverse range of personality traits and situations, and define 10 evaluation dimensions of personalized support quality. Comprehensive evaluation of 20 advanced LLMs on EmoHarbor reveals a critical insight: while these models excel at generating empathetic responses, they consistently fail to tailor support to individual user contexts. This finding reframes the central challenge, shifting research focus from merely enhancing generic empathy to developing truly user-aware emotional support. EmoHarbor provides a reproducible and scalable framework to guide the development and evaluation of more nuanced and user-aware emotional support systems.

Figures (10)

• Figure 1: Comparison of different evaluation paradigms for assessing personalized emotional support. The evaluator-centric paradigm fails to perform subjective assessments on behalf of users, whereas the user-centric paradigm can more accurately capture the quality of personalized ESC systems.
• Figure 2: Overview of the EmoHarbor Benchmark framework. It adopts the User-as-a-Judge paradigm by simulating a user’s internal world through a Chain-of-Agent architecture.
• Figure 3: Demographic and personality coverage of the benchmark user profiles, spanning gender, age, personality types, and occupations. Together, these distributions highlight the diversity and representativeness of our dataset.
• Figure 4: Distribution of counseling problem scenarios.
• Figure 5: Pairwise human evaluation of User Simulator. 'One Agent' lacks the User Thinker, generating responses directly by the User Talker. The 'Simple Profile' uses only basic demographic and counseling attributes, excluding personality, preferences, and scenario scripts. $\blacksquare$ indicates 'EmoHarbor wins'.