PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Models

TL;DR

This work identifies a fundamental limitation in existing RL-based empathy for LLMs: reward modeling often covers only a single perspective. It introduces Psychology-grounded Empathetic Reward Modeling (PERM), a bidirectional, multi-perspective framework grounded in the Empathy Cycle that jointly evaluates the supporter and seeker, plus a bystander to ensure interaction quality. Rewards from resilience, expression, reception, and bystander quality are combined to train LLMs via Group Relative Policy Optimization, leading to consistent improvements (>10%) on emotional intelligence benchmarks and strong user preferences (≈70%). The approach generalizes across backbone models, scales to real-world daily conversations, and is supported by extensive ablations and user studies; code and data are released for reproducibility.

Abstract

Large Language Models (LLMs) are increasingly deployed in human-centric applications, yet they often fail to provide substantive emotional support. While Reinforcement Learning (RL) has been utilized to enhance empathy of LLMs, existing reward models typically evaluate empathy from a single perspective, overlooking the inherently bidirectional interaction nature of empathy between the supporter and seeker as defined by Empathy Cycle theory. To address this limitation, we propose Psychology-grounded Empathetic Reward Modeling (PERM). PERM operationalizes empathy evaluation through a bidirectional decomposition: 1) Supporter perspective, assessing internal resonation and communicative expression; 2) Seeker perspective, evaluating emotional reception. Additionally, it incorporates a bystander perspective to monitor overall interaction quality. Extensive experiments on a widely-used emotional intelligence benchmark and an industrial daily conversation dataset demonstrate that PERM outperforms state-of-the-art baselines by over 10\%. Furthermore, a blinded user study reveals a 70\% preference for our approach, highlighting its efficacy in generating more empathetic responses. Our code, dataset, and models are available at https://github.com/ZhengWwwq/PERM.

Figures (12)

• Figure 1: An example illustrating the effectiveness of different reward modeling methods. Modeling only the LLM's responses leads to generic, surface-level expressions, while modeling only the user's needs yields solution-oriented but emotionally cold responses. By jointly considering both perspectives, PERM enables more authentic and comprehensive empathetic replies.
• Figure 2: Overview of PERM. We use red, blue, and purple to distinguish the empathy seeker, empathy supporter and bystander, respectively. PERM models rewards from these three perspectives and leverages the aggregated reward to optimize the LLM via RL methods.
• Figure 3: Evaluation results on the daily conversation benchmark. Higher values indicate better performance. "Avg." denotes the average score of all other dimensions.
• Figure 4: The changes in empathy judge score and bystander judge score during training under different $\lambda_\text{bys}$.
• Figure 5: Case study on EQ-Bench3. PERM enables the fine-tuned LLM to empathize with the person's psychological needs and respond with higher emotional intelligence to solve the problem more effectively.