Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue

TL;DR

This work addresses the challenge of empathetic dialogue in end-to-end SLMs by replacing sole reliance on ground-truth responses and scalar preferences with descriptive empathy signals. It introduces EmpathyEval, a descriptive natural-language evaluation model, and ReEmpathy, an end-to-end SLM that uses empathetic self-reflective alternating inference to steer responses through ongoing reflective reasoning. A newly constructed Mandarin empathetic dialogue dataset of 18,000 samples supports descriptive evaluation and training, with EmpathyEval augmented by regression heads to predict four empathy dimensions $E_{NS}$, $E_{WA}$, $E_{EU}$, and $E_{ES}$. Experiments show ReEmpathy improves empathy-sensitive dialogue and ablations confirm the contribution of reflective reasoning, offering a path toward more emotionally intelligent and paralinguistic-aware human-computer interactions.

Abstract

End-to-end Spoken Language Models (SLMs) hold great potential for paralinguistic perception, and numerous studies have aimed to enhance their capabilities, particularly for empathetic dialogue. However, current approaches largely depend on rigid supervised signals, such as ground-truth response in supervised fine-tuning or preference scores in reinforcement learning. Such reliance is fundamentally limited for modeling complex empathy, as there is no single "correct" response and a simple numerical score cannot fully capture the nuances of emotional expression or the appropriateness of empathetic behavior. To address these limitations, we sequentially introduce EmpathyEval, a descriptive natural-language-based evaluation model for assessing empathetic quality in spoken dialogues. Building upon EmpathyEval, we propose ReEmpathy, an end-to-end SLM that enhances empathetic dialogue through a novel Empathetic Self-Reflective Alternating Inference mechanism, which interleaves spoken response generation with free-form, empathy-related reflective reasoning. Extensive experiments demonstrate that ReEmpathy substantially improves empathy-sensitive spoken dialogue by enabling reflective reasoning, offering a promising approach toward more emotionally intelligent and empathy-aware human-computer interactions.

Figures (10)

• Figure 1: Distribution of dialogue emotion labels and average rating score ranges.
• Figure 2: Distribution of dialogue audio durations and assessment word counts.
• Figure 3: Overview of the proposed EmpathyEval and ReEmpathy framework. The upper-left shows the EmpathyEval architecture and its two modes: text-based and score-based assessment. The lower-left illustrates the construction of off-policy response--reflection alternating training data. The right depicts the alternating inference mechanism of ReEmpathy, where reflective tokens are periodically generated to encourage paralinguistic and empathetic reasoning.
• Figure 4: Illustration of adjusting self-attention weights between responses and reflections alternating.
• Figure 5: A/B test results comparing ReEmpathy with other optimization approaches with GPT-4 + E2V.