ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation

TL;DR

ES4R tackles the challenge of preserving affective cues in empathetic speech dialogue by introducing prepositive affective modeling prior to speech encoding. It combines intra-turn and inter-turn attention to form structured affective context, then applies speech-guided cross-modal fusion with an LLM to generate empathetic text, followed by energy-aware TTS for speech synthesis. Across AvaMERG and zero-shot MELD tests, ES4R shows improved empathy, coherence, and expressiveness, with robust performance across backbones and clear ablations validating its components. This approach demonstrates that explicitly modeling affective context before encoding can substantially enhance multimodal empathetic dialogue systems and offers practical gains for real-world conversational AI.

Abstract

Empathetic speech dialogue requires not only understanding linguistic content but also perceiving rich paralinguistic information such as prosody, tone, and emotional intensity for affective understandings. Existing speech-to-speech large language models either rely on ASR transcription or use encoders to extract latent representations, often weakening affective information and contextual coherence in multi-turn dialogues. To address this, we propose \textbf{ES4R}, a framework for speech-based empathetic response generation. Our core innovation lies in explicitly modeling structured affective context before speech encoding, rather than relying on implicit learning by the encoder or explicit emotion supervision. Specifically, we introduce a dual-level attention mechanism to capture turn-level affective states and dialogue-level affective dynamics. The resulting affective representations are then integrated with textual semantics through speech-guided cross-modal attention to generate empathetic responses. For speech output, we employ energy-based strategy selection and style fusion to achieve empathetic speech synthesis. ES4R consistently outperforms strong baselines in both automatic and human evaluations and remains robust across different LLM backbones.

Figures (5)

• Figure 1: Comparison of speech dialogue system architectures. (a) Cascaded pipeline through ASR. (b) End-to-end latent representations. (c) Ours model.
• Figure 2: The overall framework of ES4R. Stage 1: Intra-Turn and Inter-Turn attention are used to prepositive affective modeling. Stage 2: Speech-guided cross-modal fusion is used for empathetic response generation. Stage 3: Energy-based strategy selection and style fusion are applied for speech synthesis.
• Figure 3: This figure shows multi-dimensional quality assessment across different model configurations.
• Figure 4: A/B human text evaluation criteria used for the five aspects reported in Table \ref{['tab:2']}
• Figure 5: A/B human speech evaluation criteria for speech responses (DMOS-Q/C/E).