A Unified Spoken Language Model with Injected Emotional-Attribution Thinking for Human-like Interaction

TL;DR

The paper tackles the challenge of achieving semantic and emotional congruence in a single spoken language system. It introduces Injected Emotional-Attribution Thinking (IEAT) and a two-stage training regime on a GOAT-SLM backbone to align speech and text while internalizing emotional cues. Key contributions include a data construction pipeline with emotion labeling, internal reasoning injection, and end-to-end cross-modal optimization, leading to top performance on the HumDial Emotional Intelligence benchmark across tasks and languages. The work demonstrates the practicality and impact of emotion-aware internal cognition for more natural and robust multimodal human–AI interaction.

Abstract

This paper presents a unified spoken language model for emotional intelligence, enhanced by a novel data construction strategy termed Injected Emotional-Attribution Thinking (IEAT). IEAT incorporates user emotional states and their underlying causes into the model's internal reasoning process, enabling emotion-aware reasoning to be internalized rather than treated as explicit supervision. The model is trained with a two-stage progressive strategy. The first stage performs speech-text alignment and emotional attribute modeling via self-distillation, while the second stage conducts end-to-end cross-modal joint optimization to ensure consistency between textual and spoken emotional expressions. Experiments on the Human-like Spoken Dialogue Systems Challenge (HumDial) Emotional Intelligence benchmark demonstrate that the proposed approach achieves top-ranked performance across emotional trajectory modeling, emotional reasoning, and empathetic response generation under both LLM-based and human evaluations.

Figures (1)

• Figure 1: Overview of training procedure.