Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning

Dingdong Wang, Shujie Liu, Tianhua Zhang, Youjun Chen, Jinyu Li, Helen Meng

TL;DR

EmotionThinker reframes speech emotion recognition as explainable multimodal reasoning by grounding predictions in prosody, semantics, and speaker traits. It introduces EmotionCoT-35K for reasoning supervision, a prosody-enhanced base model via Prosody-Centric SFT, and a Gradual Rewarded RL regime GRPO-PTR to supervise intermediate reasoning with a trust-based gating mechanism. Empirical results show state-of-the-art emotion accuracy and superior reasoning quality on standard SER benchmarks, validated by human judgments. This approach advances SER toward interpretable, ground-truthed multimodal reasoning with practical implications for human–computer interaction and affective AI systems.

Abstract

Emotional information in speech plays a unique role in multimodal perception. However, current Speech Large Language Models (SpeechLLMs), similar to conventional speech emotion recognition (SER) systems, still treat emotion understanding as a simple classification problem. This provides limited interpretability of predictions, while leaving the LLMs' expressive and reasoning capabilities underutilized. In this work, we take the first step to reformulate SER as a deep reasoning problem through reinforcement learning (RL). We propose EmotionThinker, which is designed to generate accurate emotion predictions with interpretable explanations grounded in fine-grained acoustic cues. To achieve this, we first construct EmotionCoT-35K, an emotional reasoning dataset with Chain-of-Thought annotations and detailed captions. Second, we observe that current SpeechLLMs exhibit weak prosody perception, whereas prosodic cues constitute fundamental signals for interpreting emotions. To address this, we develop the prosody-enhanced foundation model EmotionThinker-Base, and demonstrate that prosody enhancement improves emotion understanding. Third, we introduce Group-Relative-Policy-Optimization with Progressive-Trust-aware-Reasoning-Reward (GRPO-PTR) for RL. Different from standard GRPO, which relies only on rule-based outcome rewards, GRPO-PTR progressively introduces reasoning reward, dynamically adjusts it with a trustworthiness weight reflecting the alignment between reasoning and outcome, and evaluates the overall reasoning quality with a reward model based on multi-dimensional criteria. EmotionThinker outperforms previous state-of-the-art evaluation models both in emotion accuracy and explanation quality, advancing SER toward interpretable multimodal reasoning. Project page: https://github.com/dingdongwang/EmotionThinker

EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning

TL;DR

EmotionThinker reframes speech emotion recognition as explainable multimodal reasoning by grounding predictions in prosody, semantics, and speaker traits. It introduces EmotionCoT-35K for reasoning supervision, a prosody-enhanced base model via Prosody-Centric SFT, and a Gradual Rewarded RL regime GRPO-PTR to supervise intermediate reasoning with a trust-based gating mechanism. Empirical results show state-of-the-art emotion accuracy and superior reasoning quality on standard SER benchmarks, validated by human judgments. This approach advances SER toward interpretable, ground-truthed multimodal reasoning with practical implications for human–computer interaction and affective AI systems.

Abstract

Emotional information in speech plays a unique role in multimodal perception. However, current Speech Large Language Models (SpeechLLMs), similar to conventional speech emotion recognition (SER) systems, still treat emotion understanding as a simple classification problem. This provides limited interpretability of predictions, while leaving the LLMs' expressive and reasoning capabilities underutilized. In this work, we take the first step to reformulate SER as a deep reasoning problem through reinforcement learning (RL). We propose EmotionThinker, which is designed to generate accurate emotion predictions with interpretable explanations grounded in fine-grained acoustic cues. To achieve this, we first construct EmotionCoT-35K, an emotional reasoning dataset with Chain-of-Thought annotations and detailed captions. Second, we observe that current SpeechLLMs exhibit weak prosody perception, whereas prosodic cues constitute fundamental signals for interpreting emotions. To address this, we develop the prosody-enhanced foundation model EmotionThinker-Base, and demonstrate that prosody enhancement improves emotion understanding. Third, we introduce Group-Relative-Policy-Optimization with Progressive-Trust-aware-Reasoning-Reward (GRPO-PTR) for RL. Different from standard GRPO, which relies only on rule-based outcome rewards, GRPO-PTR progressively introduces reasoning reward, dynamically adjusts it with a trustworthiness weight reflecting the alignment between reasoning and outcome, and evaluates the overall reasoning quality with a reward model based on multi-dimensional criteria. EmotionThinker outperforms previous state-of-the-art evaluation models both in emotion accuracy and explanation quality, advancing SER toward interpretable multimodal reasoning. Project page: https://github.com/dingdongwang/EmotionThinker
Paper Structure (33 sections, 7 equations, 9 figures, 7 tables)

This paper contains 33 sections, 7 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SpeechLLMs for emotion understanding.
  4. Multimodal Large Reasoning Models.
  5. Method
  6. EmotionCoT-35K
  7. Prosody-Centric Supervised Fine-Tuning (SFT)
  8. Reinforcement Learning for Emotion Reasoning
  9. Rule-based Rewards
  10. Format reward.
  11. Outcome accuracy reward.
  12. GRPO-PTR: Progressive Trust-aware Reasoning Reward
  13. Experiment
  14. Experimental Setup
  15. Main Results
  16. ...and 18 more sections

Figures (9)

  • Figure 1: EmotionThinker generates emotion predictions with explanatory reasoning by leveraging speaker traits, prosody, semantics, and logic, in contrast to traditional models that provide only categorical outputs.
  • Figure 2: EmotionCoT-35K data curation pipeline.
  • Figure 3: Architecture of EmotionThinker with the proposed GRPO-PTR framework. The upper part depicts the high-level GRPO-PTR training pipeline, where only the policy model is optimized. The lower part details PTR strategy, which progressively introduces reasoning reward to stabilize training and enhance reasoning.
  • Figure 4: Case study comparing emotional reasoning across models on a sad utterance. Red highlights mark descriptions inconsistent with the ground truth, whereas green highlights indicate correct evidence.
  • Figure 5: Prompt template used to elicit emotion reasoning traces from GPT-4o.
  • ...and 4 more figures