Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning

Jingxiang Chen, Minseok Kim, Seong-Gyun Leem, Yin Huang, Rashi Rungta, Zhicheng Ouyang, Haibin Wu, Surya Teja Appini, Ankur Bansal, Yang Bai, Yue Liu, Florian Metze, Ahmed A Aly, Anuj Kumar, Ariya Rastrow, Zhaojiang Lin

Abstract

Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio) by 8-12% on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs.

Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning

Abstract

Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio) by 8-12% on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs.
Paper Structure (32 sections, 6 equations, 7 figures, 4 tables)

This paper contains 32 sections, 6 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Emotion Recognition and Paralinguistic Modeling
  4. Paralinguistic-Aware Dialogue Systems
  5. Methodology
  6. Task Formulation
  7. Sentiment Classification
  8. Paralinguistics-Aware Response Generation
  9. Two-Stage Training Pipeline
  10. Stage 1: Supervised Fine-Tuning
  11. (SFT) Sentiment Classification
  12. (SFT) Paralinguistics-Aware Response Generation
  13. Stage 2: Reinforcement Learning with Chain-of-Thought
  14. (RL) Sentiment Classification with CoT
  15. (RL) Paralinguistics-Aware Response Generation with Reasoning
  16. ...and 17 more sections

Figures (7)

  • Figure 1: Paralinguistics-Aware LLM stage 2 overview. A multi-task RL jointly performs sentiment classification and paralinguistics-aware response generation with chain-of-thought reasoning.
  • Figure 2: Response appropriateness instruction prompt.
  • Figure 3: Response appropriateness instruction prompt. (cont'd)
  • Figure 4: Instruction prompt for sentiment classification in SFT.
  • Figure 5: Instruction prompt for sentiment classification in RL.
  • ...and 2 more figures