Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLAIF-SPA: Optimizing LLM-based Emotional Speech Synthesis via RLAIF

Qing Yang, Zhenghao Liu, Junxin Wang, Yangfan Du, Pengcheng Huang, Tong Xiao

TL;DR

RLAIF-SPA addresses the lack of emotional expressiveness in neutral TTS by integrating AI-driven feedback from ASR and LLMs into a reinforcement learning loop. It uses a Group Relative Policy Optimization framework to update a TTS policy with a composite reward $R(s_i) = - \alpha_1 R_{wer}(s_i) + \alpha_2 R_{label}(s_i)$, balancing intelligibility and prosodic-emotional alignment. Prosodic Label Alignment uses four targets (Structure, Emotion, Speed, Tone) with $R_{label}(s_i) = \sum_{k=1}^{4} w_k m_k(s_i)$, while Semantic Accuracy Feedback relies on $R_{wer}(s_i) = \text{WER}(t_i, \text{ASR}(s_i))$. Experiments on LibriSpeech and ESD show substantial improvements over baselines, including a 26.1% WER reduction and notable gains in SIM-O and subjective emotion MOS, demonstrating scalable, annotation-free emotional TTS.

Abstract

Text-To-Speech synthesis has achieved near-human quality in neutral speech, but emotional expressiveness remains a challenge. Existing methods often rely on costly emotion annotations or optimize indirect objectives that fail to capture the emotional expressiveness and perceptual naturalness of speech, leading to generated speech that is accurate but emotionally flat. To address these challenges, we propose the RLAIF-SPA framework, incorporating a Reinforcement Learning from AI Feedback (RLAIF) mechanism to employ Automatic Speech Recognition (ASR) and Large Language Model (LLM) techniques to respectively judge semantic accuracy and prosodic-emotional label alignment as a direct reward for emotional expressiveness and intelligibility optimization. Specifically, it leverages Prosodic Label Alignment to enhance expressive quality by jointly considering semantic accuracy and prosodic-emotional alignment along four fine-grained dimensions: Structure, Emotion, Speed, and Tone. In addition, it incorporates Semantic Accuracy Feedback to ensure the generation of clear and accurate speech. Experiments on the Libri Speech dataset show that RLAIF-SPA outperforms Chat-TTS, with a 26.1% reduction in WER, a 9.1% increase in SIM-O, and over 10% improvement in human evaluation.

RLAIF-SPA: Optimizing LLM-based Emotional Speech Synthesis via RLAIF

TL;DR

RLAIF-SPA addresses the lack of emotional expressiveness in neutral TTS by integrating AI-driven feedback from ASR and LLMs into a reinforcement learning loop. It uses a Group Relative Policy Optimization framework to update a TTS policy with a composite reward , balancing intelligibility and prosodic-emotional alignment. Prosodic Label Alignment uses four targets (Structure, Emotion, Speed, Tone) with , while Semantic Accuracy Feedback relies on . Experiments on LibriSpeech and ESD show substantial improvements over baselines, including a 26.1% WER reduction and notable gains in SIM-O and subjective emotion MOS, demonstrating scalable, annotation-free emotional TTS.

Abstract

Text-To-Speech synthesis has achieved near-human quality in neutral speech, but emotional expressiveness remains a challenge. Existing methods often rely on costly emotion annotations or optimize indirect objectives that fail to capture the emotional expressiveness and perceptual naturalness of speech, leading to generated speech that is accurate but emotionally flat. To address these challenges, we propose the RLAIF-SPA framework, incorporating a Reinforcement Learning from AI Feedback (RLAIF) mechanism to employ Automatic Speech Recognition (ASR) and Large Language Model (LLM) techniques to respectively judge semantic accuracy and prosodic-emotional label alignment as a direct reward for emotional expressiveness and intelligibility optimization. Specifically, it leverages Prosodic Label Alignment to enhance expressive quality by jointly considering semantic accuracy and prosodic-emotional alignment along four fine-grained dimensions: Structure, Emotion, Speed, and Tone. In addition, it incorporates Semantic Accuracy Feedback to ensure the generation of clear and accurate speech. Experiments on the Libri Speech dataset show that RLAIF-SPA outperforms Chat-TTS, with a 26.1% reduction in WER, a 9.1% increase in SIM-O, and over 10% improvement in human evaluation.
Paper Structure (10 sections, 4 equations, 2 figures, 2 tables)

This paper contains 10 sections, 4 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Training Strategy and Optimization Framework
  4. Prosodic Label Alignment
  5. Semantic Accuracy Feedback
  6. Experimental Methodology
  7. Experimental Results
  8. Effectiveness of LLM-TTS Optimized Using RLAIF-SPA
  9. Ablation Study
  10. Conclusion and Futher Work

Figures (2)

  • Figure 1: Illustration of the proposed RLAIF-SPA framework. RLAIF-SPA optimizes both emotional expressiveness and intelligibility through the AI Feedback mechanism, which is generated by AI models themselves.
  • Figure 2: Comparison of AB Preference Test results on the LibriSpeech dataset ((a), (b)) and the ESD dataset ((c), (d)). Proportions of Win (RLAIF-SPA preferred), Loss (baseline preferred), and Tie are shown for each comparison.