Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization

Xiaoxue Gao, Chen Zhang, Yiming Chen, Huayun Zhang, Nancy F. Chen

TL;DR

This work tackles the limitation of single-emotion supervision in emotional TTS by introducing Emo-DPO, which blends instruction tuning with Direct Preference Optimization (DPO) in an emotion-aware LLM-TTS framework. The model learns from pairwise emotion preferences, constructing positive and negative examples from the same text to encourage preferred emotional expressions, and employs JS-regularized DPO alongside KL and SFT losses for stable training. Across the English ESD dataset, Emo-DPO outperforms baselines on intelligibility, prosody similarity, and emotion similarity, with strong AB preferences and favorable subjective ratings, demonstrating improved cross-emotion controllability. The approach highlights the potential of integrating emotion-aware LLMs with preference-guided optimization to advance expressive and controllable TTS, with practical implications for more nuanced speech synthesis in human–computer interaction; code will be released upon acceptance.

Abstract

Current emotional text-to-speech (TTS) models predominantly conduct supervised training to learn the conversion from text and desired emotion to its emotional speech, focusing on a single emotion per text-speech pair. These models only learn the correct emotional outputs without fully comprehending other emotion characteristics, which limits their capabilities of capturing the nuances between different emotions. We propose a controllable Emo-DPO approach, which employs direct preference optimization to differentiate subtle emotional nuances between emotions through optimizing towards preferred emotions over less preferred emotional ones. Instead of relying on traditional neural architectures used in existing emotional TTS models, we propose utilizing the emotion-aware LLM-TTS neural architecture to leverage LLMs' in-context learning and instruction-following capabilities. Comprehensive experiments confirm that our proposed method outperforms the existing baselines.

Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization

TL;DR

This work tackles the limitation of single-emotion supervision in emotional TTS by introducing Emo-DPO, which blends instruction tuning with Direct Preference Optimization (DPO) in an emotion-aware LLM-TTS framework. The model learns from pairwise emotion preferences, constructing positive and negative examples from the same text to encourage preferred emotional expressions, and employs JS-regularized DPO alongside KL and SFT losses for stable training. Across the English ESD dataset, Emo-DPO outperforms baselines on intelligibility, prosody similarity, and emotion similarity, with strong AB preferences and favorable subjective ratings, demonstrating improved cross-emotion controllability. The approach highlights the potential of integrating emotion-aware LLMs with preference-guided optimization to advance expressive and controllable TTS, with practical implications for more nuanced speech synthesis in human–computer interaction; code will be released upon acceptance.

Abstract

Current emotional text-to-speech (TTS) models predominantly conduct supervised training to learn the conversion from text and desired emotion to its emotional speech, focusing on a single emotion per text-speech pair. These models only learn the correct emotional outputs without fully comprehending other emotion characteristics, which limits their capabilities of capturing the nuances between different emotions. We propose a controllable Emo-DPO approach, which employs direct preference optimization to differentiate subtle emotional nuances between emotions through optimizing towards preferred emotions over less preferred emotional ones. Instead of relying on traditional neural architectures used in existing emotional TTS models, we propose utilizing the emotion-aware LLM-TTS neural architecture to leverage LLMs' in-context learning and instruction-following capabilities. Comprehensive experiments confirm that our proposed method outperforms the existing baselines.
Paper Structure (14 sections, 5 equations, 3 figures, 2 tables)

This paper contains 14 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Emo-DPO Overview
  4. Instruction Tuning
  5. Emo-Direct Preference Optimization Training
  6. Beyond One Emotion - DPO Training
  7. Emo-DPO Training Objective
  8. Experiments
  9. Datasets and Experimental Setup
  10. Evaluation Metrics
  11. Results and Discussion
  12. Effectiveness of Emo-DPO training on LLM-TTS
  13. Ablation Study
  14. Conclusion

Figures (3)

  • Figure 1: Overview of the proposed Emo-DPO approach: (a) instruction tuning, (b) Emo-DPO training, and (c) the inference process.
  • Figure 2: Comparison of subjective evaluation results for MOS and Emotion MOS tests across cosyvoice, emospeech, and the proposed Emo-DPO models.
  • Figure 3: Comparison of subjective evaluation results from AB preference tests: 1) left: cosyvoice vs. Emo-DPO and 2) right: emospeech vs. Emo-DPO.