Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization

Jiacheng Shi, Hongfei Du, Yangfan He, Y. Alicia Hong, Ye Gao

TL;DR

The paper tackles the challenge of fine-grained emotional control in diffusion-based TTS by moving beyond endpoint supervision to dense, stepwise preferences across intermediate denoising steps. It introduces Emotion-Aware Stepwise Preference Optimization (EASPO) and the time-conditioned Emotion-Aware Stepwise Preference Model (EASPM), which score noisy intermediate states to guide generation via win/lose comparisons at each step. The approach formulates a stepwise, time-conditioned RL objective with dense rewards that align policy changes with emotional expressiveness, incorporating candidate pooling and stochastic rollouts to maintain exploration. Experiments on MSP-Podcast and ESD show that EASPO outperforms multiple baselines in emotion similarity, prosody, intelligibility, and perceptual naturalness, demonstrating effective, temporally coherent emotional shaping in diffusion TTS.

Abstract

Emotional text-to-speech seeks to convey affect while preserving intelligibility and prosody, yet existing methods rely on coarse labels or proxy classifiers and receive only utterance-level feedback. We introduce Emotion-Aware Stepwise Preference Optimization (EASPO), a post-training framework that aligns diffusion TTS with fine-grained emotional preferences at intermediate denoising steps. Central to our approach is EASPM, a time-conditioned model that scores noisy intermediate speech states and enables automatic preference pair construction. EASPO optimizes generation to match these stepwise preferences, enabling controllable emotional shaping. Experiments show superior performance over existing methods in both expressiveness and naturalness.

Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization

TL;DR

The paper tackles the challenge of fine-grained emotional control in diffusion-based TTS by moving beyond endpoint supervision to dense, stepwise preferences across intermediate denoising steps. It introduces Emotion-Aware Stepwise Preference Optimization (EASPO) and the time-conditioned Emotion-Aware Stepwise Preference Model (EASPM), which score noisy intermediate states to guide generation via win/lose comparisons at each step. The approach formulates a stepwise, time-conditioned RL objective with dense rewards that align policy changes with emotional expressiveness, incorporating candidate pooling and stochastic rollouts to maintain exploration. Experiments on MSP-Podcast and ESD show that EASPO outperforms multiple baselines in emotion similarity, prosody, intelligibility, and perceptual naturalness, demonstrating effective, temporally coherent emotional shaping in diffusion TTS.

Abstract

Emotional text-to-speech seeks to convey affect while preserving intelligibility and prosody, yet existing methods rely on coarse labels or proxy classifiers and receive only utterance-level feedback. We introduce Emotion-Aware Stepwise Preference Optimization (EASPO), a post-training framework that aligns diffusion TTS with fine-grained emotional preferences at intermediate denoising steps. Central to our approach is EASPM, a time-conditioned model that scores noisy intermediate speech states and enables automatic preference pair construction. EASPO optimizes generation to match these stepwise preferences, enabling controllable emotional shaping. Experiments show superior performance over existing methods in both expressiveness and naturalness.

Paper Structure

This paper contains 10 sections, 7 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Emotion-Aware Stepwise Preference Model
  4. Objective Function of EASPO
  5. Experiments
  6. Datasets and Experimental Setup
  7. Evaluation Metrics
  8. Main Results
  9. Ablation Study
  10. Conclusion

Figures (1)

  • Figure 1: Unlike prior DPO-based methods, EASPO avoids direct propagation of preferences across diffusion steps. At each step in EASPO, a set of candidate samples is produced, from which a suitable win–lose pair is chosen to update the diffusion model. Afterward, one sample is randomly picked to serve as the starting point for the next iteration.