DLPO: Diffusion Model Loss-Guided Reinforcement Learning for Fine-Tuning Text-to-Speech Diffusion Models

TL;DR

The results show that RLHF can enhance diffusion-based text-to-speech synthesis models, and, moreover, DLPO can better improve diffusion models in generating natural and high quality speech audios.

Abstract

Recent advancements in generative models have sparked a significant interest within the machine learning community. Particularly, diffusion models have demonstrated remarkable capabilities in synthesizing images and speech. Studies such as those by Lee et al. (2023), Black et al. (2023), Wang et al. (2023), and Fan et al. (2024) illustrate that Reinforcement Learning with Human Feedback (RLHF) can enhance diffusion models for image synthesis. However, due to architectural differences between these models and those employed in speech synthesis, it remains uncertain whether RLHF could similarly benefit speech synthesis models. In this paper, we explore the practical application of RLHF to diffusion-based text-to-speech synthesis, leveraging the mean opinion score (MOS) as predicted by UTokyo-SaruLab MOS prediction system (Saeki et al., 2022) as a proxy loss. We introduce diffusion model loss-guided RL policy optimization (DLPO) and compare it against other RLHF approaches, employing the NISQA speech quality and naturalness assessment model (Mittag et al., 2021) and human preference experiments for further evaluation. Our results show that RLHF can enhance diffusion-based text-to-speech synthesis models, and, moreover, DLPO can better improve diffusion models in generating natural and high quality speech audios.

Figures (6)

• Figure 1: (fine-tuning effectiveness) The relative effectiveness of five RL algorithms. (Top) shows the the change of reward UTMOS during initial 160 training episodes (training samples = episodes * batch size) of each RL approach while (bottom) shows the change of evaluation NISQA MOS during the same initial 160 training episodes. Left figures shows the different methods' training performance and right figures shows that DLPO increases UTMOS from 3.0 to 3.68 and increases NISQA from 3.85 to 4.12.
• Figure 2: (a) shows the mean UTMOS and NISQA scores for generated audio based on 200 unseen texts, the error bar shows the standard deviation of each result. (b) shows the proportion of raters who prefer the audios generated from the DLPO fine-tuned model or baseline model and the proportion of raters who think audios generated by DLPO fine-tuned model and baseline model are about the same (Tie).
• Figure 3: Detail steps for fine-tuning text-to-speech diffusion models with online RL learning
• Figure 4: WaveGrad2 follows a Markov process where forward diffusion $q(y_{n+1}| y_n,x)$ iteratively adds Gaussian noise to the signal starting from the waveform $y_0$. $q(y_{t+1}| y_0)$ is the noise distribution used for training. The inference denoising process progressively removes noise, starting from Gaussian noise $x_T$. This figure is adapted from chen2020wavegradho2020denoising.
• Figure :