Preference Alignment Improves Language Model-Based TTS

TL;DR

This paper addresses aligning language-model-based TTS outputs with human preferences by applying Direct Preference Optimization (DPO). It formulates the DPO objective, leveraging win-lose preference data and a Bradley–Terry formulation, to optimize the LM without explicit reward models via $L_{DPO} = \max_{\theta} \mathbb{E}[\log \sigma(\beta \cdot \log \frac{P_\theta(\mathbf{y}_{w}|\mathbf{x})}{P_{ref}(\mathbf{y}_{w}|\mathbf{x})} - \beta \cdot \log \frac{P_\theta(\mathbf{y}_{l}|\mathbf{x})}{P_{ref}(\mathbf{y}_{l}|\mathbf{x})})]$. Through extensive experiments on a 1.15B parameter LM-TTS trained on ~55k hours, the method yields consistent improvements in intelligibility (WER), speaker similarity (SPK_SIM), and proxy MOS, with some proxy scores surpassing human baselines and strong generalization to out-of-domain data. The study systematically analyzes data curation, hyperparameters, length normalization, metric selection, iterative optimization, data efficiency, and comparisons with other PA methods, demonstrating robust gains and practical applicability, including low-resource and cross-domain settings. Overall, the results indicate that Direct Preference Optimization is a practical and scalable approach to aligning TTS with human preferences, potentially enhancing deployment in real-world speech systems.

Abstract

Recent advancements in text-to-speech (TTS) have shown that language model (LM)-based systems offer competitive performance to their counterparts. Further optimization can be achieved through preference alignment algorithms, which adjust LMs to align with the preferences of reward models, enhancing the desirability of the generated content. This study presents a thorough empirical evaluation of how preference alignment algorithms, particularly Direct Preference Optimization (DPO), enhance LM-based TTS. With a 1.15B parameter LM-based TTS model, we demonstrate that preference alignment consistently improves intelligibility, speaker similarity, and proxy subjective evaluation scores, with the latter two metrics surpassing even human speech in certain evaluations. We also show preference alignment is applicable to low-resource scenarios and effectively generalized to out-of-domain applications.