Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
Yicheng Zhong, Peiji Yang, Zhisheng Wang
TL;DR
The paper tackles instability and drift in single-codebook TTS LLMs by introducing a Group Relative Policy Optimization framework that jointly optimizes intelligibility, speaker similarity, duration, and probabilistic stability through a suite of rewards. A novel LLM-assisted prosody alignment reward guides rhythmic structure, complemented by a length penalty and entropy regularization, and optionally enhanced with a Flow Matching refinement. Empirical results demonstrate improved prosodic stability, speaker similarity, and naturalness across languages and scales, with pronounced gains from the prosody-alignment term and RL data efficiency. These findings suggest that directly optimizing the autoregressive policy with structured rewards yields robust, scalable improvements in end-to-end TTS LLMs, and that RL-based policy improvements remain complementary to downstream refiners like Flow Matching.
Abstract
Recent advances in Large Language Models (LLMs) have transformed text-to-speech (TTS) synthesis, inspiring autoregressive frameworks that represent speech as sequences of discrete codec tokens. Among them, single-codebook TTS LLMs have emerged as compact and streamable architectures that jointly model semantic and acoustic integration. However, despite their efficiency, these models often exhibit unstable prosody, speaker drift, and degraded naturalness. To address these issues, we propose a multi-reward Group Relative Policy Optimization (GRPO) framework that directly optimizes the token generation policy of single-codebook TTS LLMs. Beyond standard intelligibility and speaker similarity objectives, our design integrates three rule-based rewards: a length penalty for duration consistency, an entropy regularization reward for decoding stability, and an LLM-annotated prosody alignment reward that explicitly supervises rhythm. In this prosody reward, an external reasoning LLM predicts multiple plausible pause structures via in-context learning, providing a human-preference-aligned supervisory signal for GRPO training. To assess universality, we further attach a flow-matching (FM) decoder on top of the GRPO-optimized AR backbone and observe consistent additional gains, indicating that our reinforcement optimization enhances the intrinsic AR policy. We further conduct a scalability analysis across data sizes and model scales, revealing that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.