Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GLM-TTS Technical Report

Jiayan Cui, Zhihan Yang, Naihan Li, Jiankun Tian, Xingyu Ma, Yi Zhang, Guangyu Chen, Runxuan Yang, Yuqing Cheng, Yizhi Zhou, Guochen Yu, Xiaotao Gu, Jie Tang

TL;DR

GLM-TTS introduces a production-oriented, two-stage TTS framework that combines a text-to-token autoregressive model with a token-to-waveform diffusion model, achieving strong results with only 100k hours of data. The approach integrates an optimized Whisper-VQ speech tokenizer, GRPO-based multi-reward reinforcement learning, low-cost LoRA-based voice customization, and a hybrid phoneme–text input plus the Vocos2D vocoder to balance accuracy, expressiveness, and deployability. Experimental results on Seed-TTS-eval and internal benchmarks show competitive pronunciation, timbre fidelity, and emotion expressiveness, with ablations confirming the effectiveness of Phoneme-in and Vocos2D. The work emphasizes practical deployment considerations, providing code and demos to facilitate production-ready speech synthesis and customization at scale.

Abstract

This work proposes GLM-TTS, a production-level TTS system designed for efficiency, controllability, and high-fidelity speech generation. GLM-TTS follows a two-stage architecture, consisting of a text-to-token autoregressive model and a token-to-waveform diffusion model. With only 100k hours of training data, GLM-TTS achieves state-of-the-art performance on multiple open-source benchmarks. To meet production requirements, GLM-TTS improves speech quality through an optimized speech tokenizer with fundamental frequency constraints and a GRPO-based multi-reward reinforcement learning framework that jointly optimizes pronunciation, speaker similarity, and expressive prosody. In parallel, the system enables efficient and controllable deployment via parameter-efficient LoRA-based voice customization and a hybrid phoneme-text input scheme that provides precise pronunciation control. Our code is available at https://github.com/zai-org/GLM-TTS. Real-time speech synthesis demos are provided via Z.ai (audio.z.ai), the Zhipu Qingyan app/web (chatglm.cn).

GLM-TTS Technical Report

TL;DR

GLM-TTS introduces a production-oriented, two-stage TTS framework that combines a text-to-token autoregressive model with a token-to-waveform diffusion model, achieving strong results with only 100k hours of data. The approach integrates an optimized Whisper-VQ speech tokenizer, GRPO-based multi-reward reinforcement learning, low-cost LoRA-based voice customization, and a hybrid phoneme–text input plus the Vocos2D vocoder to balance accuracy, expressiveness, and deployability. Experimental results on Seed-TTS-eval and internal benchmarks show competitive pronunciation, timbre fidelity, and emotion expressiveness, with ablations confirming the effectiveness of Phoneme-in and Vocos2D. The work emphasizes practical deployment considerations, providing code and demos to facilitate production-ready speech synthesis and customization at scale.

Abstract

This work proposes GLM-TTS, a production-level TTS system designed for efficiency, controllability, and high-fidelity speech generation. GLM-TTS follows a two-stage architecture, consisting of a text-to-token autoregressive model and a token-to-waveform diffusion model. With only 100k hours of training data, GLM-TTS achieves state-of-the-art performance on multiple open-source benchmarks. To meet production requirements, GLM-TTS improves speech quality through an optimized speech tokenizer with fundamental frequency constraints and a GRPO-based multi-reward reinforcement learning framework that jointly optimizes pronunciation, speaker similarity, and expressive prosody. In parallel, the system enables efficient and controllable deployment via parameter-efficient LoRA-based voice customization and a hybrid phoneme-text input scheme that provides precise pronunciation control. Our code is available at https://github.com/zai-org/GLM-TTS. Real-time speech synthesis demos are provided via Z.ai (audio.z.ai), the Zhipu Qingyan app/web (chatglm.cn).

Paper Structure

This paper contains 16 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data processing pipeline
  4. Text Tokenizer
  5. Speech Tokenizer
  6. SpeechLM RL-Alignment
  7. LoRA for Premium Voice Customization
  8. Phoneme-in
  9. Vocos2D
  10. Experiments and Results
  11. Speech tokenizer Evaluation
  12. Voice Cloning Results on Seed-TTS-eval
  13. Speech-LM RL-Alignment
  14. Effectiveness of Phoneme-in
  15. Vocos2D vocoder
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Overall Architecture of GLM-TTS
  • Figure 2: An overview of the data processing pipeline.
  • Figure 3: An overview of the GLM-TTS-GRPO framework.
  • Figure 4: Diagram of the Vocos2D generator architecture. FC denotes a fully connected (linear) layer, PwConv denotes point-wise convolution, and DwConv denotes depth-wise convolution.
  • Figure 5: Comparison of the original Vocos generator loss (upper) and the proposed Vocos2D generator loss (lower). DA denotes discriminator augmentation, Hinge denotes hinge loss, MPD denotes multi-period discriminator, and MRD denotes multi-resolution discriminator.