Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Muyan-TTS: A Trainable Text-to-Speech Model Optimized for Podcast Scenarios with a $50K Budget

Xin Li, Kaikai Jia, Hao Sun, Jun Dai, Ziyang Jiang

TL;DR

Muyan-TTS tackles the need for an open-source, podcast-optimized TTS system that performs robust zero-shot synthesis while allowing speaker adaptation within a budget of $50{,}000$. By replacing the AR component with a pre-trained LLM (Llama-3.2-3B) and coupling it to a VITS-based decoder, the model aligns textual and acoustic representations via a quantized audio-token space and is trained through a multi-stage pipeline, including unsupervised LLM pre-training on $>100{,}000$ podcast hours and targeted decoder fine-tuning on high-MOS data. The authors release the full training methodology, data-processing pipeline, and inference-acceleration framework, achieving competitive WER and MOS on LibriSpeech and SEED, while delivering the fastest synthesis speed among open-source baselines ($r=\frac{T_{inf}}{T_{syn}}=0.33$). The work demonstrates that high-quality podcast data and careful decoder refinement can yield perceptually natural speech with strong speaker similarity, making Muyan-TTS a practical foundation for voice-interaction and podcast applications. It also highlights remaining limitations, such as streaming constraints and multilingual coverage, guiding future enhancements in streaming-capable G2P pipelines and broader language support.

Abstract

Recent advancements in text-to-speech (TTS) models have been driven by the integration of large language models (LLMs), enhancing semantic comprehension and improving speech naturalness. However, existing LLM-based TTS models often lack open-source training code and efficient inference acceleration frameworks, limiting their accessibility and adaptability. Additionally, there is no publicly available TTS model specifically optimized for podcast scenarios, which are in high demand for voice interaction applications. To address these limitations, we introduce Muyan-TTS, an open-source trainable TTS model designed for podcast applications within a $50,000 budget. Our model is pre-trained on over 100,000 hours of podcast audio data, enabling zero-shot TTS synthesis with high-quality voice generation. Furthermore, Muyan-TTS supports speaker adaptation with dozens of minutes of target speech, making it highly customizable for individual voices. In addition to open-sourcing the model, we provide a comprehensive data collection and processing pipeline, a full training procedure, and an optimized inference framework that accelerates LLM-based TTS synthesis. Our code and models are available at https://github.com/MYZY-AI/Muyan-TTS.

Muyan-TTS: A Trainable Text-to-Speech Model Optimized for Podcast Scenarios with a $50K Budget

TL;DR

Muyan-TTS tackles the need for an open-source, podcast-optimized TTS system that performs robust zero-shot synthesis while allowing speaker adaptation within a budget of . By replacing the AR component with a pre-trained LLM (Llama-3.2-3B) and coupling it to a VITS-based decoder, the model aligns textual and acoustic representations via a quantized audio-token space and is trained through a multi-stage pipeline, including unsupervised LLM pre-training on podcast hours and targeted decoder fine-tuning on high-MOS data. The authors release the full training methodology, data-processing pipeline, and inference-acceleration framework, achieving competitive WER and MOS on LibriSpeech and SEED, while delivering the fastest synthesis speed among open-source baselines (). The work demonstrates that high-quality podcast data and careful decoder refinement can yield perceptually natural speech with strong speaker similarity, making Muyan-TTS a practical foundation for voice-interaction and podcast applications. It also highlights remaining limitations, such as streaming constraints and multilingual coverage, guiding future enhancements in streaming-capable G2P pipelines and broader language support.

Abstract

Recent advancements in text-to-speech (TTS) models have been driven by the integration of large language models (LLMs), enhancing semantic comprehension and improving speech naturalness. However, existing LLM-based TTS models often lack open-source training code and efficient inference acceleration frameworks, limiting their accessibility and adaptability. Additionally, there is no publicly available TTS model specifically optimized for podcast scenarios, which are in high demand for voice interaction applications. To address these limitations, we introduce Muyan-TTS, an open-source trainable TTS model designed for podcast applications within a $50,000 budget. Our model is pre-trained on over 100,000 hours of podcast audio data, enabling zero-shot TTS synthesis with high-quality voice generation. Furthermore, Muyan-TTS supports speaker adaptation with dozens of minutes of target speech, making it highly customizable for individual voices. In addition to open-sourcing the model, we provide a comprehensive data collection and processing pipeline, a full training procedure, and an optimized inference framework that accelerates LLM-based TTS synthesis. Our code and models are available at https://github.com/MYZY-AI/Muyan-TTS.
Paper Structure (18 sections, 2 equations, 2 figures, 6 tables)

This paper contains 18 sections, 2 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Framework
  5. Datasets
  6. Pre-training of LLM
  7. Post-training of LLM
  8. Training of Decoder
  9. Inference Framework and Acceleration
  10. Experiments
  11. Experimental Settings
  12. Experimental Results
  13. Zero-Shot TTS Synthesis
  14. Comparison with Supervised Fine-Tuned Model
  15. Synthesis Speed
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Framework of Muyan-TTS. Left is an LLM that models the parallel corpus of text (in blue) and audio (in green) tokens. Right is a SoVITS model that decodes the generated audio tokens, as well as phonemes and speaker embeddings, into the audio waveform.
  • Figure 2: Data processing pipeline.