Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoonCast: High-Quality Zero-Shot Podcast Generation

Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, Xiangyang Li

TL;DR

MoonCast tackles the challenge of generating long-form, multi-speaker podcasts from text-only inputs with unseen voices. It couples a long-context two-speaker audio model that uses discrete speech codes and a chunk-wise detokenizer with an LLM-powered script generator to imbue spontaneity into conversations. Through curriculum learning and large-scale multi-domain data, MoonCast demonstrates improvements in spontaneity, coherence, intelligibility, and speaker similarity across Chinese and English podcasts, while open-sourcing prompts and modules. The work advances zero-shot podcast generation and outlines ethical considerations and future directions for expanding language coverage and multi-person conversations.

Abstract

Recent advances in text-to-speech synthesis have achieved notable success in generating high-quality short utterances for individual speakers. However, these systems still face challenges when extending their capabilities to long, multi-speaker, and spontaneous dialogues, typical of real-world scenarios such as podcasts. These limitations arise from two primary challenges: 1) long speech: podcasts typically span several minutes, exceeding the upper limit of most existing work; 2) spontaneity: podcasts are marked by their spontaneous, oral nature, which sharply contrasts with formal, written contexts; existing works often fall short in capturing this spontaneity. In this paper, we propose MoonCast, a solution for high-quality zero-shot podcast generation, aiming to synthesize natural podcast-style speech from text-only sources (e.g., stories, technical reports, news in TXT, PDF, or Web URL formats) using the voices of unseen speakers. To generate long audio, we adopt a long-context language model-based audio modeling approach utilizing large-scale long-context speech data. To enhance spontaneity, we utilize a podcast generation module to generate scripts with spontaneous details, which have been empirically shown to be as crucial as the text-to-speech modeling itself. Experiments demonstrate that MoonCast outperforms baselines, with particularly notable improvements in spontaneity and coherence.

MoonCast: High-Quality Zero-Shot Podcast Generation

TL;DR

MoonCast tackles the challenge of generating long-form, multi-speaker podcasts from text-only inputs with unseen voices. It couples a long-context two-speaker audio model that uses discrete speech codes and a chunk-wise detokenizer with an LLM-powered script generator to imbue spontaneity into conversations. Through curriculum learning and large-scale multi-domain data, MoonCast demonstrates improvements in spontaneity, coherence, intelligibility, and speaker similarity across Chinese and English podcasts, while open-sourcing prompts and modules. The work advances zero-shot podcast generation and outlines ethical considerations and future directions for expanding language coverage and multi-person conversations.

Abstract

Recent advances in text-to-speech synthesis have achieved notable success in generating high-quality short utterances for individual speakers. However, these systems still face challenges when extending their capabilities to long, multi-speaker, and spontaneous dialogues, typical of real-world scenarios such as podcasts. These limitations arise from two primary challenges: 1) long speech: podcasts typically span several minutes, exceeding the upper limit of most existing work; 2) spontaneity: podcasts are marked by their spontaneous, oral nature, which sharply contrasts with formal, written contexts; existing works often fall short in capturing this spontaneity. In this paper, we propose MoonCast, a solution for high-quality zero-shot podcast generation, aiming to synthesize natural podcast-style speech from text-only sources (e.g., stories, technical reports, news in TXT, PDF, or Web URL formats) using the voices of unseen speakers. To generate long audio, we adopt a long-context language model-based audio modeling approach utilizing large-scale long-context speech data. To enhance spontaneity, we utilize a podcast generation module to generate scripts with spontaneous details, which have been empirically shown to be as crucial as the text-to-speech modeling itself. Experiments demonstrate that MoonCast outperforms baselines, with particularly notable improvements in spontaneity and coherence.

Paper Structure

This paper contains 32 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Zero-Shot TTS
  4. Dialogue and Conversation Generation
  5. Spontaneous TTS
  6. Method
  7. Overall
  8. Audio Modeling Module
  9. Long-Context Two-Speaker Text-to-Semantic Model
  10. Chunk-wise Autoregressive Speech Detokenizer
  11. LLM-Powered Script Generation Module
  12. Experiments and Results
  13. Experimental Settings
  14. Data Preparation
  15. Model Details
  16. ...and 17 more sections

Figures (2)

  • Figure 1: The overall pipeline of the proposed system.
  • Figure 2: The attention mask design for chunk-wise autoregressive speech detokenizer. $\mathbf{M}_i$ means the clean mel-spectrogram and $\mathbf{M}_i'$ means noisy mel-spectrogram. Yellow means allow to attend, and gray means not allowed to attend. Attention is conducted among row-wise in figure.