Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation

Yakun Song, Xiaobin Zhuang, Jiawei Chen, Zhikang Niu, Guanrou Yang, Chenpeng Du, Dongya Jia, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen

TL;DR

DiSTAR tackles zero-shot TTS by modeling speech entirely in a discrete RVQ code space. It couples a patchwise autoregressive sketcher with a discrete masked-diffusion refiner to generate within-patch content, avoiding explicit duration predictors and mitigating exposure bias. The approach supports flexible inference controls, including CFG, cross-patch bitrate pruning, and patch-level parallelism, while achieving state-of-the-art robustness, speaker similarity, and naturalness on LibriSpeech PC test-clean and SeedTTS benchmarks. This enables robust, controllable, and efficient zero-shot synthesis suitable for real-world deployment across long-form speech tasks.

Abstract

Recent attempts to interleave autoregressive (AR) sketchers with diffusion-based refiners over continuous speech representations have shown promise, but they remain brittle under distribution shift and offer limited levers for controllability. We introduce DISTAR, a zero-shot text-to-speech framework that operates entirely in a discrete residual vector quantization (RVQ) code space and tightly couples an AR language model with a masked diffusion model, without forced alignment or a duration predictor. Concretely, DISTAR drafts block-level RVQ tokens with an AR language model and then performs parallel masked-diffusion infilling conditioned on the draft to complete the next block, yielding long-form synthesis with blockwise parallelism while mitigating classic AR exposure bias. The discrete code space affords explicit control at inference: DISTAR produces high-quality audio under both greedy and sample-based decoding using classifier-free guidance, supports trade-offs between robustness and diversity, and enables variable bit-rate and controllable computation via RVQ layer pruning at test time. Extensive experiments and ablations demonstrate that DISTAR surpasses state-of-the-art zero-shot TTS systems in robustness, naturalness, and speaker/style consistency, while maintaining rich output diversity. Audio samples are provided on https://anonymous.4open.science/w/DiSTAR_demo.

DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation

TL;DR

DiSTAR tackles zero-shot TTS by modeling speech entirely in a discrete RVQ code space. It couples a patchwise autoregressive sketcher with a discrete masked-diffusion refiner to generate within-patch content, avoiding explicit duration predictors and mitigating exposure bias. The approach supports flexible inference controls, including CFG, cross-patch bitrate pruning, and patch-level parallelism, while achieving state-of-the-art robustness, speaker similarity, and naturalness on LibriSpeech PC test-clean and SeedTTS benchmarks. This enables robust, controllable, and efficient zero-shot synthesis suitable for real-world deployment across long-form speech tasks.

Abstract

Recent attempts to interleave autoregressive (AR) sketchers with diffusion-based refiners over continuous speech representations have shown promise, but they remain brittle under distribution shift and offer limited levers for controllability. We introduce DISTAR, a zero-shot text-to-speech framework that operates entirely in a discrete residual vector quantization (RVQ) code space and tightly couples an AR language model with a masked diffusion model, without forced alignment or a duration predictor. Concretely, DISTAR drafts block-level RVQ tokens with an AR language model and then performs parallel masked-diffusion infilling conditioned on the draft to complete the next block, yielding long-form synthesis with blockwise parallelism while mitigating classic AR exposure bias. The discrete code space affords explicit control at inference: DISTAR produces high-quality audio under both greedy and sample-based decoding using classifier-free guidance, supports trade-offs between robustness and diversity, and enables variable bit-rate and controllable computation via RVQ layer pruning at test time. Extensive experiments and ablations demonstrate that DISTAR surpasses state-of-the-art zero-shot TTS systems in robustness, naturalness, and speaker/style consistency, while maintaining rich output diversity. Audio samples are provided on https://anonymous.4open.science/w/DiSTAR_demo.
Paper Structure (46 sections, 12 equations, 2 figures, 6 tables)

This paper contains 46 sections, 12 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Zero-Shot Text-to-Speech
  4. Mask Diffusion Models
  5. DiSTAR
  6. Overview
  7. Formulation
  8. Patchified view of the code stream.
  9. Decomposed generator.
  10. Discrete masked diffusion over the next span.
  11. Overall architecture
  12. Aggregator
  13. Next-Patch Masked Diffusion Modeling
  14. Training and Inference
  15. Embedding initialization.
  16. ...and 31 more sections

Figures (2)

  • Figure 1: An overview of the proposed DiSTAR framework. It consists of an aggregator for RVQ code input, a causal language model backbone with text prompts, and a masked diffusion decoder, predicting patches of tokens purely on the discrete RVQ space.
  • Figure 2: Comparison results among different inference RVQ layers.