Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding Order Matters in Autoregressive Speech Synthesis

Minghui Zhao, Anton Ragni

TL;DR

This work investigates decoding order as a modelling choice in autoregressive speech synthesis and shows that left-to-right generation is not universally optimal. It introduces a Masked Diffusion Model (MDM) that enables arbitrary decoding orders and develops order-agnostic training and several adaptive decoding strategies, including Top-$K$ and duration-guided decoding, as well as a quantisation scheme for discrete inputs. The experiments on LJSpeech demonstrate that adaptive decoding often outperforms fixed orders, with duration-based strategies delivering strong objective and perceptual scores, while partial randomness can balance spectral fidelity and pitch. The study also reveals that speech can be effectively synthesised from discrete Mel representations, even at very low bit-depth, when paired with a robust vocoder like HiFi-GAN, highlighting practical avenues for efficient, flexible TTS systems.

Abstract

Autoregressive speech synthesis often adopts a left-to-right order, yet generation order is a modelling choice. We investigate decoding order through masked diffusion framework, which progressively unmasks positions and allows arbitrary decoding orders during training and inference. By interpolating between identity and random permutations, we show that randomness in decoding order affects speech quality. We further compare fixed strategies, such as \texttt{l2r} and \texttt{r2l} with adaptive ones, such as Top-$K$, finding that fixed-order decoding, including the dominating left-to-right approach, is suboptimal, while adaptive decoding yields better performance. Finally, since masked diffusion requires discrete inputs, we quantise acoustic representations and find that even 1-bit quantisation can support reasonably high-quality speech.

Decoding Order Matters in Autoregressive Speech Synthesis

TL;DR

This work investigates decoding order as a modelling choice in autoregressive speech synthesis and shows that left-to-right generation is not universally optimal. It introduces a Masked Diffusion Model (MDM) that enables arbitrary decoding orders and develops order-agnostic training and several adaptive decoding strategies, including Top- and duration-guided decoding, as well as a quantisation scheme for discrete inputs. The experiments on LJSpeech demonstrate that adaptive decoding often outperforms fixed orders, with duration-based strategies delivering strong objective and perceptual scores, while partial randomness can balance spectral fidelity and pitch. The study also reveals that speech can be effectively synthesised from discrete Mel representations, even at very low bit-depth, when paired with a robust vocoder like HiFi-GAN, highlighting practical avenues for efficient, flexible TTS systems.

Abstract

Autoregressive speech synthesis often adopts a left-to-right order, yet generation order is a modelling choice. We investigate decoding order through masked diffusion framework, which progressively unmasks positions and allows arbitrary decoding orders during training and inference. By interpolating between identity and random permutations, we show that randomness in decoding order affects speech quality. We further compare fixed strategies, such as \texttt{l2r} and \texttt{r2l} with adaptive ones, such as Top-, finding that fixed-order decoding, including the dominating left-to-right approach, is suboptimal, while adaptive decoding yields better performance. Finally, since masked diffusion requires discrete inputs, we quantise acoustic representations and find that even 1-bit quantisation can support reasonably high-quality speech.
Paper Structure (22 sections, 6 equations, 6 figures, 2 algorithms)

This paper contains 22 sections, 6 equations, 6 figures, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. METHOD
  3. Inference
  4. Order-agnostic training
  5. Quantisation
  6. Decoding strategies
  7. Controlling order stochasticity
  8. Top-K probability
  9. Duration-guided decoding
  10. EXPERIMENTS
  11. Experimental setup
  12. Evaluation metrics
  13. Results
  14. Effect of discretisation on speech quality
  15. Controlling randomness in decoding order
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Intermediate mel-spectrograms shown from left to right as generation progresses in a random order. The number beneath each frame indicates its order in generation and the rightmost frames are the final output.
  • Figure 2: Evaluation on quantisation levels
  • Figure 3: Evaluation on orders with controlled randomness
  • Figure 4: Evaluation results for single-frame decoding strategies
  • Figure 5: Breakdown of MOS scores
  • ...and 1 more figures