VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modeling

TL;DR

VocalNet-MDM demonstrates that Masked Diffusion Modeling can serve as an effective non-autoregressive paradigm for streaming speech LLMs. By combining Hierarchical Block-wise Masking to bridge training and streaming inference and Iterative Self-Distillation to compress multi-step refinement into few steps, the approach achieves 3.7×–10× decoding speedups and 34% reduction in first-chunk latency on modest data (~6k hours) while preserving strong text and speech quality. The Thinker–Talker architecture enables efficient conditioning and high-fidelity speech reconstruction via a flow-based vocoder, illustrating a scalable path toward low-latency, high-quality speech interaction. These results position MDM as a viable alternative to AR decoding for real-time spoken dialogue systems, with practical implications for responsive, multimodal assistants and dialog agents.

Abstract

Recent Speech Large Language Models~(LLMs) have achieved impressive capabilities in end-to-end speech interaction. However, the prevailing autoregressive paradigm imposes strict serial constraints, limiting generation efficiency and introducing exposure bias. In this paper, we investigate Masked Diffusion Modeling~(MDM) as a non-autoregressive paradigm for speech LLMs and introduce VocalNet-MDM. To adapt MDM for streaming speech interaction, we address two critical challenges: training-inference mismatch and iterative overhead. We propose Hierarchical Block-wise Masking to align training objectives with the progressive masked states encountered during block diffusion decoding, and Iterative Self-Distillation to compress multi-step refinement into fewer steps for low-latency inference. Trained on a limited scale of only 6K hours of speech data, VocalNet-MDM achieves a 3.7$\times$--10$\times$ decoding speedup and reduces first-chunk latency by 34\% compared to AR baselines. It maintains competitive recognition accuracy while achieving state-of-the-art text quality and speech naturalness, demonstrating that MDM is a promising and scalable alternative for low-latency, efficient speech LLMs.

Figures (7)

• Figure 1: Overview of our approach. (a) Masked Diffusion Modeling: a Thinker--Talker speech LLM architecture where speech generation is performed via MDM. (b) Iterative Self-Distillation: transferring more certain distributions from later to earlier denoising steps to enable few-step inference. (c) Block Diffusion Decoding: streaming inference that denoises tokens within fixed-size blocks over diffusion steps.
• Figure 2: Masking strategies. (a) Global Bernoulli Masking: each token masked independently with uniform probability. (b) Hierarchical Block-wise Masking: block selection followed by intra-block token masking.
• Figure 3: Latency breakdown for generating the first-chunk in VocalNet-MDM at different diffusion steps.
• Figure 4: Comparison of prediction uncertainty before and after Iterative Self-Distillation. (a) Mean confidence scores across diffusion steps. (b) Mean entropy values across diffusion steps.
• Figure 5: Prompt for AlpacaEval.