Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs

TL;DR

The paper addresses the bottleneck of discrete token representations in multimodal LLMs for TTS by introducing a frame-level continuous diffusion head that autoregressively generates speech embeddings conditioned on multimodal context. A dual-head architecture preserves the LLM's multitask capability while enabling continuous-speech synthesis, and a two-stage training regime stabilizes joint optimization by freezing the LM during diffusion-head refinement. Masked autoregressive learning mitigates exposure bias, leading to robust long-form speech generation. Evaluations on LibriSpeech(PC) show significant gains, with a 46% relative WER reduction and competitive speaker similarity and perceptual quality, highlighting the method's potential for unified foundation models that handle multiple speech and multimodal tasks.

Abstract

Unified architectures in multimodal large language models (MLLM) have shown promise in handling diverse tasks within a single framework. In the text-to-speech (TTS) task, current MLLM-based approaches rely on discrete token representations, which disregard the inherently continuous nature of speech and can lead to loss of fine-grained acoustic information. In this work, we investigate the TTS within the MLLM paradigm using continuous speech representations. We design a dual-head architecture and implement two complementary training strategies for a robust model. (1) A diffusion head generating continuous speech representations is added on the MLLM, which is on frame-level and strictly autoregressive. (2) The original language model head is retained to preserve multitask capability and to control the start and end of speech synthesis. (3) Masked training is employed to address exposure bias in autoregressive decoding. (4) To stabilize optimization, we propose a two-stage scheme where the LM is frozen in the second stage, ensuring the diffusion head learns from a fixed input distribution. Evaluations on LibriSpeech(PC) test-clean show that our approach achieves state-of-the-art autoregressive performance, with a WER of 1.95%, speaker similarity of 0.54, and UTMOS of 4.00. The two-stage training yields a 46% relative WER reduction over the one-stage training baseline. These results highlight the effectiveness of combining autoregressive modeling with continuous-token diffusion, supported by a two-stage training procedure.

Figures (2)

• Figure 1: Illustration of the proposed method. Compared to discrete token-based generation in (a), our approach (b) adopts a dual-head MLLM with a diffusion head for frame-level autoregressive generation, enabling continuous speech synthesis with natural and high-fidelity quality.
• Figure 2: Training and design details of our method. (a) Dual-head architecture: The LM head predicts speech boundaries (<s-bos> and <eos>) and the special token <continuous_speech_gen>, while the diffusion head generates continuous frame-level speech embeddings. (b) Masked training strategy: A portion of speech inputs is masked to bridge the gap between teacher-forcing training and auto-regressive inference. (c) Two-stage training: In the first stage, all components are trained jointly; in the second stage, only the Diffusion head is further optimized.