DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion

TL;DR

DiffuSpeech addresses the unreliability of direct speech generation by enabling a diffusion-based model to jointly produce internal text reasoning (thinking traces) and spoken outputs, thereby letting reasoning inform speech quality. It introduces a unified speech-text diffusion framework and the ThinkingTalk dataset to support reasoning-augmented speech generation. The approach achieves state-of-the-art speech-to-speech QA, strong TTS quality, and robust language understanding, with ablations confirming the value of both the diffusion architecture and thinking traces. This work demonstrates that diffusion-based multimodal generation, coupled with explicit thinking traces, can improve accuracy and trustworthiness in spoken dialogue systems and opens avenues for multi-modal extensions. Its ThinkingTalk dataset provides a crucial resource for advancing reasoning-augmented speech applications in real-world settings.

Abstract

Current speech language models generate responses directly without explicit reasoning, leading to errors that cannot be corrected once audio is produced. We introduce \textbf{``Silent Thought, Spoken Answer''} -- a paradigm where speech LLMs generate internal text reasoning alongside spoken responses, with thinking traces informing speech quality. To realize this, we present \method{}, the first diffusion-based speech-text language model supporting both understanding and generation, unifying discrete text and tokenized speech under a single masked diffusion framework. Unlike autoregressive approaches, \method{} jointly generates reasoning traces and speech tokens through iterative denoising, with modality-specific masking schedules. We also construct \dataset{}, the first speech QA dataset with paired text reasoning traces, containing 26K samples totaling 319 hours. Experiments show \method{} achieves state-of-the-art speech-to-speech QA accuracy, outperforming the best baseline by up to 9 points, while attaining the best TTS quality among generative models (6.2\% WER) and preserving language understanding (66.2\% MMLU). Ablations confirm that both the diffusion architecture and thinking traces contribute to these gains.

Figures (5)

• Figure 1: The "Silent Thought, Spoken Answer" paradigm. Given a spoken question, DiffuSpeech jointly generates internal text reasoning (silent thought) alongside the spoken response, with thinking traces informing answer quality. This enables more accurate answers compared to direct speech-to-speech generation.
• Figure 2: Overview of DiffuSpeech architecture and training pipeline.(a) Model Architecture: Speech input is encoded by a frozen HuBERT encoder and quantized into discrete tokens, which are combined with text tokens in a unified vocabulary. The LLaDA backbone performs masked diffusion over the combined sequence, generating both text reasoning (silent thought) and speech tokens. Speech output is synthesized by a frozen HiFi-GAN vocoder. (b) Two-Stage Training: Stage 1 performs speech-text alignment using ASR/TTS data from LibriHeavy, VoxPopuli, and CommonVoice, along with text LM data to prevent forgetting. Stage 2 fine-tunes on the ThinkingTalk dataset for instruction following with thinking traces, enabling the "Silent Thought, Spoken Answer" capability.
• Figure 3: Construction pipeline for the ThinkingTalk dataset. We rewrite existing text QA datasets into an oral style with explicit thinking traces, filter for speech suitability using an LLM judge, and synthesize high-quality audio for both user questions and assistant responses.
• Figure 4: Capability comparison across six dimensions, including speech reasoning (MMSU). DiffuSpeech achieves well-rounded performance, excelling in Speech QA while maintaining strong language understanding and competitive speech reasoning. Values are normalized to [0, 100] with higher being better (WER metrics are inverted).
• Figure 5: Training dynamics of AR vs. Diffusion for speech adaptation. (a) ASR and (b) TTS on LibriSpeech-Clean. While AR (blue) shows faster initial learning, Diffusion (red) surpasses AR mid-training and achieves lower final WER, demonstrating better speech adaptation potential.