Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

dLLM-ASR: A Faster Diffusion LLM-based Framework for Speech Recognition

Wenjie Tian, Bingshen Mu, Guobin Ma, Xuelong Geng, Zhixian Zhao, Lei Xie

TL;DR

dLLM-ASR tackles the latency of autoregressive LLM-based ASR by reimagining decoding as a prior-guided adaptive denoising process within a discrete diffusion LLM framework. The architecture couples a frozen Whisper-based speech encoder, a lightweight adapter, and a pretrained dLLM (LLaDA) decoder, and introduces a two-stage training regime with LoRA to preserve pretrained capabilities. Key innovations include using an ASR prior to initialize denoising and anchor sequence length, along with token-level adaptive denoising via confidence-based exits and length-adaptive pruning, plus an efficient speech cache and a prior initialization branch. Experiments show dLLM-ASR attains competitive WER with significantly reduced inference time, achieving about 4.44× speedups over strong AR baselines and outperforming non-prior diffusion approaches, signaling a practical diffusion-based pathway for efficient ASR.

Abstract

Automatic speech recognition (ASR) systems based on large language models (LLMs) achieve superior performance by leveraging pretrained LLMs as decoders, but their token-by-token generation mechanism leads to inference latency that grows linearly with sequence length. Meanwhile, discrete diffusion large language models (dLLMs) offer a promising alternative, enabling high-quality parallel sequence generation with pretrained decoders. However, directly applying native text-oriented dLLMs to ASR leads to a fundamental mismatch between open-ended text generation and the acoustically conditioned transcription paradigm required by ASR. As a result, it introduces unnecessary difficulty and computational redundancy, such as denoising from pure noise, inflexible generation lengths, and fixed denoising steps. We propose dLLM-ASR, an efficient dLLM-based ASR framework that formulates dLLM's decoding as a prior-guided and adaptive denoising process. It leverages an ASR prior to initialize the denoising process and provide an anchor for sequence length. Building upon this prior, length-adaptive pruning dynamically removes redundant tokens, while confidence-based denoising allows converged tokens to exit the denoising loop early, enabling token-level adaptive computation. Experiments demonstrate that dLLM-ASR achieves recognition accuracy comparable to autoregressive LLM-based ASR systems and delivers a 4.44$\times$ inference speedup, establishing a practical and efficient paradigm for ASR.

dLLM-ASR: A Faster Diffusion LLM-based Framework for Speech Recognition

TL;DR

dLLM-ASR tackles the latency of autoregressive LLM-based ASR by reimagining decoding as a prior-guided adaptive denoising process within a discrete diffusion LLM framework. The architecture couples a frozen Whisper-based speech encoder, a lightweight adapter, and a pretrained dLLM (LLaDA) decoder, and introduces a two-stage training regime with LoRA to preserve pretrained capabilities. Key innovations include using an ASR prior to initialize denoising and anchor sequence length, along with token-level adaptive denoising via confidence-based exits and length-adaptive pruning, plus an efficient speech cache and a prior initialization branch. Experiments show dLLM-ASR attains competitive WER with significantly reduced inference time, achieving about 4.44× speedups over strong AR baselines and outperforming non-prior diffusion approaches, signaling a practical diffusion-based pathway for efficient ASR.

Abstract

Automatic speech recognition (ASR) systems based on large language models (LLMs) achieve superior performance by leveraging pretrained LLMs as decoders, but their token-by-token generation mechanism leads to inference latency that grows linearly with sequence length. Meanwhile, discrete diffusion large language models (dLLMs) offer a promising alternative, enabling high-quality parallel sequence generation with pretrained decoders. However, directly applying native text-oriented dLLMs to ASR leads to a fundamental mismatch between open-ended text generation and the acoustically conditioned transcription paradigm required by ASR. As a result, it introduces unnecessary difficulty and computational redundancy, such as denoising from pure noise, inflexible generation lengths, and fixed denoising steps. We propose dLLM-ASR, an efficient dLLM-based ASR framework that formulates dLLM's decoding as a prior-guided and adaptive denoising process. It leverages an ASR prior to initialize the denoising process and provide an anchor for sequence length. Building upon this prior, length-adaptive pruning dynamically removes redundant tokens, while confidence-based denoising allows converged tokens to exit the denoising loop early, enabling token-level adaptive computation. Experiments demonstrate that dLLM-ASR achieves recognition accuracy comparable to autoregressive LLM-based ASR systems and delivers a 4.44 inference speedup, establishing a practical and efficient paradigm for ASR.
Paper Structure (23 sections, 2 equations, 2 figures, 2 tables)

This paper contains 23 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Speech Recognition Models
  4. Discrete Diffusion LLMs
  5. dLLM-ASR Framework
  6. Overall Architecture
  7. Speech Encoder
  8. Adapter
  9. dLLM Decoder
  10. Training Strategies
  11. Prior-Guided Adaptive Denoising
  12. Confidence-based Denoising
  13. Length-Adaptive Pruning
  14. Efficient Speech Cache
  15. Prior Initialization
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Overview of the proposed dLLM-ASR. On the left side of the figure is the model's architecture and training process. The input consists of the prompt, input speech, and answer, where the answer comprises the prompt and the ground-truth speech transcript. Red components indicate trainable modules, while blue ones represent frozen modules. The right side displays the transformation of the inference process. Subfigure 1 shows the baseline Whisper-LLaDA, which directly combines the speech encoder with LLaDA for ASR. Subfigures 2–4 visualize how our proposed method is built by incrementally applying confidence-based denoising, pruning, and ASR Prior initialization.
  • Figure 2: Ablation study on the confidence threshold ($\tau$) for the confidence-based allocation strategy. The results are reported on LibriSpeech (LS) clean and other test sets.