Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MDM-ASR: Bridging Accuracy and Efficiency in ASR with Diffusion-Based Non-Autoregressive Decoding

Hao Yen, Pin-Jui Ku, Ante Jukić, Sabato Marco Siniscalchi

TL;DR

This work proposes a principled NAR ASR framework based on Masked Diffusion Models and introduces Iterative Self-Correction Training that exposes the model to its own intermediate predictions to mitigate the training-inference mismatch.

Abstract

In sequence-to-sequence Transformer ASR, autoregressive (AR) models achieve strong accuracy but suffer from slow decoding, while non-autoregressive (NAR) models enable parallel decoding at the cost of degraded performance. We propose a principled NAR ASR framework based on Masked Diffusion Models to reduce this gap. A pre-trained speech encoder is coupled with a Transformer diffusion decoder conditioned on acoustic features and partially masked transcripts for parallel token prediction. To mitigate the training-inference mismatch, we introduce Iterative Self-Correction Training that exposes the model to its own intermediate predictions. We also design a Position-Biased Entropy-Bounded Confidence-based sampler with positional bias to further boost results. Experiments across multiple benchmarks demonstrate consistent gains over prior NAR models and competitive performance with strong AR baselines, while retaining parallel decoding efficiency.

MDM-ASR: Bridging Accuracy and Efficiency in ASR with Diffusion-Based Non-Autoregressive Decoding

TL;DR

This work proposes a principled NAR ASR framework based on Masked Diffusion Models and introduces Iterative Self-Correction Training that exposes the model to its own intermediate predictions to mitigate the training-inference mismatch.

Abstract

In sequence-to-sequence Transformer ASR, autoregressive (AR) models achieve strong accuracy but suffer from slow decoding, while non-autoregressive (NAR) models enable parallel decoding at the cost of degraded performance. We propose a principled NAR ASR framework based on Masked Diffusion Models to reduce this gap. A pre-trained speech encoder is coupled with a Transformer diffusion decoder conditioned on acoustic features and partially masked transcripts for parallel token prediction. To mitigate the training-inference mismatch, we introduce Iterative Self-Correction Training that exposes the model to its own intermediate predictions. We also design a Position-Biased Entropy-Bounded Confidence-based sampler with positional bias to further boost results. Experiments across multiple benchmarks demonstrate consistent gains over prior NAR models and competitive performance with strong AR baselines, while retaining parallel decoding efficiency.
Paper Structure (27 sections, 12 equations, 4 figures, 4 tables)

This paper contains 27 sections, 12 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Masked Diffusion Model
  4. Diffusion-/Flow-matching based NAR ASR
  5. Iterative Masked Refinement NAR ASR
  6. Masked Diffusion Based ASR
  7. Audio-conditioned Masked Diffusion Model
  8. Model Architecture
  9. Iterative Self-Correction Training
  10. Inference Samplers
  11. Discrete Flow-Matching Sampler
  12. Confidence-based Samplers
  13. Position-Biased EB-Conf Sampler
  14. Experimental Setup
  15. Datasets & Evaluation Metrics
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Overview of our MDM-ASR framework.
  • Figure 2: The RTFx as a function of sequence length. Shaded regions indicate a $\pm10\%$ variability band around the fitted curve to illustrate the uncertainty of the estimated trend.
  • Figure 3: WER comparison between different samplers.
  • Figure 4: Effect of the proposed ISCT. WER comparison on LS Other for 180M and 1B models under varying maximum NFEs.