Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Caio Vicentino

Abstract

We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Abstract

We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.
Paper Structure (35 sections, 3 equations, 2 figures, 5 tables)

This paper contains 35 sections, 3 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. This work.
  3. Contributions
  4. Related Work
  5. Autoregressive Language Models
  6. Diffusion Language Models
  7. Existing Comparisons
  8. Experimental Setup
  9. Controlled Variables
  10. Parameter Count Difference
  11. Autoregressive Model
  12. Masked Diffusion Language Model (MDLM)
  13. Training.
  14. Generation.
  15. Results
  16. ...and 20 more sections

Figures (2)

  • Figure 1: Convergence comparison. Dual $y$-axes reflect different loss functions; the shape of each curve (not the absolute value) is the relevant comparison. AR overfits after step 14K (blue shaded region); MDLM continues improving monotonically.
  • Figure 2: Training throughput and wall-clock time. MDLM achieves 95.5% of AR throughput, requiring only 5.1 additional minutes over 20K steps.