Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes

Fangyu Ding, Ding Ding, Sijin Chen, Kaibo Wang, Peng Xu, Zijin Feng, Haoli Bai, Kai Han, Youliang Yan, Binhang Yuan, Jiacheng Sun

Abstract

While Masked Diffusion Language Models (MDLMs) relying on token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibility remain constrained by the masking paradigm. In this paper, we propose Deletion-Insertion Diffusion language models (DID) that rigorously formulate token deletion and insertion as discrete diffusion processes, replacing the masking and unmasking processes in current MDLMs. DID improves training and inference efficiency by eliminating two major sources of computational overhead in MDLMs: the computations on non-informative 1) <MASK> tokens inherent to the paradigm, and 2) <PAD> tokens introduced in variable-length settings. Furthermore, DID offers greater flexibility by: 1) natively supporting variable-length sequences without requiring fixed-length padding, and 2) an intrinsic self-correction mechanism during generation due to insertion that dynamically adjusts token positions. To train DID, we design a score-based approach that assigns scores to token insertion operations and derive appropriate training objectives. The objectives involve subsequence counting problems, which we efficiently solve via a parallelized dynamic programming algorithm. Our experiments across fixed and variable-length settings demonstrate the advantage of DID over baselines of MDLMs and existing insertion-based LMs, in terms of modeling performance, sampling quality, and training/inference speed, without any hyperparameter tuning.

Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes

Abstract

While Masked Diffusion Language Models (MDLMs) relying on token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibility remain constrained by the masking paradigm. In this paper, we propose Deletion-Insertion Diffusion language models (DID) that rigorously formulate token deletion and insertion as discrete diffusion processes, replacing the masking and unmasking processes in current MDLMs. DID improves training and inference efficiency by eliminating two major sources of computational overhead in MDLMs: the computations on non-informative 1) <MASK> tokens inherent to the paradigm, and 2) <PAD> tokens introduced in variable-length settings. Furthermore, DID offers greater flexibility by: 1) natively supporting variable-length sequences without requiring fixed-length padding, and 2) an intrinsic self-correction mechanism during generation due to insertion that dynamically adjusts token positions. To train DID, we design a score-based approach that assigns scores to token insertion operations and derive appropriate training objectives. The objectives involve subsequence counting problems, which we efficiently solve via a parallelized dynamic programming algorithm. Our experiments across fixed and variable-length settings demonstrate the advantage of DID over baselines of MDLMs and existing insertion-based LMs, in terms of modeling performance, sampling quality, and training/inference speed, without any hyperparameter tuning.
Paper Structure (14 sections, 1 theorem, 16 equations, 2 figures, 5 tables)

This paper contains 14 sections, 1 theorem, 16 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and Related Works
  3. Continuous-Time Discrete Diffusion
  4. Insertion-Based Language Models
  5. DID: Deletion-Insertion Diffusion Language Models
  6. Forward Process: Deletion
  7. Backward Process: Insertion
  8. Training Objective: Denoising Insertion Score Entropy
  9. Efficient Parallel Dynamic Programming for Subsequence Counting Problems
  10. Simplified Model for Fixed-Length Setting
  11. Experiments
  12. DID for Fixed-Length Language Modeling
  13. DID for Variable-Length Language Modeling
  14. Conclusion

Key Result

Proposition 1

The DSE objective for the deletion-insertion process is upper bounded by the DISE objective, $\mathcal{L}^\text{DSE}_\theta({\bm{x}}_0) \le \mathcal{L}_\theta^\text{DISE}({\bm{x}}_0)$, which is defined as: where $C = K(\frac{N(\text{Ins}({\bm{x}}_t, i, v), {\bm{x}}_0)}{N({\bm{x}}_t, {\bm{x}}_0)})$ is a $\theta$-free constant, $t\sim\text{Unif}([0,1])$, and $\bm x_t\sim p_{t|0}({\bm{x}}_t|{\bm{x}}

Figures (2)

  • Figure 1: Conceptual diagram of MDLMs compared to Deletion-Insertion Diffusion language models (DID).
  • Figure 2: Cumulative distribution functions (CDFs) of generation length under different total denoising steps.

Theorems & Definitions (1)

  • Proposition 1: Denoising Insertion Score Entropy (DISE)