Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STDD:Spatio-Temporal Dynamics-Driven Token Refinement in Diffusion Language Models

Xinhao Sun, Maoliang Li, Zihao Zheng, Jiayu Chen, Hezhao Xu, Yun Liang, Xiang Chen

TL;DR

This work tackles the inefficiency of fixed-threshold remasking in diffusion language models by introducing a per-token, step-adaptive remasking framework grounded in spatio-temporal token dynamics. By defining Temporal Variance and Spatial Deviance, the method captures each token's convergence status and inter-token correlations, enabling dynamic thresholds $ au_t^i$ that adapt as decoding progresses. The approach, STDD, also includes a feasibility optimization with Suspected Fast/Slow token mechanisms to further stabilize decoding. Empirical results show substantial speedups (up to $8.9\times$) while preserving or improving accuracy across benchmarks, and STDD remains compatible with existing acceleration methods like dKV-Cache and ICE, suggesting broad applicability in diffusion-based decoding workflows.

Abstract

Unlike autoregressive language models, diffusion language models (DLMs) generate text by iteratively denoising all token positions in parallel. At each timestep, the remasking strategy of a DLM selects low-priority tokens to defer their decoding, thereby improving both efficiency and output quality. However, mainstream remasking strategies rely on a single global confidence threshold, overlooking the temporal and spatial dynamics of individual tokens. Motivated by the redundant iterations and constrained parallelism introduced by fixed-threshold remasking, we propose a novel remasking approach that dynamically detects Temporal Variance and Spatial Deviance of each token, which reflect its convergence status and inter-token correlations. Using these signals, our method adaptively adjusts the confidence threshold for every token at every step. Empirical results show that our approach significantly improves the operational efficiency of DLMs across mainstream datasets, achieving speedups of up to 8.9 times while faithfully preserving generation quality.

STDD:Spatio-Temporal Dynamics-Driven Token Refinement in Diffusion Language Models

TL;DR

This work tackles the inefficiency of fixed-threshold remasking in diffusion language models by introducing a per-token, step-adaptive remasking framework grounded in spatio-temporal token dynamics. By defining Temporal Variance and Spatial Deviance, the method captures each token's convergence status and inter-token correlations, enabling dynamic thresholds that adapt as decoding progresses. The approach, STDD, also includes a feasibility optimization with Suspected Fast/Slow token mechanisms to further stabilize decoding. Empirical results show substantial speedups (up to ) while preserving or improving accuracy across benchmarks, and STDD remains compatible with existing acceleration methods like dKV-Cache and ICE, suggesting broad applicability in diffusion-based decoding workflows.

Abstract

Unlike autoregressive language models, diffusion language models (DLMs) generate text by iteratively denoising all token positions in parallel. At each timestep, the remasking strategy of a DLM selects low-priority tokens to defer their decoding, thereby improving both efficiency and output quality. However, mainstream remasking strategies rely on a single global confidence threshold, overlooking the temporal and spatial dynamics of individual tokens. Motivated by the redundant iterations and constrained parallelism introduced by fixed-threshold remasking, we propose a novel remasking approach that dynamically detects Temporal Variance and Spatial Deviance of each token, which reflect its convergence status and inter-token correlations. Using these signals, our method adaptively adjusts the confidence threshold for every token at every step. Empirical results show that our approach significantly improves the operational efficiency of DLMs across mainstream datasets, achieving speedups of up to 8.9 times while faithfully preserving generation quality.
Paper Structure (17 sections, 6 equations, 6 figures, 3 tables)

This paper contains 17 sections, 6 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Prelimilary
  3. Diffusion Language Models
  4. Remasking Strategies in DLM
  5. Temporal-Spatial Token Dynamics
  6. Temporal Analysis for Specific Tokens
  7. Spatial Analysis for Specific Decoding Step
  8. Temporal-Spatial Integration
  9. Spatio-Temporal Dynamics-Driven Token Refinement
  10. Temporal-Informed Remasking
  11. Spatial-Informed Remasking
  12. Spatial-Temporal Feasibility Optimization
  13. Experiment
  14. Experiment Setup
  15. Overall Performance
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Current Confidence-only Strategy
  • Figure 2: Token Remasking Strategies
  • Figure 3: Token's Temporal Variance
  • Figure 4: Anomalous Tokens in Spatial Analysis
  • Figure 5: Token Distribution w/ Temporal-Spatial Dynamics
  • ...and 1 more figures