Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models

TL;DR

This work identifies initialization mismatch as a key bottleneck in diffusion-based language decoding and investigates context-aware, training-free warm-starts to shorten the diffusion trajectory. It introduces two concrete interfaces—token-level injection and embedding-level interpolation—for injecting prompt-conditioned priors from lightweight auxiliary models, along with a remasking mechanism to mitigate miscalibration. Preliminary GSM8K results show meaningful reductions in denoising iterations (roughly 35%), but also a notable accuracy gap relative to strong diffusion baselines, highlighting calibration and representation-alignment challenges. Overall, the paper lays out a research agenda for reliable warm-started diffusion decoding, emphasizing calibration, revision policies, and end-to-end throughput considerations.

Abstract

Diffusion Large Language Models (DLLMs) enable fully parallel token decoding but often remain impractical at inference time due to the many denoising iterations required to refine an information-free, fully masked initialization into coherent text. Most existing acceleration methods focus on traversing this generative trajectory more efficiently via improved solvers or sampling strategies. We advance a complementary perspective: shorten the trajectory itself by starting closer to the target distribution through context-aware initialization. We propose a training-free interface that injects prompt-conditioned priors from a lightweight auxiliary model into the diffusion initialization, and instantiate it with two mechanisms: discrete token injection and representation-level embedding interpolation. Because injected priors can be imperfect and unmask-only decoding can over-commit early, we also introduce a simple confidence-based remasking mechanism as a form of prior skepticism. Preliminary evidence on GSM8K suggests that context-aware initialization can substantially reduce denoising iterations (about 35\% fewer function evaluations in our setting), while also exposing a key open challenge: naive warm-starting can degrade final accuracy relative to strong diffusion baselines. We use these findings to motivate a research agenda around calibration, revision mechanisms, and representation alignment for reliable warm-started diffusion decoding.

Figures (4)

• Figure 1: Autoregressive models generate tokens sequentially conditioned on prior outputs, while diffusion language models decode all positions in parallel via iterative denoising. Adapted from li2025surveydiffusionlanguagemodels.
• Figure 2: Model confidence increases with denoising steps, suggesting that earlier access to reliable structure could reduce the required refinement budget. Adapted from nie2025largelanguagediffusionmodels.
• Figure 3: Token injection: initialize a subset of positions with auxiliary warm tokens (injection dropout), leaving the rest masked. Injected tokens act as a noisy prior and can be revoked by remasking.
• Figure 4: Embedding interpolation: keep the discrete initialization fully masked, but warm-start the DLLM by interpolating the mask embedding with the warm-token embedding, with injection dropout applied in embedding space.