Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Way

TL;DR

The paper tackles the rigidity of fixed-length generation in diffusion LLMs by introducing dLLM-Var, a training paradigm that enables native variable-length generation through EOS prediction, fixed EOS masking, and multi-sample packing with full attention. This approach preserves bidirectional context while enabling block diffusion and efficient KV-cache reuse, achieving substantial speedups (up to 30.1x vs a diffusion baseline and 2.4x vs autoregressive models) with competitive accuracy across diverse benchmarks. Key contributions include a novel EOS-masking schedule, multi-sample packing, and inference leveraging block diffusion under full attention, which together unlock practical, scalable diffusion-based generation. The results suggest dLLM-Var significantly advances the practicality of dLLMs for real-world tasks, including editing and self-correction capabilities, with released code and models supporting broader adoption.

Abstract

Diffusion-based large language models (dLLMs) have exhibited substantial potential for parallel text generation, which may enable more efficient generation compared to autoregressive models. However, current dLLMs suffer from fixed generation lengths, which indicates the generation lengths of dLLMs have to be determined before decoding as a hyper-parameter, leading to issues in efficiency and flexibility. To solve these problems, in this work, we propose to train a diffusion LLM with native variable generation lengths, abbreviated as dLLM-Var. Concretely, we aim to train a model to accurately predict the [EOS] token in the generated text, which makes a dLLM be able to natively infer in a block diffusion manner, while still maintaining the ability of global bi-directional (full) attention and high parallelism. Experiments on standard benchmarks demonstrate that our method achieves a 30.1x speedup over traditional dLLM inference paradigms and a 2.4x speedup relative to autoregressive models such as Qwen and Llama. Our method achieves higher accuracy and faster inference, elevating dLLMs beyond mere academic novelty and supporting their practical use in real-world applications. Codes and models have been released.

Figures (7)

• Figure 1: Overview of Probabilistic Modeling Paradigms for Text Generation: Evolution from Autoregressive to Diffusion-Based Approaches. AR & MTP (left): Low parallelism, variable generation. Vanilla dLLMs (right): High parallelism, fixed lengths. dLLM-Var (middle): variable generation lengths while maintaining parallelism.
• Figure 2: (a) LLaDA Base: generates irrelevant content without timely EOS token, complicating response extraction. (b) LLaDA Instruct under pure diffusion: fails to produce any effective text, prematurely filling trailing masks with EOS tokens.
• Figure 3: During the masking forward process of dLLM-Var, tokens in the prompt are never masked. In the response section, tokens are replaced with a <mask> token based on a probability, while the final EOS token is always masked.
• Figure 4: The inference process of dLLM-Var. For the prompt and the already generated blocks, they will be stored in the form of a KV cache to accelerate the model’s inference.
• Figure 5: Training Loss curve during our training of dLLM-Var.