Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly

TL;DR

It is revealed that DLMs possess an inherent ability to discover the correct infilling length, and a training-free method, CAL, enables DLMs to approximate the optimal length through an efficient search before formal decoding.

Abstract

Diffusion language models (DLMs) provide a bidirectional generation framework naturally suited for infilling, yet their performance is constrained by the pre-specified infilling length. In this paper, we reveal that DLMs possess an inherent ability to discover the correct infilling length. We identify two key statistical phenomena in the first-step denoising confidence: a local \textit{Oracle Peak} that emerges near the ground-truth length and a systematic \textit{Length Bias} that often obscures this signal. By leveraging this signal and calibrating the bias, our training-free method \textbf{CAL} (\textbf{C}alibrated \textbf{A}daptive \textbf{L}ength) enables DLMs to approximate the optimal length through an efficient search before formal decoding. Empirical evaluations demonstrate that CAL improves Pass@1 by up to 47.7\% over fixed-length baselines and 40.5\% over chat-based adaptive methods in code infilling, while boosting BLEU-2 and ROUGE-L by up to 8.5\% and 9.9\% in text infilling. These results demonstrate that CAL paves the way for robust DLM infilling without requiring any specialized training. Code is available at https://github.com/NiuHechang/Calibrated_Adaptive_Length.

Figures (4)

• Figure 1: Infilling sensitivity to a fixed mask length. Evaluated on HumanEval-Infilling humaneval_infilling using LLaDA-8B-Instruct. An underestimated length ($L=4$) truncates the completion, whereas an overestimated length ($L=16$) introduces redundant tokens and may yield invalid logic (e.g., division by zero) to fill the remaining masked positions. The correct solution is only recovered when the length matches the ground truth ($L=10$).
• Figure 2: Statistical Analysis of First-Step Denoising Confidence $\Phi(L)$. (a) A distinct confidence peak emerges when aligned with the ground-truth length. (b) Without length information, confidence exhibits systematic decay, necessitating calibration. Statistics are derived from 100 random HumanEval-Infilling humaneval_infilling tasks using LLaDA-8B-Base.
• Figure 3: Calibrated Adaptive Length Framework. The method operates in two stages: (1) Length Probing, where a bidirectional hill-climbing search identifies an estimated length $\hat{L}$ to approximate the Oracle length $L^*$ by maximizing the calibrated first-step confidence $\Phi_c(L)$; and (2) Formal Decoding, where the model performs standard iterative denoising initialized with the discovered length $\hat{L}$.
• Figure 4: Dynamics of the confidence landscape during bidirectional search. We compare the raw confidence $\Phi(L)$ (red) and the calibrated score $\Phi_c(L)$ (blue solid). While the raw signal is biased towards shorter lengths, our calibration against the systematic bias $B(L)$ (blue dashed) reveals a prominent Oracle Peak at the ground-truth length $L^*=10$ (green dotted line).