Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model

TL;DR

This work introduces Stable-DiffCoder, a diffusion-based block code model that reuses the Seed-Coder pipeline to achieve superior performance under the same data and compute budgets. By coupling block diffusion continual pretraining with a tailored warmup and block-wise clipped noise scheduling, the approach learns token reasoning efficiently and demonstrates strong results across code generation, reasoning, and editing benchmarks, often surpassing ~8B autoregressive baselines. The study provides a principled analysis of training-inference alignment, curriculum design, and stability improvements, showing that diffusion-based training can yield quality gains beyond autoregressive training alone and can aid low-resource languages through data augmentation. Overall, the results establish diffusion-based code modeling as a viable and competitive alternative to AR training for both base and instruction-tuned models, with practical implications for faster generation and broader language support.

Abstract

Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.

Figures (4)

• Figure 1: Benchmark performance of Stable-DiffCoder-8B-Instruct.
• Figure 2: Training dynamics under different block sizes. We compare ARDLLM and AR against a BiDLLM baseline for block sizes 32, 2, and 1 (top to bottom). Solid lines indicate training up to 100k steps, while dotted lines denote continued training after switching to BiDLLM. The horizontal dashed line marks the fixed baseline reference (0-step AR).
• Figure 3: As shown by the black solid line, we initialize from the pre-annealing checkpoint of Seed-Coder and perform CPT with small-block DLLM to obtain Stable-DiffCoder-Base, aiming to study efficient knowledge acquisition in diffusion-based models. The dashed line denotes an alternative pipeline for larger blocks, where new knowledge is first compressed using an AR model or a small-block DLLM before being transferred to the large-block diffusion setting.
• Figure 4: Comparison of training stability before and after applying warmup. The left figure shows the behavior without warmup, and the right figure shows the behavior with warmup. When warmup is used, both the training loss and gradient norm become significantly more stable, and they quickly decrease to a level comparable to that of the AR continual pretraining stage. BiDLLM refers to a purely bidirectional masked diffusion model, BlockDLLM to a block-masked diffusion model, and no-shift indicates that token logit shifting is disabled, which is the configuration adopted in our final model.

Theorems & Definitions (1)

• Definition 3.1: Token Reasoning Knowledge