Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DARE: Diffusion Large Language Models Alignment and Reinforcement Executor

Jingyi Yang, Yuxian Jiang, Xuhao Hu, Shuang Cheng, Biqing Qi, Jing Shao

Abstract

Diffusion large language models (dLLMs) are emerging as a compelling alternative to dominant autoregressive models, replacing strictly sequential token generation with iterative denoising and parallel generation dynamics. However, their open-source ecosystem remains fragmented across model families and, in particular, across post-training pipelines, where reinforcement learning objectives, rollout implementations and evaluation scripts are often released as paper-specific codebases. This fragmentation slows research iteration, raises the engineering burden of reproduction, and makes fair comparison across algorithms difficult. We present \textbf{DARE} (\textbf{d}LLMs \textbf{A}lignment and \textbf{R}einforcement \textbf{E}xecutor), an open framework for post-training and evaluating dLLMs. Built on top of verl~\cite{sheng2024hybridflow} and OpenCompass~\cite{2023opencompass}, DARE unifies supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and dLLM-specific reinforcement learning under a shared execution stack for both masked and block diffusion language models. Across representative model families including LLaDA, Dream, SDAR, and LLaDA2.x, DARE provides broad algorithmic coverage, reproducible benchmark evaluation, and practical acceleration. Extensive empirical results position that DARE serves as a reusable research substrate for developing, comparing, and deploying post-training methods for current and emerging dLLMs.

DARE: Diffusion Large Language Models Alignment and Reinforcement Executor

Abstract

Diffusion large language models (dLLMs) are emerging as a compelling alternative to dominant autoregressive models, replacing strictly sequential token generation with iterative denoising and parallel generation dynamics. However, their open-source ecosystem remains fragmented across model families and, in particular, across post-training pipelines, where reinforcement learning objectives, rollout implementations and evaluation scripts are often released as paper-specific codebases. This fragmentation slows research iteration, raises the engineering burden of reproduction, and makes fair comparison across algorithms difficult. We present \textbf{DARE} (\textbf{d}LLMs \textbf{A}lignment and \textbf{R}einforcement \textbf{E}xecutor), an open framework for post-training and evaluating dLLMs. Built on top of verl~\cite{sheng2024hybridflow} and OpenCompass~\cite{2023opencompass}, DARE unifies supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and dLLM-specific reinforcement learning under a shared execution stack for both masked and block diffusion language models. Across representative model families including LLaDA, Dream, SDAR, and LLaDA2.x, DARE provides broad algorithmic coverage, reproducible benchmark evaluation, and practical acceleration. Extensive empirical results position that DARE serves as a reusable research substrate for developing, comparing, and deploying post-training methods for current and emerging dLLMs.

Paper Structure

This paper contains 25 sections, 3 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Diffusion Language Models
  4. Post-Training Frameworks and Reinforcement Learning for dLLMs
  5. DARE Framework
  6. Overview
  7. Model Families
  8. Unified Training Executor
  9. Abstracted as worker, dataflow, workflow.
  10. Shared workflow.
  11. Integrated RL Algorithms.
  12. Algorithm plug-in points.
  13. Acceleration Backends
  14. Training-side optimization.
  15. Rollout-side optimization for MDLMs.
  16. ...and 10 more sections

Figures (3)

  • Figure 1: High-level view of DARE. The framework connects dLLM families, model-aware acceleration backends, a unified post-training executor, and an OpenCompass-based evaluation platform under one shared execution stack.
  • Figure 2: Attention backend comparison across training-side and rollout-side for the masked dLLMs pipeline. (a) shows that optimized attention backends substantially reduce SFT iteration latency relative to eager and sdpa ones. (b) shows that rollout-side backend choice has an even larger effect, with Fast-dLLM-based execution yielding the lowest rollout latency. Together, these results motivate DARE's decoupled optimization of training and rollout paths.
  • Figure 3: Training-curves across different tasks, algorithms and backbones.