Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding

Xiangzhong Luo, Yilin An, Zhicheng Yu, Weichen Liu, Xu Yang

Abstract

Diffusion-based large language models (dLLMs) have shown promising performance across various reasoning tasks, establishing themselves as an alternative to autoregressive large language models (LLMs). Unlike autoregressive LLMs that generate one token per step based on all previous tokens, dLLMs theoretically enable parallel generation of multiple tokens at each decoding step. However, recent dLLMs still favor one-token-per-step generation in practice, as directly decoding multiple masked tokens often leads to degraded generation quality and stability. This reveals a substantial gap between the theoretical parallelism and practical performance of dLLMs. To bridge this gap, we introduce an adaptive parallel decoding approach, namely DiCo, which features a three-phase divide-and-conquer paradigm to unleash the inherent parallelism of dLLMs. During the Divide phase, DiCo first explores the input masked sequence and identifies masked tokens as seed tokens, which are then expanded to construct a set of local clusters. During the Conquer phase, DiCo performs parallel decoding across different local clusters constructed in the Divide phase. The divide-and-conquer process repeatedly alternates between the Divide and Conquer phases until convergence. During the Finalize phase, DiCo decodes the remaining few masked tokens using an effective fine-grained compound decoding scheme to finalize the generation. Extensive experiments demonstrate that DiCo can achieve significant inference speedups while maintaining competitive generation quality.

Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding

Abstract

Diffusion-based large language models (dLLMs) have shown promising performance across various reasoning tasks, establishing themselves as an alternative to autoregressive large language models (LLMs). Unlike autoregressive LLMs that generate one token per step based on all previous tokens, dLLMs theoretically enable parallel generation of multiple tokens at each decoding step. However, recent dLLMs still favor one-token-per-step generation in practice, as directly decoding multiple masked tokens often leads to degraded generation quality and stability. This reveals a substantial gap between the theoretical parallelism and practical performance of dLLMs. To bridge this gap, we introduce an adaptive parallel decoding approach, namely DiCo, which features a three-phase divide-and-conquer paradigm to unleash the inherent parallelism of dLLMs. During the Divide phase, DiCo first explores the input masked sequence and identifies masked tokens as seed tokens, which are then expanded to construct a set of local clusters. During the Conquer phase, DiCo performs parallel decoding across different local clusters constructed in the Divide phase. The divide-and-conquer process repeatedly alternates between the Divide and Conquer phases until convergence. During the Finalize phase, DiCo decodes the remaining few masked tokens using an effective fine-grained compound decoding scheme to finalize the generation. Extensive experiments demonstrate that DiCo can achieve significant inference speedups while maintaining competitive generation quality.
Paper Structure (18 sections, 1 theorem, 17 equations, 7 figures, 3 tables)

This paper contains 18 sections, 1 theorem, 17 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Generation Process of dLLMs
  5. Theoretical Inspirations
  6. Empirical Observations
  7. Motivations
  8. DiCo: Divide and Conquer
  9. Overview
  10. Phase 1: How to Divide
  11. Phase 2: How to Conquer
  12. Phase 3: How to Finalize
  13. Experiments
  14. Experimental Settings
  15. Main Results
  16. ...and 3 more sections

Key Result

Theorem 4.1

For a given dLLM, let $\mathbf{x} = (\mathbf{x}_0, \ldots, \mathbf{x}_{n-1})$ denote its decoding trajectory of length $n$, where $\mathbf{x}_{i}$ represents the token predicted at position $i$ during decoding. For each position $i \in [0,\ldots,n-1]$, its prediction probability under greedy decodin We define $\epsilon = \max_{0\le i\le n-1} \epsilon_i$. Based on the above, we can derive that the

Figures (7)

  • Figure 1: Comparisons between autoregressive LLMs and non-autoregressive dLLMs, where each box denotes a specific token.
  • Figure 2: Illustration of the performance collapse of naive parallel decoding on GSM8K (left) and Math-500 (right).
  • Figure 3: Visualization of two representative decoding trajectories of LLaDA-8B-Instruct on GSM8K. The blue points represent the predicted confidence of masked tokens, while the red points denote unmasked tokens. (best viewed when zoomed in)
  • Figure 4: Visualization of attention patterns in LLaDA-8B-Instruct during the decoding process (accumulated on GSM8K).
  • Figure 5: Ablation results of the number of seek tokens $N$ on HumanEval under the non-AR (left) and semi-AR (right) settings.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Theorem 4.1
  • proof
  • proof