The Flexibility Trap: Why Arbitrary Order Limits Reasoning Potential in Diffusion Language Models

TL;DR

This paper questions the assumed advantage of arbitrary-order generation in diffusion language models, showing that unrestricted order narrows rather than expands reasoning potential due to entropy-driven bypass of high-uncertainty forks. It introduces JustGRPO, a minimalist RL approach that restricts training to autoregressive order while preserving the diffusion model's parallel decoding at inference. Empirically, JustGRPO achieves strong reasoning performance (e.g., $89.1\%$ on GSM8K and $45.1\%$ on MATH) and robust behavior across generation budgets, outperforming diffusion-specific RL methods. The findings advocate re-evaluating the value of order arbitrariness in diffusion models and highlight the practicality of AR-focused training with GRPO for scalable reasoning tasks.

Abstract

Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior reasoning potential for general tasks like mathematics and coding. Consequently, numerous works have leveraged reinforcement learning (RL) to elicit the reasoning capability of dLLMs. In this paper, we reveal a counter-intuitive reality: arbitrary order generation, in its current form, narrows rather than expands the reasoning boundary of dLLMs. We find that dLLMs tend to exploit this order flexibility to bypass high-uncertainty tokens that are crucial for exploration, leading to a premature collapse of the solution space. This observation challenges the premise of existing RL approaches for dLLMs, where considerable complexities, such as handling combinatorial trajectories and intractable likelihoods, are often devoted to preserving this flexibility. We demonstrate that effective reasoning is better elicited by intentionally forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, JustGRPO, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs. Project page: https://nzl-thu.github.io/the-flexibility-trap

Figures (12)

• Figure 1: Less flexibility unlocks better reasoning potential.Left: We observe a counter-intuitive phenomenon where restricting dLLMs to standard Autoregressive (AR) order expands the reasoning solution space. Right: Motivated by this, we propose "JustGRPO". By foregoing complex arbitrary-order adaptations and adopting standard GRPO, we effectively elicit the reasoning capability of dLLMs.
• Figure 2: Confronting vs. bypassing uncertainty. (a) AR order preserves reasoning space by forcing decisions at uncertain tokens. (b) Arbitrary order bypasses uncertainty and resolves easier tokens first. Once future context is established, the original forks collapse, prematurely narrowing the solution space.
• Figure 3: Reasoning potential measured by Pass@$k$. While arbitrary order is competitive in single-shot settings ($k=1$), it exhibits notably flatter scaling curves compared to AR Order.
• Figure 4: Solution space coverage measured by Pass@$1024$. The reasoning traces generated by arbitrary order are largely a strict subset of those generated by AR Order.
• Figure 5: Frequently bypassed tokens in arbitrary order, measured on MATH-500, are typically logical connectors and transition words.