Thinking Out of Order: When Output Order Stops Reflecting Reasoning Order in Diffusion Language Models

TL;DR

The paper investigates how output order constraints interact with internal reasoning in language models. It demonstrates that masked diffusion language models (MDLMs) can maintain reasoning accuracy even when prompts require an answer-first output, unlike autoregressive models, by decoupling computation order from output position. Using ReasonOrderQA, a controlled benchmark with graded complexity, the authors show that diffusion-based generation stabilizes simpler reasoning tokens earlier and defers complex final answers, enabling order-robust reasoning. They identify two breakdown conditions—insufficient token complexity differences and large generation lengths—and analyze how sampling strategies affect robustness, providing practical guidance for order-agnostic generation in real-world tasks.

Abstract

Autoregressive (AR) language models enforce a fixed left-to-right generation order, creating a fundamental limitation when the required output structure conflicts with natural reasoning (e.g., producing answers before explanations due to presentation or schema constraints). In such cases, AR models must commit to answers before generating intermediate reasoning, and this rigid constraint forces premature commitment. Masked diffusion language models (MDLMs), which iteratively refine all tokens in parallel, offer a way to decouple computation order from output structure. We validate this capability on GSM8K, Math500, and ReasonOrderQA, a benchmark we introduce with controlled difficulty and order-level evaluation. When prompts request answers before reasoning, AR models exhibit large accuracy gaps compared to standard chain-of-thought ordering (up to 67% relative drop), while MDLMs remain stable ($\leq$14% relative drop), a property we term "order robustness". Using ReasonOrderQA, we present evidence that MDLMs achieve order robustness by stabilizing simpler tokens (e.g., reasoning steps) earlier in the diffusion process than complex ones (e.g., final answers), enabling reasoning tokens to stabilize before answer commitment. Finally, we identify failure conditions where this advantage weakens, outlining the limits required for order robustness.

Figures (13)

• Figure 1: Masked Diffusion models reason before answering—even when asked to answer first. Under "answer-first" prompting, AR models must commit to answers before reasoning (generation order = output order). Masked diffusion LMs decode reasoning tokens first regardless of output position, enabling order robustness.
• Figure 2: ReasonOrderQA benchmark design. Each problem contains (1) a noisy context with hidden variable definitions, (2) an arithmetic formula of varying complexity (D1-D4), and (3) evaluation under two generation orders (Standard: reasoning→answer, Reverse: answer→reasoning). This design decouples retrieval difficulty from reasoning complexity.
• Figure 3: Order robustness across diffusion and autoregressive models. We compare LLaDA (diffusion, trained from scratch), Dream (diffusion, distilled from AR), and Qwen (autoregressive) on GSM8K, Math500, and ReasonOrderQA. Orange bars show Answer-First accuracy; blue portions show additional gain from CoT-First. LLaDA exhibits strong order robustness with minimal CoT-First gain (+2% on GSM8K). Dream, despite being a diffusion model, shows weaker robustness (+46% gain), suggesting that distillation from AR models may preserve order-sensitive behavior. Qwen shows the largest gap (+67%), consistent with AR's structural dependence on generation order.
• Figure 4: Convergence trajectories on GSM8K. Blue and orange lines show LLaDA Diffusion under CoT-First and Answer-First prompting, respectively; gray dashed line shows Qwen2.5-7B final accuracy under Answer-First. Diffusion trajectories overlap almost perfectly regardless of output order, gradually improving to $\sim$55% accuracy. The Qwen baseline achieves only $\sim$23% under Answer-First, confirming the AR model's order sensitivity.
• Figure 5: Answer token confidence decreases with problem difficulty. For each problem, we compute the mean confidence across all diffusion steps for answer tokens (located via the "Answer:" delimiter). Each box shows the distribution across problems within that difficulty level. D1 exhibits the highest mean confidence ($\sim$0.99), while D4 shows the lowest ($\sim$0.55). Variance also increases with difficulty. See Figure \ref{['fig:answer_confidence_boxplot']} for Math500 results.