RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance
Tianlang Chen, Minkai Xu, Jure Leskovec, Stefano Ermon
TL;DR
This work tackles the challenge of enhancing reasoning in diffusion-based large language models without resorting to additional reward-model training. It introduces Reward-Free Guidance (RFG), which defines a trajectory-level reward as a log-likelihood ratio between a policy dLLM and a reference dLLM, and derives per-step process rewards to guide the denoising trajectory. The authors prove that their stepwise guidance yields a reward-guided sampling distribution and demonstrate strong, training-free improvements across math reasoning and code generation benchmarks using diverse model families and post-training methods. The approach is practical and broadly applicable, achieving notable gains without extra reward data or tuning, and offering a flexible framework for future alignment and reasoning enhancements in diffusion-based systems.
Abstract
Diffusion large language models (dLLMs) have shown great potential in large-scale language modeling, and there is an increasing interest in further improving the capacity to solve complex problems by guiding the reasoning process step by step. Common practice for autoregressive language models typically learns a process reward model with dense annotation for each intermediate step. However, this is challenging for dLLMs where the generation is in an any-order fashion and intermediate states are partially masked sentences. To this end, in this paper, we propose reward-free guidance (RFG), a principled method for guiding the reasoning trajectory of dLLMs without explicit process reward. The key idea of RFG is to parameterize the process reward by log-likelihood ratios of the enhanced and reference dLLMs, where the enhanced model can be easily obtained by any off-the-shelf dLLM that has been post-trained with reinforcement learning (RL) or supervised fine-tuning (SFT). We provide theoretical justification that RFG induces the reward-guided sampling distribution with no additional reward. We conduct comprehensive experiments on four challenging mathematical reasoning and code generation benchmarks using a diverse suite of dLLMs enhanced with various post-training methods. RFG consistently yields significant improvements across all tasks and model types, achieving accuracy gains of up to 9.2%. These findings establish RFG as a general training-free framework that scales test-time reasoning without reliance on external reward models.