Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off
Zhaochun Li, Chen Wang, Jionghao Bai, Shisheng Cui, Ge Lan, Zhou Zhao, Yue Wang
TL;DR
This work reframes exploration in RL with verifiable rewards for LLMs as a distribution-level problem, introducing Distribution-Centric Policy Optimization (DCPO). DCPO uses a virtual high-entropy target, REINFORCE as a regularizer, and double importance sampling to regulate entropy on-policy, achieving stable exploration and improved reasoning across seven benchmarks. Through controlled experiments contrasting sample-centric and distribution-centric hypotheses, the authors show that the gradient of the target distribution—not rare samples—drives entropy dynamics, supporting a distribution-centric EE view. Empirically, DCPO outperforms GRPO and entropy-based baselines by about 20% on average, with stronger gains on difficult contest-style tasks and robust Pass@128 improvements, demonstrating a principled path to controllable exploration in RLVR. A key insight is the Precision–Prediction trade-off, balancing exact sampling against distribution-guided optimization to achieve the desired EE balance.
Abstract
The exploration-exploitation (EE) trade-off is a central challenge in reinforcement learning (RL) for large language models (LLMs). With Group Relative Policy Optimization (GRPO), training tends to be exploitation driven: entropy decreases monotonically, samples convergence, and exploration fades. Most existing fixes are \textbf{sample-centric}: they seek or bonus rare samples, assuming exploration comes from novel trajectories and tokens. These heuristics depend on the "luck" of informative samples, lack principled control of the policy, and often yield limited or inconsistent gains. In this work, we are the first to introduce a \textbf{distribution-centric} perspective for RL, in which exploration is always guided by a "better" target distribution, and reveal that a policy's ability to resist entropy collapse is governed by the distribution itself rather than individual samples. Building on this insight, we propose Distribution-Centric Policy Optimization (DCPO), which reformulates entropy regulation as distribution-level regularization. DCPO achieves controllable entropy fully on-policy without sampling from external distributions, enabling efficient exploration while maintaining training stability. Across multiple models and seven benchmarks, DCPO improves over GRPO by about 20\% on average. Overall, DCPO replaces sample-level heuristics with distribution-level principles, offering a theoretically grounded and flexible framework for controllable exploration and a stronger EE trade-off. The code is available in https://github.com/597358816/DCPO.