DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping

TL;DR

DeepPlanner identifies persistent planning-token entropy as a bottleneck in end-to-end deep research agents and introduces two end-to-end reinforcement learning mechanisms to scale planning: entropy-based advantage shaping (EAS) and selective advantage upweighting (SAU). Built on the GRPO token-level framework, these methods amplify learning on uncertain planning decisions and on high-quality, complex rollouts without triggering entropy collapse. Experiments across seven deep research benchmarks show state-of-the-art performance with substantially lower training budgets (3,072 queries and 8 rollouts per query), and ablations demonstrate explicit planning benefits and the complementary effects of EAS and SAU. The work advances practical planning capabilities for deep research agents, enabling more reliable, scalable, and efficient long-horizon information synthesis and tool integration.

Abstract

Large language models (LLMs) augmented with multi-step reasoning and action generation abilities have shown promise in leveraging external tools to tackle complex tasks that require long-horizon planning. However, existing approaches either rely on implicit planning in the reasoning stage or introduce explicit planners without systematically addressing how to optimize the planning stage. As evidence, we observe that under vanilla reinforcement learning (RL), planning tokens exhibit significantly higher entropy than other action tokens, revealing uncertain decision points that remain under-optimized. To address this, we propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research agents. Our approach shapes token-level advantage with an entropy-based term to allocate larger updates to high entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts. Extensive experiments across seven deep research benchmarks demonstrate that DeepPlanner improves planning quality and achieves state-of-the-art results under a substantially lower training budget.

Figures (12)

• Figure 1: When the agent is instructed to explicitly write a research plan and then execute it, we observe that the planning stage exhibits much higher token entropy (0.78) than other stages (0.32) during RL training.
• Figure 2: The overview of DeepPlanner. For each rollout, GRPO token-level advantages are augmented with $\alpha \times$ token entropy (clipped to avoid sign flips). Within each group, rollout(s) that reach the correct answer with fewer tool calls receive a higher weight (tool-call counts gate complexity).
• Figure 3: In the vanilla GRPO framework, planning-stage entropy is markedly higher than other execution-stage entropy, revealing underdeveloped planning capacity cui2025entropymechanismreinforcementlearning. With the advantage shaping mechanisms, our DeepPlanner effectively transforms the planning entropy to the performance on deep research tasks without inducing global entropy collapse.
• Figure 4: The ratio $\frac{\mathcal{H}_{i,t}}{|A_{i,t}|}$ indicates that our approach does not over-encourage during the training process.
• Figure 5: The average number of tool calls between different approaches in both the training and test sets.