Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs

TL;DR

This work reframes TravelPlanner as a multi-step, tool-augmented MDP $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$ with $\gamma=1$, where an LLM-based agent must produce a schema-compliant itinerary via tool calls. The authors propose Planner-R1, leveraging reward shaping via a multi-stage reward $r$ that combines $r_{schema}, r_{cs}, r_{hard}, r_{pass}$ with stage-specific weights $\lambda$, enabling dense guidance early and sparse end goals later while preserving the same optimal policy. Optimization uses GRPO, a clipped PPO-style objective, to update policy parameters from beam-like trajectories collected during interaction with seven planning tools, balancing constraint satisfaction and plan quality. Empirically, smaller 8B models become highly efficient and competitive under dense rewards, while 32B models achieve state-of-the-art results (final-pass of 56.9% on TravelPlanner’s official test) with robust generalization to out-of-domain benchmarks like NaturalPlan, Multi-IF, and $\tau$-Bench; overall, reward shaping proves a decisive lever for scaling agentic RL with smaller models and reduced compute/memory footprints. The work also contributes system-level optimizations for memory management and provides extensive qualitative analyses of failure modes and tool-use trajectories, underscoring practical viability for efficient, generalizable agentic reasoning on real-world planning tasks.

Abstract

We investigated Agentic RL with large language models on the \textsc{TravelPlanner} benchmark. Our approach, \textsc{Planner-R1}, achieved a \textbf{56.9\%} final-pass rate with only 180 training queries, a $2.7\times$ improvement over GPT-5's $21.2\%$ baseline and the strongest agentic result on the public leaderboard. A central finding was that smaller models (8B) were highly responsive to reward shaping: with dense process-level signals, they reached competitive performance while being $3.5\times$ more compute-efficient and $1.5\times$ more memory-efficient than 32B models. Larger models were more robust under sparse rewards but exhibited smaller relative gains from shaping and higher variance across runs. While curriculum learning offered no significant benefit, shaped rewards consistently amplified learning dynamics, making 8B models the most efficient setting for agentic RL. Crucially, these gains did not come at the cost of overfitting: fine-tuned models mostly maintained or exceeded baseline performance on out-of-domain tasks, including \textsc{Multi-IF}, \textsc{NaturalPlan}, and $τ$-\textsc{Bench}. These results establish reward shaping as a decisive lever for scaling agentic RL, highlight the competitive strength of smaller models, and demonstrate that efficiency can be achieved without sacrificing generalization.

Figures (4)

• Figure 1: Final-pass rate on the leaderboard test set for tool-use travel planning. Our Planner-R1 models outperformed SOTA LLMs reaching $56.9\%$ average final pass rate.
• Figure 2: MDP Visualization. $x_i$ represent the $i$th token, while $a_t$ represents the action the agent took at time $t$. Notice that initial prompts and tool responses contain tokens, but they dont increase the time step $t$.
• Figure 3: Performance of 8B and 32B Planner-R1 during training based on learning steps (left) and training FLOPS (right). The horizontal dashed line highlights 90% of the maximum average performance of 32B models, while vertical dashed lines show the required FLOPs to reach that performance by both 8B and 32B models.
• Figure 4: Progression of top 5 failures for 8B (left) and 32B (right) Planner-R1 during training