Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use

Qihao Wang, Mingzhe Lu, Jiayue Wu, Yue Hu, Yanbing Liu

TL;DR

PEARL tackles the challenge of long-horizon, multi-hop tool use by large language models through a decoupled two-stage framework: offline exploration for reliable tool execution and online reinforcement learning to train a Planner that generates complete tool plans before execution. The Planner is optimized with Group Relative Policy Optimization (GRPO) using a planning-centric reward, while an offline-exploration-grounded Executor carries out the plan step-by-step. Empirical results on ToolHop and T-Eval show state-of-the-art performance, with a ToolHop SR of $56.5\%$ and an IER of $3.8\%$, and a T-Eval SR of $77.0\%$ with an IER of $1.0\%$, indicating strong generalization and reliability. The work also demonstrates that the learned planning strategy generalizes across executors and even boosts the performance of other strong models when guided by the PEARL Planner, highlighting the practical impact for robust, autonomous LLM-based agents in complex tool-enabled tasks.

Abstract

Large Language Models show great potential with external tools, but face significant challenges in complex, multi-turn tool invocation. They often exhibit weak planning, tool hallucination, erroneous parameter generation, and struggle with robust interaction. To tackle these issues, we present PEARL, a novel framework to enhance LLM planning and execution for sophisticated tool use. PEARL adopts a two-stage approach: an offline phase where the agent explores tools to learn valid usage patterns and failure conditions, and an online reinforcement learning phase. In the online phase, a dedicated Planner is trained via group Relative Policy Optimization (GRPO) with a carefully designed reward function that provides distinct signals for planning quality. Experiments on the ToolHop and T-Eval benchmarks show PEARL significantly outperforms existing methods, achieving a new state-of-the-art success rate of \textbf{56.5\%} on ToolHop while maintaining a low invocation error rate. Our work marks a key advance in addressing the complex planning challenges of tool use, contributing to the development of more robust and reliable LLM-based agents.

PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use

TL;DR

PEARL tackles the challenge of long-horizon, multi-hop tool use by large language models through a decoupled two-stage framework: offline exploration for reliable tool execution and online reinforcement learning to train a Planner that generates complete tool plans before execution. The Planner is optimized with Group Relative Policy Optimization (GRPO) using a planning-centric reward, while an offline-exploration-grounded Executor carries out the plan step-by-step. Empirical results on ToolHop and T-Eval show state-of-the-art performance, with a ToolHop SR of and an IER of , and a T-Eval SR of with an IER of , indicating strong generalization and reliability. The work also demonstrates that the learned planning strategy generalizes across executors and even boosts the performance of other strong models when guided by the PEARL Planner, highlighting the practical impact for robust, autonomous LLM-based agents in complex tool-enabled tasks.

Abstract

Large Language Models show great potential with external tools, but face significant challenges in complex, multi-turn tool invocation. They often exhibit weak planning, tool hallucination, erroneous parameter generation, and struggle with robust interaction. To tackle these issues, we present PEARL, a novel framework to enhance LLM planning and execution for sophisticated tool use. PEARL adopts a two-stage approach: an offline phase where the agent explores tools to learn valid usage patterns and failure conditions, and an online reinforcement learning phase. In the online phase, a dedicated Planner is trained via group Relative Policy Optimization (GRPO) with a carefully designed reward function that provides distinct signals for planning quality. Experiments on the ToolHop and T-Eval benchmarks show PEARL significantly outperforms existing methods, achieving a new state-of-the-art success rate of \textbf{56.5\%} on ToolHop while maintaining a low invocation error rate. Our work marks a key advance in addressing the complex planning challenges of tool use, contributing to the development of more robust and reliable LLM-based agents.
Paper Structure (26 sections, 5 equations, 1 figure, 4 tables)

This paper contains 26 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Tool Use
  4. Planning and Reasoning in LLMs
  5. Reinforcement Learning for LLMs
  6. Method
  7. Task Formulation
  8. A Two-Stage Framework: Planning and Execution
  9. Stage 1: Strategic Plan Generation
  10. Stage 2: Informed Execution
  11. Offline Tool Exploration.
  12. Online Step-by-Step Execution.
  13. Planner Optimization with GRPO
  14. Group-wise Advantage Normalization.
  15. Policy Optimization Objective.
  16. ...and 11 more sections

Figures (1)

  • Figure 1: The PEARL framework's two-stage architecture. Stage 1 (Plan Generation): A dedicated Planner is trained via reinforcement learning (GRPO) with a planning-centric reward to create a multi-step plan. Stage 2 (Informed Execution): An Executor first learns tool usage through an offline exploration phase and then reliably executes the generated plan step-by-step.