Multi-Agent Tool-Integrated Policy Optimization

TL;DR

The paper tackles the challenge of enabling robust, knowledge-intensive reasoning with LLMs by addressing context length and tool response noise in single-agent TIP. It introduces MATPO, a multi-agent-in-one-model reinforcement learning framework that trains planner and worker roles within a single LLM instance through role-specific prompts and a principled credit assignment mechanism. The authors provide a theoretical basis and concrete implementation, demonstrating that MATPO outperforms single-agent baselines on multiple benchmarks with an average relative improvement of 18.38% and improved robustness to noisy tool outputs. The work offers practical guidance for stable multi-agent RL training and highlights future directions for expanding agent roles and improving RL infrastructure for multi-turn tool integration.

Abstract

Large language models (LLMs) increasingly rely on multi-turn tool-integrated planning for knowledge-intensive and complex reasoning tasks. Existing implementations typically rely on a single agent, but they suffer from limited context length and noisy tool responses. A natural solution is to adopt a multi-agent framework with planner- and worker-agents to manage context. However, no existing methods support effective reinforcement learning post-training of tool-integrated multi-agent frameworks. To address this gap, we propose Multi-Agent Tool-Integrated Policy Optimization (MATPO), which enables distinct roles (planner and worker) to be trained within a single LLM instance using role-specific prompts via reinforcement learning. MATPO is derived from a principled credit assignment mechanism across planner and worker rollouts. This design eliminates the need to deploy multiple LLMs, which would be memory-intensive, while preserving the benefits of specialization. Experiments on GAIA-text, WebWalkerQA, and FRAMES show that MATPO consistently outperforms single-agent baselines by an average of 18.38% relative improvement in performance and exhibits greater robustness to noisy tool outputs. Our findings highlight the effectiveness of unifying multiple agent roles within a single LLM and provide practical insights for stable and efficient multi-agent RL training.

Figures (6)

• Figure 1: Multi-agent framework. At each step, the planner-agent creates and assigns new subtasks to worker-agents; the planner-agent generates successive subtasks or final answers based on the worker-agents’ responses.
• Figure 2: Visualization of a single-agent multi-turn TIP rollout. The LLM solves a query through iterative planning and tool-use. At each step, it plans a tool call, executes it with the parsed parameters, and uses the tool response to decide the next move, continuing until it is confident enough to produce a final answer.
• Figure 3: Visualization of a multi-agent multi-turn TIP rollout. At each step, the planner agent generates and assigns a subtask to the worker agent, which completes it via multi-turn TIP and returns the result. The planner agent then decides whether to generate a new subtask or produce the final answer based on this response.
• Figure 4: An illustration of the implementation of MATPO.
• Figure 5: Test accuracy on three benchmarks across different training steps. Models are trained on the MusiQue dataset musique.