Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data

TL;DR

This paper proposes Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption, and provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.

Abstract

Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabilities, but typically under tightly controlled training setups. It often depends on carefully constructed task-solution pairs and substantial human supervision, which creates a fundamental obstacle to open-ended self-evolution toward superintelligent systems. In this paper, we propose Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption. Initialized from the same base LLM, Tool-R0 co-evolves a Generator and a Solver with complementary rewards: one proposes targeted challenging tasks at the other's competence frontier and the other learns to solve them with real-world tool calls. This creates a self-evolving cycle that requires no pre-existing tasks or datasets. Evaluation on different tool-use benchmarks show that Tool-R0 yields 92.5 relative improvement over the base model and surpasses fully supervised tool-calling baselines under the same setting. Our work further provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.

Figures (14)

• Figure 1: Tool-R0 self-evolution loop.
• Figure 2: Tool-R0 Self-Evolution Framework. A base LLM is initialized into two roles: a Generator and a Solver. The Generator synthesizes challenging tool-calling tasks (question, tool menu, and gold tool-call), targeting the frozen Solver's competence frontier with designed rewards ($r_{\text{fmt}} + r_{\text{valid}} + r_{\text{curr}}$). Generated tasks are filtered and ranked easy-to-hard into a curriculum pool. The Solver trains on this curated data to predict tool calls ($r_{\text{fmt}} + r_{\text{acc}}$), completing a self-evolving cycle that requires no pre-existing human datasets.
• Figure 3: Difficulty reward ($r_{\text{diff}}$).
• Figure 4: Self-play coverage analysis.
• Figure 5: Self-play convergence for extended iterations across model scales.