Towards Agentic Self-Learning LLMs in Search Environment

TL;DR

This work tackles scaling self-learning for open-domain LLM-based agents without human-curated data or predefined rewards. It introduces Agentic Self-Learning (ASL), a closed-loop framework unifying task generation, problem solving, and verification through three co-evolving roles: Prompt Generator, Policy Model, and Generative Reward Model. Key findings show that generative rewards outperform rule-based ones, and co-evolving the GRM with the policy yields stronger performance; increasing agent-generated task data substantially boosts capabilities, and the GRM becomes the main bottleneck, improvable with continual training and occasional real verification data. Empirically, ASL surpasses strong RLVR baselines and remains robust under zero-labeled-data conditions, demonstrating improved sample efficiency and scalability for open-domain agent learning. The authors release their data and code to support reproducibility and further research.

Abstract

We study whether self-learning can scale LLM-based agents without relying on human-curated datasets or predefined rule-based rewards. Through controlled experiments in a search-agent setting, we identify two key determinants of scalable agent training: the source of reward signals and the scale of agent task data. We find that rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning, and that co-evolving the GRM with the policy further boosts performance. Increasing the volume of agent task data-even when synthetically generated-substantially enhances agentic capabilities. Building on these insights, we propose \textbf{Agentic Self-Learning} (ASL), a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone. ASL coordinates a Prompt Generator, a Policy Model, and a Generative Reward Model to form a virtuous cycle of harder task setting, sharper verification, and stronger solving. Empirically, ASL delivers steady, round-over-round gains, surpasses strong RLVR baselines (e.g., Search-R1) that plateau or degrade, and continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness. We further show that GRM verification capacity is the main bottleneck: if frozen, it induces reward hacking and stalls progress; continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling. This work establishes reward source and data scale as critical levers for open-domain agent learning and demonstrates the efficacy of multi-role co-evolution for scalable, self-improving agents. The data and code of this paper are released at https://github.com/forangel2014/Towards-Agentic-Self-Learning

Figures (15)

• Figure 1: The framework of Agentic Self-Learning (ASL).
• Figure 2: Performance comparison between rule-based, fixed GRM, and policy-shared GRM reward functions. A co-evolving GRM yields the best reward quality and downstream policy performance.
• Figure 3: Effect of scaling the amount of GRM-rewarded agentic data on downstream accuracy. Larger amounts of verifiably judged agent-generated data substantially improve policy performance.
• Figure 4: The test accuracy of LLM agents trained by ASL and baseline methods.
• Figure 5: Comparison across iterations. Blue: accuracy of a fixed evaluator (Search-r1, trained 1,024 steps) on 2,000 questions generated in each iteration—lower values indicate harder generated problems. Green: GRM accuracy on the verify set constructed from $(q, a^+; correct)$ and GPT-4o-mini–derived negatives $(q, a^-; wrong)$—higher is better judgment. Yellow: policy model solving accuracy—steadily improving. These patterns show the three roles co-evolve under ASL.