Search Self-play: Pushing the Frontier of Agent Capability without Supervision

TL;DR

Search Self-Play (SSP) introduces a self-supervised paradigm to train deep search agents without human-annotated data by letting a single LLM play dual roles: task proposer and problem solver. The proposer crafts ground-truth-grounded, increasingly difficult search queries, while the solver is challenged to answer them using multi-turn reasoning and tool calls; a RAG-based verification step uses the proposer's retrieved documents to confirm answerability. Through a min-max objective with cooperation, the proposer and solver co-evolve, guided by PAYOFF signals and an LLM-based judge, leading to progressive improvements in search, reasoning, and verification capabilities. Experiments across seven benchmarks demonstrate consistent improvements across model sizes and training regimes, including from-scratch and continual training, without external supervision, highlighting SSP as a scalable path for agentic LLM training.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become the mainstream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers to provide accurate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be controlled to provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously as both a task proposer and a problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries to handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, we collect all the searching results from the proposer's trajectory as external knowledge, then conduct retrieval-augmentation generation (RAG) to test whether the proposed query can be correctly answered with all necessary search documents provided. In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision under both from-scratch and continuous RL training setups. The code is at https://github.com/Qwen-Applications/SSP.

Figures (7)

• Figure 1: Performance gains of deep search agents trained via Search Self-play (SSP) across various agentic benchmarks. Our SSP method uniformly surpasses multiple strong open-source baselines without any agentic data annotation and additional supervision.
• Figure 2: Examples of Search Self-play with a given ground-truth "Dr. Will Boyd". Provided with the ground-truth, the proposer iteratively uses search tools to excavate implicit factual evidence, then generates a challenging search question. Then the solver leverages all search results from the proposer's trajectory as the RAG materials to predict the answer without searching, to validate the question's correctness. Once verified, the solver follows the ordinary deep search pipeline to explore the solution via multi-turn agentic rollout.
• Figure 3: Training dynamics of different SSP variants. (a) shows the in-game reward. (b) and (c) display the evaluation accuracy on the held-out NQ and 2Wiki datasets over training steps.
• Figure 4: Diversity and length distribution of pre-defined answer set ${\mathcal{D}}$.
• Figure 5: Training dynamics of SSP with Qwen2.5-7B-Base. (a) The agent learns to use the search tool more frequently. (b) Solver response length increases while prompt length remains stable. (c, d) Evaluation scores on both GeneralQA and Multi-HopQA datasets show continuous improvement during training.