SearchGym: Bootstrapping Real-World Search Agents via Cost-Effective and High-Fidelity Environment Simulation

TL;DR

SearchGym introduces a high-fidelity offline simulation to bootstrap robust search agents, addressing the cost and data misalignment challenges of training with live web APIs. It builds a verifiable knowledge graph and aligned document corpus through a rigorous data-generation pipeline, enabling strictly solvable reasoning tasks. A curriculum-based SearchGym-RL framework leverages a two-stage progression and a GRPO objective to cultivate hierarchical reasoning with purified feedback. Across Llama and Qwen families, agents trained in SearchGym demonstrate strong Sim-to-Real generalization, with zero API costs and competitive to superior web-based baselines, validating high-fidelity simulation as a scalable, cost-efficient path for developing capable search agents.

Abstract

Search agents have emerged as a pivotal paradigm for solving open-ended, knowledge-intensive reasoning tasks. However, training these agents via Reinforcement Learning (RL) faces a critical dilemma: interacting with live commercial Web APIs is prohibitively expensive, while relying on static data snapshots often introduces noise due to data misalignment. This misalignment generates corrupted reward signals that destabilize training by penalizing correct reasoning or rewarding hallucination. To address this, we propose SearchGym, a simulation environment designed to bootstrap robust search agents. SearchGym employs a rigorous generative pipeline to construct a verifiable knowledge graph and an aligned document corpus, ensuring that every reasoning task is factually grounded and strictly solvable. Building on this controllable environment, we introduce SearchGym-RL, a curriculum learning methodology that progressively optimizes agent policies through purified feedback, evolving from basic interactions to complex, long-horizon planning. Extensive experiments across the Llama and Qwen families demonstrate strong Sim-to-Real generalization. Notably, our Qwen2.5-7B-Base model trained within SearchGym surpasses the web-enhanced ASearcher baseline across nine diverse benchmarks by an average relative margin of 10.6%. Our results validate that high-fidelity simulation serves as a scalable and highly cost-effective methodology for developing capable search agents.

Figures (5)

• Figure 1: An overview of the SearchGym pipeline. From a verified knowledge graph (left), we sample reasoning paths of varying structures. These paths are verbalized into complex, solvable Question-Answering pairs (right), categorized as Simple, Parallel, or Combo QA. This creates a closed-loop environment where the LLM agent interacts with a search engine to retrieve evidence and solve the tasks. The legend defines the visual elements.
• Figure 2: Scalability analysis of SearchGym-RL. Left: Impact of maximum reasoning depth (hops) on downstream performance. Right: Performance trajectories across varying percentages of the training corpus. Standard QA represents the average score across single and multi-hop benchmarks; Challenging QA denotes the average on GAIA and xbench-DeepSearch.
• Figure 3: Comparison of training dynamics between our method stage one and Search-R1. The Training Avg Score (left) represents the mean reward achieved for each rollout within a training batch. Our approach demonstrates stable, monotonic convergence towards a high reward, whereas Search-R1 exhibits significant volatility and eventual policy collapse. Note that the Search-R1 training curves are derived from their official public logs searchr1_wandb.
• Figure 4: Qualitative comparison on a GAIA task. The SearchGym-RL agent (Qwen2.5-7B-Base) (Left) correctly extracts "Marquette, Michigan" from search results to anchor the reasoning chain (Jacques Marquette $\to$ Marquette Building $\to$ Holabird & Root). In contrast, ASearcher-Web-Qwen2.5-7B-Base (Right) succumbs to context bias: despite retrieving the correct entity, it ignores the evidence and hallucinates "Chicago" as the target city, leading to a hallucinated trajectory and task failure.
• Figure 5: Qualitative case study on a multi-hop reasoning task from SearchGymBench. The query requires an agent to sequentially identify a birth city, its country, and the country's official language. (Right) The SearchGym agent(Qwen2.5-7B-Base), trained in our high-fidelity simulation, effectively decomposes the problem, executes a correct chain of searches (Gholir Hayami $\to$ Riverside $\to$ Ythrel $\to$ Fhalth), and grounds the final answer in retrieved evidence. (Left & Center) In contrast, baseline agents fail to sustain the search process. ASearcher-Web-QwQ (Left) terminates prematurely, hallucinating "English" based on the document's metadata rather than its content. The kimi-k2 baseline (Center) similarly fabricates a fictional context ("Republic of Valeria") when faced with ambiguity, highlighting the tendency of standard agents to rely on parametric fabrication rather than rigorous information retrieval.