WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection

TL;DR

This work tackles open-domain, multi-hop QA by addressing shallow tool-use and error accumulation in agentic RAG. It introduces WebSeer, a self-reflective reinforcement learning framework with a unified two-stage pipeline that combines cold-start supervised refinement and iterative RL, enabling a single 14B model to perform deeper, more robust tool use in real web environments. The system leverages three tools (web search, webpage reading, and code execution) and a rejection-sampling data collection method to foster long-horizon, reflective reasoning, achieving state-of-the-art results on HotpotQA and SimpleQA and strong generalization to out-of-domain data. These results demonstrate a substantial advance in robust, self-verifying search agents capable of deep, adaptable reasoning in dynamic web contexts.

Abstract

Search agents have achieved significant advancements in enabling intelligent information retrieval and decision-making within interactive environments. Although reinforcement learning has been employed to train agentic models capable of more dynamic interactive retrieval, existing methods are limited by shallow tool-use depth and the accumulation of errors over multiple iterative interactions. In this paper, we present WebSeer, a more intelligent search agent trained via reinforcement learning enhanced with a self-reflection mechanism. Specifically, we construct a large dataset annotated with reflection patterns and design a two-stage training framework that unifies cold start and reinforcement learning within the self-reflection paradigm for real-world web-based environments, which enables the model to generate longer and more reflective tool-use trajectories. Our approach substantially extends tool-use chains and improves answer accuracy. Using a single 14B model, we achieve state-of-the-art results on HotpotQA and SimpleQA, with accuracies of 72.3% and 90.0%, respectively, and demonstrate strong generalization to out-of-distribution datasets. The code is available at https://github.com/99hgz/WebSeer

Figures (7)

• Figure 1: Comparasion between different search agent: Search-r1jin2025searchr1trainingllmsreason faces shallow search and error buildup, while WebSeer trained using our self-reflection paradigm significantly improve tool calls and maintain rigorous reasoning.
• Figure 2: Overview of our two-stage training framework. In the first stage, we construct self-reflective, tool-augmented reasoning trajectories. In the second stage, we apply SRRL, allowing the model to iteratively refine and resubmit answers with F1-based feedback.
• Figure 3: Evaluation on three harder benchmarks. Qwen2.5-14B and WebSeer use web search engine, while Search-r1 relies on local RAG.
• Figure 4: Accuracy and average tool call times for Qwen2.5 Models on HotpotQA and SimpleQA
• Figure 5: Tool call distributions on HotpotQA across three training stages: Pre-SFT, Post-SFT, and Post-RL.