Reinforcement Learning Foundations for Deep Research Systems: A Survey

TL;DR

This survey advances a training-centric view of RL for deep research systems, arguing that end-to-end RL of a Planner within a hierarchical Planner–Coordinator–Executors stack offers robust long-horizon reasoning and tool orchestration beyond what SFT/DPO can sustain. It structures the literature into three pillars: data synthesis and curation to produce challenging, verifiable tasks; RL methods that improve stability, efficiency, and multimodal integration; and agentic RL training frameworks that address scalability and reproducibility. The authors synthesize patterns, bottlenecks, and design choices across data, algorithms, and systems, offering practical guidance on constructing complex queries, reward design, and framework selection, while highlighting evaluation benchmarks for holistic agentic performance. The work emphasizes that a well-designed training regime—combined with modular architectures and solid evaluation—can unlock robust, auditable, tool-rich agents capable of open-web reasoning and long-form synthesis in real-world settings.

Abstract

Deep research systems, agentic AI that solve complex, multi-step tasks by coordinating reasoning, search across the open web and user files, and tool use, are moving toward hierarchical deployments with a Planner, Coordinator, and Executors. In practice, training entire stacks end-to-end remains impractical, so most work trains a single planner connected to core tools such as search, browsing, and code. While SFT imparts protocol fidelity, it suffers from imitation and exposure biases and underuses environment feedback. Preference alignment methods such as DPO are schema and proxy-dependent, off-policy, and weak for long-horizon credit assignment and multi-objective trade-offs. A further limitation of SFT and DPO is their reliance on human defined decision points and subskills through schema design and labeled comparisons. Reinforcement learning aligns with closed-loop, tool-interaction research by optimizing trajectory-level policies, enabling exploration, recovery behaviors, and principled credit assignment, and it reduces dependence on such human priors and rater biases. This survey is, to our knowledge, the first dedicated to the RL foundations of deep research systems. It systematizes recent work along three axes: (i) data synthesis and curation; (ii) RL methods for agentic research covering stability, sample efficiency, long context handling, reward and credit design, multi-objective optimization, and multimodal integration; and (iii) agentic RL training systems and frameworks. We also cover agent architecture and coordination, as well as evaluation and benchmarks, including recent QA, VQA, long-form synthesis, and domain-grounded, tool-interaction tasks. We distill recurring patterns, surface infrastructure bottlenecks, and offer practical guidance for training robust, transparent deep research agents with RL.

Figures (3)

• Figure 1: Illustration of the hierarchical deep research system architecture.
• Figure 2: The organizational structure of the survey and representative papers under each branch.
• Figure 3: Illustration of QA Task Complexity Levels.