Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents

Yuhang Zhou, Kai Zheng, Qiguang Chen, Mengkang Hu, Qingfeng Sun, Can Xu, Jingjing Chen

TL;DR

This work challenges the prevailing assumption that online reinforcement learning is essential for high-performance deep research agents by showing that a carefully designed offline training pipeline can achieve competitive results. It introduces DeepForge, an open-source framework that synthesizes large-scale, multi-hop deep research tasks without heavy preprocessing, and releases a substantial resource suite (66k QA pairs, 33k SFT trajectories, 21k DPO pairs). The OffSeeker 8B model, trained entirely offline via SFT and Direct Preference Optimization, demonstrates strong performance across six benchmarks, matching or approaching 30B-parameter online-RL systems while drastically reducing API costs and improving reproducibility. Together, these contributions advance accessible, cost-effective development of deep research agents and open avenues for broader academic participation in this area.

Abstract

Deep research agents have shown remarkable potential in handling long-horizon tasks. However, state-of-the-art performance typically relies on online reinforcement learning (RL), which is financially expensive due to extensive API calls. While offline training offers a more efficient alternative, its progress is hindered by the scarcity of high-quality research trajectories. In this paper, we demonstrate that expensive online reinforcement learning is not all you need to build powerful research agents. To bridge this gap, we introduce a fully open-source suite designed for effective offline training. Our core contributions include DeepForge, a ready-to-use task synthesis framework that generates large-scale research queries without heavy preprocessing; and a curated collection of 66k QA pairs, 33k SFT trajectories, and 21k DPO pairs. Leveraging these resources, we train OffSeeker (8B), a model developed entirely offline. Extensive evaluations across six benchmarks show that OffSeeker not only leads among similar-sized agents but also remains competitive with 30B-parameter systems trained via heavy online RL.

OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents

TL;DR

This work challenges the prevailing assumption that online reinforcement learning is essential for high-performance deep research agents by showing that a carefully designed offline training pipeline can achieve competitive results. It introduces DeepForge, an open-source framework that synthesizes large-scale, multi-hop deep research tasks without heavy preprocessing, and releases a substantial resource suite (66k QA pairs, 33k SFT trajectories, 21k DPO pairs). The OffSeeker 8B model, trained entirely offline via SFT and Direct Preference Optimization, demonstrates strong performance across six benchmarks, matching or approaching 30B-parameter online-RL systems while drastically reducing API costs and improving reproducibility. Together, these contributions advance accessible, cost-effective development of deep research agents and open avenues for broader academic participation in this area.

Abstract

Deep research agents have shown remarkable potential in handling long-horizon tasks. However, state-of-the-art performance typically relies on online reinforcement learning (RL), which is financially expensive due to extensive API calls. While offline training offers a more efficient alternative, its progress is hindered by the scarcity of high-quality research trajectories. In this paper, we demonstrate that expensive online reinforcement learning is not all you need to build powerful research agents. To bridge this gap, we introduce a fully open-source suite designed for effective offline training. Our core contributions include DeepForge, a ready-to-use task synthesis framework that generates large-scale research queries without heavy preprocessing; and a curated collection of 66k QA pairs, 33k SFT trajectories, and 21k DPO pairs. Leveraging these resources, we train OffSeeker (8B), a model developed entirely offline. Extensive evaluations across six benchmarks show that OffSeeker not only leads among similar-sized agents but also remains competitive with 30B-parameter systems trained via heavy online RL.
Paper Structure (46 sections, 4 equations, 6 figures, 4 tables)

This paper contains 46 sections, 4 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. DeepForge: Large Scale Deep-Research Data Synthesis
  3. Scalable Entity Expansion
  4. Complex Question Generation
  5. Difficulty Analysis of Synthesized Tasks
  6. Methodology
  7. Overall Agent Framework
  8. Supervised Fine-tuning
  9. SFT Trajectory Construction.
  10. Training Objective.
  11. Direct Preference Optimization
  12. Preference Data Construction.
  13. Training Objective.
  14. Preliminary: Online RL Cost Analysis
  15. Experiment Setup.
  16. ...and 31 more sections

Figures (6)

  • Figure 1: Overview of our DeepForge data synthesis pipeline. DeepForge comprises two main stages: (a) Scalable Entity Expansion and (b) Complex Question Generation. After synthesizing complex deep search tasks, we further deploy an agent framework to collect high-quality trajectories.
  • Figure 2: Comparison of tool calling turns distribution between DeepForge-generated tasks and BrowseComp-en with DeepSeek-v3.1.
  • Figure 3: Analysis of GRPO training results. Left: Model performance (score) on BrowseComp-ZH during GRPO training. Middle: Cumulative API expenditure (in USD) throughout training. Right: Time consumption (minutes) of each training step.
  • Figure 4: Pass@1 accuracy (%) on BrowseComp-zh for different SFT models and dataset sizes. For our model, all tasks are generated by DeepForge.
  • Figure 5: OffSeeker test accuracy (%) on GAIA and BrowseComp-ZH with different context window sizes.
  • ...and 1 more figures