Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents

Xuan-Phi Nguyen, Shrey Pandit, Revanth Gangi Reddy, Austin Xu, Silvio Savarese, Caiming Xiong, Shafiq Joty

TL;DR

This work presents SFR-DeepResearch, a framework that turns reasoning-optimized LLMs into autonomous single-agent DR systems using a minimal set of tools. It introduces an inference scaffolding with internal memory management and a synthetic-data RL pipeline that leverages a length-normalized advantage to stabilize long-horizon tool use. The approach yields competitive results across DR benchmarks, notably 28.7% on Humanity's Last Exam for the best model (SFR-DR-20B), and provides detailed analyses of workflow design, tool usage, and response lengths. The findings highlight the importance of controlling context, thoughtful tool invocation, and model-specific training setups to realize robust, scalable agentic DR behavior with moderate data and compute.

Abstract

Equipping large language models (LLMs) with complex, interleaved reasoning and tool-use capabilities has become a key focus in agentic AI research, especially with recent advances in reasoning-oriented (``thinking'') models. Such capabilities are key to unlocking a number of important applications. One such application is Deep Research (DR), which requires extensive search and reasoning over many sources. Our work in this paper focuses on the development of native Autonomous Single-Agent models for DR featuring minimal web crawling and Python tool integration. Unlike multi-agent systems, where agents take up pre-defined roles and are told what to do at each step in a static workflow, an autonomous single-agent determines its next action dynamically based on context, without manual directive. While prior work has proposed training recipes for base or instruction-tuned LLMs, we focus on continual reinforcement learning (RL) of reasoning-optimized models to further enhance agentic skills while preserving reasoning ability. Towards this end, we propose a simple RL recipe with entirely synthetic data, which we apply to various open-source LLMs. Our best variant SFR-DR-20B achieves up to 28.7% on Humanity's Last Exam benchmark. In addition, we conduct key analysis experiments to provide more insights into our methodologies.

SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents

TL;DR

This work presents SFR-DeepResearch, a framework that turns reasoning-optimized LLMs into autonomous single-agent DR systems using a minimal set of tools. It introduces an inference scaffolding with internal memory management and a synthetic-data RL pipeline that leverages a length-normalized advantage to stabilize long-horizon tool use. The approach yields competitive results across DR benchmarks, notably 28.7% on Humanity's Last Exam for the best model (SFR-DR-20B), and provides detailed analyses of workflow design, tool usage, and response lengths. The findings highlight the importance of controlling context, thoughtful tool invocation, and model-specific training setups to realize robust, scalable agentic DR behavior with moderate data and compute.

Abstract

Equipping large language models (LLMs) with complex, interleaved reasoning and tool-use capabilities has become a key focus in agentic AI research, especially with recent advances in reasoning-oriented (``thinking'') models. Such capabilities are key to unlocking a number of important applications. One such application is Deep Research (DR), which requires extensive search and reasoning over many sources. Our work in this paper focuses on the development of native Autonomous Single-Agent models for DR featuring minimal web crawling and Python tool integration. Unlike multi-agent systems, where agents take up pre-defined roles and are told what to do at each step in a static workflow, an autonomous single-agent determines its next action dynamically based on context, without manual directive. While prior work has proposed training recipes for base or instruction-tuned LLMs, we focus on continual reinforcement learning (RL) of reasoning-optimized models to further enhance agentic skills while preserving reasoning ability. Towards this end, we propose a simple RL recipe with entirely synthetic data, which we apply to various open-source LLMs. Our best variant SFR-DR-20B achieves up to 28.7% on Humanity's Last Exam benchmark. In addition, we conduct key analysis experiments to provide more insights into our methodologies.

Paper Structure

This paper contains 18 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Deep Research Agents
  4. Agentic Tool-integrated Reinforcement Learning
  5. SFR-DeepResearch
  6. Agentic Inference Scaffolding
  7. Tools
  8. Single-agent Agentic Workflow
  9. Training Data Synthesis Pipeline
  10. End-to-End Reinforcement Learning Recipe
  11. Experiments
  12. Main Evaluation
  13. Analysis
  14. Effectiveness of Modified Agentic Workflow
  15. More Tool Calls Do Not Equate Better Scores
  16. ...and 3 more sections

Figures (3)

  • Figure 1: An example tool calling trajectory by our SFR-DR agentic workflow, catered for QwQ-32B and Qwen3 models. The multi-turn interaction is framed as a single-turn contextual question answering problem, where there is always only 1 user turn. The previous tool calls and responses are packed as memory and placed in the user turn together with the question.
  • Figure 2: Average training trajectory lengths of SFR-DR-8B agents over the course of RL training with and without our proposed length normalization (\ref{['sec:end_to_end_rl']}). Without the normalization term, tool usage rapidly increases but the performance on HLE phan2025humanity collapses. On the other hand, our length normalization regulates tool usage, improving performance.
  • Figure 3: Comparison of (a) average tool usage and (b) average step-level response lengths (tokens) on HLE across different models.