Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving and Evaluating Open Deep Research Agents

Doaa Allabadi, Kyle Bradbury, Jordan M. Malof

TL;DR

The paper tackles evaluating Open Deep Research Agents (DRAs) under realistic, open benchmarks. It introduces BrowseComp-Small (BC-Small), a computationally tractable subset of BrowseComp, and presents ODR+, an open-source DRA that adds sub-question decomposition, iterative sub-solution search, and structured response synthesis to the baseline ODR. On BC-Small, ODR+ achieves 10% exact-match accuracy on the test set, surpassing both the original open-source baseline and several closed-source systems, with ablation studies confirming the contribution of each module. The work provides an open, reproducible framework to study and improve DRAs, with implications for scalable evaluation and development in open research communities. $D_{ ext{max}}=6$, $T_{ ext{max}}=210$ seconds, $k=3$, and $N_{ ext{query}}=3$ are among the key settings guiding the evaluation.$

Abstract

We focus here on Deep Research Agents (DRAs), which are systems that can take a natural language prompt from a user, and then autonomously search for, and utilize, internet-based content to address the prompt. Recent DRAs have demonstrated impressive capabilities on public benchmarks however, recent research largely involves proprietary closed-source systems. At the time of this work, we only found one open-source DRA, termed Open Deep Research (ODR). In this work we adapt the challenging recent BrowseComp benchmark to compare ODR to existing proprietary systems. We propose BrowseComp-Small (BC-Small), comprising a subset of BrowseComp, as a more computationally-tractable DRA benchmark for academic labs. We benchmark ODR and two other proprietary systems on BC-Small: one system from Anthropic and one system from Google. We find that all three systems achieve 0% accuracy on the test set of 60 questions. We introduce three strategic improvements to ODR, resulting in the ODR+ model, which achieves a state-of-the-art 10% success rate on BC-Small among both closed-source and open-source systems. We report ablation studies indicating that all three of our improvements contributed to the success of ODR+.

Improving and Evaluating Open Deep Research Agents

TL;DR

The paper tackles evaluating Open Deep Research Agents (DRAs) under realistic, open benchmarks. It introduces BrowseComp-Small (BC-Small), a computationally tractable subset of BrowseComp, and presents ODR+, an open-source DRA that adds sub-question decomposition, iterative sub-solution search, and structured response synthesis to the baseline ODR. On BC-Small, ODR+ achieves 10% exact-match accuracy on the test set, surpassing both the original open-source baseline and several closed-source systems, with ablation studies confirming the contribution of each module. The work provides an open, reproducible framework to study and improve DRAs, with implications for scalable evaluation and development in open research communities. , seconds, , and are among the key settings guiding the evaluation.$

Abstract

We focus here on Deep Research Agents (DRAs), which are systems that can take a natural language prompt from a user, and then autonomously search for, and utilize, internet-based content to address the prompt. Recent DRAs have demonstrated impressive capabilities on public benchmarks however, recent research largely involves proprietary closed-source systems. At the time of this work, we only found one open-source DRA, termed Open Deep Research (ODR). In this work we adapt the challenging recent BrowseComp benchmark to compare ODR to existing proprietary systems. We propose BrowseComp-Small (BC-Small), comprising a subset of BrowseComp, as a more computationally-tractable DRA benchmark for academic labs. We benchmark ODR and two other proprietary systems on BC-Small: one system from Anthropic and one system from Google. We find that all three systems achieve 0% accuracy on the test set of 60 questions. We introduce three strategic improvements to ODR, resulting in the ODR+ model, which achieves a state-of-the-art 10% success rate on BC-Small among both closed-source and open-source systems. We report ablation studies indicating that all three of our improvements contributed to the success of ODR+.

Paper Structure

This paper contains 14 sections, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. The BrowseComp-Small Benchmark
  4. Open Deep Research
  5. Open Deep Research Plus (ODR+)
  6. Question Decomposition
  7. Iterative Sub-Solution Search
  8. Response Synthesis
  9. Numerical Experiments
  10. ODR+ Development and Hyperparameter Settings
  11. Evaluation Methodology
  12. Main Results
  13. Ablation Studies
  14. Conclusions

Figures (2)

  • Figure 1: ODR system architecture illustrating the three main steps: (1) Web URL Search; (2) Content Extraction; and (3) Response Generation.
  • Figure 2: ODR+ system architecture, illustrating three major steps: (1) Question decomposition; (2) an iterative Sub-solution Search step, which seeks internet-based evidence to address each sub-question; and (3) Response Synthesis, where a structured response, denoted $r_{+}$, is generated for the user based upon a summary of evidence from the internet.