ScholarGym: Benchmarking Deep Research Workflows on Academic Literature Retrieval

TL;DR

ScholarGym provides a deterministic simulation environment to evaluate deep research workflows on academic literature, decoupling query planning, tool invocation, and relevance assessment within a static corpus of 570K papers and 2,536 ground-truth queries. It formalizes the iterative workflow with a memory mechanism, enabling long-horizon planning and reproducible analysis. Experiments reveal that extended thinking induces a precision-recall trade-off, with planning quality and calibration of relevance assessment as dual bottlenecks for open models, while proprietary systems excel in planning. The work demonstrates clear value for RL-based planning development in a noise-free setting and releases the environment and benchmarks to the community to advance reproducible evaluation in automated literature retrieval.

Abstract

Tool-augmented large language models have advanced from single-turn question answering to deep research workflows that iteratively plan queries, invoke external tools, and synthesize information to address complex information needs. Evaluating such workflows presents a fundamental challenge: reliance on live APIs introduces non-determinism, as tool invocations may yield different results across runs due to temporal drift, rate limiting, and evolving backend states. This variance undermines reproducibility and invalidates cross-system comparisons. We present ScholarGym, a simulation environment for reproducible evaluation of deep research workflows on academic literature. The environment decouples workflow components into query planning, tool invocation, and relevance assessment, enabling fine-grained analysis of each stage under controlled conditions. Built on a static corpus of 570K papers with deterministic retrieval, ScholarGym provides 2,536 queries with expert-annotated ground truth. Experiments across diverse backbone models reveal how reasoning capabilities, planning strategies, and selection mechanisms interact over iterative refinement.

Figures (15)

• Figure 1: Overview of ScholarGym. Each iteration comprises three stages connected by directed information flow: solid red arrows indicate stage inputs, and dashed orange arrows indicate outputs. Query Planning receives the subquery tree $\mathcal{M}_{t-1}$, experience buffer $\mathcal{B}_{t-1}$, and feedback $\mathcal{O}_{t-1}$ from the previous iteration, then outputs newly generated child nodes and an updated experience buffer. The resulting subqueries are passed to Tool Invocation, which parameterizes retrieval calls and executes them against the corpus; retrieved candidates undergo relevance-based ranking before selection. Relevance Assessment evaluates ranked papers and produces feedback to guide the next iteration.
• Figure 2: Recall trajectories over 25 iterations (Qwen3 models). Dashed line: default $T{=}5$. Solid lines show cumulative recall (left axis); dashed lines show per-iteration $\Delta$Recall (right axis).
• Figure 3: Per-iteration GT Discard Rate (%) on Test-Fast. Darker cells indicate higher discard rates of ground-truth papers during relevance assessment.
• Figure 4: Recall and Precision trajectories across 5 iterations on Test-Fast (sparse retrieval, Abstract-only).
• Figure 5: Avg.Distance trajectories across iterations. Higher values indicate queries that rank ground-truth papers earlier in retrieval results.