Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities

TL;DR

End-to-end benchmarks conflate retrieval with reasoning, making it difficult to diagnose whether errors arise from evidence access or from reasoning over provided information. DeR2 provides a controlled sandbox with four regimes (Instruction-only, Concepts, Related-only, Full-set), a frozen document library, and expert concept annotations to isolate grounding from reasoning. The study introduces a rigorous two-phase validation and a collaboration-driven data collection process to ensure novelty, tractability, and reproducibility, and it demonstrates substantial headroom and regime-dependent failure modes across diverse foundation models. This framework enables principled model selection, targeted debugging, and future research directions toward robust retrieval-infused reasoning in frontier scientific domains.

Abstract

Despite strong performance on existing benchmarks, it remains unclear whether large language models can reason over genuinely novel scientific information. Most evaluations score end-to-end RAG pipelines, where reasoning is confounded with retrieval and toolchain choices, and the signal is further contaminated by parametric memorization and open-web volatility. We introduce DeR2, a controlled deep-research sandbox that isolates document-grounded reasoning while preserving core difficulties of deep search: multi-step synthesis, denoising, and evidence-based conclusion making. DeR2 decouples evidence access from reasoning via four regimes--Instruction-only, Concepts (gold concepts without documents), Related-only (only relevant documents), and Full-set (relevant documents plus topically related distractors)--yielding interpretable regime gaps that operationalize retrieval loss vs. reasoning loss and enable fine-grained error attribution. To prevent parametric leakage, we apply a two-phase validation that requires parametric failure without evidence while ensuring oracle-concept solvability. To ensure reproducibility, each instance provides a frozen document library (drawn from 2023-2025 theoretical papers) with expert-annotated concepts and validated rationales. Experiments across a diverse set of state-of-the-art foundation models reveal substantial variation and significant headroom: some models exhibit mode-switch fragility, performing worse with the Full-set than with Instruction-only, while others show structural concept misuse, correctly naming concepts but failing to execute them as procedures.

Figures (6)

• Figure 1: Closed-book QA measures intrinsic knowledge, while standard RAG entangles retrieval and reasoning—distractors can hide whether errors come from bad retrieval or failed evidence-based reasoning. DeR$^{2}$ decouples the two by evaluating the same question under controlled inputs (instruction/concepts/related/full), isolating failure causes.
• Figure 2: (a) Existing benchmarks. Closed-book QA only evaluates intrinsic (parametric) knowledge. RAG pipelines couple retrieval and reasoning end-to-end, creating a reasoning–retrieval tangle that masks the true cause of failure. (b) Decoupled Retrieval and Reasoning Benchmark. DeR$^{2}$ provides a controlled setting that decouples retrieval from reasoning by comparing Instruction-only, Full DocSet (related + noise), Related-only, and Concepts-only conditions, enabling diagnosis of grounding, mode-switching, and concept execution errors.
• Figure 3: Pipeline for constructing DeR$^{2}$ data: PhD annotators gather theoretical papers from academic platforms, extracting instructions, concepts, CoT, and concise answers. These questions are designed to be challenging and represent cutting-edge academic problems. During the difficulty control phase, offline AI models must fail to answer correctly three times without concepts, but have a chance to answer correctly when provided with the relevant concepts. When the difficulty control conditions are satisfied, retrieve the related documents and noise documents from the references of the source paper. Finally, the reviewer conducts logical quality inspection and format quality inspection on the submitted data, and verifies the difficulty control conditions again.
• Figure 4: Distribution of problem domains and answer types, showing the disciplinary coverage of the benchmark and the structural diversity of target answers.
• Figure 5: Structural and contextual complexity of the dataset. (a) Reports the distributions of reasoning step counts and required concept counts per problem, reflecting multi-step and multi-concept reasoning demands. (b) Presents the distributions of related documents, noise documents, and total documents per instance, characterizing the degree of retrieval difficulty and noise exposure in the document sets. Together, these statistics illustrate the controlled diversity of DeR² across reasoning depth, conceptual load, and retrieval conditions.