When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering

TL;DR

The paper conducts a controlled diagnostic study to determine whether iterative retrieval-reasoning (Iterative RAG) can outperform an ideal Gold Context baseline in scientific multi-hop QA, using ChemKGMultiHopQA across eleven LLMs. It shows that Iterative RAG consistently yields gains beyond static oracle evidence, with non-reasoning models often benefiting most and deeper hops presenting both opportunities and challenges. A comprehensive diagnostic framework analyzes retrieval coverage, anchor propagation, stopping calibration, and synthesis, revealing failure modes such as final-hop coverage gaps, composition errors, and distractor latch, along with measures of efficiency and procedural compliance. The findings imply that aligning retrieval dynamics with a model’s reasoning trajectory is key to robust, scalable scientific QA, and they offer practical guidance for deploying and diagnosing iterative retrieval systems in specialized domains.

Abstract

Retrieval-Augmented Generation (RAG) extends large language models (LLMs) beyond parametric knowledge, yet it is unclear when iterative retrieval-reasoning loops meaningfully outperform static RAG, particularly in scientific domains with multi-hop reasoning, sparse domain knowledge, and heterogeneous evidence. We provide the first controlled, mechanism-level diagnostic study of whether synchronized iterative retrieval and reasoning can surpass an idealized static upper bound (Gold Context) RAG. We benchmark eleven state-of-the-art LLMs under three regimes: (i) No Context, measuring reliance on parametric memory; (ii) Gold Context, where all oracle evidence is supplied at once; and (iii) Iterative RAG, a training-free controller that alternates retrieval, hypothesis refinement, and evidence-aware stopping. Using the chemistry-focused ChemKGMultiHopQA dataset, we isolate questions requiring genuine retrieval and analyze behavior with diagnostics spanning retrieval coverage gaps, anchor-carry drop, query quality, composition fidelity, and control calibration. Across models, Iterative RAG consistently outperforms Gold Context, with gains up to 25.6 percentage points, especially for non-reasoning fine-tuned models. Staged retrieval reduces late-hop failures, mitigates context overload, and enables dynamic correction of early hypothesis drift, but remaining failure modes include incomplete hop coverage, distractor latch trajectories, early stopping miscalibration, and high composition failure rates even with perfect retrieval. Overall, staged retrieval is often more influential than the mere presence of ideal evidence; we provide practical guidance for deploying and diagnosing RAG systems in specialized scientific settings and a foundation for more reliable, controllable iterative retrieval-reasoning frameworks.

Figures (26)

• Figure 1: The Iterative RAG System: A training-free controller alternates between targeted retrieval and partial answer updates.
• Figure 2: Models' accuracy Distribution of models' accuracy in three setups of No Context (Parametric memory), Gold context, and Iterative retrieval and reasoning, shown in blue, red and green, respectively. Horizontal bars shows the results of t-test statistical analysis (Significance: *** p<0.001, , ** p<0.01)
• Figure 3: Partition of Solvability. This heatmap classifies correct answers by the necessary condition for success: internal knowledge (Parametric), static evidence (Gold-Dependent), or dynamic retrieval (Iterative-Exclusive).
• Figure 4: Recoveries vs. Regressions from Gold Context to Iterative RAG. Green bars count recoveries (Gold incorrect $\rightarrow$ Iterative correct) and red bars count regressions (Gold correct $\rightarrow$ Iterative incorrect) per model; the black line (top) and the condensed panel (bottom) show the net gain questions count (= recoveries $-$ regressions). The plot quantifies iteration’s overall benefit: models like GPT–4o and Llama 3.3 Instruct post the largest net gains, while Mistral Large 2402 shows the smallest due to higher regressions.
• Figure 5: Parametric Suppression Rate (PSR). The plot illustrates the proportion of questions answered correctly in the No Context setting that are suppressed (answered incorrectly) in Iterative RAG. Mistral Large 2402 exhibits the highest suppression rate (14.1%), indicating a strong tendency to prioritize retrieved noise over correct internal weights. In contrast, Claude 3.7 Sonnet is highly robust (2.7%), effectively filtering irrelevant context to preserve its parametric knowledge.