Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stop-RAG: Value-Based Retrieval Control for Iterative RAG

Jaewan Park, Solbee Cho, Jay-Yoon Lee

TL;DR

Stop-RAG presents a value-based stopping controller for iterative retrieval-augmented generation by framing stopping as a finite-horizon MDP and training a full-width forward-view Q(λ) network. The method learns forward-looking estimates of the benefit of continued retrieval, enabling adaptive termination that balances recall and noise within modular pipelines compatible with black-box LLMs. Empirical results on MuSiQue, HotpotQA, and 2WikiMultihopQA show consistent gains over fixed-iteration and prompting-based baselines, with ablations confirming the advantages of the Q(λ) targets and lambda annealing. The work highlights adaptive stopping as a crucial component for effective agentic AI and demonstrates the practical impact of value-based control in RAG systems.

Abstract

Iterative retrieval-augmented generation (RAG) enables large language models to answer complex multi-hop questions, but each additional loop increases latency, costs, and the risk of introducing distracting evidence, motivating the need for an efficient stopping strategy. Existing methods either use a predetermined number of iterations or rely on confidence proxies that poorly reflect whether more retrieval will actually help. We cast iterative RAG as a finite-horizon Markov decision process and introduce Stop-RAG, a value-based controller that adaptively decides when to stop retrieving. Trained with full-width forward-view Q($λ$) targets from complete trajectories, Stop-RAG learns effective stopping policies while remaining compatible with black-box APIs and existing pipelines. On multi-hop question-answering benchmarks, Stop-RAG consistently outperforms both fixed-iteration baselines and prompting-based stopping with LLMs. These results highlight adaptive stopping as a key missing component in current agentic systems, and demonstrate that value-based control can improve the accuracy of RAG systems.

Stop-RAG: Value-Based Retrieval Control for Iterative RAG

TL;DR

Stop-RAG presents a value-based stopping controller for iterative retrieval-augmented generation by framing stopping as a finite-horizon MDP and training a full-width forward-view Q(λ) network. The method learns forward-looking estimates of the benefit of continued retrieval, enabling adaptive termination that balances recall and noise within modular pipelines compatible with black-box LLMs. Empirical results on MuSiQue, HotpotQA, and 2WikiMultihopQA show consistent gains over fixed-iteration and prompting-based baselines, with ablations confirming the advantages of the Q(λ) targets and lambda annealing. The work highlights adaptive stopping as a crucial component for effective agentic AI and demonstrates the practical impact of value-based control in RAG systems.

Abstract

Iterative retrieval-augmented generation (RAG) enables large language models to answer complex multi-hop questions, but each additional loop increases latency, costs, and the risk of introducing distracting evidence, motivating the need for an efficient stopping strategy. Existing methods either use a predetermined number of iterations or rely on confidence proxies that poorly reflect whether more retrieval will actually help. We cast iterative RAG as a finite-horizon Markov decision process and introduce Stop-RAG, a value-based controller that adaptively decides when to stop retrieving. Trained with full-width forward-view Q() targets from complete trajectories, Stop-RAG learns effective stopping policies while remaining compatible with black-box APIs and existing pipelines. On multi-hop question-answering benchmarks, Stop-RAG consistently outperforms both fixed-iteration baselines and prompting-based stopping with LLMs. These results highlight adaptive stopping as a key missing component in current agentic systems, and demonstrate that value-based control can improve the accuracy of RAG systems.
Paper Structure (31 sections, 16 equations, 1 figure, 4 tables, 1 algorithm)

This paper contains 31 sections, 16 equations, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Iterative RAG
  4. Adaptive RAG
  5. Preliminaries
  6. Q-learning
  7. Stop-RAG
  8. Iterative RAG as an MDP
  9. Stop-RAG training
  10. Specialization of Q($\boldsymbol{\lambda}$) to iterative RAG
  11. Offline dataset construction
  12. Q($\boldsymbol{\lambda}$) training via function approximation
  13. Experiments
  14. Evaluation setup
  15. Datasets
  16. ...and 16 more sections

Figures (1)

  • Figure 1: Illustration of the iterative RAG pipeline formulated as a finite-horizon MDP. Blue nodes denote non-terminal states while red nodes denote terminal states. At each step, the agent can either $\mathtt{STOP}$ (terminate to the absorbing state $\mathtt{TERM}$) or $\mathtt{CONT}$ (perform a RAG iteration). In addition, states that reach the maximum iteration limit are also treated as terminal. A RAG iteration consists of query generation ($q_k$), retrieval and reranking ($\vb{d}_k$), and intermediate answer generation ($a_k$). Rewards are given only at terminal states, reflecting the answer generation quality at that point.