Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications

Minhua Lin, Zongyu Wu, Zhichao Xu, Hui Liu, Xianfeng Tang, Qi He, Charu Aggarwal, Hui Liu, Xiang Zhang, Suhang Wang

TL;DR

This survey identifies RL-based agentic search as a paradigm shift from static RAG toward autonomous, multi-step information seeking. It categorizes RL roles into retrieval control, query optimization, reasoning–retrieval integration, multi-agent collaboration, and tool/knowledge integration, and surveys the optimization strategies (training regimes and reward design) and the scope (agent-, module-, and system-level). It collates representative methods, benchmarks, and applications, and discusses challenges in memory, trust, generalization, and cross-domain deployment. The work emphasizes the practical potential of RL-driven agentic search for reliable, scalable, and efficient information retrieval across domains and modalities, and provides a roadmap for future research. The accompanying repository offers a curated collection of RL-based agentic search papers to foster reproducibility and collaboration.

Abstract

The advent of large language models (LLMs) has transformed information access and reasoning through open-ended natural language interaction. However, LLMs remain limited by static knowledge, factual hallucinations, and the inability to retrieve real-time or domain-specific information. Retrieval-Augmented Generation (RAG) mitigates these issues by grounding model outputs in external evidence, but traditional RAG pipelines are often single turn and heuristic, lacking adaptive control over retrieval and reasoning. Recent advances in agentic search address these limitations by enabling LLMs to plan, retrieve, and reflect through multi-step interaction with search environments. Within this paradigm, reinforcement learning (RL) offers a powerful mechanism for adaptive and self-improving search behavior. This survey provides the first comprehensive overview of \emph{RL-based agentic search}, organizing the emerging field along three complementary dimensions: (i) What RL is for (functional roles), (ii) How RL is used (optimization strategies), and (iii) Where RL is applied (scope of optimization). We summarize representative methods, evaluation protocols, and applications, and discuss open challenges and future directions toward building reliable and scalable RL driven agentic search systems. We hope this survey will inspire future research on the integration of RL and agentic search. Our repository is available at https://github.com/ventr1c/Awesome-RL-based-Agentic-Search-Papers.

A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications

TL;DR

This survey identifies RL-based agentic search as a paradigm shift from static RAG toward autonomous, multi-step information seeking. It categorizes RL roles into retrieval control, query optimization, reasoning–retrieval integration, multi-agent collaboration, and tool/knowledge integration, and surveys the optimization strategies (training regimes and reward design) and the scope (agent-, module-, and system-level). It collates representative methods, benchmarks, and applications, and discusses challenges in memory, trust, generalization, and cross-domain deployment. The work emphasizes the practical potential of RL-driven agentic search for reliable, scalable, and efficient information retrieval across domains and modalities, and provides a roadmap for future research. The accompanying repository offers a curated collection of RL-based agentic search papers to foster reproducibility and collaboration.

Abstract

The advent of large language models (LLMs) has transformed information access and reasoning through open-ended natural language interaction. However, LLMs remain limited by static knowledge, factual hallucinations, and the inability to retrieve real-time or domain-specific information. Retrieval-Augmented Generation (RAG) mitigates these issues by grounding model outputs in external evidence, but traditional RAG pipelines are often single turn and heuristic, lacking adaptive control over retrieval and reasoning. Recent advances in agentic search address these limitations by enabling LLMs to plan, retrieve, and reflect through multi-step interaction with search environments. Within this paradigm, reinforcement learning (RL) offers a powerful mechanism for adaptive and self-improving search behavior. This survey provides the first comprehensive overview of \emph{RL-based agentic search}, organizing the emerging field along three complementary dimensions: (i) What RL is for (functional roles), (ii) How RL is used (optimization strategies), and (iii) Where RL is applied (scope of optimization). We summarize representative methods, evaluation protocols, and applications, and discuss open challenges and future directions toward building reliable and scalable RL driven agentic search systems. We hope this survey will inspire future research on the integration of RL and agentic search. Our repository is available at https://github.com/ventr1c/Awesome-RL-based-Agentic-Search-Papers.
Paper Structure (72 sections, 6 equations, 4 figures, 6 tables)

This paper contains 72 sections, 6 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Preliminary
  3. Large Language Models as Agents
  4. From Traditional IR to Agentic Search
  5. Traditional IR
  6. RAG
  7. Agentic Search
  8. Basics of Reinforcement Learning
  9. On-policy Optimization
  10. Off-policy Optimization
  11. RL-based Agentic Search
  12. Comparison with Pre-RL Agentic Search
  13. Formalization.
  14. What RL is for: Functional Roles in Agentic Search
  15. Retrieval Control
  16. ...and 57 more sections

Figures (4)

  • Figure 1: Overview of RL-based Agentic Search.
  • Figure 2: Overview of RL components
  • Figure 3: Illustrative framework of RL-based agentic search. RL intervenes at multiple decision points— controlling when to retrieve (retrieval control), how to formulate queries (query optimization), how to integrate evidence into reasoning (reasoning-retrieval integration), and which tools or knowledge sources to use (tool and knowledge integration).
  • Figure 4: Overview of RL for multi-agent collaboration. (a) Planner--executor architecture: a central planner coordinates specialized executor agents for task decomposition and dynamic subtask allocation. (b) Cooperative multi-agent system: multiple agents jointly optimize shared objectives through communication, coordination, and reward sharing.