Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers

TL;DR

The paper tackles the challenge of enabling large language models to perform multi-hop information retrieval by introducing ExSearch, an agentic framework where an LLM alternates between thinking, triggering an external retriever, and recording fine-grained evidence. It casts the retrieval-reasoning loop as a latent trajectory and optimizes it with a Generalized EM algorithm, deriving an ELBO that jointly trains retrieval and answer generation. Theoretical convergence guarantees accompany empirical results across four knowledge-intensive benchmarks, with ExSearch outperforming direct-reasoning and RAG baselines and demonstrating strong retrieval recall. ExSearch-Zoo extends the framework to diverse backbones and richer actions, showing broad applicability and scalability. The approach promises improved factual grounding and interpretability for open-domain QA and knowledge-intensive tasks, while highlighting avenues for future multimodal and tool-augmented extensions.

Abstract

Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning. To enable LLM with this capability, EXSEARCH adopts a Generalized Expectation-Maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated data, progressively improving itself for search. We further theoretically analyze this training process, establishing convergence guarantees. Extensive experiments on four knowledge-intensive benchmarks show that EXSEARCH substantially outperforms baselines, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce EXSEARCH-Zoo, an extension that extends our method to broader scenarios, to facilitate future work.

Figures (11)

• Figure 1: Performance on HotpotQA dataset when applying our ExSearch to different LLMs.
• Figure 2: Overall framework of the Expectation-Maximization process in ExSearch. The E-step samples search trajectories and assigns each a weight based on its likelihood leading to a correct answer. The special token End marks the completion of reasoning. The M-step trains the LLM, encouraging the LLM to generate more supportive search trajectories and accurate answers.
• Figure 3: Training convergence of Qwen-2.5-7B and Llama-3.1-8B, where we report the Exact Match score for checkpoints in each iteration.
• Figure 4: Performance for Qwen2.5-7B that is initially empowered by different amounts of warm-up data.
• Figure 5: Performance of ExSearch when extended with the document re-ranking action.

Theorems & Definitions (15)

• Lemma 3.1: Non-decreasing Optimization
• proof
• Theorem 3.2: Monotone Convergence Theorem
• Lemma C.1: Monotonic Improvement
• proof
• Lemma C.2: Boundedness
• proof
• Theorem C.3: Convergence of ExSearch
• Remark C.1: Tightness via KL Divergence
• Proposition 1: Variational Lower Bound as a Proxy for Metric Maximization