QAgent: A modular Search Agent with Interactive Query Understanding

TL;DR

QAgent presents a modular, agentic RAG framework that treats query understanding as an interactive, multi-round process to improve retrieval quality and downstream answers. It introduces a two-stage RL training regime to balance end-to-end retrieval/utilization with generalization when deployed as a submodule, showing improvements over baselines on open-domain QA tasks. The approach demonstrates strong end-to-end QA gains and robust submodule performance, with analysis on information utilization, retriever generalization, and model-size effects, highlighting practical benefits for real-world systems. The work advances practical deployment of search agents by emphasizing query understanding, modularity, and feedback-driven optimization, while candidly noting limitations in diversity control, robustness, and the need for larger-model validation.

Abstract

Large language models (LLMs) excel at natural language tasks but are limited by their static parametric knowledge, especially in knowledge-intensive task. Retrieval-augmented generation (RAG) mitigates this by integrating external information. However, (1) traditional RAG struggles with complex query understanding, and (2) even search agents trained with reinforcement learning (RL), despite their promise, still face generalization and deployment challenges. To address these limitations, we propose QAgent, a unified agentic RAG framework that employs a search agent for adaptive retrieval. This agent optimizes its understanding of the query through interactive reasoning and retrieval. To facilitate real-world application, we focus on modular search agent for query understanding that are plug-and-play in complex systems. Secifically, the agent follows a multi-step decision process trained with RL to maximize retrieval quality and support accurate downstream answers. We further analyze the strengths and weaknesses of end-to-end RL and propose a strategy that focuses on effective retrieval, thereby enhancing generalization in LLM applications. Experiments show QAgent excels at QA and serves as a plug-and-play module for real-world deployment.

Figures (11)

• Figure 1: Different methods applied to the system.
• Figure 2: The proposed QAgent framework, with the left representing the system and the right representing our agent design, including "plan-search-information-reflect". In addition, the illustration shows a two-stage training framework, with the first stage using end-to-end RL and the second stage introducing generalized training. During training, the searched content tokens are masked to exclude them from loss calculations.
• Figure 3: Illustration of query understanding.
• Figure 4: A case study of Agentic RL training (Search-R1), where both the fast ascent phase and the slow convergence phase are tested.
• Figure 5: Illustration of the combined gain. The left is the experimental results of HotpotQA, and the right is that of 2WikiMHQ.