SmartSearch: Process Reward-Guided Query Refinement for Search Agents

TL;DR

SmartSearch tackles the problem of low-quality intermediate search queries in LLM-based search agents by introducing process rewards and query refinement. It leverages Dual-Level Credit Assessment to provide fine-grained feedback on query novelty and usefulness, and uses a query refinement module to iteratively improve weak queries and regenerate subsequent steps. A three-stage curriculum—Query Quality Screened Imitation Learning, Query Generation Alignment via Direct Preference Optimization, and Query Aware Policy Optimization—enables progressive internalization of high-quality querying behavior. Across six benchmarks, SmartSearch consistently outperforms baselines in accuracy and search efficiency, and shows strong generalization to web exploration tasks, underscoring its practical impact for knowledge-intensive retrieval systems. The approach is supported by extensive ablations and quantitative analyses, and the codebase is publicly available for reproducibility and extension ($ ext{code at } https://github.com/MYVAE/SmartSearch$).

Abstract

Large language model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems by incorporating information retrieval capabilities. Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked. As a result, the generated queries often remain inaccurate, leading to unexpected retrieval results and ultimately limiting search agents' overall effectiveness. To mitigate this issue, we introduce SmartSearch, a framework built upon two key mechanisms: (1) Process rewards, which provide fine-grained supervision for the quality of each intermediate search query through Dual-Level Credit Assessment. (2) Query refinement, which promotes the optimization of query generation by selectively refining low-quality search queries and regenerating subsequent search rounds based on these refinements. To enable the search agent to progressively internalize the ability to improve query quality under the guidance of process rewards, we design a three-stage curriculum learning framework. This framework guides the agent through a progression from imitation, to alignment, and ultimately to generalization. Experimental results show that SmartSearch consistently surpasses existing baselines, and additional quantitative analyses further confirm its significant gains in both search efficiency and query quality. The code is available at https://github.com/MYVAE/SmartSearch.

Figures (6)

• Figure 1: An example from ASearcher gaobeyond dataset demonstrating how low-quality intermediate search queries lead to unexpected retrieval results and derail the entire trajectory.
• Figure 2: An overview of the two key mechanisms in SmartSearch: the process rewards (a) and the query refinement (b).
• Figure 3: The overall framework of query-oriented three-stage curriculum learning, including Query Quality Screened Imitation Learning, Query Generation Alignment, and Query Generation Alignment.
• Figure 4: F1 score training dynamics for different algorithms.
• Figure 5: Left: Search query quality comparison. Right: Search efficiency comparison.