Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration
Bowei He, Minda Hu, Zenan Xu, Hongru Wang, Licheng Zong, Yankai Chen, Chen Ma, Xue Liu, Pluto Zhou, Irwin King
TL;DR
The paper addresses fragility and sampling inefficiency in search-integrated reasoning caused by multi-scale credit assignment. It introduces Search-R2, an Actor–Refiner framework in which an Actor generates initial reasoning with tool calls and a Meta-Refiner surgically repairs errors via a cut-and-regenerate mechanism, guided by a hybrid global-local reward and trained with GRPO. The authors formalize the interaction as a smoothed mixture policy and prove conditions under which selective correction strictly improves performance over baselines, while empirically showing gains across seven QA benchmarks and multiple model scales with modest overhead. The work demonstrates that joint optimization of generation and targeted refinement yields more accurate reasoning traces and higher-quality search behavior, offering a practical path toward robust, search-based reasoning in real-world applications.
Abstract
Search-integrated reasoning enables language agents to transcend static parametric knowledge by actively querying external sources. However, training these agents via reinforcement learning is hindered by the multi-scale credit assignment problem: existing methods typically rely on sparse, trajectory-level rewards that fail to distinguish between high-quality reasoning and fortuitous guesses, leading to redundant or misleading search behaviors. To address this, we propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention, with both components jointly optimized during training. Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories, and a Meta-Refiner, which selectively diagnoses and repairs flawed steps via a 'cut-and-regenerate' mechanism. To provide fine-grained supervision, we introduce a hybrid reward design that couples outcome correctness with a dense process reward quantifying the information density of retrieved evidence. Theoretically, we formalize the Actor-Refiner interaction as a smoothed mixture policy, proving that selective correction yields strict performance gains over strong baselines. Extensive experiments across various general and multi-hop QA datasets demonstrate that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales, achieving superior reasoning accuracy with minimal overhead.