BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search

TL;DR

The paper addresses the unreliability of RL-based agentic search systems that tend to ignore their reasoning boundaries and avoid admitting IDK. It introduces Boundary-Aware Policy Optimization (BAPO), which couples a boundary-aware reward with an adaptive reward modulator to encourage IDK only when the problem lies beyond the model's boundary while preventing reward hacking during exploration. Empirical results across four multi-hop QA benchmarks show substantial improvements in reliability (a balance between accuracy and IDK refusals) with modest costs to accuracy and strong generalization across model scales. The approach demonstrates practical impact by enabling more trustworthy agentic search deployments in knowledge-intensive tasks, while also outlining limitations and avenues for future work.

Abstract

RL-based agentic search enables LLMs to solve complex questions via dynamic planning and external search. While this approach significantly enhances accuracy with agent policies optimized via large-scale reinforcement learning, we identify a critical gap in reliability: these agents fail to recognize their reasoning boundaries and rarely admit ``I DON'T KNOW'' (IDK) even when evidence is insufficient or reasoning reaches its limit. The lack of reliability often leads to plausible but unreliable answers, introducing significant risks in many real-world scenarios. To this end, we propose Boundary-Aware Policy Optimization (BAPO), a novel RL framework designed to cultivate reliable boundary awareness without compromising accuracy. BAPO introduces two key components: (i) a group-based boundary-aware reward that encourages an IDK response only when the reasoning reaches its limit, and (ii) an adaptive reward modulator that strategically suspends this reward during early exploration, preventing the model from exploiting IDK as a shortcut. Extensive experiments on four benchmarks demonstrate that BAPO substantially enhances the overall reliability of agentic search.

Figures (7)

• Figure 1: When the agentic search model produces wrong answers, its lengthy and complex reasoning makes it difficult for users to verify. To ensure reliability, the model should explicitly state when information is insufficient and that no answer is available.
• Figure 2: Evaluation results of accuracy, precision and IDK rate ($\rho_{\textit{IDK}}$) of models before and after RL. The sharp drop in $\rho_{\textit{IDK}}$ coupled with the narrowing gap between accuracy and precision, indicates a diminished boundary awareness after RL.
• Figure 3: Left: Validation accuracy under different reward settings during RL training. Right:IDK rate ($\rho_{\textit{IDK}}$) under the modified reward during RL training.
• Figure 4: The overall framework of BAPO. Its training process begins with (a)agentic reasoning, where the model generates a group of rollouts for each question by interleaving thought processes with search environment interactions. These rollouts are then passed to the (b)reward computation module, which is comprised of (b.1)correctness reward $\mathcal{R}^{\textit{Correct}}$ derived from the correctness of format and outcome, (b.2)boundary-aware reward $\mathcal{R}^{\textit{IDK}}$ designed to incentivize IDK responses when no correct rollout exists within the group, and (b.3) adaptive reward modulator adaptively disabling $\mathcal{R}^{\textit{IDK}}$ based on IDK ratio at the exploration stage and diversity of rollouts at the plateau stage.
• Figure 5: Upper: Dynamics of correctness reward $\mathcal{R}^\textit{Correct}$ and boundary-aware reward $\mathcal{R}^\textit{IDK}$. Bottom: Dynamics of IDK ratio $\rho_\textit{IDK}$ during BAPO training on Qwen2.5-14B-Instruct.