Agentic Reinforcement Learning for Search is Unsafe

TL;DR

This work evaluates the safety of agentic reinforcement learning (RL) trained search models and finds that while these models inherit refusal behaviors from instruction tuning, their safety is fragile under targeted attacks. Using PPO-based RL for search on Qwen and Llama models (local and web search), the authors demonstrate two jailbreak-style attacks—Search and Multi-search—that induce harmful, request-mirroring searches and outputs by triggering searches before refusal or by iterative prefilled searches. The results show substantial safety degradation across refusal, answer safety, and search safety, revealing a fundamental conflict in RL objectives that reward effective querying without accounting for harmful content. The study highlights the urgency of designing safety-aware RL pipelines that explicitly optimize for safe search, to prevent easy exploitation in deployed agentic search systems.

Abstract

Agentic reinforcement learning (RL) trains large language models to autonomously call tools during reasoning, with search as the most common application. These models excel at multi-step reasoning tasks, but their safety properties are not well understood. In this study, we show that RL-trained search models inherit refusal from instruction tuning and often deflect harmful requests by turning them into safe queries. However, this safety is fragile. Two simple attacks, one that forces the model to begin response with search (Search attack), another that encourages models to repeatedly search (Multi-search attack), trigger cascades of harmful searches and answers. Across two model families (Qwen, Llama) with both local and web search, these attacks lower refusal rates by up to 60.0%, answer safety by 82.5%, and search-query safety by 82.4%. The attacks succeed by triggering models to generate harmful, request-mirroring search queries before they can generate the inherited refusal tokens. This exposes a core weakness of current RL training: it rewards continued generation of effective queries without accounting for their harmfulness. As a result, RL search models have vulnerabilities that users can easily exploit, making it urgent to develop safety-aware agentic RL pipelines optimising for safe search.

Figures (15)

• Figure 1: The safety of agentic RL search models is brittle. We show an example harmful request (money laundering) and reasoning traces from the instruction-tuned (IT)-search Qwen with attacks. (a→b) The IT-search model inherits refusal from instruction-tuning and often blocks harmful requests by turning into safe searches; (b→c) Search attack: forcing a single <search> token at the start of response triggers a harmful search and answer; (c→d) Multi-search attack: iteratively forcing <search> at the start produces cascades of harmful searches and a detailed harmful answer.
• Figure 2: Drops in safety metrics under attacks. (1) Both attacks were effective: search attacks reduced all safety metrics from IT-search, while multi-search attacks further reduced refusal and answer safety. (2) Search safety was consistently below refusal and answer safety, suggesting that RL training naturally produces some harmful searches. For each attack, we reported drops for the most effective prompt/prefill variant (lowest combined refusal+answer safety). Base-search models (black lines) show the lowest expected safety levels. Table \ref{['tab:percentage_drops']} (Appendix) reports the corresponding percentage drops.
• Figure 3: Both attacks triggered cascades of harmful searches. For Qwen-2.5-7B (local search), both attacks had consistently lower search safety scores than IT-Search across five query positions; web shows similar patterns. Scores were averaged across attack variants. The per-variant search safety plots are in Figure \ref{['fig:qwen_search_safety_search']}, \ref{['fig:qwen_search_safety_multi_search']} (Appendix).
• Figure 4: Comparison of three RL training frameworks. They differ slightly in the reward design: Search-R1 adopted a purely outcome-based reward (exact matches); R1-searcher added a bonus reward is search is used; ARPO added partial rewards to reasoning and tool-use steps via credit assignment (advantage attribution).
• Figure 5: ReAct-style system prompt. The prompt instructs models to interleave reasoning, tool use, and final answers within <think>, <search>, and <answer> tokens jin2025searchr1trainingllmsreasonchen2025researchlearningreasonsearch.