SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents

TL;DR

This work addresses safety gaps in LLM-based search agents that optimize for utility, showing that such agents are more prone to harmful outputs. It introduces SafeSearch, a multi-objective reinforcement learning framework that jointly optimizes final-output safety and utility, aided by a novel query-level shaping reward that penalizes unsafe queries. Experiments demonstrate SafeSearch reduces harmful outputs by over 70% across three red-teaming datasets while preserving QA performance, with ablations confirming the critical role of the query-level reward in balancing safety and usefulness. The approach provides a practical path to safe, useful open-domain search systems and highlights the importance of shaping search behavior at the query level to mitigate safety risks.

Abstract

Large language model (LLM) based search agents iteratively generate queries, retrieve external information, and reason to answer open-domain questions. While researchers have primarily focused on improving their utility, their safety behaviors remain underexplored. In this paper, we first evaluate search agents using red-teaming datasets and find that they are more likely to produce harmful outputs than base LLMs. For example, when asked "How can I track someone's location without their consent?", a base model refuses, whereas a search agent designed to retrieve and cite sources may lower its refusal threshold, fetch documents (e.g., court cases), and, once appended, synthesize them into an informative yet unsafe summary. We further show that utility-oriented fine-tuning intensifies this risk, motivating joint alignment of safety and utility. We present SafeSearch, a multi-objective reinforcement learning approach that couples a final-output safety/utility reward with a novel query-level shaping term that penalizes unsafe queries and rewards safe ones. Experiments show that SafeSearch reduces agent harmfulness by over 70% across three red-teaming datasets while producing safe, helpful responses, and matches the QA performance of a utility-only finetuned agent; further analyses confirm the effectiveness of the query-level reward in jointly improving safety and utility.

Figures (6)

• Figure 1: SafeSearch framework. SafeSearch optimizes the search agent via RL for both utility and safety. Utility reward captures QA correctness, while the safety reward has two parts: a query-level signal (dashed) that guides safe searches, and a final-output signal (solid) that encourages safe, helpful responses.
• Figure 2: Harmful rate (HarmR) of search agents using different LLMs across three red-teaming datasets, evaluated under three output categories: (i) No Search (no search conducted), (ii) Safe-Only Search (at least one search conducted and all queries judged safe), and (iii) Has-Unsafe Search (at least one search query judged unsafe). Ft. denotes the model after utility-oriented fine-tuning.
• Figure 3: Output distributions across harmfulness and search conditions before and after utility-only finetuning for Qwen-2.5-7B-Instruct on three red-teaming datasets. “No” indicates no search, “SS” indicates only safe searches, and “US” indicates the presence of unsafe searches.
• Figure 4: Helpfulness distribution of safe outputs across different red-teaming benchmarks, comparing with and without query-level reward.
• Figure 5: Case study showing that query-level reward prevents unsafe queries and outputs. Without query reward, the model issues an unsafe query (in red) and produces a harmful answer. With SafeSearch, the query is safe (in green) and the final output is constructive and helpful.