OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!

TL;DR

Operational safety for LLM-based agents is crucial for deployment but remains inadequately solved. OffTopicEval provides a multilingual, multi-domain benchmark to measure an agent’s ability to accept in-domain queries while refusing out-of-domain ones, including direct and adversarial adaptive OOD scenarios. Across 20 open-weight models and multiple languages, the study finds pervasive operational safety gaps, with adaptive OOD suffering the largest refusals drop, and shows prompt-based steering (Q-ground and P-ground) as a practical method to substantially improve refusals. The work demonstrates the need for domain-specific safety mechanisms and offers a foundation for future research to build more robust, reliable agent systems.

Abstract

Large Language Model (LLM) safety is one of the most pressing challenges for enabling wide-scale deployment. While most studies and global discussions focus on generic harms, such as models assisting users in harming themselves or others, enterprises face a more fundamental concern: whether LLM-based agents are safe for their intended use case. To address this, we introduce operational safety, defined as an LLM's ability to appropriately accept or refuse user queries when tasked with a specific purpose. We further propose OffTopicEval, an evaluation suite and benchmark for measuring operational safety both in general and within specific agentic use cases. Our evaluations on six model families comprising 20 open-weight LLMs reveal that while performance varies across models, all of them remain highly operationally unsafe. Even the strongest models - Qwen-3 (235B) with 77.77% and Mistral (24B) with 79.96% - fall far short of reliable operational safety, while GPT models plateau in the 62-73% range, Phi achieves only mid-level scores (48-70%), and Gemma and Llama-3 collapse to 39.53% and 23.84%, respectively. While operational safety is a core model alignment issue, to suppress these failures, we propose prompt-based steering methods: query grounding (Q-ground) and system-prompt grounding (P-ground), which substantially improve OOD refusal. Q-ground provides consistent gains of up to 23%, while P-ground delivers even larger boosts, raising Llama-3.3 (70B) by 41% and Qwen-3 (30B) by 27%. These results highlight both the urgent need for operational safety interventions and the promise of prompt-based steering as a first step toward more reliable LLM-based agents.

Figures (7)

• Figure 1: Region of operational safety as defined by policies. A general-purpose AI operates within generic safety policies (yellow circle), whereas a purpose-specific assistant introduces further restrictions (cyan and purple circles), thereby narrowing the region of allowed queries, the assistant’s Operational Safety. On the right, we illustrate how an agent may initially refuse an OOD query, but an adversarial transformation can succeed in eliciting a response. We show examples of ChatGPT-5 and Claude-Opus-4 being operationally unsafe in \ref{['fig:attack-gpt5', 'fig:attack-opus']}.
• Figure 2: t-SNE spread of in-domain (Left), out-of-domain (center), all three in (Right). Right plot ID denotes questions inside medischeduler domain, OOD samples are taken from MMLU-math domain.
• Figure 3: t-SNE visualization of multilingual (English, Hindi, Chinese) in-domain (ID), direct out-of-domain (OOD), and adaptive OOD queries from the MediScheduler assistant.
• Figure 4: t-SNE visualization comparing Medischeduler ID questions with OOD questions from multiple domains. After transformation, the OOD questions shift closer to the ID distribution. However, they do not fully overlap with ID questions, as being too close would risk losing the original semantic distinctions of the OOD queries.
• Figure 5: Refusal Rate (left axis) and Flip Rate (right axis, bars+lines) for the BankHelper assistant. The left panel shows direct OOD queries, while the right panel shows adaptive queries.