Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety

Vamshi Krishna Bonagiri, Ponnurangam Kumaragurum, Khanh Nguyen, Benjamin Plaut

TL;DR

The paper addresses safety in multi-turn LLM agents operating with tools by proposing a quitting mechanism as an uncertainty-aware behavior. It extends the agent action space to include a quit action and compares Baseline, Simple Quit, and Specified Quit prompts across the ToolEmu benchmark (144 scenarios, 36 toolkits, 9 risk types) using 12 LLMs. Findings show that Specified Quit achieves the largest safety gains (averaging $+0.39$, with proprietary models around $+0.64$) with only a negligible decrease in helpfulness ($-0.03$), establishing quitting as an effective first-line defense in high-stakes settings. The results support a simple, prompt-based intervention that can be deployed without retraining and have potential regulatory implications for safer autonomous agents in real-world deployments.

Abstract

As Large Language Model (LLM) agents increasingly operate in complex environments with real-world consequences, their safety becomes critical. While uncertainty quantification is well-studied for single-turn tasks, multi-turn agentic scenarios with real-world tool access present unique challenges where uncertainties and ambiguities compound, leading to severe or catastrophic risks beyond traditional text generation failures. We propose using "quitting" as a simple yet effective behavioral mechanism for LLM agents to recognize and withdraw from situations where they lack confidence. Leveraging the ToolEmu framework, we conduct a systematic evaluation of quitting behavior across 12 state-of-the-art LLMs. Our results demonstrate a highly favorable safety-helpfulness trade-off: agents prompted to quit with explicit instructions improve safety by an average of +0.39 on a 0-3 scale across all models (+0.64 for proprietary models), while maintaining a negligible average decrease of -0.03 in helpfulness. Our analysis demonstrates that simply adding explicit quit instructions proves to be a highly effective safety mechanism that can immediately be deployed in existing agent systems, and establishes quitting as an effective first-line defense mechanism for autonomous agents in high-stakes applications.

Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety

TL;DR

The paper addresses safety in multi-turn LLM agents operating with tools by proposing a quitting mechanism as an uncertainty-aware behavior. It extends the agent action space to include a quit action and compares Baseline, Simple Quit, and Specified Quit prompts across the ToolEmu benchmark (144 scenarios, 36 toolkits, 9 risk types) using 12 LLMs. Findings show that Specified Quit achieves the largest safety gains (averaging , with proprietary models around ) with only a negligible decrease in helpfulness (), establishing quitting as an effective first-line defense in high-stakes settings. The results support a simple, prompt-based intervention that can be deployed without retraining and have potential regulatory implications for safer autonomous agents in real-world deployments.

Abstract

As Large Language Model (LLM) agents increasingly operate in complex environments with real-world consequences, their safety becomes critical. While uncertainty quantification is well-studied for single-turn tasks, multi-turn agentic scenarios with real-world tool access present unique challenges where uncertainties and ambiguities compound, leading to severe or catastrophic risks beyond traditional text generation failures. We propose using "quitting" as a simple yet effective behavioral mechanism for LLM agents to recognize and withdraw from situations where they lack confidence. Leveraging the ToolEmu framework, we conduct a systematic evaluation of quitting behavior across 12 state-of-the-art LLMs. Our results demonstrate a highly favorable safety-helpfulness trade-off: agents prompted to quit with explicit instructions improve safety by an average of +0.39 on a 0-3 scale across all models (+0.64 for proprietary models), while maintaining a negligible average decrease of -0.03 in helpfulness. Our analysis demonstrates that simply adding explicit quit instructions proves to be a highly effective safety mechanism that can immediately be deployed in existing agent systems, and establishes quitting as an effective first-line defense mechanism for autonomous agents in high-stakes applications.
Paper Structure (25 sections, 2 equations, 3 figures, 3 tables)

This paper contains 25 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Agent Safety
  4. Uncertainty Quantification in LLMs
  5. Methodology
  6. Preliminary: LLM agents
  7. Evaluation Framework: ToolEmu
  8. Experimental Setup
  9. Quitting as a Safety Action
  10. Prompting Strategies and Models
  11. Baseline Prompt:
  12. Simple Quit Prompt:
  13. Specified Quit Prompt:
  14. Results and Analysis
  15. Discussion and Conclusion
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Motivating example demonstrating the value of quitting in LLM agents. When asked to grant delivery access to Alice Smith, an agent without quitting capability (A) proceeds with ambiguous information, potentially granting access to the wrong person. In contrast, an agent with quitting capability (B) recognizes the uncertainty and asks for clarification, preventing potential security risks. This illustrates how quitting serves as an effective proxy for uncertainty-aware decision making in multi-turn tasks.
  • Figure 2: Analysis of Quitting Behavior Across Models. (a) The safety-helpfulness trade-off, showing the change in scores relative to the baseline. (b) The relationship between quit rate and change in safety. (c) The relationship between quit rate and change in helpfulness.
  • Figure 3: Agent trajectories for the ambiguous "Jason" task. The agent must decide whether to share sensitive door access history. (a) The baseline agent makes a risky assumption. (b) The simple quit agent behaves identically to baseline. (c) The specified quit agent correctly identifies the risk and halts the task.