Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety

Xuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang

Abstract

Large Language Model (LLM) agents increasingly act through external tools, making their safety contingent on tool-call workflows rather than text generation alone. While recent benchmarks evaluate agents across diverse environments and risk categories, a fundamental question remains unanswered: how complete are existing test suites, and what unsafe interaction patterns persist even after an agent passes the benchmark? We propose SafeAudit, a meta-audit framework that addresses this gap through two contributions. First, an LLM-based enumerator that systematically generates test cases by enumerating valid tool-call workflows and diverse user scenarios. Second, we introduce rule-resistance, a non-semantic, quantitative metric that distills compact safety rules from existing benchmarks and identifies unsafe interaction patterns that remain uncovered under those rules. Across 3 benchmarks and 12 environments, SafeAudit uncovers more than 20% residual unsafe behaviors that existing benchmarks fail to expose, with coverage growing monotonically as the testing budget increases. Our results highlight significant completeness gaps in current safety evaluation and motivate meta-auditing as a necessary complement to benchmark-based agent safety testing.

Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety

Abstract

Large Language Model (LLM) agents increasingly act through external tools, making their safety contingent on tool-call workflows rather than text generation alone. While recent benchmarks evaluate agents across diverse environments and risk categories, a fundamental question remains unanswered: how complete are existing test suites, and what unsafe interaction patterns persist even after an agent passes the benchmark? We propose SafeAudit, a meta-audit framework that addresses this gap through two contributions. First, an LLM-based enumerator that systematically generates test cases by enumerating valid tool-call workflows and diverse user scenarios. Second, we introduce rule-resistance, a non-semantic, quantitative metric that distills compact safety rules from existing benchmarks and identifies unsafe interaction patterns that remain uncovered under those rules. Across 3 benchmarks and 12 environments, SafeAudit uncovers more than 20% residual unsafe behaviors that existing benchmarks fail to expose, with coverage growing monotonically as the testing budget increases. Our results highlight significant completeness gaps in current safety evaluation and motivate meta-auditing as a necessary complement to benchmark-based agent safety testing.
Paper Structure (30 sections, 1 equation, 6 figures, 8 tables, 1 algorithm)

This paper contains 30 sections, 1 equation, 6 figures, 8 tables, 1 algorithm.

Table of Contents

  1. SafeAudit
  2. Problem Formulation
  3. Setting.
  4. Two challenges.
  5. Challenge 1: Enumerating Unsafe Interaction Patterns
  6. Challenge 2: Measuring Test Set Completeness
  7. Experiments
  8. Experimental Setup
  9. Main Results
  10. Coverage Scaling
  11. Novelty Analysis
  12. Generalization to Stronger Models
  13. Ablation Study & Sensitivity Test
  14. Discussion and Conclusion
  15. Appendix
  16. ...and 15 more sections

Figures (6)

  • Figure 2: Overview of SafeAudit. The upper panel illustrates how our enumerator generates concrete test cases by instantiating tool-calling chains with diverse user scenarios. The lower panel shows the rule-resistance evaluation: we apply the rules in the compact rule set sequentially and compute the final uncovered rate based on the remaining uncovered test cases.
  • Figure 3: Uncovered rate of two environments from ASB and AgentHarm. We use GPT-5-mini to generate test cases, and GPT-4o-mini is the backbone LLM of the target agent.
  • Figure 4: Novelty analysis of unsafe interaction patterns revealed by test cases of different methods.
  • Figure 5: Case study. The agent maintains procedural order but fails at referential consistency. While it correctly replies to Thread A, it extracts the payment link from Thread B, violating the "same email" constraint. This is a novel interaction pattern not covered by existing benchmarks.
  • Figure 6: UR performance across 5 environments of ASB for 5 LLM agents.
  • ...and 1 more figures