Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents

Hanjun Luo, Shenyu Dai, Chiming Ni, Xinfeng Li, Guibin Zhang, Kun Wang, Tongliang Liu, Hanan Salam

TL;DR

AgentAuditor introduces a training-free, memory-augmented framework that elevates LLM-based evaluators to human-like safety and security assessment of autonomous agents. It combines a three-stage memory construction process with retrieval-augmented reasoning to dynamically guide evaluations, alongside ASSEBench, a large-scale benchmark addressing both safety and security with nuanced ambiguity handling. Across extensive experiments, AgentAuditor achieves state-of-the-art, near-human performance on multiple datasets and demonstrates strong robustness, ablation-driven insights, and favorable resource trade-offs compared to fine-tuning approaches. The work provides a scalable, domain-agnostic path toward more reliable evaluation of agent behaviors, with potential applications in agent defense and cross-domain safety assessment.

Abstract

Despite the rapid advancement of LLM-based agents, the reliable evaluation of their safety and security remains a significant challenge. Existing rule-based or LLM-based evaluators often miss dangers in agents' step-by-step actions, overlook subtle meanings, fail to see how small issues compound, and get confused by unclear safety or security rules. To overcome this evaluation crisis, we introduce AgentAuditor, a universal, training-free, memory-augmented reasoning framework that empowers LLM evaluators to emulate human expert evaluators. AgentAuditor constructs an experiential memory by having an LLM adaptively extract structured semantic features (e.g., scenario, risk, behavior) and generate associated chain-of-thought reasoning traces for past interactions. A multi-stage, context-aware retrieval-augmented generation process then dynamically retrieves the most relevant reasoning experiences to guide the LLM evaluator's assessment of new cases. Moreover, we developed ASSEBench, the first benchmark designed to check how well LLM-based evaluators can spot both safety risks and security threats. ASSEBench comprises 2293 meticulously annotated interaction records, covering 15 risk types across 29 application scenarios. A key feature of ASSEBench is its nuanced approach to ambiguous risk situations, employing "Strict" and "Lenient" judgment standards. Experiments demonstrate that AgentAuditor not only consistently improves the evaluation performance of LLMs across all benchmarks but also sets a new state-of-the-art in LLM-as-a-judge for agent safety and security, achieving human-level accuracy. Our work is openly accessible at https://github.com/Astarojth/AgentAuditor.

AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents

TL;DR

AgentAuditor introduces a training-free, memory-augmented framework that elevates LLM-based evaluators to human-like safety and security assessment of autonomous agents. It combines a three-stage memory construction process with retrieval-augmented reasoning to dynamically guide evaluations, alongside ASSEBench, a large-scale benchmark addressing both safety and security with nuanced ambiguity handling. Across extensive experiments, AgentAuditor achieves state-of-the-art, near-human performance on multiple datasets and demonstrates strong robustness, ablation-driven insights, and favorable resource trade-offs compared to fine-tuning approaches. The work provides a scalable, domain-agnostic path toward more reliable evaluation of agent behaviors, with potential applications in agent defense and cross-domain safety assessment.

Abstract

Despite the rapid advancement of LLM-based agents, the reliable evaluation of their safety and security remains a significant challenge. Existing rule-based or LLM-based evaluators often miss dangers in agents' step-by-step actions, overlook subtle meanings, fail to see how small issues compound, and get confused by unclear safety or security rules. To overcome this evaluation crisis, we introduce AgentAuditor, a universal, training-free, memory-augmented reasoning framework that empowers LLM evaluators to emulate human expert evaluators. AgentAuditor constructs an experiential memory by having an LLM adaptively extract structured semantic features (e.g., scenario, risk, behavior) and generate associated chain-of-thought reasoning traces for past interactions. A multi-stage, context-aware retrieval-augmented generation process then dynamically retrieves the most relevant reasoning experiences to guide the LLM evaluator's assessment of new cases. Moreover, we developed ASSEBench, the first benchmark designed to check how well LLM-based evaluators can spot both safety risks and security threats. ASSEBench comprises 2293 meticulously annotated interaction records, covering 15 risk types across 29 application scenarios. A key feature of ASSEBench is its nuanced approach to ambiguous risk situations, employing "Strict" and "Lenient" judgment standards. Experiments demonstrate that AgentAuditor not only consistently improves the evaluation performance of LLMs across all benchmarks but also sets a new state-of-the-art in LLM-as-a-judge for agent safety and security, achieving human-level accuracy. Our work is openly accessible at https://github.com/Astarojth/AgentAuditor.

Paper Structure

This paper contains 68 sections, 12 equations, 10 figures, 21 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. AgentAuditor: Our Framework
  4. Stage 1: Feature Memory Construction
  5. Stage 2: Reasoning Memory Construction
  6. Stage 3: RAG-based Memory Augmented Reasoning
  7. ASSEBench: Agent Safety & Security Evaluator Benchmark
  8. Structure of ASSEBench
  9. Construction of ASSEBench
  10. Experiments and Results
  11. Experimental Setup
  12. Empirical Results
  13. Ablation Studies
  14. Human Evaluation
  15. Limitation
  16. ...and 53 more sections

Figures (10)

  • Figure 1: Illustration of the overview of AgentAuditor and advantage across baselines.
  • Figure 2: Workflow of AgentAuditor.
  • Figure 3: ASSEBench-safety (three sub-datasets) covers 17 scenarios, 9 risk types, and 14 behavior modes. ASSEBench-Security contains 12 scenarios, 6 risk types, and 12 behavior modes. The complete system and more statistics are provided in Appendix \ref{['apx:statistics']}.
  • Figure 4: Human evaluation results of AgentAuditor.
  • Figure 5: Examples and detailed illustrations of the 4 primary challenges in automated agent safety and security evaluations.
  • ...and 5 more figures