SafeEvalAgent: Toward Agentic and Self-Evolving Safety Evaluation of LLMs

TL;DR

This work addresses the safety evaluation gap created by static benchmarks in rapidly evolving AI policy landscapes. It introduces SafeEvalAgent, a multi-agent, self-evolving framework that ingests unstructured policy documents to forge a living safety benchmark, guided by a Regulation-to-Knowledge Transformation pipeline and a test-suite generation loop. Through a self-evolving evaluation involving an Evaluator and an Analyst, the system iteratively uncovers deeper vulnerabilities, outperforming static baselines in exposing risk areas across EU AI Act, NIST RMF, and MAS FEAT. Empirical results show significant safety declines as tests intensify (e.g., GPT-5 EU Act safety dropping from $72.50\%$ to $36.36\%$), highlighting the necessity of dynamic, regulation-grounded assessment for responsible AI deployment.

Abstract

The rapid integration of Large Language Models (LLMs) into high-stakes domains necessitates reliable safety and compliance evaluation. However, existing static benchmarks are ill-equipped to address the dynamic nature of AI risks and evolving regulations, creating a critical safety gap. This paper introduces a new paradigm of agentic safety evaluation, reframing evaluation as a continuous and self-evolving process rather than a one-time audit. We then propose a novel multi-agent framework SafeEvalAgent, which autonomously ingests unstructured policy documents to generate and perpetually evolve a comprehensive safety benchmark. SafeEvalAgent leverages a synergistic pipeline of specialized agents and incorporates a Self-evolving Evaluation loop, where the system learns from evaluation results to craft progressively more sophisticated and targeted test cases. Our experiments demonstrate the effectiveness of SafeEvalAgent, showing a consistent decline in model safety as the evaluation hardens. For instance, GPT-5's safety rate on the EU AI Act drops from 72.50% to 36.36% over successive iterations. These findings reveal the limitations of static assessments and highlight our framework's ability to uncover deep vulnerabilities missed by traditional methods, underscoring the urgent need for dynamic evaluation ecosystems to ensure the safe and responsible deployment of advanced AI.

Figures (3)

• Figure 1: Overview of SafeEvalAgent. It first transforms regulations into a testable knowledge base via the Specialist agent, then generates a comprehensive test suite with the Generator agent, and finally performs a self-evolving evaluation process in which the Evaluator, Analyst, and Generator agents collaborate and adapt to uncover deeper vulnerabilities.
• Figure 2: Validation of the Specialist Agent’s ($\mathcal{A}_{S}$) knowledge structuring capability. The heatmaps display the semantic similarity between the explanation fields of atomic rules extracted by $\mathcal{A}_{S}$ from documents: (a) the NIST AI RMF, (b) the EU AI Act, and (c) the MAS FEAT. The outlined regions group rules that belong to the same high-level dimension. The pronounced clusters of high similarity (darker colors) within these outlines demonstrate strong intra-cluster coherence.
• Figure 3: LLMs safety rates during the evaluation across three regulatory frameworks. The consistent decline demonstrates the efficacy of our Self-evolving Evaluation loop.