SoK: Measuring What Matters for Closed-Loop Security Agents

TL;DR

The paper introduces CLASP, a comprehensive framework that aligns security lifecycles with agentic capabilities to enable principled evaluation of closed-loop autonomous security agents. It provides a taxonomy of five security functions and six agentic capabilities, with detailed rubrics and a systematic survey of 21 works to diagnose gaps and strengths. A Closed-Loop Capability (CLC) Score is proposed to balance end-to-end efficacy with operational efficiency, enabling meaningful comparisons beyond raw outcomes. The work aims to catalyze end-to-end benchmarks and enterprise-ready, capability-attributed evaluations for deploying robust, safe autonomous security agents.

Abstract

Cybersecurity is a relentless arms race, with AI driven offensive systems evolving faster than traditional defenses can adapt. Research and tooling remain fragmented across isolated defensive functions, creating blind spots that adversaries exploit. Autonomous agents capable of integrating, exploit confirmation, remediation, and validation into a single closed loop offer promise, but the field lacks three essentials: a framework defining the agentic capabilities of security systems across security life cycle, a principled method for evaluating closed loop agents, and a benchmark for measuring their performance in practice. We introduce CLASP: the Closed-Loop Autonomous Security Performance framework which aligns the security lifecycle (reconnaissance, exploitation, root cause analysis, patch synthesis, validation) with core agentic capabilities (planning, tool use, memory, reasoning, reflection & perception) providing a common vocabulary and rubric for assessing agentic capabilities in security tasks. By applying CLASP to 21 representative works, we map where systems demonstrate strengths, and where capability gaps persist. We then define the Closed-Loop Capability (CLC) Score, a composite metric quantifying both degree of loop closure and operational effectiveness, and outline the requirements for a closed loop benchmark. Together, CLASP and the CLC Score, provide the vocabulary, diagnostics, and measurements needed to advance both function level performance and measure closed loop security agents.