Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs

Francesco Balassone, Víctor Mayoral-Vilches, Stefan Rass, Martin Pinzger, Gaetano Perrone, Simon Pietro Romano, Peter Schartner

TL;DR

This paper empirically contrasts AI-driven offense and defense in Attack/Defense CTFs using CAI's parallel execution to run autonomous red and blue agents across 23 battlegrounds with 15-minute windows. Under unconstrained conditions, defensive patching significantly outperformed offensive initial access ($p=0.0193$), but when defense faced operational requirements like maintaining availability or preventing all intrusions, the advantage disappeared, indicating that defense effectiveness is highly sensitive to success criteria. The authors also provide taxonomy-based analyses linking outcomes to MITRE ATT&CK, CWE, and CAPEC categories, highlighting patterns in vulnerability types and attack vectors while noting small sample sizes limit broad generalizations. The work argues against a universal AI attacker advantage and underscores the need for constraint-aware testing and open, reusable Cybersecurity AI frameworks to sustain defense parity as offensive automation accelerates. Overall, the study establishes a rigorous framework for evaluating AI security capabilities and motivates practical deployment considerations for defender-led AI tooling.

Abstract

We empirically evaluate whether AI systems are more effective at attacking or defending in cybersecurity. Using CAI (Cybersecurity AI)'s parallel execution framework, we deployed autonomous agents in 23 Attack/Defense CTF battlegrounds. Statistical analysis reveals defensive agents achieve 54.3% unconstrained patching success versus 28.3% offensive initial access (p=0.0193), but this advantage disappears under operational constraints: when defense requires maintaining availability (23.9%) and preventing all intrusions (15.2%), no significant difference exists (p>0.05). Exploratory taxonomy analysis suggests potential patterns in vulnerability exploitation, though limited sample sizes preclude definitive conclusions. This study provides the first controlled empirical evidence challenging claims of AI attacker advantage, demonstrating that defensive effectiveness critically depends on success criteria, a nuance absent from conceptual analyses but essential for deployment. These findings underscore the urgency for defenders to adopt open-source Cybersecurity AI frameworks to maintain security equilibrium against accelerating offensive automation.

Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs

TL;DR

This paper empirically contrasts AI-driven offense and defense in Attack/Defense CTFs using CAI's parallel execution to run autonomous red and blue agents across 23 battlegrounds with 15-minute windows. Under unconstrained conditions, defensive patching significantly outperformed offensive initial access (), but when defense faced operational requirements like maintaining availability or preventing all intrusions, the advantage disappeared, indicating that defense effectiveness is highly sensitive to success criteria. The authors also provide taxonomy-based analyses linking outcomes to MITRE ATT&CK, CWE, and CAPEC categories, highlighting patterns in vulnerability types and attack vectors while noting small sample sizes limit broad generalizations. The work argues against a universal AI attacker advantage and underscores the need for constraint-aware testing and open, reusable Cybersecurity AI frameworks to sustain defense parity as offensive automation accelerates. Overall, the study establishes a rigorous framework for evaluating AI security capabilities and motivates practical deployment considerations for defender-led AI tooling.

Abstract

We empirically evaluate whether AI systems are more effective at attacking or defending in cybersecurity. Using CAI (Cybersecurity AI)'s parallel execution framework, we deployed autonomous agents in 23 Attack/Defense CTF battlegrounds. Statistical analysis reveals defensive agents achieve 54.3% unconstrained patching success versus 28.3% offensive initial access (p=0.0193), but this advantage disappears under operational constraints: when defense requires maintaining availability (23.9%) and preventing all intrusions (15.2%), no significant difference exists (p>0.05). Exploratory taxonomy analysis suggests potential patterns in vulnerability exploitation, though limited sample sizes preclude definitive conclusions. This study provides the first controlled empirical evidence challenging claims of AI attacker advantage, demonstrating that defensive effectiveness critically depends on success criteria, a nuance absent from conceptual analyses but essential for deployment. These findings underscore the urgency for defenders to adopt open-source Cybersecurity AI frameworks to maintain security equilibrium against accelerating offensive automation.
Paper Structure (27 sections, 11 figures, 12 tables)

This paper contains 27 sections, 11 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Research Contributions
  4. Methodology
  5. CAI Parallel Execution Architecture
  6. Data Collection and Evaluation Framework
  7. Statistical Methods
  8. Experimental Setup
  9. Agent Setup
  10. Results
  11. Statistical Analysis
  12. Security Taxonomy Analysis
  13. Performance by Vulnerability Type
  14. Attack Technique Effectiveness
  15. Attack Pattern Correlation
  16. ...and 12 more sections

Figures (11)

  • Figure 1: Core research question: Evaluating AI effectiveness in offensive versus defensive cybersecurity roles.
  • Figure 2: CAI Parallel Execution Architecture: The system bifurcates from a single CAI instance through the /parallel command, enabling concurrent Red Team (offensive) and Blue Team (defensive) operations within the same target environment. Both agents operate simultaneously on the shared platform.
  • Figure 3: CAI Attack/Defense setup overview showing the experimental architecture with two competing teams.
  • Figure 4: Comparative analysis of offensive and defensive AI performance using identical experimental constraints and Claude Sonnet 4 (claude-sonnet-4-20250514) for both offensive and defensive agents. Initial access rates (28.3% average) show substantially lower success compared to defensive capabilities (60.9% detection, 54.3% patching).
  • Figure 5: Scoring distribution analysis reveals distinct patterns: Own Points (offensive) concentrate near zero while Availability Points (defensive) show broader, higher-value distributions.
  • ...and 6 more figures