Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations

Shahin Honarvar, Amber Gorzynski, James Lee-Jones, Harry Coppock, Marek Rei, Joseph Ryan, Alastair F. Donaldson

TL;DR

The study addresses how to evaluate agentic LLMs in cybersecurity beyond single-instance benchmarks by introducing CTF families—semantics-preserving transformations of base challenges—and a dedicated tool, Evolve-CTF, to generate them. Through large-scale experiments with 13 model configurations across 16 Python CTFs from Cybench and Intercode, the authors quantify robustness to transformations, especially under composed obfuscations, and examine tool-use behavior. Key findings show models are robust to identifier renaming and simple insertions but struggle with combined transformations and PyObfuscator-based obfuscation, while explicit reasoning provides limited gains. The work yields a principled framework and dataset for future LLM evaluations in cybersecurity, enabling deeper insights into robustness, generalisation, and tool-assisted reasoning, with open-source plans for Evolve-CTF and experimental data.

Abstract

Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag (CTF) benchmarks. However, existing pointwise benchmarks have limited ability to shed light on the robustness and generalisation abilities of agents across alternative versions of the source code. We introduce CTF challenge families, whereby a single CTF is used as the basis for generating a family of semantically-equivalent challenges via semantics-preserving program transformations. This enables controlled evaluation of agent robustness to source code transformations while keeping the underlying exploit strategy fixed. We introduce a new tool, Evolve-CTF, that generates CTF families from Python challenges using a range of transformations. Using Evolve-CTF to derive families from Cybench and Intercode challenges, we evaluate 13 agentic LLM configurations with tool access. We find that models are remarkably robust to intrusive renaming and code insertion-based transformations, but that composed transformations and deeper obfuscation affect performance by requiring more sophisticated use of tools. We also find that enabling explicit reasoning has little effect on solution success rates across challenge families. Our work contributes a valuable technique and tool for future LLM evaluations, and a large dataset characterising the capabilities of current state-of-the-art models in this domain.

Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations

TL;DR

The study addresses how to evaluate agentic LLMs in cybersecurity beyond single-instance benchmarks by introducing CTF families—semantics-preserving transformations of base challenges—and a dedicated tool, Evolve-CTF, to generate them. Through large-scale experiments with 13 model configurations across 16 Python CTFs from Cybench and Intercode, the authors quantify robustness to transformations, especially under composed obfuscations, and examine tool-use behavior. Key findings show models are robust to identifier renaming and simple insertions but struggle with combined transformations and PyObfuscator-based obfuscation, while explicit reasoning provides limited gains. The work yields a principled framework and dataset for future LLM evaluations in cybersecurity, enabling deeper insights into robustness, generalisation, and tool-assisted reasoning, with open-source plans for Evolve-CTF and experimental data.

Abstract

Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag (CTF) benchmarks. However, existing pointwise benchmarks have limited ability to shed light on the robustness and generalisation abilities of agents across alternative versions of the source code. We introduce CTF challenge families, whereby a single CTF is used as the basis for generating a family of semantically-equivalent challenges via semantics-preserving program transformations. This enables controlled evaluation of agent robustness to source code transformations while keeping the underlying exploit strategy fixed. We introduce a new tool, Evolve-CTF, that generates CTF families from Python challenges using a range of transformations. Using Evolve-CTF to derive families from Cybench and Intercode challenges, we evaluate 13 agentic LLM configurations with tool access. We find that models are remarkably robust to intrusive renaming and code insertion-based transformations, but that composed transformations and deeper obfuscation affect performance by requiring more sophisticated use of tools. We also find that enabling explicit reasoning has little effect on solution success rates across challenge families. Our work contributes a valuable technique and tool for future LLM evaluations, and a large dataset characterising the capabilities of current state-of-the-art models in this domain.
Paper Structure (35 sections, 21 figures, 7 tables)

This paper contains 35 sections, 21 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Transformations Supported by Evolve-CTF
  4. Experimental Design
  5. CTFs Used for Family Generation
  6. LLMs Under Evaluation
  7. Evaluation Protocol
  8. Results and Discussion
  9. Difficulty of CTF Families Across Models
  10. Per-Model Solvability and Impact of Reasoning
  11. Solution Strategies and Use of Tools
  12. Token Usage and Reasons for Failure
  13. Related Work
  14. Conclusion and Future Work
  15. Additional Graphs and Tables of Results
  16. ...and 20 more sections

Figures (21)

  • Figure 1: Excerpt from the Dynastic CTF from the Cybench suite
  • Figure 2: Transforming an original CTF (Orig) leads to a CTF family, based on our five kinds of transformation ($T_1$--$T_5$), optionally preceded by renaming ($R$) and/or followed by obfuscation ($O$)
  • Figure 3: Heatmap showing mean solvability score across models for CTF family instances. Each cell shows the mean of 65 $\{0, 1\}$ outcomes, across 13 model configurations and 5 repeat runs.
  • Figure 4: Heatmap showing mean solvability per model for all instances featuring a particular transformation. Each cell shows the mean of 80 {0,1} outcomes (due to 16 families and 5 repeats).
  • Figure 5: Mean token for each instance across all successful solution attempts. Grey indicate that an instance was never solved.
  • ...and 16 more figures