Hacking CTFs with Plain Agents

TL;DR

This paper investigates the hacking capabilities of large language models on a high-school level benchmark (InterCode-CTF). It shows that plain prompting, expanded tool use, and multiple attempts using ReAct and Plan strategies can reach 95% task completion, surpassing prior state-of-the-art results (e.g., 29% and 72%). The key finding is that complex harnessing is not necessary; simple prompt engineering suffices to unlock strong performance. The results emphasize the need for stronger evaluation and harder benchmarks for AI risk assessment in cybersecurity.

Abstract

We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%). Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing.

Figures (8)

• Figure 1: Performance increase over weeks of agent design improvements.
• Figure 2: Plan&Solve agent
• Figure 3: ReAct agent
• Figure 4: ReAct&Plan agent
• Figure 5: Tree of Thoughts agent