D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security

TL;DR

D-CIPHER introduces a dynamic multi-agent framework for autonomous CTF solving by separating responsibilities into a Planner, multiple heterogeneous Executors, and an Auto-prompter. The system leverages a shared container environment and function-calling interactions to enable ongoing collaboration, dynamic feedback, and longer-horizon reasoning beyond single-agent loops. Empirical evaluation across NYU CTF Bench, Cybench, and HackTheBox demonstrates state-of-the-art performance and substantial MITRE ATT&CK technique coverage, while ablation studies illuminate the contributions of the Auto-prompter and Planner-Executor architecture. The work further extends evaluation by mapping CTFs to MITRE ATT&CK techniques, underscoring D-CIPHER’s offensive security capability and its potential impact on automated cybersecurity workflows, with candid discussion of limitations and ethical considerations.

Abstract

Large Language Models (LLMs) have been used in cybersecurity such as autonomous security analysis or penetration testing. Capture the Flag (CTF) challenges serve as benchmarks to assess automated task-planning abilities of LLM agents for cybersecurity. Early attempts to apply LLMs for solving CTF challenges used single-agent systems, where feedback was restricted to a single reasoning-action loop. This approach was inadequate for complex CTF tasks. Inspired by real-world CTF competitions, where teams of experts collaborate, we introduce the D-CIPHER LLM multi-agent framework for collaborative CTF solving. D-CIPHER integrates agents with distinct roles with dynamic feedback loops to enhance reasoning on complex tasks. It introduces the Planner-Executor agent system, consisting of a Planner agent for overall problem-solving along with multiple heterogeneous Executor agents for individual tasks, facilitating efficient allocation of responsibilities among the agents. Additionally, D-CIPHER incorporates an Auto-prompter agent to improve problem-solving by auto-generating a highly relevant initial prompt. We evaluate D-CIPHER on multiple CTF benchmarks and LLM models via comprehensive studies to highlight the impact of our enhancements. Additionally, we manually map the CTFs in NYU CTF Bench to MITRE ATT&CK techniques that apply for a comprehensive evaluation of D-CIPHER's offensive security capability. D-CIPHER achieves state-of-the-art performance on three benchmarks: 22.0% on NYU CTF Bench, 22.5% on Cybench, and 44.0% on HackTheBox, which is 2.5% to 8.5% better than previous work. D-CIPHER solves 65% more ATT&CK techniques compared to previous work, demonstrating stronger offensive capability.

Figures (10)

• Figure 1: Overview of D-CIPHER. The Auto-prompter, Planner, and heterogeneous Executors all collaborate and interact to solve the CTF.
• Figure 2: Workflow of the D-CIPHER multi-agent system. Execution starts with the Auto-prompter which explores the CTF and produces a dynamic, relevant prompt. The Planner proceeds with exploration and delegates specific tasks to the Executors. Each Executor starts with a fresh conversation history to focus on the delegated task, while the Planner maintains overall context and drives the problem solving.
• Figure 3: Auto-prompter generated prompt vs. hard-coded template for the collision_course CTF. Auto-prompter's dynamic prompt captures the approach tailored for this CTF.
• Figure 4: Planner and Executors interact for the collision_course cryptography CTF. Planner drives the problem solving, while each Executor focuses on delegated tasks and implements specific MITRE ATT&CKs.
• Figure 5: % solved by category for D-CIPHER on NYU CTF Bench.