Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting

TL;DR

CyberTeam introduces an embodied, function-guided benchmark to evaluate large language models in blue-team threat hunting. By structuring threat-hunting workflows as a dependency chain and mapping tasks to nine embodied functions across 30 objectives, the framework enables end-to-end, pipelined reasoning that balances precision with flexibility. Empirical results show that embodied function calling outperforms traditional prompting strategies, especially for complex, multi-step tasks, while revealing limitations in dependency resolution and resilience to semantic noise. The benchmark provides a practical foundation for integrating AI-assisted threat hunting into real-world security operations and guides future research on robust, modular LLM-enabled defense workflows.

Abstract

As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models (LLMs) offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue team threat-hunting scenarios remains insufficiently explored. In this paper, we present CYBERTEAM, a benchmark designed to guide LLMs in blue teaming practice. CYBERTEAM constructs an embodied environment in two stages. First, it models realistic threat-hunting workflows by capturing the dependencies among analytical tasks from threat attribution to incident response. Next, each task is addressed through a set of embodied functions tailored to its specific analytical requirements. This transforms the overall threat-hunting process into a structured sequence of function-driven operations, where each node represents a discrete function and edges define the execution order. Guided by this framework, LLMs are directed to perform threat-hunting tasks through modular steps. Overall, CYBERTEAM integrates 30 tasks and 9 embodied functions, guiding LLMs through pipelined threat analysis. We evaluate leading LLMs and state-of-the-art cybersecurity agents, comparing CYBERTEAM's embodied function-calling against fundamental elicitation strategies. Our results offer valuable insights into the current capabilities and limitations of LLMs in threat hunting, laying the foundation for the practical adoption in real-world cybersecurity applications.

Figures (8)

• Figure 1: A threat hunting example equipped with the embodied functions. Function names: NER--named entity recognition, REX--regex parsing, MAP--text mapping, RAG--retrieval-augmented generation, CLS--classification, SUM--summarization.
• Figure 2: An threat-hunting example demonstrating a dependency chain of analytical tasks, where each task is completed through a sequence of embodied functions executed by LLMs autonomously.
• Figure 3: Threat-hunting performance (scaled to 100%) on individual tasks, evaluating under GPT-4o across various elicitation strategies: ICL, CoT, ToT, and using embodied functions (Emb). Results for additional LLMs are provided in Appendix \ref{['app:expt']}.
• Figure 4: Evaluation of LLM performance in selecting the correct (a) task dependencies, and (b–e) embodied functions for specific analytical targets.
• Figure 5: LLM performance when input threat logs are perturbed with token-level noise (solid line) or semantic-level noise (dashed line).