Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence

Md Tanvirul Alam, Dipkamal Bhusal, Salman Ahmad, Nidhi Rastogi, Peter Worth

TL;DR

AthenaBench introduces a dynamic CTI benchmark that continuously updates via live sources like MITRE ATT&CK and the NVD to evaluate LLMs on six knowledge- and reasoning-intensive tasks. It extends prior work by adding a Risk Mitigation Strategy task, refining data curation, duplicate removal, and unified scoring, and by releasing a compact AthenaBench-Mini for rapid evaluation. Across twelve LLMs, proprietary models show stronger performance overall but still struggle with attribution and mitigation reasoning, highlighting gaps in current CTI-tailored automation. The benchmark provides a foundation for ongoing assessment and advancement of LLM-enabled CTI workflows.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in natural language reasoning, yet their application to Cyber Threat Intelligence (CTI) remains limited. CTI analysis involves distilling large volumes of unstructured reports into actionable knowledge, a process where LLMs could substantially reduce analyst workload. CTIBench introduced a comprehensive benchmark for evaluating LLMs across multiple CTI tasks. In this work, we extend CTIBench by developing AthenaBench, an enhanced benchmark that includes an improved dataset creation pipeline, duplicate removal, refined evaluation metrics, and a new task focused on risk mitigation strategies. We evaluate twelve LLMs, including state-of-the-art proprietary models such as GPT-5 and Gemini-2.5 Pro, alongside seven open-source models from the LLaMA and Qwen families. While proprietary LLMs achieve stronger results overall, their performance remains subpar on reasoning-intensive tasks, such as threat actor attribution and risk mitigation, with open-source models trailing even further behind. These findings highlight fundamental limitations in the reasoning capabilities of current LLMs and underscore the need for models explicitly tailored to CTI workflows and automation.

AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence

TL;DR

AthenaBench introduces a dynamic CTI benchmark that continuously updates via live sources like MITRE ATT&CK and the NVD to evaluate LLMs on six knowledge- and reasoning-intensive tasks. It extends prior work by adding a Risk Mitigation Strategy task, refining data curation, duplicate removal, and unified scoring, and by releasing a compact AthenaBench-Mini for rapid evaluation. Across twelve LLMs, proprietary models show stronger performance overall but still struggle with attribution and mitigation reasoning, highlighting gaps in current CTI-tailored automation. The benchmark provides a foundation for ongoing assessment and advancement of LLM-enabled CTI workflows.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in natural language reasoning, yet their application to Cyber Threat Intelligence (CTI) remains limited. CTI analysis involves distilling large volumes of unstructured reports into actionable knowledge, a process where LLMs could substantially reduce analyst workload. CTIBench introduced a comprehensive benchmark for evaluating LLMs across multiple CTI tasks. In this work, we extend CTIBench by developing AthenaBench, an enhanced benchmark that includes an improved dataset creation pipeline, duplicate removal, refined evaluation metrics, and a new task focused on risk mitigation strategies. We evaluate twelve LLMs, including state-of-the-art proprietary models such as GPT-5 and Gemini-2.5 Pro, alongside seven open-source models from the LLaMA and Qwen families. While proprietary LLMs achieve stronger results overall, their performance remains subpar on reasoning-intensive tasks, such as threat actor attribution and risk mitigation, with open-source models trailing even further behind. These findings highlight fundamental limitations in the reasoning capabilities of current LLMs and underscore the need for models explicitly tailored to CTI workflows and automation.

Paper Structure

This paper contains 21 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Methodology: Benchmark Design
  4. CTI Knowledge Test (CKT)
  5. Root Cause Mapping (RCM)
  6. Vulnerability Severity Prediction (VSP)
  7. Threat Actor Attribution (TAA)
  8. Risk Mitigation Strategy (RMS)
  9. Attack Technique Extraction (ATE)
  10. Experiments and Results
  11. Experimental Settings
  12. Evaluation metrics
  13. AthenaBench-Mini
  14. Results
  15. Results with Web Search Capability
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Overview of LLM benchmarking tasks in AthenaBench