CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence
Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, Nidhi Rastogi
TL;DR
CTIBench introduces a CTI-specific benchmark suite to rigorously evaluate LLMs on memory, understanding, problem-solving, and reasoning within cyber threat intelligence. The framework comprises five tasks (CTI-MCQ, CTI-RCM, CTI-VSP, CTI-ATE, CTI-TAA) with curated data from MITRE, NVD, CWE, and ATT&CK, plus a BFS-based threat actor attribution evaluation. Across five state-of-the-art models, GPT-4 generally excels, with Gemini-1.5 leading in VSP and LLAMA-70B showing competitive performance, while smaller models lag on nuanced CTI tasks. The study also documents task-specific insights, knowledge-cutoff effects, and compute costs, highlighting the practical challenges of deploying LLMs in CTI workflows and offering a publicly available benchmark for future improvements.
Abstract
Cyber threat intelligence (CTI) is crucial in today's cybersecurity landscape, providing essential insights to understand and mitigate the ever-evolving cyber threats. The recent rise of Large Language Models (LLMs) have shown potential in this domain, but concerns about their reliability, accuracy, and hallucinations persist. While existing benchmarks provide general evaluations of LLMs, there are no benchmarks that address the practical and applied aspects of CTI-specific tasks. To bridge this gap, we introduce CTIBench, a benchmark designed to assess LLMs' performance in CTI applications. CTIBench includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape. Our evaluation of several state-of-the-art models on these tasks provides insights into their strengths and weaknesses in CTI contexts, contributing to a better understanding of LLM capabilities in CTI.