SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligence

TL;DR

SEvenLLM tackles data scarcity and evaluation gaps in cyber threat intelligence by building a bilingual instruction corpus and a dedicated CTI benchmark. It trains open-source backbones on SEvenLLM-Instruct using a bilingual multitask objective across 28 tasks, formalized by the loss $\mathcal{L}_{all} = -\sum_{i=1}^{m} \mathbb{E}_{q_{k}^{L_i},a_{k},t_{k}\in D_i}[\log P(a_{k},t_{k}|q_{t_k}; \theta)]$, with $D_{all} = \{D_i\}_{i=1}^{m}$. The SEvenLLM-Bench provides a comprehensive evaluation across Rouge-L, semantic similarity, GPT-4 scoring, and MCQ, showing performance gains for bilingual CTI tasks, especially in mid-to-large backbones and with multi-task instruction tuning. While results demonstrate clear benefits for CTI analytics and incident response, the work acknowledges language coverage limitations and ethical considerations, outlining a path toward broader multilingual CTI support and responsible deployment.

Abstract

To address the increasing complexity and frequency of cybersecurity incidents emphasized by the recent cybersecurity threat reports with over 10 billion instances, cyber threat intelligence (CTI) plays a critical role in the modern cybersecurity landscape by offering the insights required to understand and combat the constantly evolving nature of cyber threats. Inspired by the powerful capability of large language models (LLMs) in handling complex tasks, in this paper, we introduce a framework to benchmark, elicit, and improve cybersecurity incident analysis and response abilities in LLMs for Security Events (SEvenLLM). Specifically, we create a high-quality bilingual instruction corpus by crawling cybersecurity raw text from cybersecurity websites to overcome the lack of effective data for information extraction. Then, we design a pipeline to auto-select tasks from the tasks pool and convert the raw text into supervised corpora comprised of question and response. The instruction dataset SEvenLLM-Instruct is used to train cybersecurity LLMs with the multi-task learning objective (27 well-designed tasks) for augmenting the analysis of cybersecurity events. Extensive experiments in our curated benchmark (SEvenLLM-bench) demonstrate that SEvenLLM performs more sophisticated threat analysis and fortifies defenses against the evolving landscape of cyber threats.

Figures (4)

• Figure 1: Comparison between GPT-3.5 and our proposed model SEvenLLM.
• Figure 2: Overview of SEvenLLM. By crawling different formats of files from the Internet, we collect bilingual (English and Chinese) collection of cybersecurity incident reports. First, we adopt LLMs to produce potential tasks and refine them to create a task pool. Given raw cybersecurity texts, we use Select-Instruct select a proper task and generate the query and its answer. The open-source LLMs are further fine-tuned on SEvenLLM-Instruct with multi-task learning objectives tailored for CTI. A curated CTI evaluation benchmark SEvenLLM-Bench is created to compare SEvenLLM with other baselines.
• Figure 3: Evaluation results of different instruction data sizes.
• Figure 4: Comparison between SEvenLLM with Llama-2-Chat.