Toward Cybersecurity-Expert Small Language Models

TL;DR

CyberPal 2.0 delivers domain-specific, cybersecurity-expert small language models (4B–20B) trained on SecKnowledge 2.0, an enriched, document-grounded instruction dataset produced via expert-in-the-loop formatting and multi-step grounding. The approach achieves frontier-like performance on threat management benchmarks, with 7–14% average gains over strong baselines and top results on CTI-RCM for the 20B variant, while enabling on-premises deployment through efficient 8-bit/4-bit quantization. Key contributions include the data- enrichment pipeline, expert-driven format generation, LLM-guided grounding, and rigorous LLM-as-a-Judge evaluation that aligns with human expert judgments. The findings underscore the practical impact of high-quality, domain-focused data pipelines for cybersecurity tasks and suggest strong potential for enterprise adoption in secure, privacy-preserving settings.

Abstract

Large language models (LLMs) are transforming everyday applications, yet deployment in cybersecurity lags due to a lack of high-quality, domain-specific models and training datasets. To address this gap, we present CyberPal 2.0, a family of cybersecurity-expert small language models (SLMs) ranging from 4B-20B parameters. To train CyberPal 2.0, we generate an enriched chain-of-thought cybersecurity instruction dataset built with our data enrichment and formatting pipeline, SecKnowledge 2.0, which integrates expert-in-the-loop steering of reasoning formats alongside LLM-driven multi-step grounding, yielding higher-fidelity, task-grounded reasoning traces for security tasks. Across diverse cybersecurity benchmarks, CyberPal 2.0 consistently outperforms its baselines and matches or surpasses various open and closed-source frontier models, while remaining a fraction of their size. On core cyber threat intelligence knowledge tasks, our models outperform almost all tested frontier models, ranking second only to Sec-Gemini v1. On core threat-investigation tasks, such as correlating vulnerabilities and bug tickets with weaknesses, our best 20B-parameter model outperforms GPT-4o, o1, o3-mini, and Sec-Gemini v1, ranking first, while our smallest 4B-parameter model ranks second.

Figures (15)

• Figure 1: Comparing our models (blue) against frontier models such as Sec-Gemini v1, o1, and o3-mini (gray) on key benchmarks.
• Figure 2: 3-step reformatting and enrichment pipeline overview - experts define task formats for the seed dataset, relevant evidence is retrieved from documents or web sources, answers are reformatted and enriched into structured, knowledge-grounded outputs with LLMaaJudge verification (see \ref{['pipeline_quality_assessment']}).
• Figure 3: Q&A example from SecKnowledge (green), and our improved answer (orange).
• Figure 4: Domain composition of each data source in the evaluation benchmarks.
• Figure 5: Ablation studies (using 4B parameters models) comparing CyberPal 2.0-4B (Ours) against Baseline Reformattingfan2024reformatted and SecKnowledgelevi2025cyberpal.