RedSage: A Cybersecurity Generalist LLM

TL;DR

RedSage introduces an open-source cybersecurity assistant built through domain-aware continual pretraining on a large-scale cybersecurity corpus (CyberFineWeb) and agentic post-training using a curated seed (RedSage-Seed), followed by 266K multi-turn dialogues for supervised fine-tuning and a robust benchmark (RedSage-Bench) with 30K MCQs and 240 FRQs. At 8B parameters, RedSage achieves state-of-the-art results on cybersecurity benchmarks and strong general-task performance, highlighting the value of combining large-scale domain data with agentic data augmentation and alignment. The work demonstrates that domain-specific pretraining plus agentic augmentation can improve both domain expertise and general reasoning, and it provides open data, models, and code to enable reproducible, privacy-preserving cybersecurity AI. The practical impact lies in enabling on-premise deployment on consumer GPUs, reducing reliance on private APIs while maintaining high performance in real-world security workflows.

Abstract

Cybersecurity operations demand assistant LLMs that support diverse workflows without exposing sensitive data. Existing solutions either rely on proprietary APIs with privacy risks or on open models lacking domain adaptation. To bridge this gap, we curate 11.8B tokens of cybersecurity-focused continual pretraining data via large-scale web filtering and manual collection of high-quality resources, spanning 28.6K documents across frameworks, offensive techniques, and security tools. Building on this, we design an agentic augmentation pipeline that simulates expert workflows to generate 266K multi-turn cybersecurity samples for supervised fine-tuning. Combined with general open-source LLM data, these resources enable the training of RedSage, an open-source, locally deployable cybersecurity assistant with domain-aware pretraining and post-training. To rigorously evaluate the models, we introduce RedSage-Bench, a benchmark with 30K multiple-choice and 240 open-ended Q&A items covering cybersecurity knowledge, skills, and tool expertise. RedSage is further evaluated on established cybersecurity benchmarks (e.g., CTI-Bench, CyberMetric, SECURE) and general LLM benchmarks to assess broader generalization. At the 8B scale, RedSage achieves consistently better results, surpassing the baseline models by up to +5.59 points on cybersecurity benchmarks and +5.05 points on Open LLM Leaderboard tasks. These findings demonstrate that domain-aware agentic augmentation and pre/post-training can not only enhance cybersecurity-specific expertise but also help to improve general reasoning and instruction-following. All models, datasets, and code are publicly available.

Figures (9)

• Figure 1: RedSage Overview: a Cybersecurity LLM trained with domain-specific pre-training and post-training, and evaluated on benchmarks spanning knowledge, skills, and tools.
• Figure 3: Taxonomy of RedSage Seed&Bench dataset. It spans knowledge, practical offensive skills, and tool expertise (CLI and Kali Linux).
• Figure 4: Hierarchical taxonomy for the benchmark showing Knowledge, Skill, and Tools with their subcategories.
• Figure 5: RedSage Data Collection and Augmentation Pipeline
• Figure 6: RedSage data pipeline combining large-scale text collection, curated cybersecurity resources, and agentic augmentation for supervised fine-tuning and benchmark generation. Best viewed in Zoom.[increase font size of headers slightly. Keep both versions of the figures, we will select one.]