Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

Paul Kassianik, Baturay Saglam, Alexander Chen, Blaine Nelson, Anu Vellore, Massimo Aufiero, Fraser Burch, Dhruv Kedia, Avi Zohary, Sajana Weerawardhena, Aman Priyanshu, Adam Swanda, Amy Chang, Hyrum Anderson, Kojin Oshiba, Omar Santos, Yaron Singer, Amin Karbasi

TL;DR

Foundation-Sec-8B presents a cybersecurity-specialized LLM built on Llama 3.1-8B, achieved via continued pretraining on a curated cybersecurity corpus. It demonstrates competitive performance against larger models on CTI-focused benchmarks (CTIBench, CyberMetric, SecBench) while preserving general knowledge on MMLU, and shows practical utility in SOC automation, threat modeling, and security engineering enablement. The work provides public checkpoints to accelerate adoption and emphasizes that domain-aware pretraining can enable smaller models to approach or match larger, general-purpose LLMs in domain-specific cybersecurity tasks. Together, these results highlight the value of targeted data curation and continued pretraining for deploying effective defense-oriented LLMs at modest parameter scales.

Abstract

As transformer-based large language models (LLMs) increasingly permeate society, they have revolutionized domains such as software engineering, creative writing, and digital arts. However, their adoption in cybersecurity remains limited due to challenges like scarcity of specialized training data and complexity of representing cybersecurity-specific knowledge. To address these gaps, we present Foundation-Sec-8B, a cybersecurity-focused LLM built on the Llama 3.1 architecture and enhanced through continued pretraining on a carefully curated cybersecurity corpus. We evaluate Foundation-Sec-8B across both established and new cybersecurity benchmarks, showing that it matches Llama 3.1-70B and GPT-4o-mini in certain cybersecurity-specific tasks. By releasing our model to the public, we aim to accelerate progress and adoption of AI-driven tools in both public and private cybersecurity contexts.

Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

TL;DR

Foundation-Sec-8B presents a cybersecurity-specialized LLM built on Llama 3.1-8B, achieved via continued pretraining on a curated cybersecurity corpus. It demonstrates competitive performance against larger models on CTI-focused benchmarks (CTIBench, CyberMetric, SecBench) while preserving general knowledge on MMLU, and shows practical utility in SOC automation, threat modeling, and security engineering enablement. The work provides public checkpoints to accelerate adoption and emphasizes that domain-aware pretraining can enable smaller models to approach or match larger, general-purpose LLMs in domain-specific cybersecurity tasks. Together, these results highlight the value of targeted data curation and continued pretraining for deploying effective defense-oriented LLMs at modest parameter scales.

Abstract

As transformer-based large language models (LLMs) increasingly permeate society, they have revolutionized domains such as software engineering, creative writing, and digital arts. However, their adoption in cybersecurity remains limited due to challenges like scarcity of specialized training data and complexity of representing cybersecurity-specific knowledge. To address these gaps, we present Foundation-Sec-8B, a cybersecurity-focused LLM built on the Llama 3.1 architecture and enhanced through continued pretraining on a carefully curated cybersecurity corpus. We evaluate Foundation-Sec-8B across both established and new cybersecurity benchmarks, showing that it matches Llama 3.1-70B and GPT-4o-mini in certain cybersecurity-specific tasks. By releasing our model to the public, we aim to accelerate progress and adoption of AI-driven tools in both public and private cybersecurity contexts.
Paper Structure (52 sections, 7 figures, 2 tables)

This paper contains 52 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Continued Pretraining
  4. Benchmarks
  5. Security Benchmarks
  6. CTIBench
  7. CyberMetric
  8. SecBench
  9. Non-Security Benchmarks
  10. MMLU
  11. LLMs for Security
  12. WhiteRabbitNeo-V2
  13. Primus
  14. SecurityLLM
  15. SecGemini
  16. ...and 37 more sections

Figures (7)

  • Figure 1: Overview of core results on the selected cybersecurity benchmarks. Foundation-Sec-8B shows significant improvement over Llama 3.1-8B while matching or surpassing GPT-4o-mini in cyber threat intelligence knowledge.
  • Figure 2: Three-stage data collection and processing pipeline: (1) Scraping Stage—a wide-net scraper gathers 4 TiB of raw web content; (2) Parallelized Processing Stage—language filtering, quality filtering, and relevancy classification prune the data; (3) Training Preparation Stage—deduplication, PII replacement, and tokenization produce 25 GiB (about 5 billion tokens) of final training data.
  • Figure 3: Relevancy classifier on evaluation set
  • Figure 4: Few-shot prompt format used with pretrained models for MCQA tasks. The model is expected to respond in the format "Answer: X," where X is one of A, B, C, or D.
  • Figure 5: Few-shot prompt format used with pretrained models for the CWE ID mapping task (CTIBench-RCM). The model is expected to respond in the format "Answer: CWE-X," where X is a unique CWE ID.
  • ...and 2 more figures