Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs

Md Tanvirul Alam, Aritran Piplai, Ionut Cardei, Nidhi Rastogi, Peter J Worth

TL;DR

Minerva introduces reinforcement learning with verifiable rewards (RLVR) for cyber threat intelligence (CTI) LLMs. To tackle reward sparsity, it adds MinervaRL, which uses hardness-gated answer-conditioned reasoning (ACR) and periodic distillation to ground learning on long-tail CTI identifiers and structured outputs. The authors curate Minerva-CTI, a 16-task dataset with deterministic verifiers, and demonstrate consistent gains across 12 CTI benchmarks on four open-weight backbones compared to instruction-tuned and SFT baselines. The approach improves both accuracy and robustness in producing verifiable CTI artifacts, suggesting verifiable rewards plus distillation can effectively specialize open LLMs to analyst-facing CTI tasks with scalable training dynamics. The work highlights practical impact for defense-oriented CTI pipelines while noting limitations and directions for multilinguality, stronger verifier filters, and broader task coverage.

Abstract

Cyber threat intelligence (CTI) analysts routinely convert noisy, unstructured security artifacts into standardized, automation-ready representations. Although large language models (LLMs) show promise for this task, existing approaches remain brittle when producing structured CTI outputs and have largely relied on supervised fine-tuning (SFT). In contrast, CTI standards and community-maintained resources define canonical identifiers and schemas that enable deterministic verification of model outputs. We leverage this structure to study reinforcement learning with verifiable rewards (RLVR) for CTI tasks. We introduce \textit{Minerva}, a unified dataset and training pipeline spanning multiple CTI subtasks, each paired with task-specific verifiers that score structured outputs and identifier predictions. To address reward sparsity during rollout, we propose a lightweight self-training mechanism that generates additional verified trajectories and distills them back into the model. Experiments across LLM backbones show consistent improvements in accuracy and robustness over SFT across multiple benchmarks.

Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs

TL;DR

Minerva introduces reinforcement learning with verifiable rewards (RLVR) for cyber threat intelligence (CTI) LLMs. To tackle reward sparsity, it adds MinervaRL, which uses hardness-gated answer-conditioned reasoning (ACR) and periodic distillation to ground learning on long-tail CTI identifiers and structured outputs. The authors curate Minerva-CTI, a 16-task dataset with deterministic verifiers, and demonstrate consistent gains across 12 CTI benchmarks on four open-weight backbones compared to instruction-tuned and SFT baselines. The approach improves both accuracy and robustness in producing verifiable CTI artifacts, suggesting verifiable rewards plus distillation can effectively specialize open LLMs to analyst-facing CTI tasks with scalable training dynamics. The work highlights practical impact for defense-oriented CTI pipelines while noting limitations and directions for multilinguality, stronger verifier filters, and broader task coverage.

Abstract

Cyber threat intelligence (CTI) analysts routinely convert noisy, unstructured security artifacts into standardized, automation-ready representations. Although large language models (LLMs) show promise for this task, existing approaches remain brittle when producing structured CTI outputs and have largely relied on supervised fine-tuning (SFT). In contrast, CTI standards and community-maintained resources define canonical identifiers and schemas that enable deterministic verification of model outputs. We leverage this structure to study reinforcement learning with verifiable rewards (RLVR) for CTI tasks. We introduce \textit{Minerva}, a unified dataset and training pipeline spanning multiple CTI subtasks, each paired with task-specific verifiers that score structured outputs and identifier predictions. To address reward sparsity during rollout, we propose a lightweight self-training mechanism that generates additional verified trajectories and distills them back into the model. Experiments across LLM backbones show consistent improvements in accuracy and robustness over SFT across multiple benchmarks.
Paper Structure (87 sections, 4 theorems, 17 equations, 8 figures, 7 tables, 1 algorithm)

This paper contains 87 sections, 4 theorems, 17 equations, 8 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for CTI.
  4. CTI benchmarks for LLM evaluation.
  5. RLVR for LLMs.
  6. Minerva-CTI Dataset
  7. Methodology
  8. Reinforcement Learning with Verifiable Rewards
  9. MinervaRL
  10. Notation.
  11. Step 1 (Algorithm \ref{['alg:minervarl']}): RLVR rollouts + GRPO update.
  12. Step 2 (Algorithm \ref{['alg:minervarl']}): Buffer hard examples for ACR.
  13. Step 3 (Algorithm \ref{['alg:minervarl']}): Deferred ACR generation + filtering.
  14. Step 4 (Algorithm \ref{['alg:minervarl']}): SFT distillation.
  15. Expanding Empirical Support with MinervaRL
  16. ...and 72 more sections

Key Result

Lemma 6.1

Fix a prompt $x$ and an answer $a\in\mathcal{A}$. Let $p=p_\theta(a\mid x)$. If $p \ge \varepsilon_{k,\zeta}$, then the probability that $k$ independent rollouts miss $a$ is at most $\zeta$: Equivalently, if $a$ is never observed in $k$ rollouts, then with confidence at least $1-\zeta$ we have $p \le \varepsilon_{k,\zeta}$.

Figures (8)

  • Figure 1: MinervaRL overview.
  • Figure 2: Prompt template used for pairwise response-quality judging (both responses are verifier-correct).
  • Figure 3: Pairwise preference heatmap from the CTI judge. Each cell (row A, column B) shows the fraction of comparisons where A is preferred over B (ties excluded). Higher values indicate stronger preference for the row model.
  • Figure 4: Fraction of training prompts with no successful rollout (max reward $=0$) over training steps for Llama-8B-Instruct. Curves are smoothed with a 5-step rolling mean.
  • Figure 5: Acceptance rates of the ACR self-distillation pipeline over training for Llama-8B-Instruct: fraction of batches passing the heuristic filter, passing the machine-learning (ML) filter, and meeting unique identifier (UID) coverage (i.e., accepted traces span a minimum number of distinct UIDs).
  • ...and 3 more figures

Theorems & Definitions (4)

  • Lemma 6.1: Finite-sample detectability bound
  • Theorem 6.2: Small-budget sampling barrier for on-policy RLVR (per-prompt)
  • Theorem 6.5: Empirical support seeding via answer-conditioned trace distillation
  • Corollary 6.6: Expected cycles to cross the detectability threshold