Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CyberNER: A Harmonized STIX Corpus for Cybersecurity Named Entity Recognition

Yasir Ech-Chammakhy, Anas Motii, Anass Rabii, Oussama Azrara, Jaafar Chbili

TL;DR

This work tackles the fragmentation of cybersecurity NER data by introducing CyberNER, a harmonized STIX 2.1–based corpus derived from CyNER, DNRTI, APTNER, and Attacker. It presents a principled schema harmonization methodology, mapping diverse source tags to 21 STIX entity types across 609,922 tokens, and demonstrates that naive concatenation of datasets severely degrades performance. Empirical results show a substantial relative F1 improvement of about 30% over the naive baseline when training on CyberNER, with strong cross-dataset generalization across diverse CTI domains. The authors publicly release CyberNER to provide a standardized benchmark for robust, interoperable cybersecurity NER models and outline future work on expanding datasets, ablation studies, and extending to relation extraction and knowledge graphs.

Abstract

Extracting structured intelligence via Named Entity Recognition (NER) is critical for cybersecurity, but the proliferation of datasets with incompatible annotation schemas hinders the development of comprehensive models. While combining these resources is desirable, we empirically demonstrate that naively concatenating them results in a noisy label space that severely degrades model performance. To overcome this critical limitation, we introduce CyberNER, a large-scale, unified corpus created by systematically harmonizing four prominent datasets (CyNER, DNRTI, APTNER, and Attacker) onto the STIX 2.1 standard. Our principled methodology resolves semantic ambiguities and consolidates over 50 disparate source tags into 21 coherent entity types. Our experiments show that models trained on CyberNER achieve a substantial performance gain, with a relative F1-score improvement of approximately 30% over the naive concatenation baseline. By publicly releasing the CyberNER corpus, we provide a crucial, standardized benchmark that enables the creation and rigorous comparison of more robust and generalizable entity extraction models for the cybersecurity domain.

CyberNER: A Harmonized STIX Corpus for Cybersecurity Named Entity Recognition

TL;DR

This work tackles the fragmentation of cybersecurity NER data by introducing CyberNER, a harmonized STIX 2.1–based corpus derived from CyNER, DNRTI, APTNER, and Attacker. It presents a principled schema harmonization methodology, mapping diverse source tags to 21 STIX entity types across 609,922 tokens, and demonstrates that naive concatenation of datasets severely degrades performance. Empirical results show a substantial relative F1 improvement of about 30% over the naive baseline when training on CyberNER, with strong cross-dataset generalization across diverse CTI domains. The authors publicly release CyberNER to provide a standardized benchmark for robust, interoperable cybersecurity NER models and outline future work on expanding datasets, ablation studies, and extending to relation extraction and knowledge graphs.

Abstract

Extracting structured intelligence via Named Entity Recognition (NER) is critical for cybersecurity, but the proliferation of datasets with incompatible annotation schemas hinders the development of comprehensive models. While combining these resources is desirable, we empirically demonstrate that naively concatenating them results in a noisy label space that severely degrades model performance. To overcome this critical limitation, we introduce CyberNER, a large-scale, unified corpus created by systematically harmonizing four prominent datasets (CyNER, DNRTI, APTNER, and Attacker) onto the STIX 2.1 standard. Our principled methodology resolves semantic ambiguities and consolidates over 50 disparate source tags into 21 coherent entity types. Our experiments show that models trained on CyberNER achieve a substantial performance gain, with a relative F1-score improvement of approximately 30% over the naive concatenation baseline. By publicly releasing the CyberNER corpus, we provide a crucial, standardized benchmark that enables the creation and rigorous comparison of more robust and generalizable entity extraction models for the cybersecurity domain.

Paper Structure

This paper contains 15 sections, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Schema Harmonization Methodology
  4. Source Dataset Analysis
  5. Target Taxonomy Design using STIX 2.1
  6. Schema Mapping and Dataset Construction
  7. CyberNER Corpus
  8. Statistics
  9. Dataset Benchmarking
  10. Experimental Setup
  11. Baseline Performance on Source Datasets
  12. Baseline: The Failure of Naive Concatenation
  13. Performance Evaluation on CyberNER
  14. Discussion
  15. Conclusion

Figures (4)

  • Figure 1: Distribution of Annotated Entities by Unified STIX Type in the CyberNER Corpus.
  • Figure 2: Best Performing Model and Score for Each Source Dataset across Different Evaluation Metrics (Accuracy, F1, Precision, Recall). Models were evaluated independently on original, non-harmonized data.
  • Figure 3: Overall Performance (Micro F1-Score) of models trained and evaluated on the unified CyberNER test set.
  • Figure 4: Source-Specific Performance (F1-Score) on test subsets for models trained on the unified CyberNER corpus.