Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain

Eugene Jang, Jian Cui, Dayeon Yim, Youngjin Jin, Jin-Woo Chung, Seungwon Shin, Yongjae Lee

TL;DR

The paper tackles the challenge of pretraining language models for cybersecurity CTI, where non-linguistic elements (NLEs) like URLs and hashes complicate standard MLM-based self-supervision. It introduces a domain-customized pretraining approach combining selective masking with NLE token classification (NLEC) and demonstrates their efficacy on a cybersecurity corpus, culminating in the CyBERTuned model. Empirical results show that Mask-Semis + NLEC consistently outperforms common NLE-replacement techniques on downstream cybersecurity tasks and maintains solid probing performance, while replacement strategies can harm linguistic understanding near NLEs. The work highlights the value of preserving informative NLEs during pretraining and provides public release of CyBERTuned, training resources, and code to facilitate adoption in cyber threat intelligence pipelines.

Abstract

Cybersecurity information is often technically complex and relayed through unstructured text, making automation of cyber threat intelligence highly challenging. For such text domains that involve high levels of expertise, pretraining on in-domain corpora has been a popular method for language models to obtain domain expertise. However, cybersecurity texts often contain non-linguistic elements (such as URLs and hash values) that could be unsuitable with the established pretraining methodologies. Previous work in other domains have removed or filtered such text as noise, but the effectiveness of these methods have not been investigated, especially in the cybersecurity domain. We propose different pretraining methodologies and evaluate their effectiveness through downstream tasks and probing tasks. Our proposed strategy (selective MLM and jointly training NLE token classification) outperforms the commonly taken approach of replacing non-linguistic elements (NLEs). We use our domain-customized methodology to train CyBERTuned, a cybersecurity domain language model that outperforms other cybersecurity PLMs on most tasks.

Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain

TL;DR

The paper tackles the challenge of pretraining language models for cybersecurity CTI, where non-linguistic elements (NLEs) like URLs and hashes complicate standard MLM-based self-supervision. It introduces a domain-customized pretraining approach combining selective masking with NLE token classification (NLEC) and demonstrates their efficacy on a cybersecurity corpus, culminating in the CyBERTuned model. Empirical results show that Mask-Semis + NLEC consistently outperforms common NLE-replacement techniques on downstream cybersecurity tasks and maintains solid probing performance, while replacement strategies can harm linguistic understanding near NLEs. The work highlights the value of preserving informative NLEs during pretraining and provides public release of CyBERTuned, training resources, and code to facilitate adoption in cyber threat intelligence pipelines.

Abstract

Cybersecurity information is often technically complex and relayed through unstructured text, making automation of cyber threat intelligence highly challenging. For such text domains that involve high levels of expertise, pretraining on in-domain corpora has been a popular method for language models to obtain domain expertise. However, cybersecurity texts often contain non-linguistic elements (such as URLs and hash values) that could be unsuitable with the established pretraining methodologies. Previous work in other domains have removed or filtered such text as noise, but the effectiveness of these methods have not been investigated, especially in the cybersecurity domain. We propose different pretraining methodologies and evaluate their effectiveness through downstream tasks and probing tasks. Our proposed strategy (selective MLM and jointly training NLE token classification) outperforms the commonly taken approach of replacing non-linguistic elements (NLEs). We use our domain-customized methodology to train CyBERTuned, a cybersecurity domain language model that outperforms other cybersecurity PLMs on most tasks.
Paper Structure (26 sections, 2 figures, 6 tables)

This paper contains 26 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Non-linguistic Elements
  5. Leveraging NLE Spans in Pretraining
  6. Pretraining the Models
  7. Pretraining Strategies
  8. Cybersecurity Corpus
  9. Pretraining Setup
  10. Experiments
  11. Downstream Tasks
  12. Probing Tasks
  13. Results
  14. CyBERTuned Experiments
  15. Pretraining CyBERTuned
  16. ...and 11 more sections

Figures (2)

  • Figure 1: A threat report excerpt after tokenization and masking with 15% probability. The tokens inside the SHA hash and URL are highlighted. Masked tokens are indicated by a gray bar.
  • Figure 2: The overall architecture of CyBERTuned. NLE spans from the original input is used in the masking step and for NLE classification.