Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Ethically Grounded LLM-Based Approach to Insider Threat Synthesis and Detection

Haywood Gelman, John D. Hastings, David Kenley

TL;DR

Insider threats in corporate logs are difficult to study due to privacy constraints on real data. The authors propose an ethically grounded three-phase framework that uses LLMs (Claude Sonnet 3.7 and GPT-4o) to dynamically synthesize SIEM-like syslog messages with an imbalanced insider-threat rate of $1\%$, and to evaluate detection performance. Across multiple runs, Sonnet 3.7 achieved higher accuracy and much lower false-alarm rates than GPT-4o (average accuracy about $0.899$ vs $0.560$, ROC AUC about $0.994$ vs $0.960$), while MCC remained marginal (≈$0.29$ vs $0.11$). This work demonstrates the feasibility and privacy-preserving value of LLM-driven data synthesis for insider threat analysis and provides a foundation for more robust, ethically informed detection pipelines.

Abstract

Insider threats are a growing organizational problem due to the complexity of identifying their technical and behavioral elements. A large research body is dedicated to the study of insider threats from technological, psychological, and educational perspectives. However, research in this domain has been generally dependent on datasets that are static and limited access which restricts the development of adaptive detection models. This study introduces a novel, ethically grounded approach that uses the large language model (LLM) Claude Sonnet 3.7 to dynamically synthesize syslog messages, some of which contain indicators of insider threat scenarios. The messages reflect real-world data distributions by being highly imbalanced (1% insider threats). The syslogs were analyzed for insider threats by both Sonnet 3.7 and GPT-4o, with their performance evaluated through statistical metrics including accuracy, precision, recall, F1, specificity, FAR, MCC, and ROC AUC. Sonnet 3.7 consistently outperformed GPT-4o across nearly all metrics, particularly in reducing false alarms and improving detection accuracy. The results show strong promise for the use of LLMs in synthetic dataset generation and insider threat detection.

An Ethically Grounded LLM-Based Approach to Insider Threat Synthesis and Detection

TL;DR

Insider threats in corporate logs are difficult to study due to privacy constraints on real data. The authors propose an ethically grounded three-phase framework that uses LLMs (Claude Sonnet 3.7 and GPT-4o) to dynamically synthesize SIEM-like syslog messages with an imbalanced insider-threat rate of , and to evaluate detection performance. Across multiple runs, Sonnet 3.7 achieved higher accuracy and much lower false-alarm rates than GPT-4o (average accuracy about vs , ROC AUC about vs ), while MCC remained marginal (≈ vs ). This work demonstrates the feasibility and privacy-preserving value of LLM-driven data synthesis for insider threat analysis and provides a foundation for more robust, ethically informed detection pipelines.

Abstract

Insider threats are a growing organizational problem due to the complexity of identifying their technical and behavioral elements. A large research body is dedicated to the study of insider threats from technological, psychological, and educational perspectives. However, research in this domain has been generally dependent on datasets that are static and limited access which restricts the development of adaptive detection models. This study introduces a novel, ethically grounded approach that uses the large language model (LLM) Claude Sonnet 3.7 to dynamically synthesize syslog messages, some of which contain indicators of insider threat scenarios. The messages reflect real-world data distributions by being highly imbalanced (1% insider threats). The syslogs were analyzed for insider threats by both Sonnet 3.7 and GPT-4o, with their performance evaluated through statistical metrics including accuracy, precision, recall, F1, specificity, FAR, MCC, and ROC AUC. Sonnet 3.7 consistently outperformed GPT-4o across nearly all metrics, particularly in reducing false alarms and improving detection accuracy. The results show strong promise for the use of LLMs in synthetic dataset generation and insider threat detection.

Paper Structure

This paper contains 20 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Synthetic Syslog Generation
  4. LLM Insider Threat Syslog Analysis
  5. Statistical Evaluation
  6. Experimental Configuration
  7. SysGen
  8. SysAnalyzer
  9. CGStatsGen
  10. Results
  11. Accuracy, Precision, and False Alarm Rate
  12. Specificity, False Alarm Rate, Recall, and F1
  13. Matthews Correlational Coefficient
  14. ROC AUC
  15. Discussion
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Insider Threat Syslog Research Design
  • Figure 2: Insider Threat Syslog Research Design, Phase 1
  • Figure 3: Insider Threat Syslog Research Design, Phase 2
  • Figure 4: Insider Threat Syslog Research Design, Phase 3
  • Figure 5: ROC Curve Comparison