Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PIDSMaker: Building and Evaluating Provenance-based Intrusion Detection Systems

Tristan Bilot, Baoxiang Jiang, Thomas Pasquier

TL;DR

PIDSMaker tackles reproducibility and engineering overhead in provenance-based intrusion detection by introducing an open-source, modular framework that unifies eight state-of-the-art PIDSs with standardized preprocessing and ground-truth labels. It provides a YAML-driven configuration system, seven-stage pipeline design, on-disk caching, and utilities for ablations, hyperparameter tuning, and instability measurement, accompanied by preprocessed datasets for apples-to-apples evaluation. The framework supports rapid prototyping, fair comparisons, and shared evaluation through integrated visualization and W&B tracking, addressing core methodological gaps identified in prior work. By reducing re-implementation effort and promoting standardized benchmarks, PIDSMaker aims to accelerate progress in provenance-based IDS research and community adoption.

Abstract

Recent provenance-based intrusion detection systems (PIDSs) have demonstrated strong potential for detecting advanced persistent threats (APTs) by applying machine learning to system provenance graphs. However, evaluating and comparing PIDSs remains difficult: prior work uses inconsistent preprocessing pipelines, non-standard dataset splits, and incompatible ground-truth labeling and metrics. These discrepancies undermine reproducibility, impede fair comparison, and impose substantial re-implementation overhead on researchers. We present PIDSMaker, an open-source framework for developing and evaluating PIDSs under consistent protocols. PIDSMaker consolidates eight state-of-the-art systems into a modular, extensible architecture with standardized preprocessing and ground-truth labels, enabling consistent experiments and apples-to-apples comparisons. A YAML-based configuration interface supports rapid prototyping by composing components across systems without code changes. PIDSMaker also includes utilities for ablation studies, hyperparameter tuning, multi-run instability measurement, and visualization, addressing methodological gaps identified in prior work. We demonstrate PIDSMaker through concrete use cases and release it with preprocessed datasets and labels to support shared evaluation for the PIDS community.

PIDSMaker: Building and Evaluating Provenance-based Intrusion Detection Systems

TL;DR

PIDSMaker tackles reproducibility and engineering overhead in provenance-based intrusion detection by introducing an open-source, modular framework that unifies eight state-of-the-art PIDSs with standardized preprocessing and ground-truth labels. It provides a YAML-driven configuration system, seven-stage pipeline design, on-disk caching, and utilities for ablations, hyperparameter tuning, and instability measurement, accompanied by preprocessed datasets for apples-to-apples evaluation. The framework supports rapid prototyping, fair comparisons, and shared evaluation through integrated visualization and W&B tracking, addressing core methodological gaps identified in prior work. By reducing re-implementation effort and promoting standardized benchmarks, PIDSMaker aims to accelerate progress in provenance-based IDS research and community adoption.

Abstract

Recent provenance-based intrusion detection systems (PIDSs) have demonstrated strong potential for detecting advanced persistent threats (APTs) by applying machine learning to system provenance graphs. However, evaluating and comparing PIDSs remains difficult: prior work uses inconsistent preprocessing pipelines, non-standard dataset splits, and incompatible ground-truth labeling and metrics. These discrepancies undermine reproducibility, impede fair comparison, and impose substantial re-implementation overhead on researchers. We present PIDSMaker, an open-source framework for developing and evaluating PIDSs under consistent protocols. PIDSMaker consolidates eight state-of-the-art systems into a modular, extensible architecture with standardized preprocessing and ground-truth labels, enabling consistent experiments and apples-to-apples comparisons. A YAML-based configuration interface supports rapid prototyping by composing components across systems without code changes. PIDSMaker also includes utilities for ablation studies, hyperparameter tuning, multi-run instability measurement, and visualization, addressing methodological gaps identified in prior work. We demonstrate PIDSMaker through concrete use cases and release it with preprocessed datasets and labels to support shared evaluation for the PIDS community.
Paper Structure (24 sections, 1 equation, 5 figures)

This paper contains 24 sections, 1 equation, 5 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Provenance Graphs
  4. Common PIDS Architecture
  5. Framework Design
  6. Design Principles
  7. Pipeline Stages
  8. On-Disk Caching
  9. Datasets and Ground Truth
  10. Supported Systems
  11. Configuration Files
  12. Minimum System Requirements
  13. Use Cases
  14. Running Existing Systems
  15. Designing Model Variants
  16. ...and 9 more sections

Figures (5)

  • Figure 1: PIDSMaker architecture with seven pipeline stages connected by data flow. Each stage contains configurable components.
  • Figure 2: Component modularity in PIDSMaker. Rows represent the eight supported systems; columns represent a subset of components. The black arrow demonstrates creating a custom system variant by selecting components from different systems. This composition can be specified through YAML or CLI.
  • Figure 3: Batching strategies in PIDSMaker. Global: flatten all graphs and repartition into fixed-size chunks. Intra-graph: partition within each graph. Inter-graph: group multiple graphs into a mini-batch.
  • Figure 4: Distribution of (normalized) predicted anomaly scores for each attack in a dataset (E3-CADETS dataset).
  • Figure 5: Anomaly scores predicted for top-ranked nodes (E3-CADETS dataset).