Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Far Should We Need to Go : Evaluate Provenance-based Intrusion Detection Systems in Industrial Scenarios

Yue Xiao, Ling Jiang, Sen Nie, Ding Li, Shi Wu, Ke Xu, Qi Li

Abstract

Provenance-based Intrusion Detection Systems (PIDSes) have been widely used to detect Advanced Persistent Threats (APTs). Although many studies achieve high performance in the evaluations of their original papers, their performance in industrial scenarios remains unclear. To fill this gap, we conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios. We first analyze the differences between the data from DARPA datasets and that collected in industrial scenarios, identifying three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity. We then build several datasets to evaluate five state-of-the-art PIDSes. The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities. Based on the evaluation results and our industrial practices, we provide several insights to solve or explain the above problems. For example, we propose a method to mitigate the high false positives, which reduces manual effort by 2/3. Finally, we propose several research suggestions to improve PIDSes.

How Far Should We Need to Go : Evaluate Provenance-based Intrusion Detection Systems in Industrial Scenarios

Abstract

Provenance-based Intrusion Detection Systems (PIDSes) have been widely used to detect Advanced Persistent Threats (APTs). Although many studies achieve high performance in the evaluations of their original papers, their performance in industrial scenarios remains unclear. To fill this gap, we conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios. We first analyze the differences between the data from DARPA datasets and that collected in industrial scenarios, identifying three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity. We then build several datasets to evaluate five state-of-the-art PIDSes. The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities. Based on the evaluation results and our industrial practices, we provide several insights to solve or explain the above problems. For example, we propose a method to mitigate the high false positives, which reduces manual effort by 2/3. Finally, we propose several research suggestions to improve PIDSes.
Paper Structure (22 sections, 5 equations, 11 figures, 9 tables)

This paper contains 22 sections, 5 equations, 11 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Introduction to PIDS
  4. New Characteristics in Industry
  5. Measurement Setup
  6. Research Questions
  7. Measurement Settings
  8. Measurement Results
  9. The portability of PIDSes across multiple source inputs (RQ1)
  10. The performance of PIDSes to detect real-world attacks (RQ2)
  11. The false positive rate of PIDSes on a host with ever-changing activities (RQ3)
  12. Time overhead of PIDSes (RQ4)
  13. Insights, Suggestions, and Future Directions
  14. How can we improve the deteciton performance of PIDSes?
  15. How can we reduce false positives?
  16. ...and 7 more sections

Figures (11)

  • Figure 1: The pipeline of attack detection with a PIDS.
  • Figure 2: The distribution of unknown behaviors for each process in DARPA-E3 and a real-world host.
  • Figure 3: The attack in the sandboxs on different platforms.
  • Figure 4: An attack utilises known vulnerabilities in postgreSQL to launch mining.
  • Figure 5: An attack invades the victim before any PIDS is deployed and tries to steal the sensitive information.
  • ...and 6 more figures