Eclectic Rule Extraction for Explainability of Deep Neural Network based Intrusion Detection Systems

TL;DR

The paper tackles trust issues in X-IDS by proposing eclectic rule extraction as a white-box surrogate within a Hybrid X-IDS architecture. It introduces an eclectic RE algorithm that builds human-readable rules from hidden layers of a DNN and demonstrates near-DNN-level predictive alignment on UNSW-NB15 and CIC-IDS-2017. It systematically analyzes speed-accuracy-explainability trade-offs by varying leaves, layers, and data subsets, showing scalable, customizable explainability. The work advances practical explainability for IDS by enabling trustworthy global rulesets and potential firewall rule translation.

Abstract

This paper addresses trust issues created from the ubiquity of black box algorithms and surrogate explainers in Explainable Intrusion Detection Systems (X-IDS). While Explainable Artificial Intelligence (XAI) aims to enhance transparency, black box surrogate explainers, such as Local Interpretable Model-Agnostic Explanation (LIME) and SHapley Additive exPlanation (SHAP), are difficult to trust. The black box nature of these surrogate explainers makes the process behind explanation generation opaque and difficult to understand. To avoid this problem, one can use transparent white box algorithms such as Rule Extraction (RE). There are three types of RE algorithms: pedagogical, decompositional, and eclectic. Pedagogical methods offer fast but untrustworthy white-box explanations, while decompositional RE provides trustworthy explanations with poor scalability. This work explores eclectic rule extraction, which strikes a balance between scalability and trustworthiness. By combining techniques from pedagogical and decompositional approaches, eclectic rule extraction leverages the advantages of both, while mitigating some of their drawbacks. The proposed Hybrid X-IDS architecture features eclectic RE as a white box surrogate explainer for black box Deep Neural Networks (DNN). The presented eclectic RE algorithm extracts human-readable rules from hidden layers, facilitating explainable and trustworthy rulesets. Evaluations on UNSW-NB15 and CIC-IDS-2017 datasets demonstrate the algorithm's ability to generate rulesets with 99.9% accuracy, mimicking DNN outputs. The contributions of this work include the hybrid X-IDS architecture, the eclectic rule extraction algorithm applicable to intrusion detection datasets, and a thorough analysis of performance and explainability, demonstrating the trade-offs involved in rule extraction speed and accuracy.

Figures (3)

• Figure 1: The composition of a rule. A rule consists of terms that are concatenated together to form a rule. Rules can then be combined in a set to form a ruleset.
• Figure 2: Architecture for a surrogate explainer X-IDS. It features four total phases. In the Pre-Modeling phase, the datasets are feature engineered to be compatible with the neural network, and model parameters can be selected. The model is trained and tested in the Modeling phase. Here we record important quality and performative metrics. The trained model and dataset can then be used to extract a ruleset. Lastly, we generate statistics for the ruleset and rules to aid the user in their understanding.
• Figure 3: These charts compare the speed up versus accuracy loss for the UNSW-NB15 and CIC-IDS-2017 rulesets. Ground truth accuracy is the rulesets label versus the testing datasets labels. Model prediction accuracy is the rulesets labels versus the models' predicted outputs. Figures (a)(b)(d) are split into two charts with different scales. Points on each of the figures are labeled with their experiment name.