Deployment Challenges of Industrial Intrusion Detection Systems

TL;DR

The paper tackles the gap between promising IIDS research results and real-world ICS deployments by examining data requirements and hyperparameter tunability. It demonstrates that supervised IIDS demand enormous amounts of attack data for high performance, making them impractical in live facilities, while OCC-based IIDS depend on carefully tuned hyperparameters, with transferability across deployments proving limited. Through extensive experiments across four IIDS and three ICS datasets, the authors quantify data needs and sensitivity to hyperparameters, revealing that deployment readiness cannot be assumed from abstract detection metrics alone. They propose enhanced evaluation protocols and closer collaboration with ICS stakeholders to steer future IIDS toward deployable, real-world cybersecurity solutions for industrial control systems.

Abstract

With the escalating threats posed by cyberattacks on Industrial Control Systems (ICSs), the development of customized Industrial Intrusion Detection Systems (IIDSs) received significant attention in research. While existing literature proposes effective IIDS solutions evaluated in controlled environments, their deployment in real-world industrial settings poses several challenges. This paper highlights two critical yet often overlooked aspects that significantly impact their practical deployment, i.e., the need for sufficient amounts of data to train the IIDS models and the challenges associated with finding suitable hyperparameters, especially for IIDSs training only on genuine ICS data. Through empirical experiments conducted on multiple state-of-the-art IIDSs and diverse datasets, we establish the criticality of these issues in deploying IIDSs. Our findings show the necessity of extensive malicious training data for supervised IIDSs, which can be impractical considering the complexity of recording and labeling attacks in actual industrial environments. Furthermore, while other IIDSs circumvent the previous issue by requiring only benign training data, these can suffer from the difficulty of setting appropriate hyperparameters, which likewise can diminish their performance. By shedding light on these challenges, we aim to enhance the understanding of the limitations and considerations necessary for deploying effective cybersecurity solutions in ICSs, which might be one reason why IIDSs see few deployments.

Figures (6)

• Figure 1: An IIDS learns the repetitive patterns of an ICS's behavior to indicate anomalies. This requires finding a suitable hyperparameter, such as the threshold for MinMax visualized here Wolsing2022Can, which influences the alert decision of an IIDS.
• Figure 2: Gradually increasing the amount of attack samples within the training data reveals that both RF and SVM require lots of data to yield satisfying detection performance. For a simple attack, cf. Fig. \ref{['fig:training:easy']}, the RF requires only about three samples.
• Figure 3: Reducing the amount of benign training data likewise diminishes the detection rate but not all IIDS experience an equal performance reduction. The training data Seq2SeqNN on SWaT and WADI is reduced in steps of $10.000$ in contrast to $1.000$ for the others whereas all IIDS on BATADAL are sampled in steps of $100$.
• Figure 4: An IIDS' performance depends on an optimal choice of hyperparameter. While MinMax or Invariant yield satisfying results in F1 score on SWaT for the majority of configurations, obtaining a good configuration for TABOR is challenging. Thus, judging the expectable performance of an IIDS by a single number can be misleading.
• Figure 5: The impact of hyperparameters can vary significantly between approaches. While on the SWaT for MinMax (upper plot), one parameter is decisive for the entire performance, suitable configurations for TABOR (lower plot) are more challenging to obtain as several parameters influence each other.