CND-IDS: Continual Novelty Detection for Intrusion Detection Systems
Sean Fuhrman, Onat Gungor, Tajana Rosing
TL;DR
This work tackles the problem of intrusion detection in environments with continually evolving cyber attacks and scarce labeling by introducing CND-IDS, a continual novelty detection framework. It combines a continual feature extractor (CFE) with a PCA-based novelty detector, trained over unlabeled data streams using a joint loss $L_{CND}=L_{CS}+\lambda_R L_R+\lambda_{CL} L_{CL}$, and uses PCA reconstruction error as the anomaly score. By training on evolving experiences and exploiting clean normal data $N_c$, CND-IDS detects zero-day and unseen attacks without attack labels, achieving up to $6.1\times$ F-score improvements and $6.5\times$ forward transfer over state-of-the-art unsupervised continual learning baselines, while maintaining low inference overhead. The approach demonstrates strong performance across realistic IIoT and network intrusion datasets, indicating significant practical impact for adaptive, label-efficient IDS in dynamic networks. Specifically, the method leverages $L_{CS}$ with pseudo-labels via K-Means and a triplet margin, reconstruction loss $L_R$, and latent regularization $L_{CL}$, along with PCA-based novelty scoring $FRE=\lVert h-T^{-1}(l)\rVert^2$, to robustly separate normal from anomalous data in a changing data stream.
Abstract
Intrusion detection systems (IDS) play a crucial role in IoT and network security by monitoring system data and alerting to suspicious activities. Machine learning (ML) has emerged as a promising solution for IDS, offering highly accurate intrusion detection. However, ML-IDS solutions often overlook two critical aspects needed to build reliable systems: continually changing data streams and a lack of attack labels. Streaming network traffic and associated cyber attacks are continually changing, which can degrade the performance of deployed ML models. Labeling attack data, such as zero-day attacks, in real-world intrusion scenarios may not be feasible, making the use of ML solutions that do not rely on attack labels necessary. To address both these challenges, we propose CND-IDS, a continual novelty detection IDS framework which consists of (i) a learning-based feature extractor that continuously updates new feature representations of the system data, and (ii) a novelty detector that identifies new cyber attacks by leveraging principal component analysis (PCA) reconstruction. Our results on realistic intrusion datasets show that CND-IDS achieves up to 6.1x F-score improvement, and up to 6.5x improved forward transfer over the SOTA unsupervised continual learning algorithm. Our code will be released upon acceptance.