Benchmarking the Benchmark -- Analysis of Synthetic NIDS Datasets
Siamak Layeghy, Marcus Gallagher, Marius Portmann
TL;DR
This paper interrogates whether synthetic NIDS datasets, widely used to benchmark ML-based intrusion detection systems, reflect real-world benign traffic. It analyzes nine traffic features across three synthetic datasets (UNSW-NB15, CIC-IDS2017, TON-IoT) and two real-world NetFlow datasets (ISP, UQ), employing boxplots, CDFs, Wasserstein distances, and four dimensionality-reduction embeddings to quantify distributional differences. The study finds that real-world datasets are similar to each other, synthetic datasets are similar within their group, but substantial distributional gaps exist between synthetic and real-world traffic, with CIC-IDS being the closest to real-world data while TON-IoT and UNSW-NB15 are more distant. These results challenge the assumption that ML models trained on synthetic benchmarks will generalize to real networks and motivate the development of more realistic benchmarking datasets that match real-world traffic distributions. The authors propose a methodology and metrics to assess realism and outline future work on constructing NIDS benchmarks that better align with large-scale real traffic.
Abstract
Network Intrusion Detection Systems (NIDSs) are an increasingly important tool for the prevention and mitigation of cyber attacks. A number of labelled synthetic datasets generated have been generated and made publicly available by researchers, and they have become the benchmarks via which new ML-based NIDS classifiers are being evaluated. Recently published results show excellent classification performance with these datasets, increasingly approaching 100 percent performance across key evaluation metrics such as accuracy, F1 score, etc. Unfortunately, we have not yet seen these excellent academic research results translated into practical NIDS systems with such near-perfect performance. This motivated our research presented in this paper, where we analyse the statistical properties of the benign traffic in three of the more recent and relevant NIDS datasets, (CIC, UNSW, ...). As a comparison, we consider two datasets obtained from real-world production networks, one from a university network and one from a medium size Internet Service Provider (ISP). Our results show that the two real-world datasets are quite similar among themselves in regards to most of the considered statistical features. Equally, the three synthetic datasets are also relatively similar within their group. However, and most importantly, our results show a distinct difference of most of the considered statistical features between the three synthetic datasets and the two real-world datasets. Since ML relies on the basic assumption of training and test datasets being sampled from the same distribution, this raises the question of how well the performance results of ML-classifiers trained on the considered synthetic datasets can translate and generalise to real-world networks. We believe this is an interesting and relevant question which provides motivation for further research in this space.