Systematic Evaluation of Synthetic Data Augmentation for Multi-class NetFlow Traffic

TL;DR

This study tackles imbalanced, multi-class NetFlow attack classification in NIDS by systematically evaluating both classical resampling techniques and modern generative augmentation across multiple classifiers and three benchmark datasets. Using a standardized pipeline with six classifiers and 42 resampling combinations, the authors find that resampling rarely yields robust improvements, with XGBoost often performing best on baseline data and only a few classifier-dataset pairs showing modest gains. The per-class analysis reveals that improvements are inconsistent and frequently come at the expense of other classes, underscoring the limited reliability of resampling as a general solution for class imbalance in NIDS. Overall, the work emphasizes prioritizing classifier choice and hyperparameter tuning over arbitrary resampling, and it highlights the need for broader datasets and thorough data-quality evaluation of augmented data for future progress.

Abstract

The detection of cyber-attacks in computer networks is a crucial and ongoing research challenge. Machine learning-based attack classification offers a promising solution, as these models can be continuously updated with new data, enhancing the effectiveness of network intrusion detection systems (NIDS). Unlike binary classification models that simply indicate the presence of an attack, multi-class models can identify specific types of attacks, allowing for more targeted and effective incident responses. However, a significant drawback of these classification models is their sensitivity to imbalanced training data. Recent advances suggest that generative models can assist in data augmentation, claiming to offer superior solutions for imbalanced datasets. Classical balancing methods, although less novel, also provide potential remedies for this issue. Despite these claims, a comprehensive comparison of these methods within the NIDS domain is lacking. Most existing studies focus narrowly on individual methods, making it difficult to compare results due to varying experimental setups. To close this gap, we designed a systematic framework to compare classical and generative resampling methods for class balancing across multiple popular classification models in the NIDS domain, evaluated on several NIDS benchmark datasets. Our experiments indicate that resampling methods for balancing training data do not reliably improve classification performance. Although some instances show performance improvements, the majority of results indicate decreased performance, with no consistent trend in favor of a specific resampling technique enhancing a particular classifier.

Figures (3)

• Figure 1: The main components of the experimental setup for the comparison of resampling methods
• Figure 2: Delta MCC values per model: MCC of original (None) resampling - scores of resampling strategy on the y-axis.
• Figure 3: Metric scores plotted as mean values of all models. (numeric class codes explained in Table \ref{['tab:class_codes']}