Evaluating MCC for Low-Frequency Cyberattack Detection in Imbalanced Intrusion Detection Data

TL;DR

This study addresses the unreliability of accuracy for evaluating intrusion detection under extreme class imbalance by comparing accuracy with Matthews Correlation Coefficient (MCC) on low-frequency attacks drawn from CICIDS-derived subsets. It contrasts baseline and meta-classifiers within WEKA across three attack types, revealing that MCC exposes weaknesses accuracy hides and that ensemble meta-classifiers generally achieve higher MCC for minority-class detection. The results argue for imbalance-aware evaluation in IDS research and demonstrate MCC's practical value for model ranking and deployment decisions. The work also outlines limitations and avenues for extending the analysis to larger, more diverse datasets and modern architectures.

Abstract

In many real-world network environments, several types of cyberattacks occur at very low rates compared to benign traffic, making them difficult for intrusion detection systems (IDS) to detect reliably. This imbalance causes traditional evaluation metrics, such as accuracy, to often overstate model performance in these conditions, masking failures on minority attack classes that are most important in practice. In this paper, we evaluate a set of base and meta classifiers on low-traffic attacks in the CSE-CIC-IDS2018 dataset and compare their reliability in terms of accuracy and Matthews Correlation Coefficient (MCC). The results show that accuracy consistently inflates performance, while MCC provides a more accurate assessment of a classifier's performance across both majority and minority classes. Meta-classification methods, such as LogitBoost and AdaBoost, demonstrate more effective minority class detection when measured by MCC, revealing trends that accuracy fails to capture. These findings establish the need for imbalance-aware evaluation and make MCC a more trustworthy metric for IDS research involving low-traffic cyberattacks.