Individual Packet Features are a Risk to Model Generalisation in ML-Based Intrusion Detection

TL;DR

This work critiques the reliance on Individual Packet Features (IPF) for ML-based IoT intrusion detection by showing that IPF can produce misleadingly high accuracy due to information leakage and low data complexity. Through literature review and experiments on the IoT-NID dataset, it demonstrates that session-based identifiers and simple features can inflate performance within cross-validation but fail to generalize across datasets or real deployments. The authors advocate for incorporating packet interactions and contextual, flow-based/window-based features to improve robustness and generalization in IoT security. The study provides empirical evidence and practical guidance for designing more reliable IDS that withstand dataset shifts in diverse IoT environments.

Abstract

Machine learning is increasingly used for intrusion detection in IoT networks. This paper explores the effectiveness of using individual packet features (IPF), which are attributes extracted from a single network packet, such as timing, size, and source-destination information. Through literature review and experiments, we identify the limitations of IPF, showing they can produce misleadingly high detection rates. Our findings emphasize the need for approaches that consider packet interactions for robust intrusion detection. Additionally, we demonstrate that models based on IPF often fail to generalize across datasets, compromising their reliability in diverse IoT environments.

Figures (8)

• Figure 1: Legitimate and malicious use of 3-way handshake
• Figure 2: An 80/20 split for a hypothetical dataset containing 40 samples (4 session, 40 network packets), showcasing the division into 80% training data and 20% testing data for model development and evaluation, respectively. This example illustrates how information leakage takes place. In the scenario involving the source IP attribute, information leakage occurs because this address uniquely identifies both the attacker and the benign source, remaining constant across both the training and test sets. If we examine other features such as source port numbers and ID numbers, they vary across different sessions but exhibit correlations within each session (port numbers remain constant, ID numbers increment sequentially). Consequently, if packets from the same session are found in both the training and test sets, it can result in information leakage. Benign: Malicious:
• Figure 3: Comparison of CV and isolated data performance of features on some attacks in the IoT-NID dataset with DT.
• Figure 4: Visualization of high-achieving decision tree models. (a) HTTP Flood model using dport feature, (b) Brute-Force model using dport feature.
• Figure 5: The distribution of packet size malicious data (HTTP Flood) in four different datasets koroniotis2019towardsCICferrag2022edgeIoTNID.