Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IoT Device Identification with Machine Learning: Common Pitfalls and Best Practices

Kahraman Kostas, Rabia Yasa Kostas

TL;DR

The paper tackles the challenge of IoT device identification using machine learning by dissecting the pipeline into method selection, data handling, feature extraction, and evaluation to reveal systemic pitfalls. It analyzes scope definitions, labeling integrity, data leakage risks, and the impact of feature choices on overfitting and generalizability. It advocates practical guidelines, such as aligning identification granularity with suitable features, avoiding proxy labels, enforcing clean train-test splits before augmentation, sanitizing inputs, and preferring modular One-vs-Rest architectures with diverse algorithm benchmarking. The result is a set of actionable recommendations to improve reproducibility, scalability, and robustness of IoT security models in real-world heterogeneous networks.

Abstract

This paper critically examines the device identification process using machine learning, addressing common pitfalls in existing literature. We analyze the trade-offs between identification methods (unique vs. class based), data heterogeneity, feature extraction challenges, and evaluation metrics. By highlighting specific errors, such as improper data augmentation and misleading session identifiers, we provide a robust guideline for researchers to enhance the reproducibility and generalizability of IoT security models.

IoT Device Identification with Machine Learning: Common Pitfalls and Best Practices

TL;DR

The paper tackles the challenge of IoT device identification using machine learning by dissecting the pipeline into method selection, data handling, feature extraction, and evaluation to reveal systemic pitfalls. It analyzes scope definitions, labeling integrity, data leakage risks, and the impact of feature choices on overfitting and generalizability. It advocates practical guidelines, such as aligning identification granularity with suitable features, avoiding proxy labels, enforcing clean train-test splits before augmentation, sanitizing inputs, and preferring modular One-vs-Rest architectures with diverse algorithm benchmarking. The result is a set of actionable recommendations to improve reproducibility, scalability, and robustness of IoT security models in real-world heterogeneous networks.

Abstract

This paper critically examines the device identification process using machine learning, addressing common pitfalls in existing literature. We analyze the trade-offs between identification methods (unique vs. class based), data heterogeneity, feature extraction challenges, and evaluation metrics. By highlighting specific errors, such as improper data augmentation and misleading session identifiers, we provide a robust guideline for researchers to enhance the reproducibility and generalizability of IoT security models.
Paper Structure (20 sections, 5 figures)

This paper contains 20 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Familiarizing with Methods
  3. Defining the Scope
  4. Identification Approaches and Trade-offs
  5. Familiarizing with Data
  6. Data Acquisition and Privacy
  7. Labeling Integrity and Non-IP Devices
  8. Handling Imbalance and Data Leakage
  9. Familiarizing with Feature Extraction
  10. Efficiency in Preliminary Analysis
  11. Eliminating Identifying Features (Overfitting Risks)
  12. Sanitizing Raw Data Inputs
  13. Familiarizing with Machine Learning
  14. Scalability and Model Architecture
  15. Algorithm Selection and the "No Free Lunch" Theorem
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Steps of device identification process.
  • Figure 2: Labelling the Aalto dataset according to 3 identification approacheskostas2024behaviour.
  • Figure 3: The number of packets produced by the devices in the Aalto dataset, illustrating severe class imbalance.
  • Figure 4: The fields contained in a network packet and their byte equivalents.
  • Figure 5: Interpretability levels of common machine learning methods.