Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Machine Learning Models for IoT Malware Detection under Data Scarcity and Drift

Jake Lyon, Ehsan Saeedizade, Shamik Sengupta

TL;DR

The paper tackles IoT malware detection under data scarcity and evolving threat landscapes. It systematically compares four supervised models—Logistic Regression, Random Forest, LightGBM, and MLP—on the IoT-23 dataset, evaluating binary and multiclass tasks, data-size sensitivity, and temporal drift with a rolling-window approach. Key findings show tree-based models, especially Random Forest, deliver strong performance with limited data and exhibit greater temporal robustness, while performance degrades over time as malware diversity increases, underscoring the need for periodic retraining and adaptive strategies. The work provides deployment-relevant guidance for lightweight, scalable IoT security solutions and motivates future development of adaptive learning techniques to sustain effectiveness in dynamic environments.

Abstract

The rapid expansion of the Internet of Things (IoT) in domains such as smart cities, transportation, and industrial systems has heightened the urgency of addressing their security vulnerabilities. IoT devices often operate under limited computational resources, lack robust physical safeguards, and are deployed in heterogeneous and dynamic networks, making them prime targets for cyberattacks and malware applications. Machine learning (ML) offers a promising approach to automated malware detection and classification, but practical deployment requires models that are both effective and lightweight. The goal of this study is to investigate the effectiveness of four supervised learning models (Random Forest, LightGBM, Logistic Regression, and a Multi-Layer Perceptron) for malware detection and classification using the IoT-23 dataset. We evaluate model performance in both binary and multiclass classification tasks, assess sensitivity to training data volume, and analyze temporal robustness to simulate deployment in evolving threat landscapes. Our results show that tree-based models achieve high accuracy and generalization, even with limited training data, while performance deteriorates over time as malware diversity increases. These findings underscore the importance of adaptive, resource-efficient ML models for securing IoT systems in real-world environments.

Benchmarking Machine Learning Models for IoT Malware Detection under Data Scarcity and Drift

TL;DR

The paper tackles IoT malware detection under data scarcity and evolving threat landscapes. It systematically compares four supervised models—Logistic Regression, Random Forest, LightGBM, and MLP—on the IoT-23 dataset, evaluating binary and multiclass tasks, data-size sensitivity, and temporal drift with a rolling-window approach. Key findings show tree-based models, especially Random Forest, deliver strong performance with limited data and exhibit greater temporal robustness, while performance degrades over time as malware diversity increases, underscoring the need for periodic retraining and adaptive strategies. The work provides deployment-relevant guidance for lightweight, scalable IoT security solutions and motivates future development of adaptive learning techniques to sustain effectiveness in dynamic environments.

Abstract

The rapid expansion of the Internet of Things (IoT) in domains such as smart cities, transportation, and industrial systems has heightened the urgency of addressing their security vulnerabilities. IoT devices often operate under limited computational resources, lack robust physical safeguards, and are deployed in heterogeneous and dynamic networks, making them prime targets for cyberattacks and malware applications. Machine learning (ML) offers a promising approach to automated malware detection and classification, but practical deployment requires models that are both effective and lightweight. The goal of this study is to investigate the effectiveness of four supervised learning models (Random Forest, LightGBM, Logistic Regression, and a Multi-Layer Perceptron) for malware detection and classification using the IoT-23 dataset. We evaluate model performance in both binary and multiclass classification tasks, assess sensitivity to training data volume, and analyze temporal robustness to simulate deployment in evolving threat landscapes. Our results show that tree-based models achieve high accuracy and generalization, even with limited training data, while performance deteriorates over time as malware diversity increases. These findings underscore the importance of adaptive, resource-efficient ML models for securing IoT systems in real-world environments.
Paper Structure (16 sections, 3 figures, 7 tables)

This paper contains 16 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Data Collection
  5. Data Preprocessing
  6. Model Training and Evaluation
  7. Evaluation and Analysis
  8. Experimental Setup
  9. Dataset
  10. Feature Selection
  11. ML models
  12. Performance Metrics
  13. Performance Evaluation
  14. Training Size Analysis
  15. Temporal Analysis
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Methodology workflow
  • Figure 2: Label distribution of the sampled dataset
  • Figure 3: Temporal evaluation of F1 scores using a rolling-window setup, where models are trained on earlier months and tested on later ones, for (a) binary classification and (b) multiclass classification tasks across test months.