Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models for Real-World IoT Device Identification

Rameen Mahmood, Tousif Ahmed, Sai Teja Peddinti, Danny Yuxing Huang

TL;DR

This work reframes IoT device identification as a language-model inference problem over heterogeneous network metadata, leveraging an ensemble of LLMs to generate high-fidelity vendor pseudolabels for the IoT Inspector dataset. It then instruction-tunes a quantized LLaMA 3.1 8B model in a two-phase curriculum to achieve deployment-grade generalization, demonstrating 98.25% top-1 accuracy across 2,015 vendors and strong resilience to missing fields, drift, and adversarial manipulation. External evaluation on the Mon(IoT)r testbed confirms robust cross-domain generalization, while feature attribution and qualitative analyses highlight interpretability and semi-structured reasoning as core strengths. The approach offers a scalable, interpretable foundation for real-world device identification at scale, with potential to improve inventory, privacy, and security in open-world IoT deployments.

Abstract

The rapid expansion of IoT devices has outpaced current identification methods, creating significant risks for security, privacy, and network accountability. These challenges are heightened in open-world environments, where traffic metadata is often incomplete, noisy, or intentionally obfuscated. We introduce a semantic inference pipeline that reframes device identification as a language modeling task over heterogeneous network metadata. To construct reliable supervision, we generate high-fidelity vendor labels for the IoT Inspector dataset, the largest real-world IoT traffic corpus, using an ensemble of large language models guided by mutual-information and entropy-based stability scores. We then instruction-tune a quantized LLaMA3.18B model with curriculum learning to support generalization under sparsity and long-tail vendor distributions. Our model achieves 98.25% top-1 accuracy and 90.73% macro accuracy across 2,015 vendors while maintaining resilience to missing fields, protocol drift, and adversarial manipulation. Evaluation on an independent IoT testbed, coupled with explanation quality and adversarial stress tests, demonstrates that instruction-tuned LLMs provide a scalable and interpretable foundation for real-world device identification at scale.

Large Language Models for Real-World IoT Device Identification

TL;DR

This work reframes IoT device identification as a language-model inference problem over heterogeneous network metadata, leveraging an ensemble of LLMs to generate high-fidelity vendor pseudolabels for the IoT Inspector dataset. It then instruction-tunes a quantized LLaMA 3.1 8B model in a two-phase curriculum to achieve deployment-grade generalization, demonstrating 98.25% top-1 accuracy across 2,015 vendors and strong resilience to missing fields, drift, and adversarial manipulation. External evaluation on the Mon(IoT)r testbed confirms robust cross-domain generalization, while feature attribution and qualitative analyses highlight interpretability and semi-structured reasoning as core strengths. The approach offers a scalable, interpretable foundation for real-world device identification at scale, with potential to improve inventory, privacy, and security in open-world IoT deployments.

Abstract

The rapid expansion of IoT devices has outpaced current identification methods, creating significant risks for security, privacy, and network accountability. These challenges are heightened in open-world environments, where traffic metadata is often incomplete, noisy, or intentionally obfuscated. We introduce a semantic inference pipeline that reframes device identification as a language modeling task over heterogeneous network metadata. To construct reliable supervision, we generate high-fidelity vendor labels for the IoT Inspector dataset, the largest real-world IoT traffic corpus, using an ensemble of large language models guided by mutual-information and entropy-based stability scores. We then instruction-tune a quantized LLaMA3.18B model with curriculum learning to support generalization under sparsity and long-tail vendor distributions. Our model achieves 98.25% top-1 accuracy and 90.73% macro accuracy across 2,015 vendors while maintaining resilience to missing fields, protocol drift, and adversarial manipulation. Evaluation on an independent IoT testbed, coupled with explanation quality and adversarial stress tests, demonstrates that instruction-tuned LLMs provide a scalable and interpretable foundation for real-world device identification at scale.
Paper Structure (42 sections, 4 equations, 6 figures, 7 tables)

This paper contains 42 sections, 4 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Research Objectives
  3. Threat Model & Motivations
  4. Related Work
  5. Challenges of IoT Device Identification
  6. Machine Learning for Fingerprinting
  7. Dataset Preprocessing
  8. Overview of Dataset
  9. Preprocessing Pipeline
  10. Method
  11. Stage 1: Labeling via Prompted LLMs
  12. Prompt Design and Output Structure
  13. Proxy CMI: Feature Ranking and Voting
  14. Ablation Study: Prompt and Model Selection
  15. Stage 2: Supervised instruction-tuning for Vendor Classification
  16. ...and 27 more sections

Figures (6)

  • Figure 1: Multi‑Stage Pipeline for Device‑Level Signature Extraction.
  • Figure 2: Curriculum-style instruction-tuning strategy.
  • Figure A.1: Proxy CMI scores for each input feature across three LLMs.
  • Figure D.1: LLM vendor predictions on dense, sparse, and semi-structured inputs. These examples illustrate the range of inference challenges: noisy data, sparse clues, and user-labeled metadata.
  • Figure D.2: LLM vendor predictions under adversarial prompt manipulations. Each scenario simulates a real-world actor attempting to spoof the user label to mislead the model.
  • ...and 1 more figures