Cracking IoT Security: Can LLMs Outsmart Static Analysis Tools?

TL;DR

This work benchmarks multiple LLMs against a symbolic static-analysis baseline (oHIT) for detecting Rule Interaction Threats (RITs) in openHAB TAC rules, across real and mutation datasets. It reveals that while LLMs achieve meaningful semantic understanding, they struggle with deep cross-rule structural reasoning, especially under rule transformations, whereas symbolic analysis remains robust. A hybrid workflow that uses oHIT for candidate threat detection and LLMs for contextual validation achieves higher precision, reducing false positives without sacrificing structural rigor. The findings advocate for hybrid architectures combining symbolic verification with LLM-derived semantic interpretation to enable dependable IoT safety analysis in real-world smart-home settings.

Abstract

Smart home IoT platforms such as openHAB rely on Trigger Action Condition (TAC) rules to automate device behavior, but the interplay among these rules can give rise to interaction threats, unintended or unsafe behaviors emerging from implicit dependencies, conflicting triggers, or overlapping conditions. Identifying these threats requires semantic understanding and structural reasoning that traditionally depend on symbolic, constraint-driven static analysis. This work presents the first comprehensive evaluation of Large Language Models (LLMs) across a multi-category interaction threat taxonomy, assessing their performance on both the original openHAB (oHC/IoTB) dataset and a structurally challenging Mutation dataset designed to test robustness under rule transformations. We benchmark Llama 3.1 8B, Llama 70B, GPT-4o, Gemini-2.5-Pro, and DeepSeek-R1 across zero-, one-, and two-shot settings, comparing their results against oHIT's manually validated ground truth. Our findings show that while LLMs exhibit promising semantic understanding, particularly on action- and condition-related threats, their accuracy degrades significantly for threats requiring cross-rule structural reasoning, especially under mutated rule forms. Model performance varies widely across threat categories and prompt settings, with no model providing consistent reliability. In contrast, the symbolic reasoning baseline maintains stable detection across both datasets, unaffected by rule rewrites or structural perturbations. These results underscore that LLMs alone are not yet dependable for safety critical interaction-threat detection in IoT environments. We discuss the implications for tool design and highlight the potential of hybrid architectures that combine symbolic analysis with LLM-based semantic interpretation to reduce false positives while maintaining structural rigor.

Figures (5)

• Figure 1: An overview of RITs
• Figure 2: LLM-oHIT Reconciliation Approach
• Figure 3: Comparison of LLM reasoning performance and scaling behavior across RIT contradiction types and prompting configurations.
• Figure 4: Comparative total accuracy across Experiments A–D for all models, illustrating performance trends under multi- and single-response conditions within six- and three-category RIT taxonomies.
• Figure 5: Comparative total accuracy across Experiments A–D (Mutation Vs oHC/IoTB dataset). for all models, illustrating performance trends under multi- and single-response conditions within six- and three-category RIT taxonomies.