LLMind 2.0: Distributed IoT Automation with Natural Language M2M Communication and Lightweight LLM Agents

TL;DR

The paper addresses IoT automation scalability in heterogeneous deployments by moving executable code generation from a centralized LLM coordinator to on-device, lightweight LLM-powered agents that communicate with humans and devices via natural language. It proposes LLMind 2.0, a distributed framework that uses a natural-language M2M interface, a RAG-based API mapping module, and a fault-tolerant coordination protocol to enable parallel, reliable task execution across devices. Key contributions include the three-step on-device code generation pipeline, a JSON-based device API description standard, and empirical validation in warehouse robotics and WiFi networking contexts, demonstrating improvements in scalability, latency, and task success rates, along with profiling of error sources. The work highlights privacy benefits from localized code generation and provides open-source resources and datasets to foster further research on language-driven, distributed IoT automation. The results indicate that natural language can effectively mediate both human-to-machine and machine-to-machine interactions in large-scale IoT ecosystems, enabling better adaptability and collaboration among diverse devices.

Abstract

Recent advances in large language models (LLMs) have generated great interest in their applications for IoT automation and device management. However, centralized approaches struggle to scale across heterogeneous, large-scale systems. We present LLMind 2.0, a distributed framework that embeds lightweight LLM-empowered device agents and adopts natural language for machine-to-machine (M2M) communication. In LLMind 2.0, a central coordinator translates human instructions into natural-language subtask descriptions, which instruct distributed device agents to generate device-specific code locally based on their proprietary APIs. Using natural language as a unified medium overcomes device heterogeneity and enables seamless device collaboration. LLMind 2.0 integrates: 1) a timeout-based deadlock avoidance protocol that coordinates distributed subtask executions, 2) a retrieval-augmented generation (RAG) mechanism for precise subtask-to-API mapping, and 3) fine-tuned lightweight LLMs for reliable, device-specific code generation. Experiments in multi-robot warehouse operations and Wi Fi network deployments show LLMind 2.0 improved scalability, reliability, and responsiveness compared to centralized baselines.

Figures (20)

• Figure 1: The LLMind 2.0 architecture addresses scalability challenges in managing diverse devices with proprietary interfaces. Device-specific agents, equipped with lightweight embedded LLMs, process the given natural-language subtask description and generate code employing their respective device-specific APIs.
• Figure 2: The processing pipeline of the centralized coordinator in LLMind 1.0 when given the shelf vacancy search task.
• Figure 3: Reliability and latency performance of LLMind 1.0 for the shelf vacancy detection task. The figure presents the success rate of generating accurate control scripts for all involved robots, along with the code generation latency and the “entire process” latency, including the time for task planning and code generation. Note that we focus on the bottleneck at the coordinator to highlight the scalability limitations of the centralized code generation approach. The above latencies do not include the times required for the robots to physically move to the shelves, inspect the shelves, and report the statuses to the coordinator.
• Figure 4: The framework of the proposed device agent.
• Figure 5: An example of a device API function in JSON format. The API function get_known_aps searches for nearby WiFi access points (APs) and returns a list of known APs. This function requires no input argument.