AutoIOT: LLM-Driven Automated Natural Language Programming for AIoT Applications

TL;DR

This work tackles the challenge of automating AIoT application development with privacy-preserving, interpretable program synthesis by leveraging large language models. It introduces AutoIOT, an LLM-driven system with three novel modules: background knowledge retrieval, chain-of-thought based automated program synthesis, and automated code improvement, enabling local execution of synthesized programs from natural language prompts. The approach is validated across heartbeat detection and HAR tasks, demonstrating competitive accuracy and substantial reductions in communication cost and execution time compared to remote-LMM baselines, along with positive user-study feedback. The work suggests that integrating up-to-date domain knowledge and iterative refinement can unlock robust, explainable AIoT solutions and potentially democratize AIoT development for a broader user base.

Abstract

The advent of Large Language Models (LLMs) has profoundly transformed our lives, revolutionizing interactions with AI and lowering the barrier to AI usage. While LLMs are primarily designed for natural language interaction, the extensive embedded knowledge empowers them to comprehend digital sensor data. This capability enables LLMs to engage with the physical world through IoT sensors and actuators, performing a myriad of AIoT tasks. Consequently, this evolution triggers a paradigm shift in conventional AIoT application development, democratizing its accessibility to all by facilitating the design and development of AIoT applications via natural language. However, some limitations need to be addressed to unlock the full potential of LLMs in AIoT application development. First, existing solutions often require transferring raw sensor data to LLM servers, which raises privacy concerns, incurs high query fees, and is limited by token size. Moreover, the reasoning processes of LLMs are opaque to users, making it difficult to verify the robustness and correctness of inference results. This paper introduces AutoIOT, an LLM-based automated program generator for AIoT applications. AutoIOT enables users to specify their requirements using natural language (input) and automatically synthesizes interpretable programs with documentation (output). AutoIOT automates the iterative optimization to enhance the quality of generated code with minimum user involvement. AutoIOT not only makes the execution of AIoT tasks more explainable but also mitigates privacy concerns and reduces token costs with local execution of synthesized programs. Extensive experiments and user studies demonstrate AutoIOT's remarkable capability in program synthesis for various AIoT tasks. The synthesized programs can match and even outperform some representative baselines.

Figures (16)

• Figure 1: Illustration of how LLMs can sense and interact with the physical world in AIoT applications.
• Figure 2: Prior work processes sensor data with LLMs.
• Figure 3: An example of direct code generation.
• Figure 4: Code generation with user intervention.
• Figure 5: The system overview and workflow of AutoIOT.