Penetrative AI: Making LLMs Comprehend the Physical World
Huatao Xu, Liying Han, Qirui Yang, Mo Li, Mani Srivastava
TL;DR
Penetrative AI investigates enabling LLMs to comprehend and act upon the physical world via IoT data, using textualized and digitized signal inputs. By prompting LLMs with expert sensor knowledge and, in some cases, reasoning examples, the approach achieves high accuracy in activity and environment inference, and, for ECG data, strong R-peak detection performance, even rivaling traditional signal-processing methods. The work distinguishes Penetrative AI from Embodied AI by focusing on LLMs as world models that interface with CPS rather than autonomous robotic agents. While promising, it also highlights challenges in knowledge boundaries, data processing, and ethical implications, calling for broader evaluation and integrated tooling in CPS contexts.
Abstract
Recent developments in Large Language Models (LLMs) have demonstrated their remarkable capabilities across a range of tasks. Questions, however, persist about the nature of LLMs and their potential to integrate common-sense human knowledge when performing tasks involving information about the real physical world. This paper delves into these questions by exploring how LLMs can be extended to interact with and reason about the physical world through IoT sensors and actuators, a concept that we term "Penetrative AI". The paper explores such an extension at two levels of LLMs' ability to penetrate into the physical world via the processing of sensory signals. Our preliminary findings indicate that LLMs, with ChatGPT being the representative example in our exploration, have considerable and unique proficiency in employing the embedded world knowledge for interpreting IoT sensor data and reasoning over them about tasks in the physical realm. Not only this opens up new applications for LLMs beyond traditional text-based tasks, but also enables new ways of incorporating human knowledge in cyber-physical systems.