Knowledge Graphs as World Models for Semantic Material-Aware Obstacle Handling in Autonomous Vehicles
Ayush Bheemaiah, Seungyong Yang
TL;DR
The paper tackles the challenge that autonomous vehicles struggle to infer obstacle material properties from sensors alone. It introduces a knowledge-graph-based world model that encodes obstacle types and material attributes (e.g., malleability, density, elasticity) to support semantic reasoning and infer actions beyond raw sensor data. In CARLA-based experiments, KG-integrated AVs demonstrated improved obstacle handling, including a $13.3\%$ increase in emergency stops under lane-change restrictions and a $6.6\%$ rise in lane-change success when changes were allowed, compared with sensor-only baselines. These results highlight the potential of KG-enabled world models to enhance decision-making in embodied AI and suggest broader applicability to robotics, healthcare, and environmental simulation.
Abstract
The inability of autonomous vehicles (AVs) to infer the material properties of obstacles limits their decision-making capacity. While AVs rely on sensor systems such as cameras, LiDAR, and radar to detect obstacles, this study suggests combining sensors with a knowledge graph (KG)-based world model to improve AVs' comprehension of physical material qualities. Beyond sensor data, AVs can infer qualities such as malleability, density, and elasticity using a semantic KG that depicts the relationships between obstacles and their attributes. Using the CARLA autonomous driving simulator, we evaluated AV performance with and without KG integration. The findings demonstrate that the KG-based method improves obstacle management, which allows AVs to use material qualities to make better decisions about when to change lanes or apply emergency braking. For example, the KG-integrated AV changed lanes for hard impediments like traffic cones and successfully avoided collisions with flexible items such as plastic bags by passing over them. Compared to the control system, the KG framework demonstrated improved responsiveness to obstacles by resolving conflicting sensor data, causing emergency stops for 13.3% more cases. In addition, our method exhibits a 6.6% higher success rate in lane-changing maneuvers in experimental scenarios, particularly for larger, high-impact obstacles. While we focus particularly on autonomous driving, our work demonstrates the potential of KG-based world models to improve decision-making in embodied AI systems and scale to other domains, including robotics, healthcare, and environmental simulation.