Spatial Reasoner: A 3D Inference Pipeline for XR Applications
Steven Häsler, Philipp Ackermann
TL;DR
Spatial Reasoner addresses the need for semantic 3D scene understanding in XR by unifying geometric bounding-box representations with a symbolic, NL-like predicate language and a pipeline-based inference model. It defines oriented bounding boxes, a $3\times3\times3$ spatial sector partition, and a comprehensive predicate library to support flexible, rule-based reasoning across XR tasks. Key contributions include cross-platform implementations (Swift, C#, Python), a log/visualization framework, and practical examples of property filtering, spatial queries, and production rules that enable real-time, semantically aware interactions. By bridging geometry with symbolic knowledge and NLP-friendly interfaces, the framework enables scalable spatial ontologies and richer integrations with ML, NLP, and rule systems in XR environments.
Abstract
Modern extended reality XR systems provide rich analysis of image data and fusion of sensor input and demand AR/VR applications that can reason about 3D scenes in a semantic manner. We present a spatial reasoning framework that bridges geometric facts with symbolic predicates and relations to handle key tasks such as determining how 3D objects are arranged among each other ('on', 'behind', 'near', etc.). Its foundation relies on oriented 3D bounding box representations, enhanced by a comprehensive set of spatial predicates, ranging from topology and connectivity to directionality and orientation, expressed in a formalism related to natural language. The derived predicates form a spatial knowledge graph and, in combination with a pipeline-based inference model, enable spatial queries and dynamic rule evaluation. Implementations for client- and server-side processing demonstrate the framework's capability to efficiently translate geometric data into actionable knowledge, ensuring scalable and technology-independent spatial reasoning in complex 3D environments. The Spatial Reasoner framework is fostering the creation of spatial ontologies, and seamlessly integrates with and therefore enriches machine learning, natural language processing, and rule systems in XR applications.