Queryable 3D Scene Representation: A Multi-Modal Framework for Semantic Reasoning and Robotic Task Planning

TL;DR

This work tackles the challenge of translating high-level human instructions into precise robotic actions by proposing 3D Queryable Scene Representation (3D QSR), a multimodal, object-centric framework that unifies semantic, geometric, and structural scene information. It combines Panoptic NeRF-based reconstruction, segmented point clouds, and a structured 3D scene graph, all linked by shared identifiers and enhanced with vision-language embeddings, enabling open-vocabulary querying and object-level retrieval. Querying is supported across point clouds, NeRF, and scene graphs, with retrieval augmented by LVLMs and FAISS-based indexing, and results are fed into a robotic task planner via a REST API in Unity. The approach is evaluated in Unity-based robotic tasks on Replica scenes and a digital wet-lab twin, demonstrating effective scene understanding, reasoning, and task planning from natural language inputs, while also exploring LLM-driven scene consolidation for iterative planning. The work highlights the potential of tightly integrated, multimodal scene representations to bridge human intent and autonomous robotic action, while acknowledging current limitations in open-world generalization and real-world deployment.

Abstract

To enable robots to comprehend high-level human instructions and perform complex tasks, a key challenge lies in achieving comprehensive scene understanding: interpreting and interacting with the 3D environment in a meaningful way. This requires a smart map that fuses accurate geometric structure with rich, human-understandable semantics. To address this, we introduce the 3D Queryable Scene Representation (3D QSR), a novel framework built on multimedia data that unifies three complementary 3D representations: (1) 3D-consistent novel view rendering and segmentation from panoptic reconstruction, (2) precise geometry from 3D point clouds, and (3) structured, scalable organization via 3D scene graphs. Built on an object-centric design, the framework integrates with large vision-language models to enable semantic queryability by linking multimodal object embeddings, and supporting object-level retrieval of geometric, visual, and semantic information. The retrieved data are then loaded into a robotic task planner for downstream execution. We evaluate our approach through simulated robotic task planning scenarios in Unity, guided by abstract language instructions and using the indoor public dataset Replica. Furthermore, we apply it in a digital duplicate of a real wet lab environment to test QSR-supported robotic task planning for emergency response. The results demonstrate the framework's ability to facilitate scene understanding and integrate spatial and semantic reasoning, effectively translating high-level human instructions into precise robotic task planning in complex 3D environments.

Figures (16)

• Figure 1: Three main parts of the workflow are: 3D QSR construction, 3D QSR query, and robotic task planning and action.
• Figure 2: Consistent semantic and instance predictions across different frames.
• Figure 3: Point cloud object instance segmentation of two Replica scenes using the proposed algorithm.
• Figure 4: Object labels refined by multi-view captioning.
• Figure 5: A sample illustration of a generated 3D scene graph