Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning

TL;DR

Problem: SLAM-based maps lack high-level abstraction and relational reasoning. Approach: ReasoningGraph constructs a five-layer hierarchical 3D scene graph enriched with open-vocabulary features and object-relational edges, coupled with a task reasoning module that fuses LLMs and VLMs to interpret scenes and decompose tasks. Contributions: online, incremental graph construction; multi-level open-vocabulary embeddings; explicit object relationships; and a VLM-LLM reasoning pipeline validated on a quadruped across diverse environments with strong object retrieval and planning capabilities. Significance: enables richer, context-aware robotic understanding and task planning in open-world settings with scalable, runtime-efficient reasoning.

Abstract

Representing and understanding 3D environments in a structured manner is crucial for autonomous agents to navigate and reason about their surroundings. While traditional Simultaneous Localization and Mapping (SLAM) methods generate metric reconstructions and can be extended to metric-semantic mapping, they lack a higher level of abstraction and relational reasoning. To address this gap, 3D scene graphs have emerged as a powerful representation for capturing hierarchical structures and object relationships. In this work, we propose an enhanced hierarchical 3D scene graph that integrates open-vocabulary features across multiple abstraction levels and supports object-relational reasoning. Our approach leverages a Vision Language Model (VLM) to infer semantic relationships. Notably, we introduce a task reasoning module that combines Large Language Models (LLM) and a VLM to interpret the scene graph's semantic and relational information, enabling agents to reason about tasks and interact with their environment more intelligently. We validate our method by deploying it on a quadruped robot in multiple environments and tasks, highlighting its ability to reason about them.

Figures (6)

• Figure 1: Task reasoning example. We deploy ReasoningGraph on a quadruped robot, which incrementally builds an open-vocabulary, relationship-aware hierarchical scene graph of the environment during autonomous exploration. Leveraging open-vocabulary and object-relational embeddings, ReasoningGraph identifies task-relevant objects and reasons about their interactions. In this example, it identifies all the objects (chairs, a table, and a trash can) that are blocking the exits.
• Figure 2: ReasoningGraph overview. a)ReasoningGraph incrementally builds a hierarchical 3D scene graph $\color{RoyalBlue}\mathcal{G}$ (c)) from RGB-D frames and poses, using an open-vocabulary detector wang2025yoloe and CLIP Radford2021CLIP embeddings for object representation. Object relations are derived from a Lu2024DeepSeekVLTR visual encoder, while Hydra hughes2022hydra reconstructs the semantic mesh ($L_1$), clusters objects ($L_2$), and detects places and rooms ($L_3$, $L_4$). Open-vocabulary features and relations are then assigned to $\color{RoyalBlue}\mathcal{G}$. b) The task reasoning module leverages two s and a . Given a task, the identifies relevant objects and formulates subtasks needing evaluation. These subtasks are evaluated for feasibility by the , with CLIP similarity used for object retrieval.
• Figure 3: System prompts for task reasoning and subtask decisor s.
• Figure 4: Guiding the with bounding boxes colors. The is provided with an image with a pair of objects and their inpainted bounding boxes, along with the task reasoning subtask prompt. In this example, the task is to reason about a door and a chair. Without including the bounding box colors in the prompt (blue), the focuses on the wrong object, reasoning about the door behind the chair. When the color information is included (green), the is guided to attend to the relevant objects.
• Figure 5: ReasoningGraph performing T1 and T3. The scene graph is built during exploration, after which the reasons about the task and the evaluates subtasks using object relations ($\mathbf{f}_{r}^{}$). In both individual experiments, a 100 $\text{SR}\%$ is achieved. In T3, we apply our room search method, achieving 100% accuracy in all 5 evaluations. We present one reasoning example per task, although in practice the reasons about each subtask.