Structured Interfaces for Automated Reasoning with 3D Scene Graphs

TL;DR

This paper tackles grounding natural language in large-scale 3D scene graphs for robotics by introducing a GraphRAG framework that exposes a Cypher-based interface to a graph database. By retrieving task-relevant subgraphs on demand, the LLM grounds language without embedding the entire graph in its context, enabling scalable instruction translation to PDDL and scene question answering. The approach demonstrates superior task success across indoor and outdoor scene graphs, with notable token efficiency and robustness to large graphs, and includes a real-world robot demonstration. The work advances grounded language understanding in robotics by decoupling reasoning from the full graph and leveraging a structured query language, with open-source code to enable adoption and extension.

Abstract

In order to provide a robot with the ability to understand and react to a user's natural language inputs, the natural language must be connected to the robot's underlying representations of the world. Recently, large language models (LLMs) and 3D scene graphs (3DSGs) have become a popular choice for grounding natural language and representing the world. In this work, we address the challenge of using LLMs with 3DSGs to ground natural language. Existing methods encode the scene graph as serialized text within the LLM's context window, but this encoding does not scale to large or rich 3DSGs. Instead, we propose to use a form of Retrieval Augmented Generation to select a subset of the 3DSG relevant to the task. We encode a 3DSG in a graph database and provide a query language interface (Cypher) as a tool to the LLM with which it can retrieve relevant data for language grounding. We evaluate our approach on instruction following and scene question-answering tasks and compare against baseline context window and code generation methods. Our results show that using Cypher as an interface to 3D scene graphs scales significantly better to large, rich graphs on both local and cloud-based models. This leads to large performance improvements in grounded language tasks while also substantially reducing the token count of the scene graph content. A video supplement is available at https://www.youtube.com/watch?v=zY_YI9giZSA.

Figures (4)

• Figure 1: An outdoor 3D scene graph, where the operational area traversed by the robot is approximately 1$km$. An example generated mesh is shown on the left below the robot's trajectory, with zoomed-in selections of key parts of the graph's ontology. The graph has 314 objects, 15944 places, and 124 regions. Encoding this entire graph in the context window of an LLM exceeds the context window size of most contemporary models. Moreover, if we want the robot to follow a command to navigate to the fire hydrant, which is absent from the graph but near the known rock, we need a mechanism of using language to modify the scene graph to add the hydrant, which requires reasoning about where "near the rock" would be in the graph. That reasoning in turn involves identifying an object and a set of relevant places through a question expressed in natural language. Generating a plan requires reasoning about which parts of the scene graph are relevant for the navigation problem, which involves identifying the new object and a much larger set of relevant places.
• Figure 2: An illustration of the pipeline for using Cypher queries to a graph database as a tool for the LLM to use to retrieve relevant scene graph information. Given the prompt and the user input, the LLM chooses whether to interact with the graph database or generate a final response. The LLM can interact with the database zero or more times, up to some maximum number of interactions.
• Figure 3: We demonstrate grounding a user's natural language instructions to a Cypher query to correct the object label in a 3DSG, producing a PDDL goal based on the updated object semantics, and then executing a plan to achieve the goal onboard a robot.
• Figure 4: EBNF definition for the SLDP equality comparison language. The CNAME terminal matches alphanumeric strings with a leading letter (valid identifiers in C), and SIGNED_FLOAT and INT match floating point nubmers and integers respectively. Whitespace is ignored when parsing the language.