Relational Scene Graphs for Object Grounding of Natural Language Commands

TL;DR

This work addresses the problem of grounding natural language commands in robotic environments by combining Large Language Models (LLMs) with 3D scene graphs (3DSGs). It introduces an LLM-based grounding pipeline that reasons over serialized 3DSGs and a Vision-Language Model (VLM) pipeline that augments 3DSGs with open-vocabulary spatial edges from robot-captured images. Across two datasets (VLA-3D and REACT), the study shows that explicit spatial edges improve object grounding performance, with GPT-5 achieving up to 99.5% accuracy on VLA-3D, while open-vocabulary edges do not consistently outperform closed-vocabulary edges given current generation reliability. The findings suggest spatial relations are important for grounding and point to future work on robust edge extraction, subgraph reasoning, and retrieval-augmented strategies to scale grounding to larger scenes.

Abstract

Robots are finding wider adoption in human environments, increasing the need for natural human-robot interaction. However, understanding a natural language command requires the robot to infer the intended task and how to decompose it into executable actions, and to ground those actions in the robot's knowledge of the environment, including relevant objects, agents, and locations. This challenge can be addressed by combining the capabilities of Large language models (LLMs) to understand natural language with 3D scene graphs (3DSGs) for grounding inferred actions in a semantic representation of the environment. However, many 3DSGs lack explicit spatial relations between objects, even though humans often rely on these relations to describe an environment. This paper investigates whether incorporating open- or closed-vocabulary spatial relations into 3DSGs can improve the ability of LLMs to interpret natural language commands. To address this, we propose an LLM-based pipeline for target object grounding from open-vocabulary language commands and a vision language model (VLM)-based pipeline to add open-vocabulary spatial edges to 3DSGs from images captured while mapping. Finally, two LLMs are evaluated in a study assessing their performance on the downstream task of target object grounding. Our study demonstrates that explicit spatial relations improve the ability of LLMs to ground objects. Moreover, open-vocabulary relation generation with VLMs proves feasible from robot-captured images, but their advantage over closed-vocabulary relations is found to be limited.

Figures (5)

• Figure 1: Based on images captured while mapping, spatial edges are added to a 3dsg. When a user command is given, the command is grounded through the 3dsg, leveraging the added spatial edges.
• Figure 2: Overview of the llm-based pipeline for object grounding in 3dsg.
• Figure 3: Example image used for edge generation, where the generated edge is: both are positioned around the same table, facing each other
• Figure 4: Overview of the vlm-based pipeline for relation generation.
• Figure 5: Example image used for edge generation, where the generated edge is: left side of the table, closer to the camera