Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SceneGPT: A Language Model for 3D Scene Understanding

Shivam Chandhok

TL;DR

SceneGPT demonstrates that pre-trained language models contain priors useful for 3D scene reasoning and that simple in-context prompts can unlock these capabilities without 3D supervision. The system builds an open-vocabulary 3D scene graph from RGBD frames using ConceptGraphs, converts it to JSON, and queries an LLM (GPT-4) with a grounded prompt to answer user questions. Qualitative results show capabilities in object semantics, affordances, geometry and spatial relations, while noting limitations tied to context length and perception accuracy. The work suggests a practical path to open-domain 3D understanding using LLM priors and prompts instead of costly 3D supervision.

Abstract

Building models that can understand and reason about 3D scenes is difficult owing to the lack of data sources for 3D supervised training and large-scale training regimes. In this work we ask - How can the knowledge in a pre-trained language model be leveraged for 3D scene understanding without any 3D pre-training. The aim of this work is to establish whether pre-trained LLMs possess priors/knowledge required for reasoning in 3D space and how can we prompt them such that they can be used for general purpose spatial reasoning and object understanding in 3D. To this end, we present SceneGPT, an LLM based scene understanding system which can perform 3D spatial reasoning without training or explicit 3D supervision. The key components of our framework are - 1) a 3D scene graph, that serves as scene representation, encoding the objects in the scene and their spatial relationships 2) a pre-trained LLM that can be adapted with in context learning for 3D spatial reasoning. We evaluate our framework qualitatively on object and scene understanding tasks including object semantics, physical properties and affordances (object-level) and spatial understanding (scene-level).

SceneGPT: A Language Model for 3D Scene Understanding

TL;DR

SceneGPT demonstrates that pre-trained language models contain priors useful for 3D scene reasoning and that simple in-context prompts can unlock these capabilities without 3D supervision. The system builds an open-vocabulary 3D scene graph from RGBD frames using ConceptGraphs, converts it to JSON, and queries an LLM (GPT-4) with a grounded prompt to answer user questions. Qualitative results show capabilities in object semantics, affordances, geometry and spatial relations, while noting limitations tied to context length and perception accuracy. The work suggests a practical path to open-domain 3D understanding using LLM priors and prompts instead of costly 3D supervision.

Abstract

Building models that can understand and reason about 3D scenes is difficult owing to the lack of data sources for 3D supervised training and large-scale training regimes. In this work we ask - How can the knowledge in a pre-trained language model be leveraged for 3D scene understanding without any 3D pre-training. The aim of this work is to establish whether pre-trained LLMs possess priors/knowledge required for reasoning in 3D space and how can we prompt them such that they can be used for general purpose spatial reasoning and object understanding in 3D. To this end, we present SceneGPT, an LLM based scene understanding system which can perform 3D spatial reasoning without training or explicit 3D supervision. The key components of our framework are - 1) a 3D scene graph, that serves as scene representation, encoding the objects in the scene and their spatial relationships 2) a pre-trained LLM that can be adapted with in context learning for 3D spatial reasoning. We evaluate our framework qualitatively on object and scene understanding tasks including object semantics, physical properties and affordances (object-level) and spatial understanding (scene-level).
Paper Structure (22 sections, 2 figures)

This paper contains 22 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. 3D Scene Representations
  4. LLMs and prompting techniques
  5. Spatial understanding with LLMs
  6. Method
  7. Overview
  8. Scene Representation
  9. Graph Structure
  10. LLM Readable Representation
  11. Scenegraph generation
  12. LLM Interface and In-context Learning
  13. LLM Interface
  14. Chain of thought Output Structure
  15. In-Context prompting:
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Overall pipeline for our SceneGPT model. Our framework takes as input RGBD frames of a scene and generates a semantically rich 3D scene graph representation. This is converted to a LLM readable format (json) and fed to a large language model like GPT-4. The system prompt creates an interface between the scene representation and LLM and takes in the user query, scene graph json and in-context examples which guide the output of the LLM. The large language model generates a reasonable response grounded to the input scene to address the user query.
  • Figure 2: Qualitative results for our SceneGPT 3D understanding system on object understanding queries(top row), geometric understanding queries (middle), and spatial understanding queries (bottom)