Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding

William Chen, Siyi Hu, Rajat Talak, Luca Carlone

TL;DR

This paper investigates leveraging large language models (LMs) and vision–language models (VLMs) to impart common-sense priors for abstract 3D scene understanding, focusing on room classification within 3D scene graphs. It compares language-only and vision–language paradigms across zero-shot, embedding-based, and structured-data approaches, as well as zero-shot and fine-tuned VLM methods, using Matterport3D-derived scene graphs and real-scene graphs. Key findings show that language-informed methods—including structured-language and embedding-based strategies—achieve high accuracy (up to ~69–70%), with strong generalization and transfer across object label spaces, and that VLMs provide complementary gains when combined with object-label information. The results highlight the promise of scalable, data-efficient reasoning about high-level semantic concepts in robotic spatial perception, while noting compute constraints and suggesting avenues for leveraging larger models and embodied reasoning in future work.

Abstract

Abstract semantic 3D scene understanding is a problem of critical importance in robotics. As robots still lack the common-sense knowledge about household objects and locations of an average human, we investigate the use of pre-trained language models to impart common sense for scene understanding. We introduce and compare a wide range of scene classification paradigms that leverage language only (zero-shot, embedding-based, and structured-language) or vision and language (zero-shot and fine-tuned). We find that the best approaches in both categories yield $\sim 70\%$ room classification accuracy, exceeding the performance of pure-vision and graph classifiers. We also find such methods demonstrate notable generalization and transfer capabilities stemming from their use of language.

Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding

TL;DR

This paper investigates leveraging large language models (LMs) and vision–language models (VLMs) to impart common-sense priors for abstract 3D scene understanding, focusing on room classification within 3D scene graphs. It compares language-only and vision–language paradigms across zero-shot, embedding-based, and structured-data approaches, as well as zero-shot and fine-tuned VLM methods, using Matterport3D-derived scene graphs and real-scene graphs. Key findings show that language-informed methods—including structured-language and embedding-based strategies—achieve high accuracy (up to ~69–70%), with strong generalization and transfer across object label spaces, and that VLMs provide complementary gains when combined with object-label information. The results highlight the promise of scalable, data-efficient reasoning about high-level semantic concepts in robotic spatial perception, while noting compute constraints and suggesting avenues for leveraging larger models and embodied reasoning in future work.

Abstract

Abstract semantic 3D scene understanding is a problem of critical importance in robotics. As robots still lack the common-sense knowledge about household objects and locations of an average human, we investigate the use of pre-trained language models to impart common sense for scene understanding. We introduce and compare a wide range of scene classification paradigms that leverage language only (zero-shot, embedding-based, and structured-language) or vision and language (zero-shot and fine-tuned). We find that the best approaches in both categories yield room classification accuracy, exceeding the performance of pure-vision and graph classifiers. We also find such methods demonstrate notable generalization and transfer capabilities stemming from their use of language.
Paper Structure (28 sections, 6 equations, 4 figures, 5 tables)

This paper contains 28 sections, 6 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Language-only Methods
  4. Query Strings and Informativeness
  5. Zero-shot Approach
  6. Embedding-based Approach
  7. Structured-data Approach
  8. Vision-and-language Methods
  9. Zero-shot Approach
  10. Fine-tuning Approach
  11. Experimental Setup
  12. Datasets
  13. Baselines
  14. Language-only Trial Specifications
  15. Vision-and-language Trial Specifications
  16. ...and 13 more sections

Figures (4)

  • Figure 1: 3D scene graph example. We use (V)LMs to attach high-level labels to nodes (e.g., to label rooms) using lower-level information (e.g., contained objects).
  • Figure 2: Zero-shot accuracies on all data for all conditions, by room label.
  • Figure 3: Example visualization of room label predictions on the real scene graph.
  • Figure 4: Example egocentric images from each room of the uHumans2 apartment, along with their corresponding ID, ground truth label, and contained objects (as detected by Hydra).