Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenSU3D: Open World 3D Scene Understanding using Foundation Models

Rafay Mohiuddin, Sai Manoj Prakhya, Fiona Collins, Ziyuan Liu, André Borrmann

TL;DR

A novel, scalable approach for constructing open set, instance-level 3D scene representations, advancing open world understanding of 3D environments and introducing fusion schemes for feature vectors to enhance their contextual knowledge and performance on complex queries is presented.

Abstract

In this paper, we present a novel, scalable approach for constructing open set, instance-level 3D scene representations, advancing open world understanding of 3D environments. Existing methods require pre-constructed 3D scenes and face scalability issues due to per-point feature vector learning, limiting their efficacy with complex queries. Our method overcomes these limitations by incrementally building instance-level 3D scene representations using 2D foundation models, efficiently aggregating instance-level details such as masks, feature vectors, names, and captions. We introduce fusion schemes for feature vectors to enhance their contextual knowledge and performance on complex queries. Additionally, we explore large language models for robust automatic annotation and spatial reasoning tasks. We evaluate our proposed approach on multiple scenes from ScanNet and Replica datasets demonstrating zero-shot generalization capabilities, exceeding current state-of-the-art methods in open world 3D scene understanding.

OpenSU3D: Open World 3D Scene Understanding using Foundation Models

TL;DR

A novel, scalable approach for constructing open set, instance-level 3D scene representations, advancing open world understanding of 3D environments and introducing fusion schemes for feature vectors to enhance their contextual knowledge and performance on complex queries is presented.

Abstract

In this paper, we present a novel, scalable approach for constructing open set, instance-level 3D scene representations, advancing open world understanding of 3D environments. Existing methods require pre-constructed 3D scenes and face scalability issues due to per-point feature vector learning, limiting their efficacy with complex queries. Our method overcomes these limitations by incrementally building instance-level 3D scene representations using 2D foundation models, efficiently aggregating instance-level details such as masks, feature vectors, names, and captions. We introduce fusion schemes for feature vectors to enhance their contextual knowledge and performance on complex queries. Additionally, we explore large language models for robust automatic annotation and spatial reasoning tasks. We evaluate our proposed approach on multiple scenes from ScanNet and Replica datasets demonstrating zero-shot generalization capabilities, exceeding current state-of-the-art methods in open world 3D scene understanding.
Paper Structure (33 sections, 6 equations, 5 figures, 5 tables)

This paper contains 33 sections, 6 equations, 5 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. Overview of Contributions
  3. Literature Review
  4. Foundation & Large Language Models
  5. 3D Scene Segmentation
  6. Open Vocab. 3D Scene Understanding
  7. 3D Spatial Reasoning
  8. Recent Related Works
  9. METHOD
  10. Per-Image Feature Extraction
  11. 2D to 3D Fusion & Tracking
  12. Post Processing
  13. Feature Fusion
  14. Instance Retrieval & Segmentation
  15. Spatial Reasoning
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Open World 3D Scene Understanding Pipeline. Our method takes a sequence of RGB-D images and constructs a 3D scene representation for open vocabulary instance retrieval, open set annotation, segmentation, and spatial reasoning.
  • Figure 2: Feature Extraction Module. For each instance in an image assigns a unique ID and extracts name, bounding box, detailed caption, prediction score, and CLIP clip features.
  • Figure 3: 2D-3D Fusion and Tracking. Tracks IDs of projected 3D semantic masks, associated with instance in image by assessing overlap in 3D space; tracked IDs are recorded and updated mask-projections concatenated.
  • Figure 4: Text-Instance Similarity Heatmaps. Cosine similarity for text queries using ConceptGraph conceptgraph, OpenMask3D openins3d and our method. $\blacksquare$: max, $\blacksquare$: min similarity.
  • Figure : Query: "plant on shelf"