Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Space3D-Bench: Spatial 3D Question Answering Benchmark

Emilia Szymanska, Mihai Dusmanu, Jan-Willem Buurlage, Mahdi Rad, Marc Pollefeys

TL;DR

This work tackles robust indoor 3D spatial question answering by identifying gaps in existing benchmarks and models. It introduces Space3D-Bench, a dataset of $1000$ questions tied to $13$ Replica scenes with multi-modal inputs, guided by a GIS-inspired indoor spatial taxonomy. A Vision-Language Model–based automatic assessment framework validates answers, complemented by a user study to ensure reliability. As a baseline, the paper presents RAG3D-Chat, a Retrieval-Augmented Generation system that integrates multiple data modules and a planner, achieving $67%$ accuracy on the dataset and highlighting substantial room for improvement in 3D spatial reasoning and scene understanding.

Abstract

Answering questions about the spatial properties of the environment poses challenges for existing language and vision foundation models due to a lack of understanding of the 3D world notably in terms of relationships between objects. To push the field forward, multiple 3D Q&A datasets were proposed which, overall, provide a variety of questions, but they individually focus on particular aspects of 3D reasoning or are limited in terms of data modalities. To address this, we present Space3D-Bench - a collection of 1000 general spatial questions and answers related to scenes of the Replica dataset which offers a variety of data modalities: point clouds, posed RGB-D images, navigation meshes and 3D object detections. To ensure that the questions cover a wide range of 3D objectives, we propose an indoor spatial questions taxonomy inspired by geographic information systems and use it to balance the dataset accordingly. Moreover, we provide an assessment system that grades natural language responses based on predefined ground-truth answers by leveraging a Vision Language Model's comprehension of both text and images to compare the responses with ground-truth textual information or relevant visual data. Finally, we introduce a baseline called RAG3D-Chat integrating the world understanding of foundation models with rich context retrieval, achieving an accuracy of 67% on the proposed dataset.

Space3D-Bench: Spatial 3D Question Answering Benchmark

TL;DR

This work tackles robust indoor 3D spatial question answering by identifying gaps in existing benchmarks and models. It introduces Space3D-Bench, a dataset of questions tied to Replica scenes with multi-modal inputs, guided by a GIS-inspired indoor spatial taxonomy. A Vision-Language Model–based automatic assessment framework validates answers, complemented by a user study to ensure reliability. As a baseline, the paper presents RAG3D-Chat, a Retrieval-Augmented Generation system that integrates multiple data modules and a planner, achieving accuracy on the dataset and highlighting substantial room for improvement in 3D spatial reasoning and scene understanding.

Abstract

Answering questions about the spatial properties of the environment poses challenges for existing language and vision foundation models due to a lack of understanding of the 3D world notably in terms of relationships between objects. To push the field forward, multiple 3D Q&A datasets were proposed which, overall, provide a variety of questions, but they individually focus on particular aspects of 3D reasoning or are limited in terms of data modalities. To address this, we present Space3D-Bench - a collection of 1000 general spatial questions and answers related to scenes of the Replica dataset which offers a variety of data modalities: point clouds, posed RGB-D images, navigation meshes and 3D object detections. To ensure that the questions cover a wide range of 3D objectives, we propose an indoor spatial questions taxonomy inspired by geographic information systems and use it to balance the dataset accordingly. Moreover, we provide an assessment system that grades natural language responses based on predefined ground-truth answers by leveraging a Vision Language Model's comprehension of both text and images to compare the responses with ground-truth textual information or relevant visual data. Finally, we introduce a baseline called RAG3D-Chat integrating the world understanding of foundation models with rich context retrieval, achieving an accuracy of 67% on the proposed dataset.
Paper Structure (13 sections, 8 figures)

This paper contains 13 sections, 8 figures.

Table of Contents

  1. Introduction
  2. Literature Overview
  3. Benchmark
  4. Dataset
  5. Taxonomy of Questions
  6. Automatic Assessment
  7. Baseline
  8. Modules
  9. Experimental Results
  10. Evaluation of the Automatic Assessment
  11. Baseline Results
  12. Limitations & Future Work
  13. Conclusions

Figures (8)

  • Figure 1: Questions from Space3D-Bench with answers by RAG3D-Chat. The dataset supports a variety of spatial tasks, including object location, measurements, pattern identification, navigation, spatial relationships, and predictions.
  • Figure 2: Distribution of detected objects across selected scenes. Only the object classes that appear more than 6 times are included in this figure.
  • Figure 3: Statistics of questions in our dataset. The dataset has a large variety of phrasings (a) and is well distributed across the question categories (b). Furthermore, the questions are overall concise with an average length of around 8 words, but some longer ones are also present (c).
  • Figure 4: Automatic assessment procedure. The left chart presents the scenario of Ground Truth Check, the right one depicts Answer Cross-Check, used respectively for indisputable data and more creative answers.
  • Figure 5: Overview of RAG3D-Chat. Based on the received question, the Semantic Kernel library orchestrates the calls of four different modules having different specializations and types of input.
  • ...and 3 more figures