Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenes

Haochen Zhang, Nader Zantout, Pujith Kachana, Ji Zhang, Wenshan Wang

TL;DR

This work addresses robust interactive referential grounding in 3D scenes when user language may be imperfect. It introduces IRef-VLA, the largest real-world benchmark combining 3D scans, scene graphs, traversable space, and a large corpus of imperfect referential statements to support ground-truth grounding and alternative generation. The authors provide baseline evaluations with standard referential grounding models and a graph-search approach for imperfect references, demonstrating generalization capabilities and highlighting domain shifts across datasets. The benchmark enables development of more robust, interactive 3D navigation systems for robotics, with practical impact in real-world situated language understanding and planning. Key metrics include object-existence accuracy and a similarity-based scoring for alternatives, complemented by a parsing accuracy measure for LLM-driven subgraph extraction.

Abstract

With the recent rise of large language models, vision-language models, and other general foundation models, there is growing potential for multimodal, multi-task robotics that can operate in diverse environments given natural language input. One such application is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the 3D spatial reasoning and semantic understanding required. Additionally, the language used may be imperfect or misaligned with the scene, further complicating the task. To address this challenge, we curate a benchmark dataset, IRef-VLA, for Interactive Referential Vision and Language-guided Action in 3D Scenes with imperfect references. IRef-VLA is the largest real-world dataset for the referential grounding task, consisting of over 11.5K scanned 3D rooms from existing datasets, 7.6M heuristically generated semantic relations, and 4.7M referential statements. Our dataset also contains semantic object and room annotations, scene graphs, navigable free space annotations, and is augmented with statements where the language has imperfections or ambiguities. We verify the generalizability of our dataset by evaluating with state-of-the-art models to obtain a performance baseline and also develop a graph-search baseline to demonstrate the performance bound and generation of alternatives using scene-graph knowledge. With this benchmark, we aim to provide a resource for 3D scene understanding that aids the development of robust, interactive navigation systems. The dataset and all source code is publicly released at https://github.com/HaochenZ11/IRef-VLA.

IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenes

TL;DR

This work addresses robust interactive referential grounding in 3D scenes when user language may be imperfect. It introduces IRef-VLA, the largest real-world benchmark combining 3D scans, scene graphs, traversable space, and a large corpus of imperfect referential statements to support ground-truth grounding and alternative generation. The authors provide baseline evaluations with standard referential grounding models and a graph-search approach for imperfect references, demonstrating generalization capabilities and highlighting domain shifts across datasets. The benchmark enables development of more robust, interactive 3D navigation systems for robotics, with practical impact in real-world situated language understanding and planning. Key metrics include object-existence accuracy and a similarity-based scoring for alternatives, complemented by a parsing accuracy measure for LLM-driven subgraph extraction.

Abstract

With the recent rise of large language models, vision-language models, and other general foundation models, there is growing potential for multimodal, multi-task robotics that can operate in diverse environments given natural language input. One such application is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the 3D spatial reasoning and semantic understanding required. Additionally, the language used may be imperfect or misaligned with the scene, further complicating the task. To address this challenge, we curate a benchmark dataset, IRef-VLA, for Interactive Referential Vision and Language-guided Action in 3D Scenes with imperfect references. IRef-VLA is the largest real-world dataset for the referential grounding task, consisting of over 11.5K scanned 3D rooms from existing datasets, 7.6M heuristically generated semantic relations, and 4.7M referential statements. Our dataset also contains semantic object and room annotations, scene graphs, navigable free space annotations, and is augmented with statements where the language has imperfections or ambiguities. We verify the generalizability of our dataset by evaluating with state-of-the-art models to obtain a performance baseline and also develop a graph-search baseline to demonstrate the performance bound and generation of alternatives using scene-graph knowledge. With this benchmark, we aim to provide a resource for 3D scene understanding that aids the development of robust, interactive navigation systems. The dataset and all source code is publicly released at https://github.com/HaochenZ11/IRef-VLA.

Paper Structure

This paper contains 19 sections, 1 equation, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Formulation
  4. Referential Grounding with Imperfect References Task
  5. Metrics
  6. Dataset Creation
  7. Overview
  8. 3D Scan Processing
  9. Scene Graph Generation
  10. Language Generation
  11. Baseline Evaluation
  12. Referential Object Grounding
  13. Experimental Setup
  14. Generalizability Results
  15. Referential Grounding with Imperfect References
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Sample region from the dataset visualized with (a) a scene graph and (b) a corresponding referential statement
  • Figure 2: Breakdown of regions from each data source
  • Figure 3: Number of statements per relation type from each dataset processed
  • Figure 4: Data processing pipeline consisting of: 3D Scan Processing, Scene Graph Generation, and Language Generation
  • Figure 5: A comparison between heuristically generated statements describing a binary spatial relation from Sr3D, Nr3D achlioptas2020referit3d, SceneVerse jia2024sceneverse, and IRef-VLA. Both chairs are close to the radiator, so using the superlative relation "closest" is the clearest way to disambiguate.
  • ...and 1 more figures