Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open-Vocabulary 3D Instruction Ambiguity Detection

Jiayu Ding, Haoran Tang, Ge Li

TL;DR

The paper tackles safety-critical instruction execution in 3D environments by introducing Open-Vocabulary 3D Instruction Ambiguity Detection, a task where a model must determine if a given instruction is unambiguous within a 3D scene. It contributes Ambi3D, a large, human-annotated benchmark grounded in 700+ scenes with ~22k instructions, and AmbiVer, a two-stage framework that first extracts structured visual evidence from multiple views and then uses a zero-shot vision-language model to adjudicate ambiguity. Experimental results show existing 3D LLMs struggle with ambiguity detection, while AmbiVer delivers robust, balanced performance and clear interpretability via its Dossier and verdict structure. The work lays the groundwork for safer, more trustworthy embodied AI by separating perception from reasoning and focusing on objective ambiguity rather than execution alone.

Abstract

In safety-critical domains, linguistic ambiguity can have severe consequences; a vague command like "Pass me the vial" in a surgical setting could lead to catastrophic errors. Yet, most embodied AI research overlooks this, assuming instructions are clear and focusing on execution rather than confirmation. To address this critical safety gap, we are the first to define Open-Vocabulary 3D Instruction Ambiguity Detection, a fundamental new task where a model must determine if a command has a single, unambiguous meaning within a given 3D scene. To support this research, we build Ambi3D, the large-scale benchmark for this task, featuring over 700 diverse 3D scenes and around 22k instructions. Our analysis reveals a surprising limitation: state-of-the-art 3D Large Language Models (LLMs) struggle to reliably determine if an instruction is ambiguous. To address this challenge, we propose AmbiVer, a two-stage framework that collects explicit visual evidence from multiple views and uses it to guide an vision-language model (VLM) in judging instruction ambiguity. Extensive experiments demonstrate the challenge of our task and the effectiveness of AmbiVer, paving the way for safer and more trustworthy embodied AI. Code and dataset available at https://jiayuding031020.github.io/ambi3d/.

Open-Vocabulary 3D Instruction Ambiguity Detection

TL;DR

The paper tackles safety-critical instruction execution in 3D environments by introducing Open-Vocabulary 3D Instruction Ambiguity Detection, a task where a model must determine if a given instruction is unambiguous within a 3D scene. It contributes Ambi3D, a large, human-annotated benchmark grounded in 700+ scenes with ~22k instructions, and AmbiVer, a two-stage framework that first extracts structured visual evidence from multiple views and then uses a zero-shot vision-language model to adjudicate ambiguity. Experimental results show existing 3D LLMs struggle with ambiguity detection, while AmbiVer delivers robust, balanced performance and clear interpretability via its Dossier and verdict structure. The work lays the groundwork for safer, more trustworthy embodied AI by separating perception from reasoning and focusing on objective ambiguity rather than execution alone.

Abstract

In safety-critical domains, linguistic ambiguity can have severe consequences; a vague command like "Pass me the vial" in a surgical setting could lead to catastrophic errors. Yet, most embodied AI research overlooks this, assuming instructions are clear and focusing on execution rather than confirmation. To address this critical safety gap, we are the first to define Open-Vocabulary 3D Instruction Ambiguity Detection, a fundamental new task where a model must determine if a command has a single, unambiguous meaning within a given 3D scene. To support this research, we build Ambi3D, the large-scale benchmark for this task, featuring over 700 diverse 3D scenes and around 22k instructions. Our analysis reveals a surprising limitation: state-of-the-art 3D Large Language Models (LLMs) struggle to reliably determine if an instruction is ambiguous. To address this challenge, we propose AmbiVer, a two-stage framework that collects explicit visual evidence from multiple views and uses it to guide an vision-language model (VLM) in judging instruction ambiguity. Extensive experiments demonstrate the challenge of our task and the effectiveness of AmbiVer, paving the way for safer and more trustworthy embodied AI. Code and dataset available at https://jiayuding031020.github.io/ambi3d/.
Paper Structure (18 sections, 5 equations, 3 figures, 4 tables)

This paper contains 18 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Task Formulation
  4. Formalizing 3D Instruction Ambiguity
  5. Task Definition
  6. Benchmark
  7. Dataset
  8. Evaluation Metrics
  9. Method
  10. Overall Architecture
  11. Perception Engine: From Pixels to Evidence
  12. Reasoning Engine: From Evidence to Verdict
  13. Experiments
  14. Implementation Details.
  15. Quantitative Analysis
  16. ...and 3 more sections

Figures (3)

  • Figure 1: This high-stakes scenario highlights a critical safety challenge where an ambiguous instruction forces a robot to choose between a harmless substance and a lethal one.
  • Figure 2: Overview of the AmbiVer framework. AmbiVer is a two-stage system composed of a perception engine and a reasoning engine. The perception stage parses an instruction into action, attribute, relation, and target components, employs an open-vocabulary grounding method to detect 2D candidates across views, and integrates them into 3D instances via ray-based fusion followed by refinement. It also generates a BEV map for scene-level context. The reasoning stage then performs multimodal evidence bundling and leverages a VLM to assess instruction ambiguity, outputting a structured verdict.
  • Figure 3: Qualitative results of our AmbiVer framework on the Ambi3D benchmark.