Hypo3D: Exploring Hypothetical Reasoning in 3D

TL;DR

Hypo3D presents the first benchmark for hypothetical reasoning in 3D Visual Question Answering, challenging models to imagine scene changes before reasoning. By anchoring directional terms to a world frame and collecting 7,727 context changes across 700 indoor scenes (14,885 QA pairs), it evaluates resilience to perception gaps when real-time data is unavailable. Across ten foundation models, humans substantially outperform machines, with improvement margins most pronounced for movement and direction-based reasoning and with notable hallucinations when changes do not affect the question. The work provides a rigorous data-generation pipeline, multi-modal scene representations, and detailed analyses, highlighting a clear path toward enhancing imaginative, context-aware reasoning in 3D understanding systems.

Abstract

The rise of vision-language foundation models marks an advancement in bridging the gap between human and machine capabilities in 3D scene reasoning. Existing 3D reasoning benchmarks assume real-time scene accessibility, which is impractical due to the high cost of frequent scene updates. To this end, we introduce Hypothetical 3D Reasoning, namely Hypo3D, a benchmark designed to evaluate models' ability to reason without access to real-time scene data. Models need to imagine the scene state based on a provided change description before reasoning. Hypo3D is formulated as a 3D Visual Question Answering (VQA) benchmark, comprising 7,727 context changes across 700 indoor scenes, resulting in 14,885 question-answer pairs. An anchor-based world frame is established for all scenes, ensuring consistent reference to a global frame for directional terms in context changes and QAs. Extensive experiments show that state-of-the-art foundation models struggle to reason in hypothetically changed scenes. This reveals a substantial performance gap compared to humans, particularly in scenarios involving movement changes and directional reasoning. Even when the context change is irrelevant to the question, models often incorrectly adjust their answers. Project website: https://matchlab-imperial.github.io/Hypo3D/

Figures (15)

• Figure 1: Overview of the Hypo3D benchmark. ① Examples of five context change types. ② Sample questions, including scale-based and direction-based questions requiring spatial reasoning, as well as semantic questions, all of which have open-ended answers. ③ The radar chart highlights a notable performance gap between models and humans, especially in direction-based questions.
• Figure 2: Example of hypothetical reasoning in a 3D scene. Given a 3D scene and an anchor-based frame description (Scene Orientation), models first align the scene to the specified frame. Then, based on a context change description and a question, models hypothetically modify the aligned scene and answer questions about the changed scene. Various models (e.g., LLMs, 2D VLMs, 3D VLMs) can tackle this task using corresponding scene representations, including scene captions, top-view maps, point clouds, and egocentric RGB-D videos.
• Figure 3: Dataset Generation Pipeline. The Hypo3D collection pipeline consists of five stages: Stage ① curates scenes (50 hours per person), Stage ② defines world frames (10 h/p), Stages ③ and ④ collect context changes and QA descriptions from human annotators (thousands of hours) and LLM, and Stage ⑤ conducts grammar checks and filters data based on semantic similarity. (25 h/p).
• Figure 4: Dataset Statistics. ① Word cloud representing context change descriptions. ② Frequency distribution of context change types across 7,727 instances. ③ Distribution of question types across change categories, with question frequency consistently highest for scale-based, then direction-based, and finally semantic.
• Figure 5: Model and human EM performance across question types. Humans consistently achieve strong performance, whereas models struggle, particularly with direction-based questions.