Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VIZOR: Viewpoint-Invariant Zero-Shot Scene Graph Generation for 3D Scene Reasoning

Vivek Madhavaram, Vartika Sengar, Arkadipta De, Charu Sharma

TL;DR

VIZOR introduces a training-free, viewpoint-invariant approach to 3D scene graph generation that defines spatial relations from each object's front-facing direction, enabling consistent reasoning across viewpoints. The method combines object segmentation, front-direction estimation, multi-view attribute extraction, and dense open-vocabulary edge reasoning, producing object-centric graphs suitable for zero-shot grounding and complex visual-language queries. Evaluations on Replica and Nr3D show competitive or superior performance to state-of-the-art zero-shot methods, with notable gains in grounding accuracy and richer graph connectivity. This work advances practical 3D scene understanding by removing reliance on annotated view-specific graphs and leveraging open-vocabulary reasoning for robust downstream tasks.

Abstract

Scene understanding and reasoning has been a fundamental problem in 3D computer vision, requiring models to identify objects, their properties, and spatial or comparative relationships among the objects. Existing approaches enable this by creating scene graphs using multiple inputs such as 2D images, depth maps, object labels, and annotated relationships from specific reference view. However, these methods often struggle with generalization and produce inaccurate spatial relationships like "left/right", which become inconsistent across different viewpoints. To address these limitations, we propose Viewpoint-Invariant Zero-shot scene graph generation for 3D scene Reasoning (VIZOR). VIZOR is a training-free, end-to-end framework that constructs dense, viewpoint-invariant 3D scene graphs directly from raw 3D scenes. The generated scene graph is unambiguous, as spatial relationships are defined relative to each object's front-facing direction, making them consistent regardless of the reference view. Furthermore, it infers open-vocabulary relationships that describe spatial and proximity relationships among scene objects without requiring annotated training data. We conduct extensive quantitative and qualitative evaluations to assess the effectiveness of VIZOR in scene graph generation and downstream tasks, such as query-based object grounding. VIZOR outperforms state-of-the-art methods, showing clear improvements in scene graph generation and achieving 22% and 4.81% gains in zero-shot grounding accuracy on the Replica and Nr3D datasets, respectively.

VIZOR: Viewpoint-Invariant Zero-Shot Scene Graph Generation for 3D Scene Reasoning

TL;DR

VIZOR introduces a training-free, viewpoint-invariant approach to 3D scene graph generation that defines spatial relations from each object's front-facing direction, enabling consistent reasoning across viewpoints. The method combines object segmentation, front-direction estimation, multi-view attribute extraction, and dense open-vocabulary edge reasoning, producing object-centric graphs suitable for zero-shot grounding and complex visual-language queries. Evaluations on Replica and Nr3D show competitive or superior performance to state-of-the-art zero-shot methods, with notable gains in grounding accuracy and richer graph connectivity. This work advances practical 3D scene understanding by removing reliance on annotated view-specific graphs and leveraging open-vocabulary reasoning for robust downstream tasks.

Abstract

Scene understanding and reasoning has been a fundamental problem in 3D computer vision, requiring models to identify objects, their properties, and spatial or comparative relationships among the objects. Existing approaches enable this by creating scene graphs using multiple inputs such as 2D images, depth maps, object labels, and annotated relationships from specific reference view. However, these methods often struggle with generalization and produce inaccurate spatial relationships like "left/right", which become inconsistent across different viewpoints. To address these limitations, we propose Viewpoint-Invariant Zero-shot scene graph generation for 3D scene Reasoning (VIZOR). VIZOR is a training-free, end-to-end framework that constructs dense, viewpoint-invariant 3D scene graphs directly from raw 3D scenes. The generated scene graph is unambiguous, as spatial relationships are defined relative to each object's front-facing direction, making them consistent regardless of the reference view. Furthermore, it infers open-vocabulary relationships that describe spatial and proximity relationships among scene objects without requiring annotated training data. We conduct extensive quantitative and qualitative evaluations to assess the effectiveness of VIZOR in scene graph generation and downstream tasks, such as query-based object grounding. VIZOR outperforms state-of-the-art methods, showing clear improvements in scene graph generation and achieving 22% and 4.81% gains in zero-shot grounding accuracy on the Replica and Nr3D datasets, respectively.
Paper Structure (15 sections, 4 equations, 6 figures, 6 tables)

This paper contains 15 sections, 4 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Proposed Method
  4. Object Segmentation & Front Direction Estimation
  5. Attribute Extraction for Graph Nodes
  6. Computing Edges between Pairs of Objects
  7. Experiments
  8. Datasets
  9. Scene Graph Construction
  10. Object Grounding based on Text Queries
  11. Complex Visual-Language Queries
  12. Qualitative Analysis
  13. Design Choice Analysis
  14. Failure Analysis
  15. Conclusion

Figures (6)

  • Figure 1: Overview: We present the advantage of view-invariant scene graphs generated by VIZOR, using object grounding as a downstream task. For a given complex query, view-invariant scene graphs (c) capture scene layout more accurately than (a) view-dependent scene graphs, which rely on annotator viewpoint, and (b) LLM-inferred graphs, which often produce overly generic relations.
  • Figure 2: Overall architecture of VIZOR. It has three major components: Object Segmentation & Front-direction prediction module (Sec 3.1), object attribute extraction module (Sec 3.2), and Relationship extraction and enrichment module (Sec 3.3). Given only an input 3D scene mesh data, it generates a 3D view-invariant scene graph that can be directly applied to a variety of tasks viz. query based open vocabulary object grounding, complex scene understanding etc.
  • Figure 3: Scene Graph Schema
  • Figure 4: Qualitative Results: (a) Input query, (b) Part of scene graph generated by VIZOR, (c) ConceptGraphs gu2024conceptgraphs, (d) VIZOR output, (e) Explanation of object grounding using VIZOR.
  • Figure 5: Comparative Scene Graph Results: Scene graph comparison between (a) 3DSSG ground truth scene graphs and (b) VIZOR generated scene graphs on 3DSSG scenes
  • ...and 1 more figures