Z3D: Zero-Shot 3D Visual Grounding from Images

TL;DR

This paper tackles zero-shot 3D visual grounding (3DVG) using multi-view images. It introduces Z3D, a universal pipeline that incorporates depth-enabled grounding, efficient view selection, zero-shot 2D segmentation via SAM3-Agent, and 2D-to-3D lifting through MaskClustering, with an optional reconstruction path via DUSt3R when depth or pose data are unavailable. The authors demonstrate state-of-the-art zero-shot performance on ScanRefer and Nr3D, highlighting the importance of high-quality 3D proposals and robust VLM-based reasoning. By enabling zero-shot 3DVG from images—and handling depth/pose availability flexibly—Z3D broadens practical 3D grounding for robotics and embodied AI, while also providing insights through ablations on component contributions.

Abstract

3D visual grounding (3DVG) aims to localize objects in a 3D scene based on natural language queries. In this work, we explore zero-shot 3DVG from multi-view images alone, without requiring any geometric supervision or object priors. We introduce Z3D, a universal grounding pipeline that flexibly operates on multi-view images while optionally incorporating camera poses and depth maps. We identify key bottlenecks in prior zero-shot methods causing significant performance degradation and address them with (i) a state-of-the-art zero-shot 3D instance segmentation method to generate high-quality 3D bounding box proposals and (ii) advanced reasoning via prompt-based segmentation, which utilizes full capabilities of modern VLMs. Extensive experiments on the ScanRefer and Nr3D benchmarks demonstrate that our approach achieves state-of-the-art performance among zero-shot methods. Code is available at https://github.com/col14m/z3d .

Figures (4)

• Figure 1: (1) Inference pipeline of Z3D given point cloud inputs. Z3D leverages MaskClustering yan2024maskclustering to predict class-agnostic 3D bounding boxes from a point cloud and images. The most relevant frames are chosen based on their CLIP embeddings and further filtered with VLM. Object masks are annotated using SAM3-Agent carion2025sam3 and lifted to 3D. The final bounding box is selected from MaskClustering proposals through a mask voting scheme. (2) Our approach flexibly accommodates different input modalities, including images, camera poses, and depth maps. When camera poses and depth maps are unavailable, they are estimated with DUSt3R wang2024dust3r.
• Figure 2: Qualitative results of Z3D on ScanRefer.
• Figure 3: Qualitative results of Z3D on Nr3D.
• Figure 4: Failure cases of Z3D on Nr3D.