SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

TL;DR

SeeGround introduces a zero-shot 3D Visual Grounding framework that leverages 2D Vision-Language Models by representing 3D scenes as a hybrid of query-aligned rendered images and 3D spatial descriptions. It builds a query-driven Perspective Adaptation Module to render viewpoints aligned with the description and a Fusion Alignment Module to fuse visual cues with spatial text, enabling precise grounding without 3D-specific training. The approach achieves state-of-the-art performance among zero-shot methods on ScanRefer and Nr3D, with notable gains and robustness to incomplete text, underscoring its potential for scalable open-vocabulary 3D understanding in AR and robotics. Overall, SeeGround demonstrates the power of cross-modal 2D-3D grounding, offering a practical, training-free pathway to robust 3D object localization in complex scenes.

Abstract

3D Visual Grounding (3DVG) aims to locate objects in 3D scenes based on textual descriptions, essential for applications like augmented reality and robotics. Traditional 3DVG approaches rely on annotated 3D datasets and predefined object categories, limiting scalability and adaptability. To overcome these limitations, we introduce SeeGround, a zero-shot 3DVG framework leveraging 2D Vision-Language Models (VLMs) trained on large-scale 2D data. SeeGround represents 3D scenes as a hybrid of query-aligned rendered images and spatially enriched text descriptions, bridging the gap between 3D data and 2D-VLMs input formats. We propose two modules: the Perspective Adaptation Module, which dynamically selects viewpoints for query-relevant image rendering, and the Fusion Alignment Module, which integrates 2D images with 3D spatial descriptions to enhance object localization. Extensive experiments on ScanRefer and Nr3D demonstrate that our approach outperforms existing zero-shot methods by large margins. Notably, we exceed weakly supervised methods and rival some fully supervised ones, outperforming previous SOTA by 7.7% on ScanRefer and 7.1% on Nr3D, showcasing its effectiveness in complex 3DVG tasks.

Figures (11)

• Figure 1: Effectiveness of SeeGround: Different from previous SoTA, our method associates 2D visual cues -- color, texture, viewpoint, spatial position, orientation, and state -- with 3D spatial text description to achieve precise scene understanding. Specifically, our method: (a) identifies the floral chair by recognizing unique color and texture cues; (b) recognizes the couch by interpreting geometric shape; (c) determines the right window by interpreting spatial relationships and perspective; (d) identifies the chair by discerning directional alignment; (e) detects the closed door by visually interpreting its state; and (f) selects the bookshelf by understanding relative positioning.
• Figure 2: Overview of the SeeGround framework. We first use a 2D-VLM to interpret the query, identifying both the target object (e.g., "laptop") and a context-providing anchor (e.g., "chair with a floral pattern"). A dynamic viewpoint is then selected based on the anchor’s position, enabling the capture of a 2D rendered image that aligns with the query’s spatial requirements. Using the Object Lookup Table ($\mathcal{OLT}$), we retrieve the 3D bounding boxes of relevant objects, project them onto the 2D image, and apply visual prompts to mark visible objects, filtering out occlusions. The image with prompts, along with the spatial descriptions and query, is then input into the 2D-VLM for precise localization of the target object. Finally, the 2D-VLM outputs the target object’s ID, which is then used to retrieve its 3D bounding box from the $\mathcal{OLT}$, providing the final, accurate 3D position in the scene.
• Figure 3: Illustrative example of different perspective selection strategies. Our "Query-Aligned" method dynamically adapts the viewpoint to match the spatial context of the query, enhancing detail and relevance of visible objects compared to static methods.
• Figure 4: Visualization of scene details from different viewpoints. The Bird's Eye View (a) captures the entire scene layout but lacks object-specific detail, while the "Query-Aligned" View (b) focuses on relevant objects from an optimal angle, revealing additional context like textures and spatial arrangement.
• Figure 5: Qualitative Results. Rendered images are presented, including the incorrectly identified objects (Orange) and correctly identified objects (Green). Key visual cues are underlined.