PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding

TL;DR

PanoGrounder tackles 3D Visual Grounding by bridging 2D vision-language models with 3D scene grounding through panoramic representations. It introduces a three-stage pipeline that (i) places a compact set of panoramic viewpoints with structure-aware camera placement, (ii) grounds text on each panorama using an augmented VLM with a lightweight multi-modal feature adapter, and (iii) lifts and fuses per-view predictions into a single 3D bounding box via visibility-aware aggregation. The method leverages multi-modal panoramas (RGB, semantic, and range) to inject geometric and semantic cues, and uses an Earth Mover’s Distance loss with cross-entropy to produce numerically coherent coordinates, plus geometric QA to train spatial reasoning. Across ScanRefer and Nr3D, with strong generalization to unseen scenes and paraphrased queries, PanoGrounder achieves state-of-the-art results and demonstrates the practicality of panorama-based 2D–3D grounding for open-vocabulary, robotic perception tasks.

Abstract

3D Visual Grounding (3DVG) is a critical bridge from vision-language perception to robotics, requiring both language understanding and 3D scene reasoning. Traditional supervised models leverage explicit 3D geometry but exhibit limited generalization, owing to the scarcity of 3D vision-language datasets and the limited reasoning capabilities compared to modern vision-language models (VLMs). We propose PanoGrounder, a generalizable 3DVG framework that couples multi-modal panoramic representation with pretrained 2D VLMs for strong vision-language reasoning. Panoramic renderings, augmented with 3D semantic and geometric features, serve as an intermediate representation between 2D and 3D, and offer two major benefits: (i) they can be directly fed to VLMs with minimal adaptation and (ii) they retain long-range object-to-object relations thanks to their 360-degree field of view. We devise a three-stage pipeline that places a compact set of panoramic viewpoints considering the scene layout and geometry, grounds a text query on each panoramic rendering with a VLM, and fuses per-view predictions into a single 3D bounding box via lifting. Our approach achieves state-of-the-art results on ScanRefer and Nr3D, and demonstrates superior generalization to unseen 3D datasets and text rephrasings.

Figures (13)

• Figure 1: Overview. Given a free-form text query and a 3D scene, PanoGrounder outputs a single 3D bounding box that localizes the referred object. Panoramic renderings serve as a 2D-3D interface with two functions: providing $360^\circ$ scene context and spatial relations between objects, and enabling direct transfer of 2D VLMs' language understanding to 3D tasks from the panoramas and the 3D model.
• Figure 2: Overall pipeline of PanoGrounder. We first select a compact set of informative panoramic viewpoints via Structure-aware Camera Placement and render multi-modal panoramas (RGB, lifted semantic features, and range). Each view is processed by an off-the-shelf VLM augmented with our multi-modal feature adapter to produce 2D bounding boxes from the text query. Finally, per-view 2D predictions are lifted and fused with visibility-aware multi-view aggregation to yield a single 3D bounding box for the referred object.
• Figure 3: Multi-Modal Feature Adapter. We inject panoramic geometry (range) and semantic (multi-view) features into selected ViT layers of the VLM. Each modality is processed by a lightweight adapter (2-layer MLP followed by a $1{\times}1$ convolution), and the resulting output is added patch-wise to the ViT tokens. The $1{\times}1$ convolution weights and bias are initialized to zero.
• Figure 4: Qualitative results on ScanRefer chen2020scanrefer. Green boxes denote ground-truth objects, and red boxes denote predictions from PanoGrounder. Across diverse scenes and query types, PanoGrounder produces accurate and spatially consistent grounding results.
• Figure 5: Prompt templates for VLM inference and text augmentation. (a) Instruction-style prompt for querying PanoGrounder with a panoramic image and a referring expression <description>. (b) LLaMA 3.3 prompt for rephrasing queries while preserving the original semantic meaning. (c) LLaMA 3.3 prompt for generating affordance-focused descriptions that explicitly include the object class name. (d) LLaMA 3.3 prompt for generating affordance-focused descriptions that exclude the object class name, so that target is specified only through its affordances and context.