Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChangingGrounding: 3D Visual Grounding in Changing Scenes

Miao Hu, Zhiwei Huang, Tai Wang, Jiangmiao Pang, Dahua Lin, Nanning Zheng, Runsen Xu

TL;DR

This work reframes 3D visual grounding for dynamic environments as an active, memory-driven task and introduces the ChangingGrounding benchmark to evaluate grounding accuracy and exploration cost when memories of past scenes guide exploration. It presents Mem-ChangingGrounder (MCG), a zero-shot method built on VLM-Grounder that leverages memory retrieval, targeted exploration, fallback, and multi-view fusion to localize 3D bounding boxes efficiently. On ChangingGrounding, MCG achieves higher localization accuracy than baselines while maintaining lower exploration costs, demonstrating the value of memory-based strategies in real-world dynamic settings. The dataset and benchmark aim to catalyze practical memory-centric 3DVG research for service robotics and related applications.

Abstract

Real-world robots localize objects from natural-language instructions while scenes around them keep changing. Yet most of the existing 3D visual grounding (3DVG) method still assumes a reconstructed and up-to-date point cloud, an assumption that forces costly re-scans and hinders deployment. We argue that 3DVG should be formulated as an active, memory-driven problem, and we introduce ChangingGrounding, the first benchmark that explicitly measures how well an agent can exploit past observations, explore only where needed, and still deliver precise 3D boxes in changing scenes. To set a strong reference point, we also propose Mem-ChangingGrounder, a zero-shot method for this task that marries cross-modal retrieval with lightweight multi-view fusion: it identifies the object type implied by the query, retrieves relevant memories to guide actions, then explores the target efficiently in the scene, falls back when previous operations are invalid, performs multi-view scanning of the target, and projects the fused evidence from multi-view scans to get accurate object bounding boxes. We evaluate different baselines on ChangingGrounding, and our Mem-ChangingGrounder achieves the highest localization accuracy while greatly reducing exploration cost. We hope this benchmark and method catalyze a shift toward practical, memory-centric 3DVG research for real-world applications. Project page: https://hm123450.github.io/CGB/ .

ChangingGrounding: 3D Visual Grounding in Changing Scenes

TL;DR

This work reframes 3D visual grounding for dynamic environments as an active, memory-driven task and introduces the ChangingGrounding benchmark to evaluate grounding accuracy and exploration cost when memories of past scenes guide exploration. It presents Mem-ChangingGrounder (MCG), a zero-shot method built on VLM-Grounder that leverages memory retrieval, targeted exploration, fallback, and multi-view fusion to localize 3D bounding boxes efficiently. On ChangingGrounding, MCG achieves higher localization accuracy than baselines while maintaining lower exploration costs, demonstrating the value of memory-based strategies in real-world dynamic settings. The dataset and benchmark aim to catalyze practical memory-centric 3DVG research for service robotics and related applications.

Abstract

Real-world robots localize objects from natural-language instructions while scenes around them keep changing. Yet most of the existing 3D visual grounding (3DVG) method still assumes a reconstructed and up-to-date point cloud, an assumption that forces costly re-scans and hinders deployment. We argue that 3DVG should be formulated as an active, memory-driven problem, and we introduce ChangingGrounding, the first benchmark that explicitly measures how well an agent can exploit past observations, explore only where needed, and still deliver precise 3D boxes in changing scenes. To set a strong reference point, we also propose Mem-ChangingGrounder, a zero-shot method for this task that marries cross-modal retrieval with lightweight multi-view fusion: it identifies the object type implied by the query, retrieves relevant memories to guide actions, then explores the target efficiently in the scene, falls back when previous operations are invalid, performs multi-view scanning of the target, and projects the fused evidence from multi-view scans to get accurate object bounding boxes. We evaluate different baselines on ChangingGrounding, and our Mem-ChangingGrounder achieves the highest localization accuracy while greatly reducing exploration cost. We hope this benchmark and method catalyze a shift toward practical, memory-centric 3DVG research for real-world applications. Project page: https://hm123450.github.io/CGB/ .
Paper Structure (40 sections, 2 equations, 11 figures, 12 tables)

This paper contains 40 sections, 2 equations, 11 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ChangingGrounding
  4. Task Formulation
  5. Evaluation Metrics
  6. Dataset and Benchmark Construction
  7. Mem-ChangingGrounder (MCG)
  8. Preliminary of VLM-Grounder
  9. Action Policy
  10. Details of MCG
  11. Experimental Results
  12. Experimental Settings
  13. Main Results
  14. Ablation Studies
  15. Discussion about Critical Limitations of 3D Visual Grounding Methods
  16. ...and 25 more sections

Figures (11)

  • Figure 1: Comparison between the previous setting of 3DVG and the ChangingGrounding task.
  • Figure 2: ChangingGrounding Dataset generation pipeline.
  • Figure 3: Workflow of Mem-ChangingGrounder (MCG). The upper part shows the overall pipeline: MCG classifies queries, retrieves memory, uses OSS and SRAS to search, applies fallback when needed, and predicts the 3D bounding box through multi-view projection. The lower part shows details of OSS, SRAS, and Multi-view Projection.
  • Figure 4: A word cloud generated from spatial-relation descriptions, visually highlighting the frequency of occurring terms.
  • Figure 5: VLMs failure in memory retrieval, the anchor object is a box.
  • ...and 6 more figures