DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Grounding

TL;DR

DenseGrounding tackles ego-centric 3D visual grounding by addressing both visual and linguistic semantic gaps. It introduces HSSE to preserve dense global scene semantics in multi-view ego-centric data and an LLM-based Language Semantic Enhancer (LSE) grounded in a Scene Information Database (SIDB) to enrich descriptions with anchors and context. The approach yields state-of-the-art accuracy on EmbodiedScan, with substantial gains on both full and mini datasets, and secured top honors in the CVPR 2024 Autonomous Grand Challenge. By enabling bidirectional cross-modal enrichment and dataset-grounded language augmentation, DenseGrounding advances robust, real-world 3D grounding for embodied AI systems.

Abstract

Enabling intelligent agents to comprehend and interact with 3D environments through natural language is crucial for advancing robotics and human-computer interaction. A fundamental task in this field is ego-centric 3D visual grounding, where agents locate target objects in real-world 3D spaces based on verbal descriptions. However, this task faces two significant challenges: (1) loss of fine-grained visual semantics due to sparse fusion of point clouds with ego-centric multi-view images, (2) limited textual semantic context due to arbitrary language descriptions. We propose DenseGrounding, a novel approach designed to address these issues by enhancing both visual and textual semantics. For visual features, we introduce the Hierarchical Scene Semantic Enhancer, which retains dense semantics by capturing fine-grained global scene features and facilitating cross-modal alignment. For text descriptions, we propose a Language Semantic Enhancer that leverages large language models to provide rich context and diverse language descriptions with additional context during model training. Extensive experiments show that DenseGrounding significantly outperforms existing methods in overall accuracy, with improvements of 5.81% and 7.56% when trained on the comprehensive full dataset and smaller mini subset, respectively, further advancing the SOTA in egocentric 3D visual grounding. Our method also achieves 1st place and receives the Innovation Award in the CVPR 2024 Autonomous Grand Challenge Multi-view 3D Visual Grounding Track, validating its effectiveness and robustness.

Figures (6)

• Figure 1: (a) illustrates how limited context due to arbitrary descriptions leads to insufficient language semantics. (b) highlights the issue of losing fine-grained semantics in sparse fusion.
• Figure 2: (a) shows overall framework, while (b) details the Language Semantic Enhancer (LSE) module, and (c) describes the Hierarchical Scene Semantic Enhancer (HSSE) module.
• Figure 3: Qualitative Analysis. Comparison of Ground Truth, our baseline, and DenseGrounding. Ground truth boxes are shown in green, baseline in red, and DenseGrounding's predictions in blue.
• Figure 4: Performance of our method on each class in validation set of EmbodiedScan
• Figure 5: Comparison of DenseGrounding and EmbodiedScan on limited data scenario.