Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignment

Xiaoqi Li, Jiaming Liu, Nuowei Han, Liang Heng, Yandong Guo, Hao Dong, Yang Liu

TL;DR

This work tackles 3D weakly supervised visual grounding by introducing a dual-branch framework that separately handles category-level and instance-level grounding. The category-level branch leverages a pre-trained detector to align object proposals with the sentence's target category while explicitly contrasting easy-to-confuse categories. The instance-level branch differentiates multiple same-category instances by exploiting noun-phrase–proposal similarities and 3D spatial relations, with training signals provided via InfoNCE losses and spatial-grounding cues. Across Nr3D, Sr3D, and ScanRef, the approach achieves state-of-the-art performance among weakly supervised methods, demonstrating strong cross-modal grounding capabilities and robust handling of category ambiguity and instance complexity.

Abstract

The 3D weakly-supervised visual grounding task aims to localize oriented 3D boxes in point clouds based on natural language descriptions without requiring annotations to guide model learning. This setting presents two primary challenges: category-level ambiguity and instance-level complexity. Category-level ambiguity arises from representing objects of fine-grained categories in a highly sparse point cloud format, making category distinction challenging. Instance-level complexity stems from multiple instances of the same category coexisting in a scene, leading to distractions during grounding. To address these challenges, we propose a novel weakly-supervised grounding approach that explicitly differentiates between categories and instances. In the category-level branch, we utilize extensive category knowledge from a pre-trained external detector to align object proposal features with sentence-level category features, thereby enhancing category awareness. In the instance-level branch, we utilize spatial relationship descriptions from language queries to refine object proposal features, ensuring clear differentiation among objects. These designs enable our model to accurately identify target-category objects while distinguishing instances within the same category. Compared to previous methods, our approach achieves state-of-the-art performance on three widely used benchmarks: Nr3D, Sr3D, and ScanRef.

3DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignment

TL;DR

This work tackles 3D weakly supervised visual grounding by introducing a dual-branch framework that separately handles category-level and instance-level grounding. The category-level branch leverages a pre-trained detector to align object proposals with the sentence's target category while explicitly contrasting easy-to-confuse categories. The instance-level branch differentiates multiple same-category instances by exploiting noun-phrase–proposal similarities and 3D spatial relations, with training signals provided via InfoNCE losses and spatial-grounding cues. Across Nr3D, Sr3D, and ScanRef, the approach achieves state-of-the-art performance among weakly supervised methods, demonstrating strong cross-modal grounding capabilities and robust handling of category ambiguity and instance complexity.

Abstract

The 3D weakly-supervised visual grounding task aims to localize oriented 3D boxes in point clouds based on natural language descriptions without requiring annotations to guide model learning. This setting presents two primary challenges: category-level ambiguity and instance-level complexity. Category-level ambiguity arises from representing objects of fine-grained categories in a highly sparse point cloud format, making category distinction challenging. Instance-level complexity stems from multiple instances of the same category coexisting in a scene, leading to distractions during grounding. To address these challenges, we propose a novel weakly-supervised grounding approach that explicitly differentiates between categories and instances. In the category-level branch, we utilize extensive category knowledge from a pre-trained external detector to align object proposal features with sentence-level category features, thereby enhancing category awareness. In the instance-level branch, we utilize spatial relationship descriptions from language queries to refine object proposal features, ensuring clear differentiation among objects. These designs enable our model to accurately identify target-category objects while distinguishing instances within the same category. Compared to previous methods, our approach achieves state-of-the-art performance on three widely used benchmarks: Nr3D, Sr3D, and ScanRef.
Paper Structure (15 sections, 3 equations, 5 figures, 2 tables)

This paper contains 15 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. 3D Supervised Visual Grounding
  4. 3D Weakly Supervised Visual Grounding
  5. METHOD
  6. Model Architecture
  7. Feature Alignment
  8. Category-level Branch
  9. Instance-level Branch
  10. Training Objectives and Inference
  11. EXPERIMENT
  12. Implementation Details
  13. Compare with Baselines
  14. Ablation Study and Analysis
  15. CONCLUSION AND LIMITATION

Figures (5)

  • Figure 1: "find the armchair that is next to the table, which is also facing the window."
  • Figure 2: "end table" v.s. "table"
  • Figure 3: "Choose the toilet paper on the wall, closest to the sinks."
  • Figure 5: Illustration of our overall framework. After obtaining features of both modalities through encoder and fusion modules, we adopt category-level and instance-level branched to realize category and instance identification. Both branches contribute jointly in inference under decision logic.
  • Figure 6: Visualization. We present the predicted results at both category and instance levels, demonstrating their effectiveness