3D-DRES: Detailed 3D Referring Expression Segmentation

TL;DR

Detailed 3D Referring Expression Segmentation (3D-DRES), a new task that provides a phrase to 3D instance mapping, aiming at enhancing fine-grained 3D vision language understanding, is introduced.

Abstract

Current 3D visual grounding tasks only process sentence level detection or segmentation, which critically fails to leverage the rich compositional contextual reasonings within natural language expressions. To address this challenge, we introduce Detailed 3D Referring Expression Segmentation (3D-DRES), a new task that provides a phrase to 3D instance mapping, aiming at enhancing fine-grained 3D vision language understanding. To support 3D-DRES, we present DetailRefer, a new dataset comprising 54,432 descriptions spanning 11,054 distinct objects. Unlike previous datasets, DetailRefer implements a pioneering phrase-instance annotation paradigm where each referenced noun phrase is explicitly mapped to its corresponding 3D elements. Additionally, we introduce DetailBase, a purposefully streamlined yet effective baseline architecture that supports dual-mode segmentation at both sentence and phrase levels. Our experimental results demonstrate that models trained on DetailRefer not only excel at phrase-level segmentation but also show surprising improvements on traditional 3D-RES benchmarks.

Figures (6)

• Figure 1: Illustration of 3D visual grounding tasks. (a) 3D Referring Expression Comprehension (3D-REC). (b) 3D Referring Expression Segmentation (3D-RES). (c) Generalized 3D Referring Expression Segmentation (3D-GRES). (d) Detailed 3D Referring Expression Segmentation (3D-DRES).
• Figure 2: Example of DetailRefer.
• Figure 3: Category distribution of phrases in DetailRefer.
• Figure 4: The overview of the Detailed 3D Referring Expression Segmentation Baseline (DetailBase).
• Figure 5: Comparison of visualization results. The 3D-STMN results are the model predictions after adaptation.