EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding

TL;DR

EDA tackles 3D visual grounding by decoupling language into semantic components and densely aligning each with point-cloud objects. It introduces two dense-aligned losses and an explicit inference mechanism, enabling robust grounding even without object names. The approach achieves state-of-the-art results on ScanRefer and SR3D/NR3D and introduces a challenging VG-w/o-ON task to test robustness. Additional analyses and implementation details support practical deployment and further research directions.

Abstract

3D visual grounding aims to find the object within point clouds mentioned by free-form natural language descriptions with rich semantic cues. However, existing methods either extract the sentence-level features coupling all words or focus more on object names, which would lose the word-level information or neglect other attributes. To alleviate these issues, we present EDA that Explicitly Decouples the textual attributes in a sentence and conducts Dense Alignment between such fine-grained language and point cloud objects. Specifically, we first propose a text decoupling module to produce textual features for every semantic component. Then, we design two losses to supervise the dense matching between two modalities: position alignment loss and semantic alignment loss. On top of that, we further introduce a new visual grounding task, locating objects without object names, which can thoroughly evaluate the model's dense alignment capacity. Through experiments, we achieve state-of-the-art performance on two widely-adopted 3D visual grounding datasets, ScanRefer and SR3D/NR3D, and obtain absolute leadership on our newly-proposed task. The source code is available at https://github.com/yanmin-wu/EDA.

Figures (9)

• Figure 1: Text-decoupled, dense aligned 3D visual grounding. Different colours in the text correspond to different decoupled components. (a) Regular 3D visual grounding: locating objects requires comprehensively considering multiple semantic cues such as appearance attributes, object names, and spatial relationships. (b) Grounding without object name: not mentioning object names, avoiding short-cuts and forcing the model to predict the target based on other attributes.
• Figure 2: Text component decoupling: (a) The query text. (b) Dependency tree analysis. (c) Decoupled into five components.
• Figure 3: The system framework. (a-c): Decouple the input text into several components to acquire the position label $L$ and features $\boldsymbol{t}$ of the decoupled text. (d-e): Transformer-based encoders for cross-modal visual-text feature extraction. (f): Decode proposal features $\mathcal{O'}$ and linearly project them as object position labels $L_{pred}$ and object features $\boldsymbol{o}$, in addition to a box prediction head for regression of the bounding box. (g-h): Visual-text feature dense alignment. Note that the additional 3D object detection procedure is optional.
• Figure 4: Qualitative results with ScanRefer texts. (a-c): Regular 3D visual grounding. (d-e): Grounding without object name.
• Figure 9: Qualitative comparison of the regular 3D VG task. Our method has a superior perception of appearance attributes.