D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding

TL;DR

D3Net addresses the lack of integrated, discriminative 3D vision-language understanding by unifying 3D object detection, dense captioning, and visual grounding in a speaker-listener framework. The speaker generates discriminative captions for object proposals, while the listener grounds these captions to refine localization; training combines MLE captioning with reinforcement learning to optimize discriminability via CIDEr rewards and listener feedback, enabling semi-supervised learning on unannotated ScanNet data. The approach achieves state-of-the-art results on ScanRefer for both 3D dense captioning and 3D visual grounding, and ablations demonstrate the importance of a strong detector, end-to-end joint optimization, and the added value of extra unannotated data. This unified, self-critical design reduces overfitting and enhances data efficiency, paving the way for leveraging large-scale unannotated 3D data in language-vision tasks.

Abstract

Recent studies on dense captioning and visual grounding in 3D have achieved impressive results. Despite developments in both areas, the limited amount of available 3D vision-language data causes overfitting issues for 3D visual grounding and 3D dense captioning methods. Also, how to discriminatively describe objects in complex 3D environments is not fully studied yet. To address these challenges, we present D3Net, an end-to-end neural speaker-listener architecture that can detect, describe and discriminate. Our D3Net unifies dense captioning and visual grounding in 3D in a self-critical manner. This self-critical property of D3Net also introduces discriminability during object caption generation and enables semi-supervised training on ScanNet data with partially annotated descriptions. Our method outperforms SOTA methods in both tasks on the ScanRefer dataset, surpassing the SOTA 3D dense captioning method by a significant margin.

Figures (10)

• Figure 1: We introduce D$^3$Net, an end-to-end neural speaker-listener architecture that can detect, describe and discriminate. D$^3$Net also enables semi-supervised training on ScanNet data with partially annotated descriptions.
• Figure 2: Prior work chen2021scan2cap struggle to produce discriminative object captions. Also, captions often appear to be template-based. In contrast, our D$^3$Net generates discriminative object captions.
• Figure 3: D$^3$Net architecture. We input point clouds into the detector to predict object proposals. Then, those proposals are fed into the speaker to generate captions that describes each object. To discriminate the object described by each caption, the listener matches the generated captions with object proposals. The captioning and localization results are back-propagated via REINFORCE williams1992simple as rewards through the dashed lines. D$^3$Net also enables end-to-end training on point clouds with no GT object descriptions (bottom blue block).
• Figure 4: Qualitative results in 3D dense captioning task from Scan2Cap chen2021scan2cap and our method. We underline the inaccurate words and mark the spatially discriminative phrases in bold. Our method qualitatively outperforms Scan2Cap in producing better object bounding boxes and more discriminative descriptions.
• Figure 5: 3D visual grounding results using 3DVG-Transformer zhao20213dvg and our method. 3DVG-Transformer fails to accurately predict object bounding boxes, while our method produces accurate bounding boxes and correctly distinguishes target objects from distractors.