Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds

TL;DR

<3-5 sentence high-level summary> SpaCap3D addresses 3D dense captioning on point clouds by explicitly modeling spatial relations between objects. It introduces a spatiality-guided Transformer with a token-to-token spatial relation learning objective and an object-centric decoder to generate per-object natural language descriptions. The method achieves state-of-the-art results on ScanRefer and Nr3D/ReferIt3D, showcasing improvements in both captioning and detection, while maintaining efficient training and inference. This spatially aware framework enhances precise, context-rich scene descriptions for applications like robotics and AR in 3D environments.

Abstract

Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to transform objects into descriptions, where we especially investigate the relative spatiality of objects in 3D scenes and design a spatiality-guided encoder via a token-to-token spatial relation learning objective and an object-centric decoder for precise and spatiality-enhanced object caption generation. Evaluated on two benchmark datasets, ScanRefer and ReferIt3D, our proposed SpaCap3D outperforms the baseline method Scan2Cap by 4.94% and 9.61% in CIDEr@0.5IoU, respectively. Our project page with source code and supplementary files is available at https://SpaCap3D.github.io/ .

Figures (10)

• Figure 1: Dense captioning for a target point-cloud object. Target and its surrounding objects are marked in green and red, respectively. (a) Point-based scene input. (b) Detected vision tokens. (c) Neighbor-to-target contribution visualization in our encoder. (d) Target-to-word contribution visualization in our decoder. Detailed explanations including the color scheme used for attention heads, can be found in the supplementary.
• Figure 2: The overview of our proposed method SpaCap3D for spatiality-guided 3D dense captioning. The encoder-decoder framework consists of an object detector to generate object proposals (i.e., tokens), a learnable function $f$ to project coordinates, a token-to-token spatial relation-guided encoder to incorporate relative 3D spatiality into tokens, and a shared object-centric decoder to generate per-object descriptions.
• Figure 3: An example of our 3D spatiality modeling of main-axis spatial relations. With respect to the couch, chair-2 is in the negative half x-axis, positive half y-axis, and on the same floor along z-axis, hence its spatial relation to couch is represented as (-1, +1, 0). As spatial relation is relative, the relation of couch to chair-2 is expressed reversely as (+1, -1, 0).
• Figure 4: Three cases when an object $o_i$ is to the positive direction of another object $o_j$ along x-/y- axis. Top view. The arrow points to the positive direction. $\alpha$ and $\beta$ are the lower and upper area limits, respectively.
• Figure 5: Detailed encoder architecture. AddNorm is omitted for brevity.