MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenes

TL;DR

This work tackles 3D dense captioning by explicitly mining multi-order spatial relations in point-cloud scenes. It introduces MORE, a two-part encoder comprising Spatial Layout Graph Convolution to encode basic first-order relations and Object-centric Triplet Attention Graphs to infer higher-order relations, feeding richer cues into a GRU-based caption decoder. Through extensive experiments on the ScanRefer benchmark, MORE consistently outperforms the previous state-of-the-art Scan2Cap, with ablations confirming the critical roles of both SLGC and OTAG and revealing that excessive stacking of graph layers can hurt performance due to over-smoothing. The approach advances descriptive, context-rich captions for 3D scenes and demonstrates strong potential for improved scene understanding in robotics and AR applications; code is publicly available.

Abstract

3D dense captioning is a recently-proposed novel task, where point clouds contain more geometric information than the 2D counterpart. However, it is also more challenging due to the higher complexity and wider variety of inter-object relations contained in point clouds. Existing methods only treat such relations as by-products of object feature learning in graphs without specifically encoding them, which leads to sub-optimal results. In this paper, aiming at improving 3D dense captioning via capturing and utilizing the complex relations in the 3D scene, we propose MORE, a Multi-Order RElation mining model, to support generating more descriptive and comprehensive captions. Technically, our MORE encodes object relations in a progressive manner since complex relations can be deduced from a limited number of basic ones. We first devise a novel Spatial Layout Graph Convolution (SLGC), which semantically encodes several first-order relations as edges of a graph constructed over 3D object proposals. Next, from the resulting graph, we further extract multiple triplets which encapsulate basic first-order relations as the basic unit, and construct several Object-centric Triplet Attention Graphs (OTAG) to infer multi-order relations for every target object. The updated node features from OTAG are aggregated and fed into the caption decoder to provide abundant relational cues, so that captions including diverse relations with context objects can be generated. Extensive experiments on the Scan2Cap dataset prove the effectiveness of our proposed MORE and its components, and we also outperform the current state-of-the-art method. Our code is available at https://github.com/SxJyJay/MORE.

Figures (4)

• Figure 1: The comparison of our proposed Multi-Order RElation mining model (MORE) with the previous method (i.e., Scan2Cap chen2021scan2cap). The core components of our MORE and the Scan2Cap are distinguished with green and blue background, respectively. Scan2Cap treats the inter-object relations as by-products derived from node feature learning in the relational graph, thus the diverse spatial relations are under-explored. While in our MORE, we model such relations via a Spatial Layout Graph Convolution (SLGC) and Object-centric Triplet Attention Graphs (OTAG) to progressively encode more complex spatial relations. For simplicity, we only show the caption of one specific object (in the dashed circle) from different models. It is clear that our method can describe more complex relations.
• Figure 2: The overall framework of our proposed method, which consists of three parts: the detection backbone, the multi-order relation encoder, and the caption decoder. Given a 3D scene represented by point clouds, the detection backbone extracts a set of object proposals. Then, based on the objects, first-order and multi-order spatial relations are progressively encoded through a novel Spatial Layout Graph Convolution (SLGC) and several Object-centric Triplet Attention Graphs (OTAG), respectively. Finally, the OTAG's output, which encapsulates rich spatial relational cues, are served as the context to aid comprehensive caption generation. To keep the figure concise, we omit part of the object nodes and their corresponding captions, as well as the attention calculation of each triplet graph.
• Figure 3: The illustration of dense captions for objects within a whole scene predicted by our method. We distinguish the captions for each object with different colors.
• Figure 4: Statistics of relational words in captions generated by different methods. "simple" and "complex" roughly represents first- and multi-order relations, respectively, and "total" is the sum of all relational words.