Contextual Modeling for 3D Dense Captioning on Point Clouds

TL;DR

This work tackles 3D dense captioning by addressing the lack of scene and background contextual information in point clouds. It introduces a coarse-to-fine contextual modeling framework with Global Context Modeling (GCM) and Local Context Modeling (LCM) that leverage clustering-derived superpoints to capture non-object details and background. Through a detector producing superpoints and candidate objects, a context identification module, and a two-layer decoder, the approach fuses global scene context with local neighborhood cues to generate more detailed and accurate object descriptions. Empirical results on ScanRefer and Nr3D show state-of-the-art performance, especially when using CiDEr-based self-critical training, demonstrating the practical value of incorporating contextual information for 3D vision-language tasks. The work lays groundwork for broader 3D language tasks and suggests that richer context can meaningfully improve scene understanding and caption quality in complex point clouds.

Abstract

3D dense captioning, as an emerging vision-language task, aims to identify and locate each object from a set of point clouds and generate a distinctive natural language sentence for describing each located object. However, the existing methods mainly focus on mining inter-object relationship, while ignoring contextual information, especially the non-object details and background environment within the point clouds, thus leading to low-quality descriptions, such as inaccurate relative position information. In this paper, we make the first attempt to utilize the point clouds clustering features as the contextual information to supply the non-object details and background environment of the point clouds and incorporate them into the 3D dense captioning task. We propose two separate modules, namely the Global Context Modeling (GCM) and Local Context Modeling (LCM), in a coarse-to-fine manner to perform the contextual modeling of the point clouds. Specifically, the GCM module captures the inter-object relationship among all objects with global contextual information to obtain more complete scene information of the whole point clouds. The LCM module exploits the influence of the neighboring objects of the target object and local contextual information to enrich the object representations. With such global and local contextual modeling strategies, our proposed model can effectively characterize the object representations and contextual information and thereby generate comprehensive and detailed descriptions of the located objects. Extensive experiments on the ScanRefer and Nr3D datasets demonstrate that our proposed method sets a new record on the 3D dense captioning task, and verify the effectiveness of our raised contextual modeling of point clouds.

Figures (4)

• Figure 1: Different pipelines for 3D dense captioning. (a) The traditional pipeline detects the candidate objects and performs the inter-object relationship modeling to generate the corresponding caption. (b) Our proposed method not only detects the candidate objects but also yields the superpoints depicting the contextual information, and thereafter performs the contextual modeling to generate the resulting caption.
• Figure 2: An overview of our proposed method, which consists of three components: a detector, a context identification module, and a caption decoder. Given a set of point clouds, the detector extracts the superpoints and candidate objects. For all superpoints and candidate objects, the context identification module specifies the target object and selects the neighboring objects and superpoints around the target object. Each layer in our caption decoder consists of two modules, namely GCM and LCM. GCM learns the global features from all objects and superpoints to obtain more complete scene information of the whole point clouds. LCM yields the local features from the neighboring objects and superpoints to enrich each object representation. Finally, the global and local features are fused together to generate the corresponding descriptions.
• Figure 3: The context identification module specifies the target object and selects the neighboring objects and superpoints close to the target object as its context.
• Figure 4: Qualitative results of Scan2Cap and our proposed method. We only highlight the ground truth bounding box (in red) to locate the target object. It can be observed that our method performs superiorly to Scan2Cap in mining spatial relations between objects and generates more detailed and vivid descriptions closer to the human annotations. Best viewed in color.