ARM3D: Attention-based relation module for indoor 3D object detection

TL;DR

This work introduces ARM3D, an attention-based relation module for indoor 3D object detection that addresses noisy proposals and ambiguous context by coupling object-aware relational reasoning with a Transformer-style attention mechanism. The two core components—an objectness-filtered relational reasoning step and a learnable attention module—produce selective, meaningful pairwise contexts that are fused with proposal features to boost detection accuracy and robustness. ARM3D is designed as a plug-and-play enhancement for detectors like VoteNet and MLCVNet, achieving substantial improvements on ScanNetV2 and SUN RGB-D, and demonstrating strong generalization across architectures and datasets. The approach highlights the value of distinguishing useful relational information from misleading context and suggests promising directions for extending attention-guided context modeling to broader 3D vision tasks.

Abstract

Relation context has been proved to be useful for many challenging vision tasks. In the field of 3D object detection, previous methods have been taking the advantage of context encoding, graph embedding, or explicit relation reasoning to extract relation context. However, there exists inevitably redundant relation context due to noisy or low-quality proposals. In fact, invalid relation context usually indicates underlying scene misunderstanding and ambiguity, which may, on the contrary, reduce the performance in complex scenes. Inspired by recent attention mechanism like Transformer, we propose a novel 3D attention-based relation module (ARM3D). It encompasses object-aware relation reasoning to extract pair-wise relation contexts among qualified proposals and an attention module to distribute attention weights towards different relation contexts. In this way, ARM3D can take full advantage of the useful relation context and filter those less relevant or even confusing contexts, which mitigates the ambiguity in detection. We have evaluated the effectiveness of ARM3D by plugging it into several state-of-the-art 3D object detectors and showing more accurate and robust detection results. Extensive experiments show the capability and generalization of ARM3D on 3D object detection. Our source code is available at https://github.com/lanlan96/ARM3D.

Figures (9)

• Figure 1: We propose an attention-based relation module (ARM3D) to reason about the most useful semantic relation contexts in 3D object detection. For example, all the objects with boxes in this figure are chairs represented as dots on the left. (a) A chair with the red box is hard to detect due to noise in point clouds and is mistakenly classified as a sofa using equal attention towards other objects. The upper left chairs in this scene have untypical structures, resulting in unclear semantic relations. (b) With unequal attention, this chair can pay more attention to the semantic relationships with objects having similar structures to filter the confusing context and thus can be classified correctly and robustly. Darker orange indicates greater attention.
• Figure 2: The 3D detection pipeline equipped with our ARM3D. With point cloud as input, the backbone networks of current proposal-based 3D detectors produce numerous proposals. These proposals are then sent into our attention-based relation module to extract the fine-grained relation features. These proposals are first selected according to their objectness, and each proposal is matched with several selected proposals to reason about their specific relation context. Darker blue means greater attention and higher weights. The relation features are concatenated with the proposal features together. The combined features of different proposals are used by the detection heads to perform classification and regression. After 3D non-maximum suppression (NMS), the pipeline outputs the final detected bounding boxes.
• Figure 3: Network architecture of ARM3D. With $N$ proposals as input, the objectness module, mainly composed of MLPs, firstly outputs binary labels to select $N_s$ proposals with high objectness. $C$ indicates the feature channels. Each proposal is matched with a certain number of selected proposals at random, and further operations, including matrix subtraction and concatenation, are performed on these object pairs to obtain their differences. Pair-wise features corresponding to the same proposal go through the MLP labelled $g_{\theta}$. The extracted pairs of features are then transposed and fed into other MLPs to reason about semantic or spatial relations: see Section \ref{['relation reasoning']}. Moreover, the original $N\times C$ proposals and pair-wise proposals go through two MLPs named Key and Query MLPs which output the matrices that are multiplied to compute the attention matrix. SoftMax activation follows, which is then multiplied by pair-wise features. Processed by the $f_\phi$ MLP, the relation module outputs relation features for each proposal.
• Figure 4: Semantic and spatial relations. The orange bounding boxes indicate semantic relations, and the blue bounding boxes show spatial relations. (a) Semantic relations in different categories between a chair and the table. (b) Semantic relations in the same category between these two chairs. (c) Vertical spatial relations between the sink and the cabinet. (d) Horizontal spatial relations between the toilet and the garbage bin beside it. Best viewed on screen.
• Figure 5: Qualitative comparison results of 3D object detection on the ScanNetV2 val set. Columns left to right: ground-truth, our method, VoteNet, VoteNet+3DRM. The detailed comparison demonstrates that our method ARM3D enables more accurate and reasonable detection.