Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection

TL;DR

This work proposes a novel backbone, termed Fore-Mamba3D, to focus on the foreground enhancement by modifying Mamba-based encoder, and emphasizes foreground-only encoding and alleviates the distance-based and causal dependencies in the linear autoregression model.

Abstract

Linear modeling methods like Mamba have been merged as the effective backbone for the 3D object detection task. However, previous Mamba-based methods utilize the bidirectional encoding for the whole non-empty voxel sequence, which contains abundant useless background information in the scenes. Though directly encoding foreground voxels appears to be a plausible solution, it tends to degrade detection performance. We attribute this to the response attenuation and restricted context representation in the linear modeling for fore-only sequences. To address this problem, we propose a novel backbone, termed Fore-Mamba3D, to focus on the foreground enhancement by modifying Mamba-based encoder. The foreground voxels are first sampled according to the predicted scores. Considering the response attenuation existing in the interaction of foreground voxels across different instances, we design a regional-to-global slide window (RGSW) to propagate the information from regional split to the entire sequence. Furthermore, a semantic-assisted and state spatial fusion module (SASFMamba) is proposed to enrich contextual representation by enhancing semantic and geometric awareness within the Mamba model. Our method emphasizes foreground-only encoding and alleviates the distance-based and causal dependencies in the linear autoregression model. The superior performance across various benchmarks demonstrates the effectiveness of Fore-Mamba3D in the 3D object detection task.

Figures (7)

• Figure 1: Comparison of previous group-based and group-free Mamba methods with our approach.
• Figure 2: The overall framework of Fore-Mamba3D . (a) Framework: the backbone of Fore-Mamba3D consists of four stages and each contains an instance selection block and a downsampling block. (b) Instance Selection Block: we predict the foreground score for each voxel and select the top-$k$ foreground voxels for further linear encoding. (c) RGSW: we utilize a regional-to-global sliding window process for iterative encoding. (d) SASFMamba: semantic-assisted and state spatial fusion modules are designed to enhance the semantic and geometric recognition of state variables.
• Figure 3: The association matrix in (a) vanilla SSM, (b, c) semantic-assisted strategy with different kernel sizes $K$ and (d) the semantic association labels. We define the semantic labels to follow a Gaussian distribution for the distance.
• Figure 4: The detailed pipeline in the SAF module, which contains the rearrangement, the 1D convolution, and the reverse process.
• Figure 5: The accuracy in foreground scores prediction. We accumulate the scores along the z-axis and visualize the heatmap in the BEV.