FSMDet: Vision-guided feature diffusion for fully sparse 3D detector

TL;DR

FSMDet tackles the sparsity bottleneck in fully sparse 3D detection by introducing vision-guided feature diffusion that leverages RGB cues to guide diffusion in a LiDAR-based pipeline. The method splits diffusion into a Shape Recover Layer that recovers visible object contours and a Self Diffusion Layer that propagates features toward the object centers, using deformable attention for robust cross-modal fusion. Empirical results on nuScenes show FSMDet achieves competitive mAP and NDS (around 70.1 mAP and 72.1 NDS) while offering substantial speedups over SOTA fusion-based models, thanks to its sparse design. The work demonstrates the viability and practical benefits of RGB-LiDAR fusion in fully sparse detectors, with potential for real-time deployment and future data-fusion research.

Abstract

Fully sparse 3D detection has attracted an increasing interest in the recent years. However, the sparsity of the features in these frameworks challenges the generation of proposals because of the limited diffusion process. In addition, the quest for efficiency has led to only few work on vision-assisted fully sparse models. In this paper, we propose FSMDet (Fully Sparse Multi-modal Detection), which use visual information to guide the LiDAR feature diffusion process while still maintaining the efficiency of the pipeline. Specifically, most of fully sparse works focus on complex customized center fusion diffusion/regression operators. However, we observed that if the adequate object completion is performed, even the simplest interpolation operator leads to satisfactory results. Inspired by this observation, we split the vision-guided diffusion process into two modules: a Shape Recover Layer (SRLayer) and a Self Diffusion Layer (SDLayer). The former uses RGB information to recover the shape of the visible part of an object, and the latter uses a visual prior to further spread the features to the center region. Experiments demonstrate that our approach successfully improves the performance of previous fully sparse models that use LiDAR only and reaches SOTA performance in multimodal models. At the same time, thanks to the sparse architecture, our method can be up to 5 times more efficient than previous SOTA methods in the inference process.

Figures (5)

• Figure 1: A Vision-Centric view for LiDAR-RGB fusion in a fully sparse framework. Our proposed FSMDet integrates image features without any parameters for 2D tasks. The feature diffusion is guided by RGB features. Prop. and D.A. stand for proposals and deformable attention. The orange arrows represent the optional steps for models in the same categories.
• Figure 2: The ray casting model for visible part ground truth generation. All calculations are performed in LiDAR(3D space) coordination. Here $c_i$ stand for the location of pixel c in 3D space.
• Figure 3: From sparse lidar signal to visible part ground truth: we first obtain the full shape objects and then reconstruct the surface. With a ray-cast model, we can paint the color information onto the object's surface and filter out the non-visible part.
• Figure 4: Proposed pipeline: We complete the diffusion of features in two steps, the SRLayer uses RGB features to recover the visible part of the contours of objects while the SDLayer diffuses the features to the center of objects. D.A. stand for the deformable attention. The operation related to color features is marked in red.
• Figure 5: Illustration of Shape Recover Layer and Self Diffusion Layer. We use the sparse signal to recover the shape of objects. After Shape recovery, we expand the boundary voxels along the ray direction, the center point will be naturally located in this area.