PV-SSD: A Multi-Modal Point Cloud Feature Fusion Method for Projection Features and Variable Receptive Field Voxel Features

TL;DR

PV-SSD tackles the information loss inherent in single-view projection and downsampling by introducing a dual-branch architecture that separately extracts projection features from a BEV-map and variable receptive field voxel features, then fuses them with MSSFA. The VR-VFE module preserves crucial local points during voxel downsampling, while the Re-voxelization step aligns voxel and BEV resolutions, enabling effective multi-modal fusion. Together with a refined detection head and loss, PV-SSD achieves competitive performance on KITTI and ONCE, especially improving small or sparse objects such as cyclists. The work demonstrates that multi-modal fusion and weight-based feature point sampling can significantly mitigate information loss and enhance 3D object detection in autonomous-driving scenarios.

Abstract

LiDAR-based 3D object detection and classification is crucial for autonomous driving. However, real-time inference from extremely sparse 3D data is a formidable challenge. To address this problem, a typical class of approaches transforms the point cloud cast into a regular data representation (voxels or projection maps). Then, it performs feature extraction with convolutional neural networks. However, such methods often result in a certain degree of information loss due to down-sampling or over-compression of feature information. This paper proposes a multi-modal point cloud feature fusion method for projection features and variable receptive field voxel features (PV-SSD) based on projection and variable voxelization to solve the information loss problem. We design a two-branch feature extraction structure with a 2D convolutional neural network to extract the point cloud's projection features in bird's-eye view to focus on the correlation between local features. A voxel feature extraction branch is used to extract local fine-grained features. Meanwhile, we propose a voxel feature extraction method with variable sensory fields to reduce the information loss of voxel branches due to downsampling. It avoids missing critical point information by selecting more useful feature points based on feature point weights for the detection task. In addition, we propose a multi-modal feature fusion module for point clouds. To validate the effectiveness of our method, we tested it on the KITTI dataset and ONCE dataset.

Figures (7)

• Figure 1: An illustration of (a) Point Cloud, (b) Point Cloud after voxelization, (c) RGB Image, (d) BEV-map, (e) Density-map, (f) Hight-map, and (g) Intensity-map. Each voxel in (b) contains 12 points, and the size of the voxel is $\left[0.1m,0.1m, 0.125m \right]$. Density-map, Height-map, and Intensity-map are the three channels of BEV-map. Each pixel in the density map, height map, and intensity map represents the normalized density of the point cloud, the maximum height of the point cloud, and the maximum intensity of the point cloud in that grid point, respectively.
• Figure 2: Overview of the PV-SSD structure. Where ‘C’ denotes concatenation operation (stitching by feature dimension). The point cloud data are transformed into BEV-map and voxel as input, the BEV-map feature is extracted with ConvNeXt-Tiny, and the voxel features are extracted with VR-VFE proposed in this paper. Multi-modal feature fusion is performed in MSSFA. Finally, the 3D detection is done in SSD Detection Head. ‘$0.1\times0.1\times4$’ refers to the size of each grid corresponding to reality after meshing the point cloud. ‘$608 \times 608 \times 3$’ refers to the size of the BEV-map generated by the projection.
• Figure 3: Overview of the VR-VFE structure. The red dashed box part is SVFE module; the blue dashed box part is VFE module; the green dashed box part is PFW-Net; the yellow dashed box part is Re-voxelization layer. In ‘$C\times V\times N$’, ‘$V$’ denotes the number of voxels, ‘$N$’ denotes that each voxel contains N point-wise, and ‘$C$’ denotes the number of features in each point-wise.
• Figure 4: Feature fusion in the backbone. The blue dashed box shows the process of reconstructing the voxels into a 4D tensor. The green dashed box part transforms the reconstructed 4D tensor into a voxel feature map in the bird's eye view. The yellow dashed box part fuses the BEV-map features with the Voxel-BEV features for feature fusion.
• Figure 5: Overview of the MSSFA structure. The purple box is the feature fusion part of Voxel-BEV features and BEV-map features; the blue box is the adjustment part for adjusting the feature map size and feature number; the yellow box is the attention fusion module.