MS23D: A 3D Object Detection Method Using Multi-Scale Semantic Feature Points to Construct 3D Feature Layer

TL;DR

MS$^{2}$3D introduces a two-stage LiDAR 3D detector that combats point-cloud sparsity and hollowness by constructing a dense 3D feature layer from multi-branch voxel features and by learning distance-aware point sampling and centroid-oriented offsets. The framework combines Multi-Scale Voxelization, a 3D Encoder, and a 2D Encoder with Semantic Feature Aggregation, leveraging Center Vote to align deep semantic points toward object centroids while preserving shallow geometric surface points. Through ablations on Center Vote schemes and distance-weighted sampling, the method demonstrates improved detection for small/low-reflectivity objects (e.g., Cyclists, Pedestrians) and robust performance in sparse scenes on KITTI and ONCE, with real-time-like speed on a modest GPU. The work advances voxel-based 3D detection by enriching the 3D feature layer with multi-scale semantic cues and targeted feature-point relocation, enabling more accurate box refinement in challenging autonomous-driving environments.

Abstract

LiDAR point clouds can effectively depict the motion and posture of objects in three-dimensional space. Many studies accomplish the 3D object detection by voxelizing point clouds. However, in autonomous driving scenarios, the sparsity and hollowness of point clouds create some difficulties for voxel-based methods. The sparsity of point clouds makes it challenging to describe the geometric features of objects. The hollowness of point clouds poses difficulties for the aggregation of 3D features. We propose a two-stage 3D object detection framework, called MS23D. (1) We propose a method using voxel feature points from multi-branch to construct the 3D feature layer. Using voxel feature points from different branches, we construct a relatively compact 3D feature layer with rich semantic features. Additionally, we propose a distance-weighted sampling method, reducing the loss of foreground points caused by downsampling and allowing the 3D feature layer to retain more foreground points. (2) In response to the hollowness of point clouds, we predict the offsets between deep-level feature points and the object's centroid, making them as close as possible to the object's centroid. This enables the aggregation of these feature points with abundant semantic features. For feature points from shallow-level, we retain them on the object's surface to describe the geometric features of the object. To validate our approach, we evaluated its effectiveness on both the KITTI and ONCE datasets.

Figures (12)

• Figure 1: Overview of voxel-based 3D object detection methods. The blue box represents the basic structure of one-stage methods, while the combination of the blue and yellow box represents the basic structure of two-stage methods.
• Figure 2: Overview of the MS$^{2}$3D structure. Here, $\left \{S_{i}\right \} _{1-4}$ denotes the Sparse Voxel Features of different sizes generated by the Multi-Scale Voxelization. Correspondingly,$\left \{F_{i}\right \} _{1-4}$ represents the Sparse Voxel Features generated by the 3D encoder, and $\left \{V_{i}\right \}$$_{1-4}$ denotes the Voxel-wise Feature generated by the Multi-Scale Voxelization.
• Figure 3: Overview of the structure of the feature encoding process in Multi-Scale Voxelization. The Center Vote operation and the calculation of $L_{vote}$ are only applied in the Multi-Scale Voxelization with the largest voxel size. The blue region in the figure represents the Weighted Mean module.
• Figure 4: Overview of the 2D encoder structure. In (a), we depict the architecture of the Base Block within the 2D encoder. Here, BN denotes the Batch Normalization operation. In (b), we present the overall structure of the 2D encoder. Notably, for $i=4$, there is no downsampling branch.
• Figure 5: Overview of the semantic point clouds and the three Center Vote schemes. In (a), the results are visualized from the bird's-eye view, while in (b), the results are visualized in 3D perspective.