Multi-scale Feature Fusion with Point Pyramid for 3D Object Detection

TL;DR

The Point Pyramid RCNN (POP-RCNN), a feature pyramid-based framework for 3D object detection on point clouds, consisting of a Point Pyramid Feature Enhancement (PPFE) module to establish connections across spatial scales and semantic depths for information exchange, is proposed.

Abstract

Effective point cloud processing is crucial to LiDARbased autonomous driving systems. The capability to understand features at multiple scales is required for object detection of intelligent vehicles, where road users may appear in different sizes. Recent methods focus on the design of the feature aggregation operators, which collect features at different scales from the encoder backbone and assign them to the points of interest. While efforts are made into the aggregation modules, the importance of how to fuse these multi-scale features has been overlooked. This leads to insufficient feature communication across scales. To address this issue, this paper proposes the Point Pyramid RCNN (POP-RCNN), a feature pyramid-based framework for 3D object detection on point clouds. POP-RCNN consists of a Point Pyramid Feature Enhancement (PPFE) module to establish connections across spatial scales and semantic depths for information exchange. The PPFE module effectively fuses multi-scale features for rich information without the increased complexity in feature aggregation. To remedy the impact of inconsistent point densities, a point density confidence module is deployed. This design integration enables the use of a lightweight feature aggregator, and the emphasis on both shallow and deep semantics, realising a detection framework for 3D object detection. With great adaptability, the proposed method can be applied to a variety of existing frameworks to increase feature richness, especially for long-distance detection. By adopting the PPFE in the voxel-based and point-voxel-based baselines, experimental results on KITTI and Waymo Open Dataset show that the proposed method achieves remarkable performance even with limited computational headroom.

Figures (4)

• Figure 1: Overview of the POP-RCNN architecture. In the first stage, the input point cloud is first voxelized and encoded with the 3D sparse convolution backbone to output $1\times$, $2\times$, $4\times$, $8\times$ voxel features. The 3D voxel features are flattened to $L\times W\times (H\times C)$, followed by a 2D RPN. The second stage of the network consists of POP-Pool, POP-Fuse and the detection head for box refinement. POP-Pool aggregates features with the guidance from proposal bounding boxes. Feature sources include raw points, voxel features ($1\times$, $2\times$, $4\times$ and $8\times$) and BEV features. POP-Fuse establishes a generalised FPN to fuse multi-scale features with cross-scale connections and shortcut connections. In the box refinement module, the fused features are processed with MLPs to compute the shared features for the regression and classification heads. With distance-aware density confidence scoring (DADCS), the final output of prediction bounding boxes is generated.
• Figure 2: Illustration of the detailed fusion scheme in POP-Fuse. For each feature node ($N^l_d$), the input nodes are determined based on the FPN structure depicted in Fig. \ref{['fig:archi']} with cross-scale connections and shortcut connections. The features from the previous depth layer ($d-1$) are processed with convolutions. The features from the neighbouring level ($l\pm 1$) are resampled with 3NN interpolation as required to match the feature resolution. In addition, the shortcut connections are fed to the current node. All input features are combined with concatenation.
• Figure 3: Illustration of Distance-Aware Density Confidence Scoring. Bounding boxes are predicted with MLPs. According to the predictions, the number of points in each bounding box is summarised and the distance to the sensor is calculated. This information is combined with the shared features according to Equation (\ref{['eq:dcs1']}) and (\ref{['eq:dcs2']}). The classification head produces the calibrated confidence scores with MLPs.
• Figure 4: Visualisations of results on KITTI val split. The original images, images with annotations and point clouds with annotations are presented at the first, second and third rows respectively. Blue boxes indicate predictions and purple boxes indicate the ground truths. In Fig. \ref{['fig:viz-c']} and \ref{['fig:viz-d']}, it is worth noting that some of the target objects are ignored by the ground truth labels, which are correctly located by the proposed method. This may lead to inferior performance due to inconsistent labelling.