Point-DETR3D: Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection

TL;DR

This paper introduces Point-DETR3D, a teacher-student framework for weakly semi-supervised 3D detection, designed to fully capitalize on point-wise supervision within a constrained instance-wise annotation budget, and proposes an explicit positional query initialization strategy to enhance the positional prior.

Abstract

Training high-accuracy 3D detectors necessitates massive labeled 3D annotations with 7 degree-of-freedom, which is laborious and time-consuming. Therefore, the form of point annotations is proposed to offer significant prospects for practical applications in 3D detection, which is not only more accessible and less expensive but also provides strong spatial information for object localization. In this paper, we empirically discover that it is non-trivial to merely adapt Point-DETR to its 3D form, encountering two main bottlenecks: 1) it fails to encode strong 3D prior into the model, and 2) it generates low-quality pseudo labels in distant regions due to the extreme sparsity of LiDAR points. To overcome these challenges, we introduce Point-DETR3D, a teacher-student framework for weakly semi-supervised 3D detection, designed to fully capitalize on point-wise supervision within a constrained instance-wise annotation budget.Different from Point-DETR which encodes 3D positional information solely through a point encoder, we propose an explicit positional query initialization strategy to enhance the positional prior. Considering the low quality of pseudo labels at distant regions produced by the teacher model, we enhance the detector's perception by incorporating dense imagery data through a novel Cross-Modal Deformable RoI Fusion (D-RoI).Moreover, an innovative point-guided self-supervised learning technique is proposed to allow for fully exploiting point priors, even in student models.Extensive experiments on representative nuScenes dataset demonstrate our Point-DETR3D obtains significant improvements compared to previous works. Notably, with only 5% of labeled data, Point-DETR3D achieves over 90% performance of its fully supervised counterpart.

Figures (5)

• Figure 1: Comparison of mAP on teacher and student models. (a) Our teacher model outperforms a large margin over the Point-DETR baseline. (b) Our student model which only uses 10% fully-labeled data achieves comparable performance with the 100% supervised paradigm on the CenterPoint baseline.
• Figure 2: Comparison of Pseudo Label Quality for Point-DETR3D (Ours) and Point-DETR. For the point cloud visualization, we show the prediction and ground truths in red and black, respectively. We can observe that Point-DETR3D can accurately detect far-distant objects, but some of these objects are regressed poorly in Point-DETR, as indicated by the red circle in the figure. Best viewed with color and zoom-in.
• Figure 3: The overall framework of Point-DETR3D. The stage 1 is the training stage of the teacher model which utilizes limited fully-labeled data and paired weakly-labeled data as the training set. The stage 2 represents the pseudo-label generation stage of the teacher model, which transforms the remaining weakly-labeled data into 3D pseudo boxes. The stage 3 is the self-supervised training process of the student model, which is trained with the full set of both labeled and pseudo annotations.
• Figure 4: Illustration of the Point-Centric Deformable RoI Cross-Modal Fusion Module. First, it projects the instance point annotations onto the image and Bird's Eye View (BEV) plane, generating RoI-wise grid points. These grid points function as reference points and can be deformed through learnable coordinate offsets. Subsequently, each object query interacts with the corresponding RoI features to assimilate both LiDAR and imagery information. Finally, an instance-wise cross-attention module is applied to aggregate instance features from both modalities.
• Figure 5: Comparison of Point-DETR3D (Ours) and Point-DETR (P-DETR) in terms of SPNDS, mAP, and Gains in different detection ranges.