Weakly Supervised Point Clouds Transformer for 3D Object Detection

TL;DR

This work tackles the high annotation cost of 3D point-cloud data by proposing a weakly supervised framework for 3D object detection (WSPCT3D). The approach combines an Unsupervised Voting Proposal Module (UVPM) that selects high-quality anchor points with a teacher–student distillation setup, where a ResNet- and Transformer-based fusion backbone processes a dense front-view XYZ-map. A pretrained image-based teacher guides the student without 3D ground-truth, enabling effective learning on KITTI. Empirical results show state-of-the-art performance among weakly supervised methods, validating the benefit of incorporating UVPM, global-local fusion, and cross-domain supervision for 3D detection without full 3D labels.

Abstract

The annotation of 3D datasets is required for semantic-segmentation and object detection in scene understanding. In this paper we present a framework for the weakly supervision of a point clouds transformer that is used for 3D object detection. The aim is to decrease the required amount of supervision needed for training, as a result of the high cost of annotating a 3D datasets. We propose an Unsupervised Voting Proposal Module, which learns randomly preset anchor points and uses voting network to select prepared anchor points of high quality. Then it distills information into student and teacher network. In terms of student network, we apply ResNet network to efficiently extract local characteristics. However, it also can lose much global information. To provide the input which incorporates the global and local information as the input of student networks, we adopt the self-attention mechanism of transformer to extract global features, and the ResNet layers to extract region proposals. The teacher network supervises the classification and regression of the student network using the pre-trained model on ImageNet. On the challenging KITTI datasets, the experimental results have achieved the highest level of average precision compared with the most recent weakly supervised 3D object detectors.

Figures (4)

• Figure 1: Overview of the Weakly Supervised Point Clouds Transformer for 3D Object Detection. There are three key points: firstly, in the teacher network, the input images are fused by CNN and the 2D proposal boxes generated by the UVPM module based on the normalized point clouds density [15] to generate 2D object bounding boxes. Secondly, the input point clouds are projected into a 2D point clouds XYZ map. The fusion module in the student network is fused with the 2D proposal boxes generated by the module from UVPM. Finally, the high-quality 3D object bounding boxes are obtained under the supervision of the teacher network.
• Figure 2: The fusion of ResNet and Self-Attention in the student network as shown in (a). The expanded structure of the Identity-block in ResNet is shown in (b), and the expanded structure of the Conv2d-block in ResNet is shown in (c).
• Figure 3: The structure of an encoder layer is shown in (a), the single-head attention mechanism of the encoder layer is shown in (b).
• Figure 4: There are three different scenes from left to right, in each scene the prediction 3D bounding boxes in the middle are the Lidar version of the VS3D, and the prediction 3D bounding boxes at the bottom are our WSPCT3D: UPM+ResNet50+SA4(4) method.