Faster Training, Fewer Labels: Self-Supervised Pretraining for Fine-Grained BEV Segmentation

TL;DR

A two-phase training strategy for fine-grained road marking segmentation that removes full supervision during pretraining and halves the amount of training data during fine-tuning while still outperforming the comparable supervised baseline model.

Abstract

Dense Bird's Eye View (BEV) semantic maps are central to autonomous driving, yet current multi-camera methods depend on costly, inconsistently annotated BEV ground truth. We address this limitation with a two-phase training strategy for fine-grained road marking segmentation that removes full supervision during pretraining and halves the amount of training data during fine-tuning while still outperforming the comparable supervised baseline model. During the self-supervised pretraining, BEVFormer predictions are differentiably reprojected into the image plane and trained against multi-view semantic pseudo-labels generated by the widely used semantic segmentation model Mask2Former. A temporal loss encourages consistency across frames. The subsequent supervised fine-tuning phase requires only 50% of the dataset and significantly less training time. With our method, the fine-tuning benefits from rich priors learned during pretraining boosting the performance and BEV segmentation quality (up to +2.5pp mIoU over the fully supervised baseline) on nuScenes. It simultaneously halves the usage of annotation data and reduces total training time by up to two thirds. The results demonstrate that differentiable reprojection plus camera perspective pseudo labels yields transferable BEV features and a scalable path toward reduced-label autonomous perception.

Figures (6)

• Figure 1: BEVFormer architecture li_bevformer_2022 with the self-supervised pretraining. Left: Camera input. Center: the BEVFormer with its transformer module containing Temporal Self-Attention into History bev and spatial Cross-Attention. Right: Output of the BEVFormer. Bottom right the differentiable reprojection and rendering. Bottom center and left 2D Reconstruction loss comparing the six reconstructed predctions $Pred_{cp}$ with the camera perspective pseudo ground truth $GT_{cp}$.
• Figure 2: Ground truth comparison for pseudo ground truth (a) and NuScenes ground truth (b) in front view (left) and bev (right). For comparison the pseudo ground truth is projected to bev using nuScenes lidar data. (road boundary; lanes; crosswalk).
• Figure 3: Relation between road distance in $m$ and image portion in $\%$. The different colors represent equally long distances each covering one quarter of the total predicted distance.
• Figure 4: Results of our self-supervised pretraining (pretrain), the temporal loss mechanism (temp) and our two phase strategy including temporal loss pretraining and supervised fine-tuning (ours($x$ epochs pretraining)). road boundary; lanes; crosswalk.
• Figure 5: miou progress evaluated on nuScenes ground truth for the supervised baseline, self-supervised pretraining, and our two phase training with different pretraining lengths including the temporal loss.