BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios

TL;DR

BEV-MAE tackles the data-labeling bottleneck in outdoor LiDAR 3D detection by introducing a BEV-guided masked autoencoder that aligns pre-training with BEV-based detectors. It uses a lightweight one-layer $3\times3$ decoder, a shared learnable point token to preserve encoder receptive fields, and dual reconstruction targets—point cloud structure via Chamfer Distance and grid density via Smooth-$\ell_1$ loss—to handle sparse outdoor data. Through extensive experiments on Waymo and nuScenes, BEV-MAE achieves state-of-the-art self-supervised and supervised improvements, enhances data efficiency, and demonstrates strong cross-dataset transfer. The approach yields practical gains for autonomous driving by leveraging unlabeled data to boost 3D object detection while reducing pre-training costs.

Abstract

Existing LiDAR-based 3D object detection methods for autonomous driving scenarios mainly adopt the training-from-scratch paradigm. Unfortunately, this paradigm heavily relies on large-scale labeled data, whose collection can be expensive and time-consuming. Self-supervised pre-training is an effective and desirable way to alleviate this dependence on extensive annotated data. In this work, we present BEV-MAE, an efficient masked autoencoder pre-training framework for LiDAR-based 3D object detection in autonomous driving. Specifically, we propose a bird's eye view (BEV) guided masking strategy to guide the 3D encoder learning feature representation in a BEV perspective and avoid complex decoder design during pre-training. Furthermore, we introduce a learnable point token to maintain a consistent receptive field size of the 3D encoder with fine-tuning for masked point cloud inputs. Based on the property of outdoor point clouds in autonomous driving scenarios, i.e., the point clouds of distant objects are more sparse, we propose point density prediction to enable the 3D encoder to learn location information, which is essential for object detection. Experimental results show that BEV-MAE surpasses prior state-of-the-art self-supervised methods and achieves a favorably pre-training efficiency. Furthermore, based on TransFusion-L, BEV-MAE achieves new state-of-the-art LiDAR-based 3D object detection results, with 73.6 NDS and 69.6 mAP on the nuScenes benchmark. The source code will be released at https://github.com/VDIGPKU/BEV-MAE

Figures (3)

• Figure 1: Performance improvement vs. Pre-training time trade-off. All entries are benchmarked by a P40 GPU. The 3D object detector is CenterPoint yin2020center. All models are pre-trained on full Waymo and then fine-tuned with 20% training samples on Waymo.
• Figure 2: Illustration of several masking strategies in the masked modeling. MAE DBLP:conf/cvpr/mae masks non-overlapping image patches. BERT devlin2018bert masks words or sentences. Point-MAE DBLP:journals/corr/pointmae uses furthest point sampling to create overlapping point patches. Our method (right) projects point clouds into a BEV plane, and masks points in non-overlapping BEV grids.
• Figure 3: Overall pipeline of BEV-MAE. We first mask point clouds with the BEV-guided masking strategy. Then, the masked points are replaced with a shared learnable point token. After extracting BEV features by a 3D encoder from visible points, we send the features to a light decoder to reconstruct masked point clouds and predict the point density of masked grids.