Masked Autoencoders for Point Cloud Self-supervised Learning

TL;DR

This work introduces Point-MAE, a masked autoencoder for self-supervised learning on point clouds that addresses location information leakage and uneven information density by masking irregular point patches at a high rate and shifting mask tokens to a lightweight decoder. It uses a pure Transformer backbone with an asymmetric encoder–decoder and a PointNet–based embedding, reconstructing masked coordinates via a Chamfer Distance loss. The method achieves state-of-the-art results on ScanObjectNN and ModelNet40, improves few-shot object classification by 1.5–2.3%, and boosts part segmentation performance, while maintaining pretraining efficiency. The findings support the viability of unified Transformer-based architectures across language, image, and point cloud modalities, driven by modular, modality-specific embeddings and task-specific heads.

Abstract

As a promising scheme of self-supervised learning, masked autoencoding has significantly advanced natural language processing and computer vision. Inspired by this, we propose a neat scheme of masked autoencoders for point cloud self-supervised learning, addressing the challenges posed by point cloud's properties, including leakage of location information and uneven information density. Concretely, we divide the input point cloud into irregular point patches and randomly mask them at a high ratio. Then, a standard Transformer based autoencoder, with an asymmetric design and a shifting mask tokens operation, learns high-level latent features from unmasked point patches, aiming to reconstruct the masked point patches. Extensive experiments show that our approach is efficient during pre-training and generalizes well on various downstream tasks. Specifically, our pre-trained models achieve 85.18% accuracy on ScanObjectNN and 94.04% accuracy on ModelNet40, outperforming all the other self-supervised learning methods. We show with our scheme, a simple architecture entirely based on standard Transformers can surpass dedicated Transformer models from supervised learning. Our approach also advances state-of-the-art accuracies by 1.5%-2.3% in the few-shot object classification. Furthermore, our work inspires the feasibility of applying unified architectures from languages and images to the point cloud.

Figures (5)

• Figure 1: Illustration of masked autoencoding. A portion of input data is masked, then an autoencoder is trained to recover the masked parts from original input data. The encoder in autoencoder is encouraged to learn high-level latent features from unmasked parts.
• Figure 2: Reconstruction examples on ShapeNet validation set. In each group, we show the original input (i.e., ground truth), masked point cloud, and reconstruction result from left to right. The masking ratio is 60%. It can be observed directly that reconstructions of key local features (such as sharp corners) are much worse than reconstructions of less important local features (such as flat surfaces).
• Figure 3: Overall scheme of our Point-MAE. On the left, we show the masking and embedding process. The input cloud is divided into point patches, which are masked randomly and then embedded. Autoencoder pre-training is shown on the right. The encoder only processes visible tokens. Mask tokens are added to the input sequence of the decoder to reconstruct masked point patches.
• Figure 4: Reconstruction results on ShapeNet validation set. The model is pre-trained with a masking ratio of 60% but can generalize well on inputs with different masking ratios. Inputs are shown in the leftmost column. In the following columns, we show the masked input (left) and reconstruction (right) with different masking ratios.
• Figure 5: Reconstructions with different masking strategies. We mainly show three different masking strategies for same inputs (leftmost). In each column, masked inputs (left) and reconstructions (right) are shown. Instances are from ShapeNet validation set.