Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Discrimination for Self-Supervised Learning on Point Clouds

Haotian Liu, Mu Cai, Yong Jae Lee

TL;DR

MaskPoint introduces a discriminative masked pretraining framework for 3D point clouds that uses occupancy-based binary classification to overcome the distribution mismatch that hampers reconstruction-based SSL. By partitioning points into masked and unmasked groups, encoding the visible groups with a Transformer, and employing a cross-attention decoder to distinguish real masked queries from randomly sampled fake points, the method learns rich, transferable representations without relying on a reconstruction objective. Evaluated across 3D classification, segmentation, few-shot learning, and detection on ShapeNet, ModelNet40, ScanObjectNN, and ScanNet, MaskPoint achieves state-of-the-art results and, notably, offers substantial pretraining speedups (e.g., 4.1x on ScanNet) over prior Transformer-based baselines. The approach demonstrates that a standard Transformer can surpass specialized 3D backbones when trained with a discriminative, occupancy-based objective, providing a scalable and effective path for self-supervised 3D understanding.

Abstract

Masked autoencoding has achieved great success for self-supervised learning in the image and language domains. However, mask based pretraining has yet to show benefits for point cloud understanding, likely due to standard backbones like PointNet being unable to properly handle the training versus testing distribution mismatch introduced by masking during training. In this paper, we bridge this gap by proposing a discriminative mask pretraining Transformer framework, MaskPoint}, for point clouds. Our key idea is to represent the point cloud as discrete occupancy values (1 if part of the point cloud; 0 if not), and perform simple binary classification between masked object points and sampled noise points as the proxy task. In this way, our approach is robust to the point sampling variance in point clouds, and facilitates learning rich representations. We evaluate our pretrained models across several downstream tasks, including 3D shape classification, segmentation, and real-word object detection, and demonstrate state-of-the-art results while achieving a significant pretraining speedup (e.g., 4.1x on ScanNet) compared to the prior state-of-the-art Transformer baseline. Code is available at https://github.com/haotian-liu/MaskPoint.

Masked Discrimination for Self-Supervised Learning on Point Clouds

TL;DR

MaskPoint introduces a discriminative masked pretraining framework for 3D point clouds that uses occupancy-based binary classification to overcome the distribution mismatch that hampers reconstruction-based SSL. By partitioning points into masked and unmasked groups, encoding the visible groups with a Transformer, and employing a cross-attention decoder to distinguish real masked queries from randomly sampled fake points, the method learns rich, transferable representations without relying on a reconstruction objective. Evaluated across 3D classification, segmentation, few-shot learning, and detection on ShapeNet, ModelNet40, ScanObjectNN, and ScanNet, MaskPoint achieves state-of-the-art results and, notably, offers substantial pretraining speedups (e.g., 4.1x on ScanNet) over prior Transformer-based baselines. The approach demonstrates that a standard Transformer can surpass specialized 3D backbones when trained with a discriminative, occupancy-based objective, providing a scalable and effective path for self-supervised 3D understanding.

Abstract

Masked autoencoding has achieved great success for self-supervised learning in the image and language domains. However, mask based pretraining has yet to show benefits for point cloud understanding, likely due to standard backbones like PointNet being unable to properly handle the training versus testing distribution mismatch introduced by masking during training. In this paper, we bridge this gap by proposing a discriminative mask pretraining Transformer framework, MaskPoint}, for point clouds. Our key idea is to represent the point cloud as discrete occupancy values (1 if part of the point cloud; 0 if not), and perform simple binary classification between masked object points and sampled noise points as the proxy task. In this way, our approach is robust to the point sampling variance in point clouds, and facilitates learning rich representations. We evaluate our pretrained models across several downstream tasks, including 3D shape classification, segmentation, and real-word object detection, and demonstrate state-of-the-art results while achieving a significant pretraining speedup (e.g., 4.1x on ScanNet) compared to the prior state-of-the-art Transformer baseline. Code is available at https://github.com/haotian-liu/MaskPoint.
Paper Structure (49 sections, 3 equations, 7 figures, 8 tables)

This paper contains 49 sections, 3 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformers.
  4. Self-supervised Learning.
  5. Mask based Pretraining.
  6. Approach
  7. Masked Point Discrimination
  8. Discarding Ambiguous Points.
  9. 3D Point Patchification.
  10. Transformer Architecture.
  11. An Information Theoretic Perspective.
  12. Why not reconstruction, as in MAE?
  13. Experiments
  14. Pretraining Datasets.
  15. Transformer Encoder.
  16. ...and 34 more sections

Figures (7)

  • Figure 1: Main Idea. We randomly partition the point cloud into masked and unmasked sets. We only feed the visible portion of the point cloud into the encoder. Then, a set of real query points are sampled from the masked points, and a set of fake query points are randomly sampled from 3D space. We train the decoder so that it distinguishes between the real and fake points. After pre-training, we discard the decoder and use the encoder for downstream tasks.
  • Figure 2: MaskPoint architecture. We first uniformly sample point groups from the point cloud, and partition them to masked and unmasked. We patchify the visible point groups to token embeddings with PointNet and feed these visible tokens into the encoder. Then, a set of real query points are sampled from the masked points, and a set of fake query points are randomly sampled from 3D space. We train the decoder so that it distinguishes between the real and fake points. After pre-training, we discard the decoder and use the encoder for downstream tasks. See Sec. \ref{['sec:dmpm']} for details.
  • Figure 3: Reconstruction results. By reformulating reconstruction as a discriminative occupancy classification task, we achieve a similar learning objective to generative reconstruction while being robust to point sampling variance. Even with a high 90% mask ratio, our approach recovers the overall shape of the original point cloud, without overfitting. Visualization toolkit: Pointflow pointflow.
  • Figure 4: Reconstruction results. We densely perform the discriminative occupancy classification task in 3D space, and visualize the predicted occupancy probability. By varying the confidence threshold $\hat{p}$, we show that our model is able to predict a continuous probability distribution of the occupancy function.
  • Figure 5: A closer look at occupancy distribution. Although there are no points present in both red and purple regions of the masked point cloud, the reconstructed probability distribution correctly reflects that of the original point cloud: a lower occupancy in red region, and a higher occupancy in purple region.
  • ...and 2 more figures