Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training

Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzhi Li, Pheng-Ann Heng

TL;DR

This work tackles the limitation of single-modality MAE pre-training by introducing Joint-MAE, a 2D-3D joint masked autoencoder for 3D point cloud pre-training. It integrates hierarchical 2D-3D embeddings with a joint encoder and a mixed-modal decoder, augmented by cross-modal strategies such as local-aligned attention and a cross-reconstruction loss to align semantic and geometric cues across modalities. Pre-training on ShapeNet with a 75% masking regime and a carefully designed architecture yields strong downstream results, including 92.4% accuracy on ModelNet40 with linear SVM and 86.07% on the hardest Split of ScanObjectNN. Ablation studies confirm the importance of mask ratio, transformer block configuration, 2D positional encodings, and pre-training, demonstrating improved 3D tasks like shape classification, few-shot learning, and part segmentation.

Abstract

Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for both 2D and 3D computer vision. However, existing MAE-style methods can only learn from the data of a single modality, i.e., either images or point clouds, which neglect the implicit semantic and geometric correlation between 2D and 3D. In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training. Joint-MAE randomly masks an input 3D point cloud and its projected 2D images, and then reconstructs the masked information of the two modalities. For better cross-modal interaction, we construct our JointMAE by two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and model-specific decoders. On top of this, we further introduce two cross-modal strategies to boost the 3D representation learning, which are local-aligned attention mechanisms for 2D-3D semantic cues, and a cross-reconstruction loss for 2D-3D geometric constraints. By our pre-training paradigm, Joint-MAE achieves superior performance on multiple downstream tasks, e.g., 92.4% accuracy for linear SVM on ModelNet40 and 86.07% accuracy on the hardest split of ScanObjectNN.

Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training

TL;DR

This work tackles the limitation of single-modality MAE pre-training by introducing Joint-MAE, a 2D-3D joint masked autoencoder for 3D point cloud pre-training. It integrates hierarchical 2D-3D embeddings with a joint encoder and a mixed-modal decoder, augmented by cross-modal strategies such as local-aligned attention and a cross-reconstruction loss to align semantic and geometric cues across modalities. Pre-training on ShapeNet with a 75% masking regime and a carefully designed architecture yields strong downstream results, including 92.4% accuracy on ModelNet40 with linear SVM and 86.07% on the hardest Split of ScanObjectNN. Ablation studies confirm the importance of mask ratio, transformer block configuration, 2D positional encodings, and pre-training, demonstrating improved 3D tasks like shape classification, few-shot learning, and part segmentation.

Abstract

Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for both 2D and 3D computer vision. However, existing MAE-style methods can only learn from the data of a single modality, i.e., either images or point clouds, which neglect the implicit semantic and geometric correlation between 2D and 3D. In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training. Joint-MAE randomly masks an input 3D point cloud and its projected 2D images, and then reconstructs the masked information of the two modalities. For better cross-modal interaction, we construct our JointMAE by two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and model-specific decoders. On top of this, we further introduce two cross-modal strategies to boost the 3D representation learning, which are local-aligned attention mechanisms for 2D-3D semantic cues, and a cross-reconstruction loss for 2D-3D geometric constraints. By our pre-training paradigm, Joint-MAE achieves superior performance on multiple downstream tasks, e.g., 92.4% accuracy for linear SVM on ModelNet40 and 86.07% accuracy on the hardest split of ScanObjectNN.
Paper Structure (15 sections, 3 figures, 3 tables)

This paper contains 15 sections, 3 figures, 3 tables.

Table of Contents

  1. Implementation Details
  2. Positional Encodings.
  3. Self-supervised Pre-training.
  4. Linear SVM.
  5. Shape Classification.
  6. Few-shot Classification.
  7. Part Segmentation
  8. Additional Ablation Study
  9. Transformer Blocks.
  10. Mask Ratio.
  11. Positional Encodings.
  12. With and Without Pre-training.
  13. Additional Visualization
  14. Input and Reconstructed Data.
  15. Local-aligned Attention.

Figures (3)

  • Figure 1: Ablations on Mask Ratio. The mask ratio 75% performs the best for Joint-MAE.
  • Figure 2: Visualization of Point Clouds and Depth Maps from Joint-MAE. In each column, we visualize the input point clouds, the masked point clouds, the reconstructed coordinates, the input projected depth maps, the masked depth maps, and the reconstructed depth maps.
  • Figure 3: Visualization of Local-aligned Attention. We visualize the attention scores with local-aligned attention. The query tokens are $A,B$ of 3D. The position in green means the attention weight is zero, while orange denotes nonzero, which are corresponding to the invalid-attention positions and valid-attention positions in Figure 3.