DAP-MAE: Domain-Adaptive Point Cloud Masked Autoencoder for Effective Cross-Domain Learning

TL;DR

This work tackles the problem of cross domain generalization for 3D point cloud analysis under data scarcity by introducing DAP-MAE, a domain adaptive masked autoencoder that jointly learns from object, face, and scene domains.It introduces a heterogeneous domain adapter to process multi domain inputs in an adaptation mode during pre training and a fusion mode during fine tuning, plus a domain feature generator that produces domain tokens to guide task specific heads via cross attention.The training objective combines a reconstruction loss $\mathcal{L}_{\text{recon}}$ and a domain contrastive loss $\mathcal{L}_{\text{con}}$, with a total loss $\mathcal{L}_{\text{total}} = w_1 \mathcal{L}_{\text{recon}} + w_2 \mathcal{L}_{\text{con}}$ and hyperparameters $w_1=100$, $w_2=0.001$.Empirical results across object classification, facial expression recognition, part segmentation and 3D object detection demonstrate state of the art or competitive performance with a single cross domain pre training, validating the effectiveness of cross domain adapters and domain feature guidance for robust cross domain 3D representation learning.

Abstract

Compared to 2D data, the scale of point cloud data in different domains available for training, is quite limited. Researchers have been trying to combine these data of different domains for masked autoencoder (MAE) pre-training to leverage such a data scarcity issue. However, the prior knowledge learned from mixed domains may not align well with the downstream 3D point cloud analysis tasks, leading to degraded performance. To address such an issue, we propose the Domain-Adaptive Point Cloud Masked Autoencoder (DAP-MAE), an MAE pre-training method, to adaptively integrate the knowledge of cross-domain datasets for general point cloud analysis. In DAP-MAE, we design a heterogeneous domain adapter that utilizes an adaptation mode during pre-training, enabling the model to comprehensively learn information from point clouds across different domains, while employing a fusion mode in the fine-tuning to enhance point cloud features. Meanwhile, DAP-MAE incorporates a domain feature generator to guide the adaptation of point cloud features to various downstream tasks. With only one pre-training, DAP-MAE achieves excellent performance across four different point cloud analysis tasks, reaching 95.18% in object classification on ScanObjectNN and 88.45% in facial expression recognition on Bosphorus.

Figures (7)

• Figure 1: (a) Existing point cloud MAEs are typically pre-trained and fine-tuned within a single domain, leading to performance degradation when applied to other tasks in different domains. (b) Pre-training MAE directly on cross-domain data could also lead to performance decrease due to the misinterpretation of out-of-domain information. (c) Our DAP-MAE collaboratively learns from point cloud data across various domains using the heterogeneous domain adapter and adopts the domain feature generator to extract diverse domain features, allowing the model to achieve high performance across multiple tasks with one pre-training in a single modality.
• Figure 1: t-SNE visualization of features extracted from three domain datasets: (a) Point cloud features extracted w/o HDA and w/o DFG; (b) Point cloud features extracted w/ HDA and w/o DFG; (c) Point cloud features extracted w/o HDA and DFG; (d) Point cloud features extracted w/ HDA and w/ Domain DFG; (e) Domain features generated after DFG.
• Figure 2: The pipeline of DAP-MAE. During the pre-training, the heterogeneous domain adapter (HDA) is set to adaptation mode to enhance the training of the Transformer-based MAE on cross-domain point cloud datasets, while the domain feature generator (DFG) is trained to extract domain features from the point clouds. In the fine-tuning phase, HDA is set to fusion mode, making comprehensive use of the feature extraction capabilities of the model trained on cross-domain data; meanwhile, the domain features extracted by DFG are used to guide the adaptation of point cloud features for downstream tasks.
• Figure 2: Comparison of classification accuracy during fine-tuning, showing how freezing or unfreezing the parameter of MLPs in HDA affects performance.
• Figure 3: Illustration of the heterogeneous domain adapter (HDA), which consists of three parallel MLPs and two modes. (a) Adaptation mode. In the pre-training, HDA adaptively selects a MLP to process the input token based on its domain. (b) Fusion mode. In the fine-tuning, HDA freezes the parameters in the three MLPs and trains two additional MLPs to generate fusion coefficients, which are used to linearly fuse the output by the three MLPs.