PointDico: Contrastive 3D Representation Learning Guided by Diffusion Models

TL;DR

PointDico addresses the challenge of learning robust 3D representations from unordered, sparse point clouds by combining diffusion-based denoising with cross-modal contrastive learning. It introduces a hierarchical pyramid conditional generator (H2 Net and DIP Net) to extract multi-scale geometric priors guided by diffusion, and leverages diffusion-guided knowledge distillation to align 3D features with 2D images and text. The approach achieves state-of-the-art results on ScanObjectNN and ShapeNetPart, with strong transfer and zero-shot performance, demonstrating the value of diffusion-contrastive fusion for 3D representation learning. This work suggests a practical pathway for richer 3D understanding by integrating generative priors with cross-modal supervision.

Abstract

Self-supervised representation learning has shown significant improvement in Natural Language Processing and 2D Computer Vision. However, existing methods face difficulties in representing 3D data because of its unordered and uneven density. Through an in-depth analysis of mainstream contrastive and generative approaches, we find that contrastive models tend to suffer from overfitting, while 3D Mask Autoencoders struggle to handle unordered point clouds. This motivates us to learn 3D representations by sharing the merits of diffusion and contrast models, which is non-trivial due to the pattern difference between the two paradigms. In this paper, we propose \textit{PointDico}, a novel model that seamlessly integrates these methods. \textit{PointDico} learns from both denoising generative modeling and cross-modal contrastive learning through knowledge distillation, where the diffusion model serves as a guide for the contrastive model. We introduce a hierarchical pyramid conditional generator for multi-scale geometric feature extraction and employ a dual-channel design to effectively integrate local and global contextual information. \textit{PointDico} achieves a new state-of-the-art in 3D representation learning, \textit{e.g.}, \textbf{94.32\%} accuracy on ScanObjectNN, \textbf{86.5\%} Inst. mIoU on ShapeNetPart.

Figures (6)

• Figure 1: Schematic illustration of our PointDico. Our PointDico performs point-to-point denoising from high-noise point clouds to pre-train different backbones. During pre-training, latent features guide multi-level point cloud recovery, while knowledge distillation paradigm is used along with 2D images and language for cross-modal contrastive learning, significantly increasing the diversity of training data.
• Figure 2: Pipeline of Our PointDico Framework. The input modals like point cloud, image, and text are encoded into tokens through respective encoders. For the 3D token embeddings, a hierarchical pyramid conditional generator is applied, combining cross-scale features for denoising. Noted that intermediate embeddings are fed into the 3D decoder via cross-attention with stop-gradient operation. These embeddings are then utilized for cross-modal contrastive learning.
• Figure 3: The directed graphical model of the diffusion process for point clouds.$\mathcal{N}$ is the number of points in the point cloud $P^0$.
• Figure 4: The illustration of the proposed mode. By leveraging a dual-channel architecture and a cross-scale interaction mechanism, DIP Net enables an efficient integration of local and global contextual information, while maintaining the flexibility to adapt its feature representation based on external conditions.
• Figure 5: Visualization results on the ShapeNet dataset. Each row visualizes the input point cloud, masked point cloud, and reconstructed point cloud. Even though we mask 60% points, PointDico still produces high-quality point clouds.