MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding

TL;DR

MM-Point tackles the challenge of learning robust 3D point cloud representations without labels by harnessing abundant information from rendered 2D multi-views. It introduces a two-stream training framework that jointly optimizes intra-modal 3D self-supervision and inter-modal 2D-3D contrast, augmented by Multi-MLP projection heads and a multi-level augmentation strategy to maximize cross-view mutual information. The approach achieves state-of-the-art results on ModelNet40 and ScanObjectNN for 3D object classification, and demonstrates strong performance in few-shot classification, part segmentation, and semantic segmentation, underscoring its strong transferability. By effectively integrating multi-view information into self-supervised 3D representation learning, MM-Point offers a scalable pre-training paradigm that enhances downstream 3D understanding in real-world scenarios.

Abstract

In perception, multiple sensory information is integrated to map visual information from 2D views onto 3D objects, which is beneficial for understanding in 3D environments. But in terms of a single 2D view rendered from different angles, only limited partial information can be provided.The richness and value of Multi-view 2D information can provide superior self-supervised signals for 3D objects. In this paper, we propose a novel self-supervised point cloud representation learning method, MM-Point, which is driven by intra-modal and inter-modal similarity objectives. The core of MM-Point lies in the Multi-modal interaction and transmission between 3D objects and multiple 2D views at the same time. In order to more effectively simultaneously perform the consistent cross-modal objective of 2D multi-view information based on contrastive learning, we further propose Multi-MLP and Multi-level Augmentation strategies. Through carefully designed transformation strategies, we further learn Multi-level invariance in 2D Multi-views. MM-Point demonstrates state-of-the-art (SOTA) performance in various downstream tasks. For instance, it achieves a peak accuracy of 92.4% on the synthetic dataset ModelNet40, and a top accuracy of 87.8% on the real-world dataset ScanObjectNN, comparable to fully supervised methods. Additionally, we demonstrate its effectiveness in tasks such as few-shot classification, 3D part segmentation and 3D semantic segmentation.

Figures (6)

• Figure 1: Schematic of MM-Point multi-modal contrastive learning. Given 2D multi-views rendered from a 3D object, the model distinguishes the views of the same object from those of different objects in the embedding space, enabling self-supervised learning of deep representation.
• Figure 2: Schematic of the MM-Point architecture. MM-Point carries out intra-modal self-supervised learning in the 3D point cloud (blue path) and cross-modal learning between 2D and 3D (other color paths), aligning 2D multi-view features with 3D point cloud features. To better utilize the information from the 2D multi-views, MM-Point introduces two strategies: 1) Multi-MLP: constructing a multi-level feature space; 2) Multi-level augmentation: establishing multi-level invariance.
• Figure 3: Mutual Information plot for the modal comparison between 3D and 2D. The yellow area signifies mutual information. The figure illustrates the impact of different strategies (a-c) on the contribution to mutual information.
• Figure 4: Schematic of the Multi-MLP strategy. Multi-modal learning enhances point cloud representation through 2D-3D interaction. Multiple 2D features and 3D features are contrasted through a multi-level feature space and adjusted via multi-path backpropagation.
• Figure 5: Schematic of Multi-level Augmentation. Multi-level augmentation are generated in an incremental manner. 2D multi-views correspond to a certain level of augmentation indicated by different colors.