PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm

TL;DR

PonderV2 presents a universal 3D pre-training framework that leverages differentiable neural rendering to learn geometry- and appearance-rich 3D representations across indoor and outdoor settings. By densifying 3D features into volumes and using a rendering decoder to supervise RGB-D outputs, the approach integrates supervision from 2D observations and foundation-model signals, enabling transfer to detection, segmentation, reconstruction, and image synthesis. The method achieves state-of-the-art results on over a dozen benchmarks and demonstrates cross-modality effectiveness across point clouds, multi-view imagery, and BEV representations, suggesting a viable path toward 3D foundation models. While scalable results are promising, the authors note the need to scale datasets and backbones further and to explore additional tasks like reconstruction and robotics control.

Abstract

In contrast to numerous NLP and 2D vision foundational models, learning a 3D foundational model poses considerably greater challenges. This is primarily due to the inherent data variability and diversity of downstream tasks. In this paper, we introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation, thereby establishing a pathway to 3D foundational models. Considering that informative 3D features should encode rich geometry and appearance cues that can be utilized to render realistic images, we propose to learn 3D representations by differentiable neural rendering. We train a 3D backbone with a devised volumetric neural renderer by comparing the rendered with the real images. Notably, our approach seamlessly integrates the learned 3D encoder into various downstream tasks. These tasks encompass not only high-level challenges such as 3D detection and segmentation but also low-level objectives like 3D reconstruction and image synthesis, spanning both indoor and outdoor scenarios. Besides, we also illustrate the capability of pre-training a 2D backbone using the proposed methodology, surpassing conventional pre-training methods by a large margin. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness. Code and models are available at https://github.com/OpenGVLab/PonderV2.

Figures (9)

• Figure 1: The radar chart of PonderV2, showing its effectiveness on over 10 benchmarks in both indoor and outdoor scenarios. Abbreviations: sem. for semantic, ins. for instance, seg. for segmentation, eff. for efficient, L.R. for limited reconstructions, L.A. for limited annotations, obj. for object, rec. for reconstruction, cam. for camera, det. for detection. The SOTA in the figure denotes the state-of-the-art performance with the same backbone as ours on validation sets. The units for different tasks are normalized based on the minimum and maximum performance for each task.
• Figure 2: The proposed unified 3D pre-training approach, termed PonderV2, is directly trained with RGB-D rendered image supervision, and can be used for various 3D downstream applications, e.g., 3D object detection, 3D semantic segmentation, 3D scene reconstruction, and image synthesis.
• Figure 3: The overall pipeline of PonderV2 with different modalities as input.(a) Sparse point cloud as input. A raw point cloud can be constructed from multi-frame RGB-D images, scene scans, or LiDAR. We apply augmentations like masking and grid sampling to create a quantized sparse tensor. A sparse backbone extracts features, serving as the encoder for pre-training and as the pre-trained weight for fine-tuning. These sparse features are then densified to a feature volume. (b) Multi-view images as input. The image encoder takes augmented multi-view images as inputs, giving out the multi-view features. The features are then zero-padded and transformed to obtain the 3D dense feature volume. (c) Rendering decoder as a unified pre-training paradiagm. A shallow dense convolutional network processes the dense feature volume to produce a dense feature volume. The rendering decoder queries this volume and uses shallow MLPs to output each point's SDF and color. Finally, these outputs are integrated to render RGB-D images, which are supervised by ground truth.
• Figure 4: Reconstructed surface by Ponder-RGBD. Our pre-training method can be easily integrated into the task of 3D reconstruction. Despite the sparsity of the input point cloud (only 2% points are used), our method can still recover precise geometric details.
• Figure 5: Illustration of the rendering results. The ground truth RGB and projected point clouds, rendered RGB, and rendered depth are shown on the left, middle, and right, respectively.