3D sans 3D Scans: Scalable Pre-training from Video-Generated Point Clouds

TL;DR

This work addresses the scarcity of large-scale 3D pre-training data by proposing a scan-free pipeline that learns 3D representations from unlabeled videos. It introduces LAM3C, a self-supervised framework built on Sinkhorn-Knopp clustering with a teacher–student EMA and two noise-regularized losses (Laplacian smoothing and noise consistency) to robustly learn from video-generated point clouds. The RoomTours dataset provides a scalable source of indoor VGPC data, enabling pre-training at tens of thousands of scenes. Empirically, LAM3C achieves competitive or superior performance to baselines that use real 3D scans on indoor semantic and instance segmentation, especially under linear probing and limited downstream data, highlighting the viability of unlabeled videos for scalable 3D-SSL.

Abstract

Despite recent progress in 3D self-supervised learning, collecting large-scale 3D scene scans remains expensive and labor-intensive. In this work, we investigate whether 3D representations can be learned from unlabeled videos recorded without any real 3D sensors. We present Laplacian-Aware Multi-level 3D Clustering with Sinkhorn-Knopp (LAM3C), a self-supervised framework that learns from video-generated point clouds from unlabeled videos. We first introduce RoomTours, a video-generated point cloud dataset constructed by collecting room-walkthrough videos from the web (e.g., real-estate tours) and generating 49,219 scenes using an off-the-shelf feed-forward reconstruction model. We also propose a noise-regularized loss that stabilizes representation learning by enforcing local geometric smoothness and ensuring feature stability under noisy point clouds. Remarkably, without using any real 3D scans, LAM3C achieves higher performance than the previous self-supervised methods on indoor semantic and instance segmentation. These results suggest that unlabeled videos represent an abundant source of data for 3D self-supervised learning.

Figures (6)

• Figure 1: Video-generated point clouds (VGPC) match or exceed real 3D scan performance without using any real 3D data. Left: Our method (LAM3C) trained solely on VGPC achieves comparable performance to methods trained on real 3D scans when fine-tuning on 10% of ScanNet. Right: Instance segmentation results on S3DIS show LAM3C outperforms self-supervised methods trained on real 3D scans and matches Sonata which uses both real and synthetic data.
• Figure 2: Laplacian smoothing loss.LAM3C resolves learning instability by using kNN graphs for video-generated point clouds and assigning distance-based weights. This reduces embedding divergence caused by noisy point clouds and promotes learning more stable representations from neighboring point clouds.
• Figure 3: Noise consistency loss. This is a constraint term stating that the same point cloud for the teacher and student models should yield the same embedding in the feature space. This enables more stable clustering even in video-generated point clouds.
• Figure 4: Overview of RoomTours construction. We segment the video into scene sequences using CLIP radford2021learning, and generate video-generated point clouds by inputting each scene sequence into $\pi^3$. Because the scenes differ from real 3D scans in coordinate system, scale, and spacing, we apply a post-processing alignment.
• Figure 5: Visualization of video-generated point clouds from the RoomTours. These are pseudo-scenes generated as video-generated point clouds from unsupervised videos collected from the web. Visually, they achieve very high-quality reconstruction of real-world indoor scenes. However, for example, the leftmost scene contains a large amount of noisy point clouds in the scene due to camera shake in the input video, resulting in blurred object boundaries and cases where walls or floors appear doubled.