Sonata: Self-Supervised Learning of Reliable Point Representations

TL;DR

Sonata addresses a reliability gap in self-supervised 3D point-cloud learning by identifying a geometric shortcut that ties representations to simple spatial cues. It introduces an encoder-only, self-distillation framework that uses coarse spatial losses, masked inputs, and progressive difficulty to prevent shortcut collapse, trained on 140k scenes with a PTv3 backbone. The results show dramatic gains in linear probing on ScanNet (e.g., from 21.8% to 72.5%), strong zero-shot semantic groupings, and state-of-the-art performance after full fine-tuning across indoor and outdoor perception tasks, with exceptional data and parameter efficiency. The work also demonstrates complementary strengths with image SSL and reveals potential for cross-modal distillation and video-based scale-up, while outlining avenues for future improvements.

Abstract

In this paper, we question whether we have a reliable self-supervised point cloud model that can be used for diverse 3D tasks via simple linear probing, even with limited data and minimal computation. We find that existing 3D self-supervised learning approaches fall short when evaluated on representation quality through linear probing. We hypothesize that this is due to what we term the "geometric shortcut", which causes representations to collapse to low-level spatial features. This challenge is unique to 3D and arises from the sparse nature of point cloud data. We address it through two key strategies: obscuring spatial information and enhancing the reliance on input features, ultimately composing a Sonata of 140k point clouds through self-distillation. Sonata is simple and intuitive, yet its learned representations are strong and reliable: zero-shot visualizations demonstrate semantic grouping, alongside strong spatial reasoning through nearest-neighbor relationships. Sonata demonstrates exceptional parameter and data efficiency, tripling linear probing accuracy (from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1% of the data compared to previous approaches. Full fine-tuning further advances SOTA across both 3D indoor and outdoor perception tasks.

Figures (11)

• Figure 1: Main properties. Sonata leads to reliable 3D self-supervised pretraining with the following superior and emerging properties: 1. Perception. Sonata advances state-of-the-art results across 3D indoor and outdoor perception tasks; 2. Linear probing. With less than 0.2% learnable parameters, Sonata achieves strong and usable linear probing performance which is 3.3$\times$ better than previous SOTA; 3. Decoder-free. Sonata moves beyond the inflexible U-Net structure, offering multi-scale representations that unchain future 3D research from previous architectural constraints. 4. Semantic awareness. Sonata reveals semantic structure in PCA and K-means visualizations. 5. Spatial reasoning. Sonata allows spatial correspondence even under strong augmentations as visualized via feature similarity.
• Figure 2: Geometric shortcut. We select a point on the sofa arm and compute pairwise similarity with other points. The similarity heatmap reveals that CSC hou2021csc collapses to surface normals, and MSC wu2023msc overfits to point height. In contrast, our Sonata can extract higher-level concepts, as can be seen by the high similarity between all sofa arms highlighted in red.
• Figure 3: The geometric shortcut is unique to 3D. When comparing the information contained in 2D image and 3D point cloud data after removing the input feature (indicated by color), it is evident that in images all information is within the input feature. Whereas point clouds retain geometric information in point positions, which is directly utilized by operators. This characteristic leads to what we term geometric shortcuts in 3D SSL.
• Figure 4: What is learned by the hierarchical backbone? We visualize PCA embeddings from different stages of a hierarchical encoder and decoder, trained for semantic segmentation. The encoder captures diverse and dispersed feature patterns, indicating a broad range of information. Notably, as the point cloud becomes coarser, accessible geometric information within point coordinates becomes increasingly global. In contrast, the decoder’s representations are more uniform and structured, suggesting a focus on refining features for task-specific outputs.
• Figure 5: Self-distillation framework of Sonata. (1) Local views (bottom left) and global views (right) are generated with dedicated spatial and photometric augmentations, while masked views are created by randomly masking out grid-based patches from the global views (top left). (2) Embeddings from local and masked views are extracted by the student, with global views processed by the teacher (top). (3) Points from local and masked views are matched with corresponding points in the global views based on their original spatial distance, allowing for the distillation of embeddings from global views to local and masked views (bottom).