SMART-PC: Skeletal Model Adaptation for Robust Test-Time Training in Point Clouds

TL;DR

SMART-PC addresses the vulnerability of 3D point cloud classifiers to distribution shifts by introducing a skeleton-based pretraining paradigm that yields robust geometric features. The framework comprises a skeletal prediction branch and a classification branch built on a shared encoder, trained with self-supervised skeletal losses and supervised classification loss. Critically, test-time adaptation can be performed without backpropagation by updating only BatchNorm statistics, enabling real-time performance while maintaining state-of-the-art accuracy on ModelNet40-C, ShapeNet-C, and ScanObjectNN-C. Extensive experiments demonstrate superior robustness and efficiency, with notable gains in online adaptation speed and competitive results under standard adaptation. The approach is implemented on the Point-MAE backbone and shows promise for extending to other 3D tasks such as segmentation and detection.

Abstract

Test-Time Training (TTT) has emerged as a promising solution to address distribution shifts in 3D point cloud classification. However, existing methods often rely on computationally expensive backpropagation during adaptation, limiting their applicability in real-world, time-sensitive scenarios. In this paper, we introduce SMART-PC, a skeleton-based framework that enhances resilience to corruptions by leveraging the geometric structure of 3D point clouds. During pre-training, our method predicts skeletal representations, enabling the model to extract robust and meaningful geometric features that are less sensitive to corruptions, thereby improving adaptability to test-time distribution shifts. Unlike prior approaches, SMART-PC achieves real-time adaptation by eliminating backpropagation and updating only BatchNorm statistics, resulting in a lightweight and efficient framework capable of achieving high frame-per-second rates while maintaining superior classification performance. Extensive experiments on benchmark datasets, including ModelNet40-C, ShapeNet-C, and ScanObjectNN-C, demonstrate that SMART-PC achieves state-of-the-art results, outperforming existing methods such as MATE in terms of both accuracy and computational efficiency. The implementation is available at: https://github.com/AliBahri94/SMART-PC.

Figures (8)

• Figure 1: The blue points represent the sampled points on the surface of spheres created using the skeletal points as centers and their corresponding radii. Each sphere illustrates the local geometric structure defined by the skeletal representation.
• Figure 2: An overview of the SMART-PC framework. The framework integrates skeletal prediction and classification tasks, leveraging skeletal representations to extract robust geometric features. During online adaptation, two strategies are employed: adaptation with the skeletal loss $\mathcal{L}_{skel}$ and backpropagation, and a lightweight, backpropagation-free approach that updates only BatchNorm statistics.
• Figure 3: Frames/Second vs. Accuracy for SMART-PC and MATE during online adaptation on the ScanObjectNN dataset. SMART-PC achieves higher frame rates and accuracy, surpassing the real-time threshold (30 Frames/Second) in the (N-24) setting.
• Figure 4: Effect of Augmentation During Adaptation in Our Method and Comparison with Other Methods on the ScanObjectNN Dataset in Online Mode.
• Figure 5: Impact of Batch Size on Mean Accuracy for the ModelNet-C Dataset in Standard Mode.