Continual Learning in 3D Point Clouds: Employing Spectral Techniques for Exemplar Selection

TL;DR

This work tackles continual learning for 3D point clouds by introducing CL3D, a framework that selects representative exemplars via spectral clustering across input points, local features, and global features. By fusing spectral embeddings from multiple domains, CL3D achieve s memory-efficient exemplar selection and improved stability-plasticity balance, delivering state-of-the-art or near-state-of-the-art results on ModelNet40, ShapeNet, and ScanNet with reduced memory. The approach is backbone-agnostic in the input space and demonstrates substantial gains in incremental accuracy, particularly in the final stages, while maintaining a smaller memory footprint. Overall, CL3D advances practical continual learning for real-world 3D perception by leveraging geometry-aware clustering and multi-domain feature fusion.

Abstract

We introduce a novel framework for Continual Learning in 3D object classification. Our approach, CL3D, is based on the selection of prototypes from each class using spectral clustering. For non-Euclidean data such as point clouds, spectral clustering can be employed as long as one can define a distance measure between pairs of samples. Choosing the appropriate distance measure enables us to leverage 3D geometric characteristics to identify representative prototypes for each class. We explore the effectiveness of clustering in the input space (3D points), local feature space (1024-dimensional points), and global feature space. We conduct experiments on the ModelNet40, ShapeNet, and ScanNet datasets, achieving state-of-the-art accuracy exclusively through the use of input space features. By leveraging the combined input, local, and global features, we have improved the state-of-the-art on ModelNet and ShapeNet, utilizing nearly half the memory used by competing approaches. For the challenging ScanNet dataset, our method enhances accuracy by 4.1% while consuming just 28% of the memory used by our competitors, demonstrating the scalability of our approach.

Figures (5)

• Figure 1: t-SNE visualization of global features of the airplane class and exemplars selected by the herding approach Rebuffi_2017_CVPR (left) contrasted with those from our CL3D method (right). Different colors denote different clusters. Our method effectively covers different subcategories of the airplane samples, demonstrating improved exemplar selection compared to the herding approach.
• Figure 2: Overview of CL3D training pipeline. Spectral embeddings may be computed from the input, the local features, and the global features. They can be simply fused together by concatenation and fed to $k$-means for clustering. Finally, the class exemplars are selected based on proximity to each cluster's centroid.
• Figure 3: The output of spectral clustering in the input space for the airplane and cup classes in the ModelNet40 dataset. The different colors within each class represent different clusters. Our model effectively distinguishes the various subtypes of both airplanes and cups.
• Figure 4: The effect of focal loss compared to Cross-Entropy loss on classification accuracy in a continual learning setting using input space spectral features with 5 samples per class on ModelNet40 wu20153d
• Figure 5: Comparing clustering-based exemplar selection with herding and random selection, reporting the average accuracy (left) and the accuracy in the last stage (right) on ModelNet wu20153d.