MIRACLE3D: Memory-efficient Integrated Robust Approach for Continual Learning on Point Clouds via Shape Model Construction

TL;DR

MIRACLE3D tackles catastrophic forgetting in 3D point-cloud continual learning with a memory-efficient, privacy-preserving approach. It constructs per-class shape models storing a mean shape and a small set of variation modes, and generates synthetic samples for rehearsal, augmented by gradient mode regularization to boost robustness. The method demonstrates state-of-the-art performance on ModelNet40, ShapeNet, and ScanNet while using a fraction of the memory required by prior methods, and it remains backbone-independent by operating in the input space. This work offers practical impact for scalable, privacy-conscious 3D continual learning in robotics, AR/VR, and autonomous systems.

Abstract

In this paper, we introduce a novel framework for memory-efficient and privacy-preserving continual learning in 3D object classification. Unlike conventional memory-based approaches in continual learning that require storing numerous exemplars, our method constructs a compact shape model for each class, retaining only the mean shape along with a few key modes of variation. This strategy not only enables the generation of diverse training samples while drastically reducing memory usage but also enhances privacy by eliminating the need to store original data. To further improve model robustness against input variations, an issue common in 3D domains due to the absence of strong backbones and limited training data, we incorporate Gradient Mode Regularization. This technique enhances model stability and broadens classification margins, resulting in accuracy improvements. We validate our approach through extensive experiments on the ModelNet40, ShapeNet, and ScanNet datasets, where we achieve state-of-the-art performance. Notably, our method consumes only 15% of the memory required by competing methods on the ModelNet40 and ShapeNet, while achieving comparable performance on the challenging ScanNet dataset with just 8.5% of the memory. These results underscore the scalability, effectiveness, and privacy-preserving strengths of our framework for 3D object classification.

Figures (4)

• Figure 1: Comparison of the overview of (a) previous methods and (b) our method. Previous methods depend on a herding strategy Rebuffi_2017_CVPRwelling2009herding, which is less effective for 3D point clouds.
• Figure 2: Overview of our method. For each old class, a compact shape model is created using the mean shape and a few key modes of variation derived from SVD, which are stored in memory. In the next session, new samples are generated by applying perturbations to the mean shape using these modes. These generated samples, along with novel class data, are processed through a shared point cloud network, trained using cross-entropy, knowledge distillation, and Gradient Mode Regularization to enhance robustness and prevent forgetting.
• Figure 3: Illustration of Gradient Mode Regularization. The figure illustrates the parameter spaces $\theta_1^*$ and $\theta_2^*$, highlighting the low-error regions for the old and novel classes, respectively. Without any regularization, model updates (blue arrow) may lead to catastrophic forgetting. Additionally, the absence of regularization can result in sub-optimal updates (red arrow), making the model sensitive to small perturbations, even within the shape space. In contrast, our method (green arrow) employs Gradient Mode Regularization to steer updates toward a more reliable region for the old classes, providing a larger margin than previous methods. This approach improves the model’s stability and robustness against perturbations during continual learning.
• Figure 4: Results of Shape Modeling: Mean shape (left), first two principal modes of variation (center), and examples of generated point cloud samples (right).