ReFu: Recursive Fusion for Exemplar-Free 3D Class-Incremental Learning

TL;DR

ReFu tackles exemplar-free 3D class-incremental learning by integrating point clouds and meshes through a fusion backbone and a Recursive Incremental Learning Mechanism (RILM) that updates a regularized auto-correlation matrix to retain prior knowledge. The framework includes RePoint and ReMesh as single-modality baselines and a fusion module with a Pointcloud-guided Mesh Attention Layer, enabling effective cross-modal feature fusion while keeping encoders frozen. The authors provide theoretical grounding via Theorem 1, showing recursive weight updates are equivalent to joint training under memory constraints, and demonstrate superior performance on ModelNet40/Manifold40 and SHREC11 datasets with strong resilience to forgetting. The work is practically impactful for 3D continual learning scenarios where exemplar storage is impractical, offering a scalable, multimodal solution with strong empirical gains.

Abstract

We introduce a novel Recursive Fusion model, dubbed ReFu, designed to integrate point clouds and meshes for exemplar-free 3D Class-Incremental Learning, where the model learns new 3D classes while retaining knowledge of previously learned ones. Unlike existing methods that either rely on storing historical data to mitigate forgetting or focus on single data modalities, ReFu eliminates the need for exemplar storage while utilizing the complementary strengths of both point clouds and meshes. To achieve this, we introduce a recursive method which continuously accumulates knowledge by updating the regularized auto-correlation matrix. Furthermore, we propose a fusion module, featuring a Pointcloud-guided Mesh Attention Layer that learns correlations between the two modalities. This mechanism effectively integrates point cloud and mesh features, leading to more robust and stable continual learning. Experiments across various datasets demonstrate that our proposed framework outperforms existing methods in 3D class-incremental learning.

Figures (3)

• Figure 1: Overall scheme of our single-modality Framework:RePoint and ReMesh, which consist of a frozen encoder, a random projection layer, the RILM memory module, and a final classifier. RILM facilitates continual learning by recursively updating the regularized auto-correlation matrix, ensuring retention of previously learned categories without forgetting. (Sec.\ref{['sec:3.4']}, Sec.\ref{['sec:3.5']}).
• Figure 2: Overview of our ReFu framework. During incremental learning, data flows progressively into the model. As outlined in Section \ref{['sec:3.3']}, our proposed fusion backbone, pre-trained on the ShapeNet dataset and frozen during learning, extracts and fuses features from point clouds and meshes. These fused features are then expanded via a random projection layer and input into the Recursive Incremental Learning Mechanism (RILM). RILM recursively updates the regularized auto-correlation matrix and classifier weights. Only the matrix and weights from the previous phase $(n-1)$ are stored, without retaining any raw data as exemplar.
• Figure 3: (a) $\sim$ (h): Test accuracy $\mathcal{A}_n$ at each incremental stage, ranging from $\mathcal{A}_1^0$ to $\mathcal{A}_N^0$. Here, $n$ represents the current phase and $N$ is the total number of phases. The results are based on the PointMAE / MeshMAE backbone. We report accuracy on point cloud datasets: (1) SHREC11 (Point) and (2) ModelNet40, and mesh datasets: (3) SHREC11 and (4) Manifold40, under 10 phase and $N$ phase settings. (i) shows the test accuracy of our proposed three frameworks, ReFu, RePoint and Remesh.