Meta-Continual Learning of Neural Fields

TL;DR

Meta-Continual Learning of Neural Fields (MCL-NF) addresses rapid adaptation of neural fields under sequential tasks while mitigating forgetting. It couples spatial-temporal modularization with optimization-based meta-learning and introduces FIM-NeRF, a sample-weighted Fisher-information loss that links to information gain and provides convergence guarantees. The approach demonstrates superior reconstruction quality and learning speed across image, audio, video, and NeRF tasks on six datasets, including city-scale NeRF rendering with reduced parameter count. This yields a scalable, memory-efficient framework suitable for edge devices and real-time, multi-modal neural-field applications.

Abstract

Neural Fields (NF) have gained prominence as a versatile framework for complex data representation. This work unveils a new problem setting termed \emph{Meta-Continual Learning of Neural Fields} (MCL-NF) and introduces a novel strategy that employs a modular architecture combined with optimization-based meta-learning. Focused on overcoming the limitations of existing methods for continual learning of neural fields, such as catastrophic forgetting and slow convergence, our strategy achieves high-quality reconstruction with significantly improved learning speed. We further introduce Fisher Information Maximization loss for neural radiance fields (FIM-NeRF), which maximizes information gains at the sample level to enhance learning generalization, with proved convergence guarantee and generalization bound. We perform extensive evaluations across image, audio, video reconstruction, and view synthesis tasks on six diverse datasets, demonstrating our method's superiority in reconstruction quality and speed over existing MCL and CL-NF approaches. Notably, our approach attains rapid adaptation of neural fields for city-scale NeRF rendering with reduced parameter requirement.

Figures (5)

• Figure 1: Illustration of the transition from traditional MSE loss to FIM-Loss in a neural network. It highlights how the FIM is used to calculate sample-specific weights. These weights are then incorporated into the final loss function, allowing the model to prioritize more informative samples.
• Figure 2: The PSNR comparison between various meta-learning methods over the adaptation steps. Our method demonstrates consistent improvement in PSNR as the number of steps increases, outperforming traditional MAML MAML and OML OML, particularly in longer adaptation sequences in all modalities and datasets.
• Figure 3: Qualitative results of image reconstruction. The first column represents the ground truth, while the remaining columns show the reconstruction results from different methods. The numbers below each image indicate the PSNR with respect to the ground truth. These images correspond to the Best results (those with the highest PSNR among 1 to 4096 steps), as presented in Table \ref{['tab:psnr_results']}. For detailed results, please refer to Table \ref{['tab:psnr_results']}.
• Figure 4: Visualization of Reconstruction Progression Over Steps for "mod" and "mim" Methods. (a) shows the progression of reconstruction for the "mod" (modularized) version, and (b) shows the "mim" (modularized with Fisher Information Maximization) version. Each row presents the reconstructed scene at different optimization steps (Step 1 to Step 4096) along with the ground truth (GT). The visualization demonstrates how the quality of reconstructed details gradually improves as the number of steps increases, highlighting the efficiency of both methods in learning the underlying structure of the scene, with "mim" exhibiting more rapid refinement.
• Figure :

Theorems & Definitions (3)

• Theorem 1: Fisher Information and KL-Divergence
• Theorem 2: Convergence of FIM-SGD
• Theorem 3: Generalization Bound for FIM Loss