CL-Splats: Continual Learning of Gaussian Splatting with Local Optimization

TL;DR

CL-Splats introduces a continual-learning framework for Gaussian Splatting that updates 3D scene reconstructions in a localized, efficient manner using sparse time-sliced views. It detects 2D changes with a DINOv2-based mask, lifts those changes into 3D via majority voting, and performs exact, region-limited optimization within dynamically pruned spheres. The approach yields higher reconstruction quality and faster convergence than state-of-the-art baselines like CL-NeRF and GaussianEditor, while enabling history recovery and batched updates with modest storage. Across synthetic and real-world datasets, CL-Splats demonstrates robust handling of object additions, removals, and movements, making it well-suited for mixed reality, robotics, and embodied AI tasks where up-to-date 3D reconstructions are critical.

Abstract

In dynamic 3D environments, accurately updating scene representations over time is crucial for applications in robotics, mixed reality, and embodied AI. As scenes evolve, efficient methods to incorporate changes are needed to maintain up-to-date, high-quality reconstructions without the computational overhead of re-optimizing the entire scene. This paper introduces CL-Splats, which incrementally updates Gaussian splatting-based 3D representations from sparse scene captures. CL-Splats integrates a robust change-detection module that segments updated and static components within the scene, enabling focused, local optimization that avoids unnecessary re-computation. Moreover, CL-Splats supports storing and recovering previous scene states, facilitating temporal segmentation and new scene-analysis applications. Our extensive experiments demonstrate that CL-Splats achieves efficient updates with improved reconstruction quality over the state-of-the-art. This establishes a robust foundation for future real-time adaptation in 3D scene reconstruction tasks.

Figures (13)

• Figure 1: We introduce CL-Splats, a simple yet effective approach for continual learning of Gaussian Splats. Our method efficiently updates scenes using a small, incremental set of views focused on changed regions. It achieves high-quality reconstructions while significantly outperforming existing continual learning methods for novel view synthesis in both reconstruction accuracy and optimization speed.
• Figure 2: Overview. Starting from an existing reconstruction, we estimate the changed regions in 2D and then lift that information into 3D. After that we only optimize a local region around the changed parts. Our optimization is faster while obtaining the same gradients as 3DGS.
• Figure 3: Sphere Pruning. We first obtain a dynamic rendering mask. Then we use this to efficiently compute the gradients. Finally, we prune gaussians that leave the 3D region of the change.
• Figure 4: Qualitative Comparison on All Datasets. This figure shows renders of our method compared to 3DGS and CL-NeRF. The views are from the test-trajectory but here we show an angle that contains the change. The red crop always shows the area that has changed while the orange crop contains an unchanged region. We can see that our method faithfully produces high quality reconstructions while 3DGS destroys unconstrained regions and CL-NeRF gives less detailed reconstructions.
• Figure 5: Visual Comparison of Ablated Components. The top row shows the optimized scene. The bottom shows the L2 difference multiplied by 50 for improved visibility.