Gaussian Mapping for Evolving Scenes

TL;DR

GaME tackles long-term dynamic changes in scenes for novel-view synthesis by online updating a 3D Gaussian Splatting map. It introduces Dynamic Scene Adaptation (DSA) to incrementally integrate additions, removals, and movements outside the camera view, and a Keyframe Management (KM) strategy to mask stale observations while preserving informative context, enabling photorealistic rendering of evolving environments. The method is evaluated on synthetic and real-world datasets, showing substantial improvements in PSNR and depth accuracy over strong baselines and demonstrating robust handling of scene changes with an open-source release. These contributions advance online, change-aware 3D reconstruction and rendering, with practical impact for AR/VR, robotics, and autonomous systems that operate in dynamic environments.

Abstract

Mapping systems with novel view synthesis (NVS) capabilities, most notably 3D Gaussian Splatting (3DGS), are widely used in computer vision and across various applications, including augmented reality, robotics, and autonomous driving. However, many current approaches are limited to static scenes. While recent works have begun addressing short-term dynamics (motion within the camera's view), long-term dynamics (the scene evolving through changes out of view) remain less explored. To overcome this limitation, we introduce a dynamic scene-adaptation mechanism that continuously updates 3DGS to reflect the latest changes. Since maintaining consistency remains challenging due to stale observations that disrupt the reconstruction process, we propose a novel keyframe management mechanism that discards outdated observations while preserving as much information as possible. We thoroughly evaluate Gaussian Mapping for Evolving Scenes (\ours) on both synthetic and real-world datasets, achieving a 29.7\% improvement in PSNR and a 3 times improvement in L1 depth error over the most competitive baseline.

Figures (5)

• Figure 1: GaME is a dense mapping method capable of novel view synthesis. Given a single-camera posed RGBD input stream of an evolving environment, i.e., structural changes happening outside of the camera view but during data capture, GaME reconstructs a consistent up-to-date 3D Gaussian map that can be rendered from previously unseen viewpoints. We showcase the high-fidelity 3D Gaussian map of a real-world environment, where our method effectively detects changes and accurately integrates them into the map. Our model accurately renders fine-grained objects that were moved during the recording, highlighted in blue, yellow, and purple.
• Figure 2: GaME Architecture. Given a segmented RGB-D input stream, the keyframe management system selects keyframes and triggers the dynamic scene adaptation (DSA) module. DSA first integrates newly observed geometry, then removes outdated geometry using covisible keyframes from the 3D Gaussian Splatting map. The keyframe manager then masks stale regions, and the mapping system uses the processed keyframes for local covisibility window optimization.
• Figure 3: Illustration of Add and Remove operations. The input disagrees with the rendered model (red). For disappearance (top), the conflicting region is projected to previous keyframes for removal (red), where complete object consistency is enforced through the object mask (blue). This allows GaME to extract complete objects even under partial observations and occlusion. When a new object appears on the scene (bottom), new Gaussians are added (red) and the area of the new object is marked as stale in previous keyframes to prevent the contamination of the optimization process.
• Figure 4: Visualizations of the outdated keyframes. After detecting the changes, our keyframe management masks out the areas of covisible keyframes observing changed geometry.
• Figure 5: Qualitative Results. Comparison across different long-term scene changes. (A) A black office chair appears in the scene; (B) the toy house and chair are moved, the picture is moved from the table to the shelf; (C) the cutlery on the table is replaced, the painting and the right chair are moved. GaME is the only method that captures the scene evolution and preserves high rendering quality.