ReCon-GS: Continuum-Preserved Gaussian Streaming for Fast and Compact Reconstruction of Dynamic Scenes

TL;DR

ReCon-GS tackles real-time online free-viewpoint video reconstruction under storage constraints by introducing Adaptive Hierarchical Motion Representation built on Anchor Gaussians. It decomposes scene dynamics into coarse-to-fine motions across multiple hierarchy levels, with a Dynamic Hierarchy Reconfiguration strategy that periodically reinitializes anchors and preserves temporal coherence via intra-hierarchical inheritance. A storage-aware optimization framework decouples geometry from appearance and uses a two-stage training (deformation fields first, then view-based densification) to balance fidelity and memory. Across three datasets, ReCon-GS achieves ~15% faster training and >50% memory reduction at equivalent rendering quality, outperforming state-of-the-art streaming methods in both quality and robustness. This framework enables practical, scalable dynamic scene reconstruction suitable for real-time streaming and immersive applications, with a flexible fidelity-memory trade-off driven by application needs.

Abstract

Online free-viewpoint video (FVV) reconstruction is challenged by slow per-frame optimization, inconsistent motion estimation, and unsustainable storage demands. To address these challenges, we propose the Reconfigurable Continuum Gaussian Stream, dubbed ReCon-GS, a novel storage-aware framework that enables high fidelity online dynamic scene reconstruction and real-time rendering. Specifically, we dynamically allocate multi-level Anchor Gaussians in a density-adaptive fashion to capture inter-frame geometric deformations, thereby decomposing scene motion into compact coarse-to-fine representations. Then, we design a dynamic hierarchy reconfiguration strategy that preserves localized motion expressiveness through on-demand anchor re-hierarchization, while ensuring temporal consistency through intra-hierarchical deformation inheritance that confines transformation priors to their respective hierarchy levels. Furthermore, we introduce a storage-aware optimization mechanism that flexibly adjusts the density of Anchor Gaussians at different hierarchy levels, enabling a controllable trade-off between reconstruction fidelity and memory usage. Extensive experiments on three widely used datasets demonstrate that, compared to state-of-the-art methods, ReCon-GS improves training efficiency by approximately 15% and achieves superior FVV synthesis quality with enhanced robustness and stability. Moreover, at equivalent rendering quality, ReCon-GS slashes memory requirements by over 50% compared to leading state-of-the-art methods.

Figures (10)

• Figure 1: The proposed ReCon-GS framework for streamable dynamic scene reconstruction achieves superior rendering qualty with substantially reduced storage. The left figures show the high-quality rendering results of ReCon-GS in a streaming fashion. The right figure is the performance comparison with previous SOTA 4d-gs4dgchicomqueensarogs3dgstream, where the radius of circle corresponds to the rendering speed.
• Figure 2: Illustration of our ReCon-GS framework. (a) We begin by generating a high‐quality base 3D Gaussian Splatting (3DGS) representation for the first frame, then embed it into an adaptively hierarchical motion presentation framework via grid‐based farthest‐point sampling. (b) Explicit Motion Composition updates the base 3DGS across successive frames. (c) and (d) Periodically, a Re‐Hierarchization stage accommodates complex object motion while preserving temporal consistency through Intra-hierarchical Deformation Inheritance. Finally, a view‐based densification further refines the 3DGS for high‐quality rendering.
• Figure 3: The PSNR Trend Comparison between Ours and HiCoM hicom across different scenarios.
• Figure 4: Qualitative results on the coffee martini scene in the N3DV Dataset. To ensure a fair comparison, we retrained their official code with the same initial sparse points.
• Figure 5: Qualitative results of our ReCon-GS under different hierarchical levels