HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting

TL;DR

HiCoM tackles the inefficiencies of online dynamic-scene reconstruction by combining a perturbation-smoothed, compact initial 3DGS representation with a Hierarchical Coherent Motion mechanism that shares motion parameters across regionally grouped Gaussians. The method supports continual refinement by adding and merging Gaussians and enables parallel training of multiple frames to dramatically reduce wall-time with minimal quality loss. Empirical results on N3DV and Meet Room show about 20% faster learning and 85% storage reduction relative to strong baselines, while maintaining competitive free-viewpoint video quality and >200 FPS rendering. The approach improves robustness and responsiveness for real-world streaming applications, though it remains focused on indoor scenes and relies on a strong initial 3DGS representation.

Abstract

The online reconstruction of dynamic scenes from multi-view streaming videos faces significant challenges in training, rendering and storage efficiency. Harnessing superior learning speed and real-time rendering capabilities, 3D Gaussian Splatting (3DGS) has recently demonstrated considerable potential in this field. However, 3DGS can be inefficient in terms of storage and prone to overfitting by excessively growing Gaussians, particularly with limited views. This paper proposes an efficient framework, dubbed HiCoM, with three key components. First, we construct a compact and robust initial 3DGS representation using a perturbation smoothing strategy. Next, we introduce a Hierarchical Coherent Motion mechanism that leverages the inherent non-uniform distribution and local consistency of 3D Gaussians to swiftly and accurately learn motions across frames. Finally, we continually refine the 3DGS with additional Gaussians, which are later merged into the initial 3DGS to maintain consistency with the evolving scene. To preserve a compact representation, an equivalent number of low-opacity Gaussians that minimally impact the representation are removed before processing subsequent frames. Extensive experiments conducted on two widely used datasets show that our framework improves learning efficiency of the state-of-the-art methods by about $20\%$ and reduces the data storage by $85\%$, achieving competitive free-viewpoint video synthesis quality but with higher robustness and stability. Moreover, by parallel learning multiple frames simultaneously, our HiCoM decreases the average training wall time to $<2$ seconds per frame with negligible performance degradation, substantially boosting real-world applicability and responsiveness.

Figures (7)

• Figure 1: The proposed HiCoM framework for streamable dynamic scene reconstruction achieves competitive rendering quality with significantly shorter training time, faster rendering speed, and substantially reduced storage and transmission requirements. The left figures show results of our HiCoM on N3DV n3dv and Meet Room streamrf datasets, where "Res” indicates video resolution. The right figure is tested on the N3DV n3dv dataset, where the radius of the circle corresponds to the average storage per frame and the method in the top left corner demonstrates the best performance.
• Figure 2: Illustration of our HiCoM framework. We first construct a compact and robust initial 3DGS representation from the first frame of the video streams with the perturbation smoothing strategy (a). Then, we learn each subsequent frame based on the previous 3DGS with our proposed Hierarchical Coherent Motion mechanism (b) and Continual Refinement (c) with additional new Gaussians. This learning process continues until the final frame.
• Figure 3: Qualitative results of Coffee Martini scene. Frames shown are the 1$^{st}$, 61$^{st}$, 121$^{st}$, 181$^{st}$, 241$^{st}$, and 300$^{th}$ from the test video. Red boxes highlight areas with significant temporal motions. Our method achieves temporal coherence more closely matching the ground truth (GT).
• Figure 4: Comparison of initial 3DGS. 3DGStream utlizes the standard 3DGS training, our HiCoM additionally incorporates noise perturbation.
• Figure 5: Qualitative results of additional scenes from N3DV dataset. Frames shown are the 1$^{st}$, 61$^{st}$, 121$^{st}$, 181$^{st}$, 241$^{st}$, and 300$^{th}$ from the test video. Red boxes highlight areas with significant temporal motions.