LoD-Structured 3D Gaussian Splatting for Streaming Video Reconstruction

TL;DR

StreamLoD-GS tackles streaming free-viewpoint video reconstruction under sparse input views by introducing a threefold solution: an anchor- and octree-based LoD-structured Gaussian Splatting representation with hierarchical Gaussian dropout for stable optimization; a GMM-based motion partitioning that concentrates refinement on dynamic regions while freezing static content; and a quantized residual refinement that dramatically reduces storage for dynamic updates. The framework streams training by initializing a canonical LoD space from the first frame and then updating only dynamic anchors and their quantized residuals for subsequent frames, guided by a reconstruction loss combining $L_1$ and $L_{D-SSIM}$. Empirical results on the N3DV and Meet Room datasets show StreamLoD-GS achieves state-of-the-art or competitive quality with substantially lower storage and faster rendering across sparse and dense views, validating its practicality for bandwidth-constrained SFVV applications. The work highlights the potential of LoD-based, anchor-driven Gaussian representations coupled with motion-aware and quantized refinements to enable real-time, high-fidelity streaming of dynamic 3D scenes.

Abstract

Free-Viewpoint Video (FVV) reconstruction enables photorealistic and interactive 3D scene visualization; however, real-time streaming is often bottlenecked by sparse-view inputs, prohibitive training costs, and bandwidth constraints. While recent 3D Gaussian Splatting (3DGS) has advanced FVV due to its superior rendering speed, Streaming Free-Viewpoint Video (SFVV) introduces additional demands for rapid optimization, high-fidelity reconstruction under sparse constraints, and minimal storage footprints. To bridge this gap, we propose StreamLoD-GS, an LoD-based Gaussian Splatting framework designed specifically for SFVV. Our approach integrates three core innovations: 1) an Anchor- and Octree-based LoD-structured 3DGS with a hierarchical Gaussian dropout technique to ensure efficient and stable optimization while maintaining high-quality rendering; 2) a GMM-based motion partitioning mechanism that separates dynamic and static content, refining dynamic regions while preserving background stability; and 3) a quantized residual refinement framework that significantly reduces storage requirements without compromising visual fidelity. Extensive experiments demonstrate that StreamLoD-GS achieves competitive or state-of-the-art performance in terms of quality, efficiency, and storage.

Figures (7)

• Figure 1: StreamLoD-GS enables fast, on-the-fly, high-fidelity reconstruction for Streaming Free-Viewpoint Video. Right: Comparison with existing methods with 5 Training Views on Meet Room li2022streaming dataset.
• Figure 2: Illustration of our proposed StreamLoD-GS. It comprises three major components: (a) LoD-Structured 3DGS with Anchor and Octree; (b) GMM-Based Motion Partitioning; (c) Quantized Residual Refinement. Bottom: The streaming training pipeline. Frames are processed sequentially: the initial frame (Time 0) undergoes LoD-structured 3DGS optimization, while subsequent frames apply motion partitioning and residual refinement to ensure temporally coherent RGB rendering throughout the stream.
• Figure 3: Qualitative Results in N3DV li2022neural and Meet Room li2022streaming Datasets with 3 training views. We demonstrate novel view results produced by 3DGStream sun20243dgstream, QUEEN girish2024queen, and our approach for comparison.
• Figure 4: Quantitative comparison on the N3DV (Scene: Coffee Martini) dataset li2022neural with 3 training views. We compare novel view results on several frames sampled at equal temporal intervals, which are produced by 3DGStream sun20243dgstream, 4DGC hu20254dgc, QUEEN girish2024queen, and our approach.
• Figure 5: Quantitative comparison on the Meet Room (Scene: Discussion) dataset li2022streaming with 3 training views. We compare novel view results on several frames sampled at equal temporal intervals, which are produced by 3DGStream sun20243dgstream, 4DGC hu20254dgc, QUEEN girish2024queen, and our approach.