MoRGS: Efficient Per-Gaussian Motion Reasoning for Streamable Dynamic 3D Scenes

Abstract

Online reconstruction of dynamic scenes aims to learn from streaming multi-view inputs under low-latency constraints. The fast training and real-time rendering capabilities of 3D Gaussian Splatting have made on-the-fly reconstruction practically feasible, enabling online 4D reconstruction. However, existing online approaches, despite their efficiency and visual quality, fail to learn per-Gaussian motion that reflects true scene dynamics. Without explicit motion cues, appearance and motion are optimized solely under photometric loss, causing per-Gaussian motion to chase pixel residuals rather than true 3D motion. To address this, we propose MoRGS, an efficient online per-Gaussian motion reasoning framework that explicitly models per-Gaussian motion to improve 4D reconstruction quality. Specifically, we leverage optical flow on a sparse set of key views as lightweight motion cues that regularize per-Gaussian motion beyond photometric supervision. To compensate for the sparsity of flow supervision, we learn a per-Gaussian motion offset field that reconciles discrepancies between projected 3D motion and observed flow across views and time. In addition, we introduce a per-Gaussian motion confidence that separates dynamic from static Gaussians and weights Gaussian attribute residual updates, thereby suppressing redundant motion in static regions for better temporal consistency and accelerating the modeling of large motions. Extensive experiments demonstrate that MoRGS achieves state-of-the-art reconstruction quality and motion fidelity among online methods, while maintaining streamable performance.

Figures (6)

• Figure 1: The proposed MoRGS framework for streamable dynamic scene reconstruction achieves superior rendering quality by explicitly modeling per-Gaussian motion. The left figures ((a),(b)) show the high-quality rendering and the corresponding Gaussian motion updates compared to 3dgstreamqueen. The right figure (c) is the performance comparison with previous state-of-the-art methods 3dgstreamqueenswift4dspacetimegaussian4dgchicom.
• Figure 2: Illustration of the MoRGS framework. (a) We incrementally update Gaussian attributes at each time step while jointly modeling per-Gaussian motion between frames. (b) Per-Gaussian motion is guided by sparse motion cues and refined by a per-Gaussian motion offset field to compensate for discrepancies in the sparse motion cues. (c) To identify dynamic Gaussians, we obtain motion masks by thresholding the motion cues and then apply a segmentation model for view consistency. The per-Gaussian motion confidence is learned from these masks to suppress redundant background motion, improve temporal consistency, and concentrate learning on large motions.
• Figure 3: Per-Gaussian Motion Visualization. We visualize per-Gaussian motion under (a) no flow guidance, (b) sparse flow guidance only, (c) sparse flow guidance with motion offset, and (d) sparse flow guidance with both motion offset and motion confidence, while (e) and (f) show the learned motion confidence map and motion offset map, respectively.
• Figure 4: Qualitative Results. A visualization of various scenes in N3DV and Meet Room dataset. We include additional video results in the supplementary material.
• Figure 5: Gaussian Motion Visualization. We observe that motion updates are confined to genuinely dynamic regions, and the recovered per-Gaussian motion closely follows the true scene dynamics.