Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction

TL;DR

This work addresses the challenge of dynamic scene reconstruction with 3D Gaussian Splatting by introducing motion-aware enhancements that harness optical flow as a 2D motion prior. It introduces cross-dimensional motion correspondence, uncertainty-aware flow augmentation, and a transient-aware deformation auxiliary to improve both iterative and deformation-based 3DGS frameworks. Extensive experiments on multi-view and monocular datasets demonstrate improved rendering quality and efficiency, with reduced model redundancy and better handling of motion. The approach highlights the potential of integrating 2D motion cues into explicit 3D representations, while acknowledging limitations related to motion blur and monocular motion uncertainty for future work.

Abstract

3D Gaussian Splatting (3DGS) has become an emerging tool for dynamic scene reconstruction. However, existing methods focus mainly on extending static 3DGS into a time-variant representation, while overlooking the rich motion information carried by 2D observations, thus suffering from performance degradation and model redundancy. To address the above problem, we propose a novel motion-aware enhancement framework for dynamic scene reconstruction, which mines useful motion cues from optical flow to improve different paradigms of dynamic 3DGS. Specifically, we first establish a correspondence between 3D Gaussian movements and pixel-level flow. Then a novel flow augmentation method is introduced with additional insights into uncertainty and loss collaboration. Moreover, for the prevalent deformation-based paradigm that presents a harder optimization problem, a transient-aware deformation auxiliary module is proposed. We conduct extensive experiments on both multi-view and monocular scenes to verify the merits of our work. Compared with the baselines, our method shows significant superiority in both rendering quality and efficiency.

Figures (6)

• Figure 1: The proposed frameworks for iterative (above) and deformation-based (below) dynamic 3DGS. We add motion-aware enhancement to both paradigms using the designed flow supervision and dynamic map. Moreover, for the deformation-based framework, a motion injector is further employed to handle the motion ambiguities by introducing auxiliary transient information to Gaussian features.
• Figure 2: Illustration of the gap between rendered flow and actual optical flow. Due to a static Gaussian in front, the 2D flow produced by the renderer is squeezed. Not only does this lead to supervision errors, but it also makes the front Gaussian incorrectly drift during optimization.
• Figure 3: Qualitative comparison on PanopticSports d3dgs (left) and Neu3DV dynerf (right). The dense optical flow maps below the images are produced by rendering the Gaussian movements (according to the Middlebury color coding baker2011database). Those render-based flows serve as a useful tool for qualitative visualization, despite being a poor choice of supervision signal.
• Figure 4: Qualitative comparison on the monocular in-the-wild scenes of HyperNeRF hypernerf dataset. We show zoomed-in details of the dynamic region for each result.
• Figure 5: Visualization of the flow map, dynamic maps, and their attention regions. The coarse dynamic map is derived directly from the flow map and shows a quite limited region of interest.