MAPo : Motion-Aware Partitioning of Deformable 3D Gaussian Splatting for High-Fidelity Dynamic Scene Reconstruction

TL;DR

MAPo addresses the challenge of high fidelity dynamic scene reconstruction by introducing a motion aware, dynamic score based partitioning of 3D Gaussian Splatting. Highly dynamic Gaussians are recursively partitioned along time with duplicated deformation networks, while low-dynamic Gaussians are treated as static to reduce cost; a cross-frame consistency loss enforces smooth transitions at partition boundaries. The approach yields state-of-the-art rendering quality on real-world dynamic datasets with competitiveStorage and rendering efficiency, particularly in regions of complex motion. This work offers a practical blueprint for scalable, high-fidelity dynamic scene reconstruction applicable to VR, AR, and robotics.

Abstract

3D Gaussian Splatting, known for enabling high-quality static scene reconstruction with fast rendering, is increasingly being applied to multi-view dynamic scene reconstruction. A common strategy involves learning a deformation field to model the temporal changes of a canonical set of 3D Gaussians. However, these deformation-based methods often produce blurred renderings and lose fine motion details in highly dynamic regions due to the inherent limitations of a single, unified model in representing diverse motion patterns. To address these challenges, we introduce Motion-Aware Partitioning of Deformable 3D Gaussian Splatting (MAPo), a novel framework for high-fidelity dynamic scene reconstruction. Its core is a dynamic score-based partitioning strategy that distinguishes between high- and low-dynamic 3D Gaussians. For high-dynamic 3D Gaussians, we recursively partition them temporally and duplicate their deformation networks for each new temporal segment, enabling specialized modeling to capture intricate motion details. Concurrently, low-dynamic 3DGs are treated as static to reduce computational costs. However, this temporal partitioning strategy for high-dynamic 3DGs can introduce visual discontinuities across frames at the partition boundaries. To address this, we introduce a cross-frame consistency loss, which not only ensures visual continuity but also further enhances rendering quality. Extensive experiments demonstrate that MAPo achieves superior rendering quality compared to baselines while maintaining comparable computational costs, particularly in regions with complex or rapid motions.

Figures (9)

• Figure 1: Overview. (a-b) Deformation-based methods often blur details in regions with complex or rapid motion. (c) Our MAPo significantly improves rendering quality in these areas. (d) Ground Truth.
• Figure 2: Rendering results of a single unified model. (a) shows the temporally averaged representation, which is visualized by directly rendering the canonical 3DGs. The regions highlighted in blue in (b) and (c) are visually close to this average. The region highlighted in red in (c) is visually distant from this average.
• Figure 3: An overview of MAPo. (a) 3DGs' deformation process. (b) Compute the dynamic score of 3DGs from history positions during training. (c) High-dynamic 3DGs are recursively temporally partitioned, and low-dynamic ones are deformed and treated as static. (d) Dynamic and static 3DGs are combined for rendering. Losses are computed on the left.
• Figure 4: Effectiveness of temporal partitioning strategy and consistency loss on a toy example. (a) A 3D curve $\mathbf{p}(t)$ simulates a dynamic trajectory. (b) A single point and a single MLP to fit $\mathbf{p}(t)$ for the entire duration; (c) Two points and two corresponding MLPs for two partitioned time segments; (d) Apply a consistency loss to (c) at the partition boundary.
• Figure 5: Qualitative comparisons against existing SOTA methods on the MeetRoom and N3DV dataset.