Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction

TL;DR

This paper tackles monocular 4D reconstruction by addressing under-constrained dynamics through explicit uncertainty modeling. It introduces USplat4D, which assigns time-varying per-Gaussian uncertainty, builds an uncertainty-aware spatio-temporal graph with key/non-key Gaussians, and performs uncertainty-guided optimization to propagate reliable motion cues. Across DyCheck, DAVIS, and Objaverse, USplat4D consistently improves motion stability and high-quality novel-view synthesis, outperforming strong Gaussian-splatting baselines like SoM and MoSca, particularly at extreme viewpoints. The approach offers a model-agnostic, principled framework for incorporating uncertainty into dynamic 3D reconstruction, with practical implications for robust 4D rendering in occluded or highly dynamic scenes.

Abstract

Reconstructing dynamic 3D scenes from monocular input is fundamentally under-constrained, with ambiguities arising from occlusion and extreme novel views. While dynamic Gaussian Splatting offers an efficient representation, vanilla models optimize all Gaussian primitives uniformly, ignoring whether they are well or poorly observed. This limitation leads to motion drifts under occlusion and degraded synthesis when extrapolating to unseen views. We argue that uncertainty matters: Gaussians with recurring observations across views and time act as reliable anchors to guide motion, whereas those with limited visibility are treated as less reliable. To this end, we introduce USplat4D, a novel Uncertainty-aware dynamic Gaussian Splatting framework that propagates reliable motion cues to enhance 4D reconstruction. Our key insight is to estimate time-varying per-Gaussian uncertainty and leverages it to construct a spatio-temporal graph for uncertainty-aware optimization. Experiments on diverse real and synthetic datasets show that explicitly modeling uncertainty consistently improves dynamic Gaussian Splatting models, yielding more stable geometry under occlusion and high-quality synthesis at extreme viewpoints.

Figures (7)

• Figure 1: We show a challenging case from the DyCheck dataset gao2022monocular, where a person casually rotates a backpack while being captured by a moving monocular camera. The goal is to reconstruct the dynamic object for arbitrary viewpoints and timestamps. The state-of-the-art dynamic Gaussian Splatting methods, e.g., Shape-of-Motion (SoM) som2025, struggle on extreme novel views far from the input trajectory, such as opposite-side views (view 1) or large angle offsets (view 2). We propose USplat4D, an Uncertainty-aware dynamic Gaussian Splatting model that produces more accurate and consistent 4D reconstruction. Left: rendered dynamic scene from our model for illustration alongside four sampled RGB inputs. Right: novel view synthesis at two extreme novel viewpoints. Please refer to supplementary video for clearer visual comparison.
• Figure 2: Overview of the proposed USplat4D. We first estimate time-varying uncertainty for each Gaussian (Section \ref{['sec:uncertainty']}). We then leverage these uncertainties to select reliable Gaussians as key nodes, while others are treated as non-key nodes for graph construction (Section \ref{['sec:graph_model']}). Finally, we optimize the spatio-temporal graph with uncertainty-weighted losses, yielding consistent 4D Gaussians (Section \ref{['sec:optimization']}). The right column shows that our approach significantly improves novel view renderings compared to vanilla optimization.
• Figure 3: Qualitative results on validation views of the DyCheck datasetgao2022monocular. We show comparisons with two strong baselines, SoM som2025 and MoSca lei2025mosca. USplat4D improves visual quality and better preserves geometry (e.g., arms in "Spin", hands in "Space-out") and pose (e.g., "Windmill"). Please zoom in for details. See supplementary for more examples and other baselines.
• Figure 4: Qualitative results on extreme novel views from DyCheck. Unlike Figure \ref{['fig:val_iphone']}, these manually sampled extreme views (red cameras) lack ground-truth. USplat4D better preserves fine structures (e.g., the dog's head in "haru-sit") and occluded regions (e.g., the hand in "spin") under large viewpoint shifts.
• Figure 5: Qualitative results on extreme novel views from DAVIS. For each case, we show an input-view rendering and compare the baseline (SoM som2025 or MoSca lei2025mosca) with our USplat4D on an extreme novel view (red). USplat4D yields clearer reconstructions under challenging conditions.