SV-GS: Sparse View 4D Reconstruction with Skeleton-Driven Gaussian Splatting

TL;DR

The paper tackles 4D articulated reconstruction from sparse temporal observations by introducing SV-GS, a skeleton-driven deformation framework built on canonical 3D Gaussian Splatting primitives. A time-dependent joint pose predictor drives coarse articulation while a skinning and a detail refinement module preserve geometry, enabling smooth motion interpolation despite sparse supervision. SV-GS achieves up to a 34% PSNR improvement over state-of-the-art methods under sparse views and matches dense monocular performance on real data using far fewer frames; crucially, a diffusion-based generative prior can replace multi-view initialization to further improve practicality. This work demonstrates that combining skeletal constraints with a learnable deformation field and priors broadens dynamic reconstruction to real-world, sparsely observed scenarios.

Abstract

Reconstructing a dynamic target moving over a large area is challenging. Standard approaches for dynamic object reconstruction require dense coverage in both the viewing space and the temporal dimension, typically relying on multi-view videos captured at each time step. However, such setups are only possible in constrained environments. In real-world scenarios, observations are often sparse over time and captured sparsely from diverse viewpoints (e.g., from security cameras), making dynamic reconstruction highly ill-posed. We present SV-GS, a framework that simultaneously estimates a deformation model and the object's motion over time under sparse observations. To initialize SV-GS, we leverage a rough skeleton graph and an initial static reconstruction as inputs to guide motion estimation. (Later, we show that this input requirement can be relaxed.) Our method optimizes a skeleton-driven deformation field composed of a coarse skeleton joint pose estimator and a module for fine-grained deformations. By making only the joint pose estimator time-dependent, our model enables smooth motion interpolation while preserving learned geometric details. Experiments on synthetic datasets show that our method outperforms existing approaches under sparse observations by up to 34% in PSNR, and achieves comparable performance to dense monocular video methods on real-world datasets despite using significantly fewer frames. Moreover, we demonstrate that the input initial static reconstruction can be replaced by a diffusion-based generative prior, making our method more practical for real-world scenarios.

Figures (10)

• Figure 1: We study the problem of 4D reconstruction from sparse observations. Our method takes the following as input: (a) A set of posed RGB images of an articulated target, captured at sparse time steps (up to 20x fewer than existing methods) from arbitrary viewpoints; (b) An annotated skeleton graph only at the first frame; (c) An initial static 3D reconstruction, derived either from multi-view images or a pre-trained image-to-3D diffusion model. Our goal is to produce a continuous 4D reconstruction of the dynamic target.
• Figure 2: Comparison of input configurations across dynamic reconstruction methods. Multi-view and monocular video methods assume small viewpoint changes and dense temporal observations, whereas our method handles sparse temporal observations with large viewpoint variations. Generative methods attempt to synthesize the full motion from a static state.
• Figure 3: Given canonical 3D Gaussians and an input skeleton, SV-GS first predicts time-dependent joint poses, regularized with $\mathcal{L}_{motion}$ for temporal smoothness. With the predicted skeleton poses, the canonical Gaussians are then transformed via Linear Blend Skinning using learnable per-bone radii and a skinning correction field. Finally, a detail deformation field refines the transformed Gaussians. All parameters are optimized by minimizing the perceptual loss between the rendered and observed images.
• Figure 4: Qualitative results on the D-NeRF dataset pumarola2021d downsampled at 0.1 intervals, yielding 11 frames per motion sequence (up to $20\times$ fewer than the original). We compare our method with SOTA methods including 4DGS Wu_2024_CVPR, SK-GS wan2024template, and RigGS yao2025riggs. Additionally, we modify RigGS yao2025riggs to take in the same skeleton input as ours. Despite all methods being initialized with the same multi-view images at $t=0$, existing methods produce noisy deformations and fail to preserve object structure given only sparse temporal observations.
• Figure 5: We show all input views from the downsampled dataset (up to $20\times$ fewer frames than the original), illustrating the challenges of establishing correspondences under sparse observations, large viewpoint changes, and self-occlusions.