4C4D: 4 Camera 4D Gaussian Splatting

Abstract

This paper tackles the challenge of recovering 4D dynamic scenes from videos captured by as few as four portable cameras. Learning to model scene dynamics for temporally consistent novel-view rendering is a foundational task in computer graphics, where previous works often require dense multi-view captures using camera arrays of dozens or even hundreds of views. We propose \textbf{4C4D}, a novel framework that enables high-fidelity 4D Gaussian Splatting from video captures of extremely sparse cameras. Our key insight lies that the geometric learning under sparse settings is substantially more difficult than modeling appearance. Driven by this observation, we introduce a Neural Decaying Function on Gaussian opacities for enhancing the geometric modeling capability of 4D Gaussians. This design mitigates the inherent imbalance between geometry and appearance modeling in 4DGS by encouraging the 4DGS gradients to focus more on geometric learning. Extensive experiments across sparse-view datasets with varying camera overlaps show that 4C4D achieves superior performance over prior art. Project page at: https://junshengzhou.github.io/4C4D.

Figures (11)

• Figure 1: Left: Our capture system records dynamic scenes using only four cameras. Right: Dynamic novel views rendered from the reconstructed 4D representations produced by Ex4DGS, 4DGS, and our method.
• Figure 2: Overview of 4C4D. We introduce a Neural Decaying Function $f_\theta$, implemented as a lightweight neural network, to adaptively control the opacity decay of Gaussians. Given key Gaussian attributes as input, $f_\theta$ predicts a factor that controls the decay of Gaussian opacities. During training, both the Neural Decaying Function and the 4D Gaussians are jointly optimized via gradient backpropagation under a photometric rendering loss.
• Figure 3: Rendered RGB and depth results from 4DGS trained with and without our Neural Decaying Function, evaluated on both training and unseen views. Without neural decaying, 4DGS can overfit the training views but fails to generalize to unseen viewpoints, due to the learned poor geometry.
• Figure 4: Visual comparisons under Neural3DV dataset.
• Figure 5: Visual comparisons under ENeRF-Outdoor dataset.