Hybrid 3D-4D Gaussian Splatting for Fast Dynamic Scene Representation

TL;DR

Hybrid 3D-4D Gaussian Splatting (3D-4DGS) targets fast, high-fidelity dynamic scene reconstruction by adaptively representing static regions with 3D Gaussians and reserving 4D Gaussians for dynamic content. The method starts with a full 4D Gaussian representation and iteratively converts temporally invariant Gaussians to 3D, reducing parameters, memory, and training time, while keeping dynamic Gaussians fully 4D to capture motion. Key contributions include a scale-based static/dynamic region identification, a robust 4D-to-3D conversion, and a unified CUDA rasterization pipeline that blends both representations in rendering. Experiments on N3V and Technicolor demonstrate near-state fidelity with significantly faster training (e.g., 12 minutes for 10s N3V clips) and reduced storage, enabling efficient dynamic scene capture at scale.

Abstract

Recent advancements in dynamic 3D scene reconstruction have shown promising results, enabling high-fidelity 3D novel view synthesis with improved temporal consistency. Among these, 4D Gaussian Splatting (4DGS) has emerged as an appealing approach due to its ability to model high-fidelity spatial and temporal variations. However, existing methods suffer from substantial computational and memory overhead due to the redundant allocation of 4D Gaussians to static regions, which can also degrade image quality. In this work, we introduce hybrid 3D-4D Gaussian Splatting (3D-4DGS), a novel framework that adaptively represents static regions with 3D Gaussians while reserving 4D Gaussians for dynamic elements. Our method begins with a fully 4D Gaussian representation and iteratively converts temporally invariant Gaussians into 3D, significantly reducing the number of parameters and improving computational efficiency. Meanwhile, dynamic Gaussians retain their full 4D representation, capturing complex motions with high fidelity. Our approach achieves significantly faster training times compared to baseline 4D Gaussian Splatting methods while maintaining or improving the visual quality.

Figures (11)

• Figure 1: Left: Rendering results on the coffee_martini scene. Right: PSNR vs. training time. The proposed method converges in 12 minutes while maintaining competitive rendering quality. All methods were evaluated under the same machine equipped with the NVIDIA RTX4090 GPU, except for 4D-Rotor GS duan20244d—whose results were estimated from iteration counts since the code is not publicly available.
• Figure 2: Distribution of the t-axis scale for Gaussians in the coffee_martini scene. Most Gaussians cluster at smaller scales, indicating dynamic content, while a minority have larger scales that suggest static regions.
• Figure 3: Overview of our hybrid 3D--4D Gaussian Splatting framework. (a) 4D Gaussians are optimized over time, and those exceeding a temporal scale threshold ($\tau$) are converted into 3D Gaussians. (b) Both 3D and 4D Gaussians are projected into screen space, assigned tile and depth keys, and sorted for rasterization. The rendered image is generated by blending static (3D) and dynamic (4D) Gaussians.
• Figure 4: Qualitative comparison on the N3V dataset. While most methods yield comparable results, our approach can preserve subtle motion cues and slightly more consistent colors in some challenging regions. Zoom in for best viewing.
• Figure 5: Visual comparison of different scale thresholds $\tau$.