Temporally Compressed 3D Gaussian Splatting for Dynamic Scenes

TL;DR

Dynamic scene reconstruction demands real-time performance with limited memory. The authors introduce Temporally Compressed 3D Gaussian Splatting (TC3DGS), which compresses dynamic 3D Gaussian representations through temporal pruning, gradient-aware mixed-precision quantization, and trajectory keypoint interpolation. TC3DGS achieves up to 67x compression with minimal visual degradation and competitive rendering speed across Panoptic Sports, Neural 3D Video, and Technicolor datasets, addressing long-sequence and complex-motion challenges. The work provides thorough ablations, implementation details, and discusses limitations and future directions for storage-efficient dynamic scene reconstruction.

Abstract

Recent advancements in high-fidelity dynamic scene reconstruction have leveraged dynamic 3D Gaussians and 4D Gaussian Splatting for realistic scene representation. However, to make these methods viable for real-time applications such as AR/VR, gaming, and rendering on low-power devices, substantial reductions in memory usage and improvements in rendering efficiency are required. While many state-of-the-art methods prioritize lightweight implementations, they struggle in handling {scenes with complex motions or long sequences}. In this work, we introduce Temporally Compressed 3D Gaussian Splatting (TC3DGS), a novel technique designed specifically to effectively compress dynamic 3D Gaussian representations. TC3DGS selectively prunes Gaussians based on their temporal relevance and employs gradient-aware mixed-precision quantization to dynamically compress Gaussian parameters. In addition, TC3DGS exploits an adapted version of the Ramer-Douglas-Peucker algorithm to further reduce storage by interpolating Gaussian trajectories across frames. Our experiments on multiple datasets demonstrate that TC3DGS achieves up to 67$\times$ compression with minimal or no degradation in visual quality. More results and videos are provided in the supplementary. Project Page: https://ahmad-jarrar.github.io/tc-3dgs/

Figures (16)

• Figure 1: Comparative Evaluation on Panoptic Dataset Joo_2019 Recent lightweight state-of-the-art methods (STG stg, and 4D-Gaussian 4dgs) struggles to reconstruct scenes in complex environments accurately. In contrast, our compression strategy effectively captures and maintains high fidelity.
• Figure 2: Overview of our Temporally Compressed 3D Gaussian Splatting for Dynamic Scenese (TC3DGS) method. Our approach involves a temporally consistent masking strategy to select relevant 3D Gaussians across frames. The masked Gaussians are then pruned and quantized using a gradient-based, parameter-aware bit-precision quantization scheme.
• Figure 3: Mask Consistency. (Left) Mask values of Gaussians trained independently for each time frame. (Right) Mask values trained with our loss.
• Figure 4: Keypoint Interpolation. In this example, we represent a position across 150 frames with only 5, 4 and 6 keypoints for $x$, $y$ and $z$, respectively, with only $0.038$ MSE. By comparison, uniformly sampling 7 keypoints increases storage and increases error to $0.089$ MSE.
• Figure 5: Our method lies on the Pareto frontier, achieving competitive performance with significantly smaller model size.