Compact 3D Gaussian Splatting for Static and Dynamic Radiance Fields

TL;DR

This work introduces Compact 3D Gaussian Splatting (c3dgs), a memory-efficient framework for static and dynamic radiance fields that achieves real-time rendering without sacrificing quality. It achieves this through three core innovations: learnable Gaussian masking to reduce the number of Gaussians, a grid-based neural field to compactly represent view-dependent color, and residual vector quantization with codebooks to compactly encode geometry and temporal attributes. End-to-end training combines rendering loss with masking and codebook losses, and post-processing with quantization and entropy coding yields substantial storage reductions (over 25× static, over 12× dynamic). Extensive experiments on real and synthetic datasets show strong performance, fast rendering, and significant parameter efficiency, establishing a practical approach for scalable 3D scene representation in both static and dynamic contexts.

Abstract

3D Gaussian splatting (3DGS) has recently emerged as an alternative representation that leverages a 3D Gaussian-based representation and introduces an approximated volumetric rendering, achieving very fast rendering speed and promising image quality. Furthermore, subsequent studies have successfully extended 3DGS to dynamic 3D scenes, demonstrating its wide range of applications. However, a significant drawback arises as 3DGS and its following methods entail a substantial number of Gaussians to maintain the high fidelity of the rendered images, which requires a large amount of memory and storage. To address this critical issue, we place a specific emphasis on two key objectives: reducing the number of Gaussian points without sacrificing performance and compressing the Gaussian attributes, such as view-dependent color and covariance. To this end, we propose a learnable mask strategy that significantly reduces the number of Gaussians while preserving high performance. In addition, we propose a compact but effective representation of view-dependent color by employing a grid-based neural field rather than relying on spherical harmonics. Finally, we learn codebooks to compactly represent the geometric and temporal attributes by residual vector quantization. With model compression techniques such as quantization and entropy coding, we consistently show over 25x reduced storage and enhanced rendering speed compared to 3DGS for static scenes, while maintaining the quality of the scene representation. For dynamic scenes, our approach achieves more than 12x storage efficiency and retains a high-quality reconstruction compared to the existing state-of-the-art methods. Our work provides a comprehensive framework for 3D scene representation, achieving high performance, fast training, compactness, and real-time rendering. Our project page is available at https://maincold2.github.io/c3dgs/.

Figures (9)

• Figure 1: Our method achieves reduced storage and faster rendering speed while maintaining high-quality renderings of 3DGS 3dgs. The core idea is to effectively remove the redundant Gaussians that do not significantly contribute to the overall performance (the sparser distribution of Gaussian points and reduced ellipsoid redundancy shown in the figure). We also introduce a more compact representation of Gaussian attributes, resulting in markedly improved storage efficiency and rendering speed.
• Figure 2: The detailed architecture of our proposed compact 3D Gaussian.
• Figure 3: The number of Gaussians during the training (Bonsai scene). '# Gaussians' denotes the number of Gaussians.
• Figure 4: The detailed process of R-VQ to represent the scale and rotation of Gaussians. In the first stage, the scale and rotation vectors are compared to codes in each codebook, with the closest code identified as the result. In the next stage, the residual between the original vector and the first stage's result is compared with another codebook. This process is repeated up to the final stage, as a result, the selected indices and the codebook from each stage collectively represent the original vector.
• Figure 5: The detailed architecture of our proposed compact Gaussian representation for dynamic scenes.