Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis

TL;DR

The work tackles memory- and bandwidth-heavy 3D Gaussian splatting for novel view synthesis by introducing a compression pipeline that uses sensitivity-aware vector clustering, quantization-aware training, and entropy encoding. It achieves up to $31\times$ memory reduction while preserving visual fidelity (approx. $0.23$ dB PSNR loss) and delivers up to $4\times$ faster rendering on hardware rasterization. The approach enables efficient streaming and rendering on low-power devices, with a GPU rasterizer-based pipeline that remains compatible with differentiable Gaussian splatting. Across multiple benchmark datasets, the method demonstrates robust performance and shows specific contributions from color and shape parameter compression, QA finetuning, and entropy-based encoding. The work opens avenues for mobile VR/AR and networked applications by reducing memory bandwidth and enabling real-time rendering of compressed 3D radiance fields.

Abstract

Recently, high-fidelity scene reconstruction with an optimized 3D Gaussian splat representation has been introduced for novel view synthesis from sparse image sets. Making such representations suitable for applications like network streaming and rendering on low-power devices requires significantly reduced memory consumption as well as improved rendering efficiency. We propose a compressed 3D Gaussian splat representation that utilizes sensitivity-aware vector clustering with quantization-aware training to compress directional colors and Gaussian parameters. The learned codebooks have low bitrates and achieve a compression rate of up to $31\times$ on real-world scenes with only minimal degradation of visual quality. We demonstrate that the compressed splat representation can be efficiently rendered with hardware rasterization on lightweight GPUs at up to $4\times$ higher framerates than reported via an optimized GPU compute pipeline. Extensive experiments across multiple datasets demonstrate the robustness and rendering speed of the proposed approach.

Figures (14)

• Figure 1: Our method achieves a $31\times$ compression at indiscernible loss in image quality and greatly improves rendering speed compared to kerbl_3d_2023. Framerates in grey and white, respectively, are taken on NVIDIA's RTX 3070M and RTX A5000 at 1080p resolution.
• Figure 2: Proposed compression pipeline. Input is an optimized 3D Gaussian scene representation. First, a sensitivity measure is computed for the Gaussian parameters, and color and shape information is compressed into separate codebooks using sensitivity-aware and scale-invariant vector clustering. Next, the compressed scene is fine-tuned on the training images to recover lost information. Finally, the Gaussians are sorted in Morton order and further compressed using entropy and run-length encoding. The shown scene is from barron_mip-nerf_2022.
• Figure 3: Histograms of maximum sensitivity to changes of SH coefficients for different scenes. Only SH coefficients of a tiny fraction of all Gaussians strongly affect image quality.
• Figure 4: 3D Gaussian splatting of synthetic scenes mildenhall_nerf_2021. Uncompressed (Baseline) vs. compressed scene.
• Figure 5: Ground truth images from the test set, results of Kerbl et al. kerbl_3d_2023 (Baseline), results using the compressed representation (Ours).