Compressing 3D Gaussian Splatting by Noise-Substituted Vector Quantization

TL;DR

<3-5 sentence high-level summary> The paper tackles the large memory footprint of 3D Gaussian Splatting (3DGS) by introducing NSVQ-GS, a noise-substituted vector quantization approach that compresses four Gaussian attributes via learnable codebooks while keeping coordinates and opacity unquantized. By training the codebooks with NSVQ, the method achieves substantial memory reductions (≈45×) with competitive PSNR/SSIM/LPIPS on standard benchmarks and preserves compatibility with popular 3DGS viewers for faster rendering. The approach avoids gradient-collapse pitfalls of straight-through estimators, enabling end-to-end optimization of quantized features. Extensive experiments show NSVQ-GS outperforms state-of-the-art SP-based GS compression baselines at similar storage budgets, validating its practicality for real-time 3D reconstruction and rendering pipelines.

Abstract

3D Gaussian Splatting (3DGS) has demonstrated remarkable effectiveness in 3D reconstruction, achieving high-quality results with real-time radiance field rendering. However, a key challenge is the substantial storage cost: reconstructing a single scene typically requires millions of Gaussian splats, each represented by 59 floating-point parameters, resulting in approximately 1 GB of memory. To address this challenge, we propose a compression method by building separate attribute codebooks and storing only discrete code indices. Specifically, we employ noise-substituted vector quantization technique to jointly train the codebooks and model features, ensuring consistency between gradient descent optimization and parameter discretization. Our method reduces the memory consumption efficiently (around $45\times$) while maintaining competitive reconstruction quality on standard 3D benchmark scenes. Experiments on different codebook sizes show the trade-off between compression ratio and image quality. Furthermore, the trained compressed model remains fully compatible with popular 3DGS viewers and enables faster rendering speed, making it well-suited for practical applications.

Figures (6)

• Figure 1: We reduce the storage requirements by implementing an advanced VQ for 3DGS. It reduces file sizes and accelerates rendering speed, while maintaining high reconstruction quality. The reported frames per second (FPS) metrics were measured using an Nvidia RTX 4070 GPU.
• Figure 2: Overview of the efficient reduction on storage requirement by our NSVQ-GS (16k). A single unit box represents 32 bits. Substituting Gaussian splats with their quantized counterparts and codebooks saves substantial memory consumption.
• Figure 3: Overview of the training process consisting of four phases. During warm-up, the model learns the 3D information by increasing the number of Gaussian splats. The pruning stage reduces the number of Gaussians while maintaining reconstruction performance. Density control is applied only until the end of the pruning phase. In the vector quantization phase, reconstruction quality initially degrades but recovers after sufficient training. Finally, the fine-tuning removes constraints imposed by noise substitution, further refining the final results.
• Figure 4: The comparison of reconstruction quality across baselines. In each sub-figure, the $x$-axis denotes the model size in Megabytes, $y$-axis denotes the metrics on the reconstruction result. Each method comprising different sub-methods is plotted as multiple connected points. Our NSVQ-GS performs best within the category of SP-based methods, whereas ML-based methods (in gray) can boost performance further while losing some of the appealing 3DGS propeties.
• Figure 5: Qualitative comparison between ground truth, CompGS (0.5k), and NSVQ-GS (0.5k) (ours). Our NSVQ-GS captures difficult sharp boundaries and straight lines better compared to CompGS (see, e.g., bicycle). This becomes clearer at stronger compression constraints (low codebook bitrates).