Image-Conditioned 3D Gaussian Splat Quantization

TL;DR

This work tackles the challenge of archiving large-scale 3D Gaussian Splatting (3DGS) scenes by introducing ICGS-Quantizer, which jointly quantizes Gaussians and attributes using shared codebooks learned across many scenes. It further enables adaptability to post-archival scene changes by conditioning decoding on one or more current images, using a coarse-to-fine fusion of image features with the latent scene representation. The approach achieves kilobyte-scale storage while maintaining high visual fidelity and delivers strong performance in both 3D scene compression and 3D scene updating, outperforming state-of-the-art methods and approaching upper-bound references with far less storage. The method's joint encoding, image-conditioned decoding, and shared-codebook design offer practical benefits for large collections and storage-constrained deployments, with public code and data anticipated to accelerate adoption.

Abstract

3D Gaussian Splatting (3DGS) has attracted considerable attention for enabling high-quality real-time rendering. Although 3DGS compression methods have been proposed for deployment on storage-constrained devices, two limitations hinder archival use: (1) they compress medium-scale scenes only to the megabyte range, which remains impractical for large-scale scenes or extensive scene collections; and (2) they lack mechanisms to accommodate scene changes after long-term archival. To address these limitations, we propose an Image-Conditioned Gaussian Splat Quantizer (ICGS-Quantizer) that substantially enhances compression efficiency and provides adaptability to scene changes after archiving. ICGS-Quantizer improves quantization efficiency by jointly exploiting inter-Gaussian and inter-attribute correlations and by using shared codebooks across all training scenes, which are then fixed and applied to previously unseen test scenes, eliminating the overhead of per-scene codebooks. This approach effectively reduces the storage requirements for 3DGS to the kilobyte range while preserving visual fidelity. To enable adaptability to post-archival scene changes, ICGS-Quantizer conditions scene decoding on images captured at decoding time. The encoding, quantization, and decoding processes are trained jointly, ensuring that the codes, which are quantized representations of the scene, are effective for conditional decoding. We evaluate ICGS-Quantizer on 3D scene compression and 3D scene updating. Experimental results show that ICGS-Quantizer consistently outperforms state-of-the-art methods in compression efficiency and adaptability to scene changes. Our code, model, and data will be publicly available on GitHub.

Figures (6)

• Figure 1: Image-conditioned scene quantization. (a) At time $t_0$, the scene is encoded and quantized as discrete codes. However, after a prolonged period, the scene may have changed. (b) Conventional methods decode the scene from the codes, but can only recover the original scene at time $t_0{<}t$. (c) Our method decodes the scene from the codes conditioned on its image(s) captured at time $t$, adapting the scene to its current illumination and appearance. We recommend watching our videos to observe the dynamic results.
• Figure 2: An overview of our proposed ICGS-Quantizer. (a) Quantization module: geometry and texture are decoupled and quantized independently. (b) Joint quantization of 3D Gaussians and their associated attributes. (c) Image-conditioned decoding of 3D Gaussians. Auxiliary neural network branches and training objectives are omitted here for simplicity.
• Figure 3: Qualitative results of 3D scene compression. Zoom in to examine details. "Large", "Medium", and "Small" indicate the number of Gaussians, balancing quality and storage.
• Figure 4: Qualitative results of 3D scene updating. For the baseline methods, "Large" and "Small" refer to configurations utilizing varying numbers of Gaussians, balancing quality and storage.
• Figure 5: Experimental results of ICGS-Quantizer for 3D scene compression ("compression") and 3D scene updating ("updating") using varying numbers of input images. Since the conditioning images are optional, "0" indicates no image is used.