Leveraging Learned Image Prior for 3D Gaussian Compression

TL;DR

This work tackles the high storage cost of 3D Gaussian Splatting (3DGS) by introducing a restoration-based compression framework that leverages learned image priors. It restores degraded renderings from initially compressed Gaussians using a dedicated image restoration network and coarse residual side information, then refines the compressed Gaussians to recover rendering quality with minimal storage increase. The approach, compatible with various initial compression methods, achieves state-of-the-art rate-distortion performance on standard 3DGS benchmarks while substantially reducing storage compared to uncompressed representations. The framework enables broader, practical deployment of 3DGS by balancing compression efficiency with high-fidelity novel-view rendering, albeit with added computational overhead and a potential path toward end-to-end optimization.

Abstract

Compression techniques for 3D Gaussian Splatting (3DGS) have recently achieved considerable success in minimizing storage overhead for 3D Gaussians while preserving high rendering quality. Despite the impressive storage reduction, the lack of learned priors restricts further advances in the rate-distortion trade-off for 3DGS compression tasks. To address this, we introduce a novel 3DGS compression framework that leverages the powerful representational capacity of learned image priors to recover compression-induced quality degradation. Built upon initially compressed Gaussians, our restoration network effectively models the compression artifacts in the image space between degraded and original Gaussians. To enhance the rate-distortion performance, we provide coarse rendering residuals into the restoration network as side information. By leveraging the supervision of restored images, the compressed Gaussians are refined, resulting in a highly compact representation with enhanced rendering performance. Our framework is designed to be compatible with existing Gaussian compression methods, making it broadly applicable across different baselines. Extensive experiments validate the effectiveness of our framework, demonstrating superior rate-distortion performance and outperforming the rendering quality of state-of-the-art 3DGS compression methods while requiring substantially less storage.

Figures (7)

• Figure 1: Visualization of compression artifacts in JPEG and Gaussian representation. We visualize the pixel-wise error maps (MSE) induced by each artifact to highlight the degradation.
• Figure 2: Illustration of overall pipeline. Given a set of Gaussians as input, we perform initial compression, followed by an image restoration network that restores the rendered images. Finally, we refine the compressed Gaussians using the restored images to enhance the rendering quality.
• Figure 3: Illustration of the proposed image restoration framework. To train the image restoration network for Gaussian compression, we construct a paired dataset consisting of degraded and original images rendered from Gaussians optimized on real-world 3D scenes. Given a degraded image and the corresponding original image, we first compute their residual. The residual is then quantized and concatenated with the degraded image. This concatenated input is fed into the image restoration network, which is designed to remove Gaussian compression artifacts and recover a high-quality image.
• Figure 4: Rate-distortion curves comparing our method with existing post-compression approaches. We optimize Gaussians following Mini-Splatting fang2024mini and compress them using each post-compression method. Results are averaged over all scenes of Mip-NeRF 360 barron2022mip.
• Figure 5: Ablation study on view-sampling. Results are averaged over all scenes of Mip-NeRF 360 barron2022mip.