Nix and Fix: Targeting 1000x Compression of 3D Gaussian Splatting with Diffusion Models

TL;DR

This work addresses the storage and bandwidth burden of 3D Gaussian Splatting (3DGS) representations in real-time rendering. It introduces NiFi, an artifact-aware diffusion-distillation framework that performs one-step restoration by projecting degraded 3DGS renders to an intermediate diffusion state $t_0$ using low-rank adapters $\phi^-$ and $\phi^+$ and distribution matching with KL divergence, guided by perceptual losses. NiFi achieves state-of-the-art perceptual restoration at extremely low rates, delivering up to $\sim$927x bitrate reduction and approaching $1000\times$ rate improvement while maintaining perceptual quality close to non-compressed baselines (down to $0.110$ MB). This approach demonstrates the potential of diffusion-based restoration for 3D graphics and sets a new benchmark for handling 3DGS artifacts in bandwidth-constrained settings.

Abstract

3D Gaussian Splatting (3DGS) revolutionized novel view rendering. Instead of inferring from dense spatial points, as implicit representations do, 3DGS uses sparse Gaussians. This enables real-time performance but increases space requirements, hindering applications such as immersive communication. 3DGS compression emerged as a field aimed at alleviating this issue. While impressive progress has been made, at low rates, compression introduces artifacts that degrade visual quality significantly. We introduce NiFi, a method for extreme 3DGS compression through restoration via artifact-aware, diffusion-based one-step distillation. We show that our method achieves state-of-the-art perceptual quality at extremely low rates, down to 0.1 MB, and towards 1000x rate improvement over 3DGS at comparable perceptual performance. The code will be open-sourced upon acceptance.

Figures (3)

• Figure 1: Artifacts resulting from 3DGS compression at different rates. Rate control is achieved through pruning ($\color{red}\bm{\times}$), quantization, and entropy coding. Notice the loss of geometry, texture, and radiance.
• Figure 2: The overall pipeline of NiFi. We create a 3DGS restoration dataset of degraded $\tilde{I}$ and high-quality $I$ frames via Artifact Synthesis, through pruning, quantization, and entropy coding at three rates. $\tilde{I}$ is mapped to the latent space and to an intermediate step $t_0$ in the diffusion trajectory. $\hat{I}$ is obtained in one step via adapter $\phi^-$ extending a frozen diffusion backbone $\epsilon_\theta$ for Artifact Restoration. $\phi^-$ is trained with Restoring Distribution Matching through the critic adapter $\phi^+$ and Perceptual Matching between $\hat{I}$ and $I$. Only Artifact Restoration is performed during inference.
• Figure 3: Qualitative results of bicycle, kitchen, truck, and playroom scenes. Backgrounds are compressed 3DGS renders with overlayed restoration results compared within the highlighted areas. The additional highlighted area in the bicycle scene shows the overemphasized high-frequency components that are introduced by our method.