PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting

TL;DR

This work addresses the storage and memory bottlenecks of large 3D-Gaussian Splatting models by introducing PUP 3D-GS, a post-hoc, Hessian-based sensitivity-pruning pipeline that ranks Gaussians via a Fisher-information-derived score over spatial parameters and applies a multi-round prune-refine strategy. A per-Gaussian block-diagonal Fisher approximation, restricted to the spatial mean $x_i$ and scale $s_i$, guides selective pruning, while patch-wise computations keep the approach computationally feasible. Empirically, pruning up to $90\%$ of Gaussians with two pruning rounds yields $3.56\times$ faster rendering and favorable image-quality metrics on Mip-NeRF 360, Tanks & Temples, and Deep Blending scenes, outperforming prior heuristic methods like LightGaussian in most datasets. The pipeline is orthogonal to additional compression techniques (e.g., Vectree Quantization) and can be adopted without retraining, enabling more scalable and efficient real-time rendering for complex 3D scenes.

Abstract

Recent advances in novel view synthesis have enabled real-time rendering speeds with high reconstruction accuracy. 3D Gaussian Splatting (3D-GS), a foundational point-based parametric 3D scene representation, models scenes as large sets of 3D Gaussians. However, complex scenes can consist of millions of Gaussians, resulting in high storage and memory requirements that limit the viability of 3D-GS on devices with limited resources. Current techniques for compressing these pretrained models by pruning Gaussians rely on combining heuristics to determine which Gaussians to remove. At high compression ratios, these pruned scenes suffer from heavy degradation of visual fidelity and loss of foreground details. In this paper, we propose a principled sensitivity pruning score that preserves visual fidelity and foreground details at significantly higher compression ratios than existing approaches. It is computed as a second-order approximation of the reconstruction error on the training views with respect to the spatial parameters of each Gaussian. Additionally, we propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model without changing its training pipeline. After pruning 90% of Gaussians, a substantially higher percentage than previous methods, our PUP 3D-GS pipeline increases average rendering speed by 3.56$\times$ while retaining more salient foreground information and achieving higher image quality metrics than existing techniques on scenes from the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets.

Figures (11)

• Figure 1: Spatial uncertainty arises from limited views because there are multiple possible 3D Gaussian locations that map to the same projected Gaussian in pixel space (a). This is reduced when multiple cameras observe previously unconstrained Gaussians (b).
• Figure 2: The average metrics over all scenes in the Mip-NeRF 360 dataset after pruning with different percentages in our two-round pipeline. PSNR and SSIM decrease and LPIPS and FPS increase with more pruning. Per-round percentages are selected in $5\%$ intervals and the model is fine-tuned for $5,000$ iterations in each round. The red dots at $(80\%,50\%)$ represent our percentages for $90\%$ compression.
• Figure 3: Visual comparison after two rounds of prune-refine using our and LightGaussian's methods. Top: bonsai from Mip-NeRF 360. Middle: room from Mip-NeRF 360. Bottom: train from Tanks & Temples. Additional visualizations are presented in Appendix \ref{['sec:appendix:scene_visuals']}.
• Figure 4: Histograms of the distribution of Gaussians over the log of their volumes for the bonsai, room, and train scenes after two rounds of prune-refine. PUP 3D-GS retains smaller Gaussians than LightGaussian, consistent with its higher rendering speed and visual fidelity.
• Figure 5: Summary results on the MipNeRF-360 bicycle scene for $2\times$ and $3\times$ prune-refine rounds. We plot the mean and standard deviation of $2\times$ for all cumulative pruning percents in Figure \ref{['fig:mipnerf_heatmap']}, computed via cubic interpolation, then do the same for $3\times$ with $10\%$ pruning intervals. The dotted red line denotes our target $90\%$ cumulative percentage; per-round results are ablated in Figure \ref{['fig:90_plot']}.