Spectrally Pruned Gaussian Fields with Neural Compensation

TL;DR

SUNDAE tackles the memory bottleneck of 3D Gaussian splatting by introducing spectral pruning on a Gaussian primitive graph and a lightweight neural compensation head. The graph-based pruning, grounded in graph signal processing, selectively removes redundant primitives while preserving essential signals; the neural compensation module then mixes remaining primitives to recover rendering quality. Across multiple datasets, SUNDAE achieves substantially lower memory footprints (e.g., 104 MB) with competitive or superior PSNR/SSIM/LPIPS and real-time-like FPS compared to denser baselines. The work demonstrates that explicit modeling of primitive relationships enables strong memory efficiency and practical rendering performance on edge devices and mobile platforms. The proposed continuous pruning option offers an alternative training strategy, though the training-then-pruning pipeline with a 50% pruning ratio delivers robust results with clear efficiency gains.

Abstract

Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered attention for its fast rendering speed and high rendering quality. However, this comes with high memory consumption, e.g., a well-trained Gaussian field may utilize three million Gaussian primitives and over 700 MB of memory. We credit this high memory footprint to the lack of consideration for the relationship between primitives. In this paper, we propose a memory-efficient Gaussian field named SUNDAE with spectral pruning and neural compensation. On one hand, we construct a graph on the set of Gaussian primitives to model their relationship and design a spectral down-sampling module to prune out primitives while preserving desired signals. On the other hand, to compensate for the quality loss of pruning Gaussians, we exploit a lightweight neural network head to mix splatted features, which effectively compensates for quality losses while capturing the relationship between primitives in its weights. We demonstrate the performance of SUNDAE with extensive results. For example, SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB memory, on the Mip-NeRF360 dataset. Codes are publicly available at https://runyiyang.github.io/projects/SUNDAE/.

Figures (6)

• Figure 1: The left panel shows vanilla 3D Gaussian splatting, which requires a large amount of storage as it does not capture the relationship between primitives. The middle panel shows our spectral pruning technique that is based upon the relationship between 3D Gaussians. The right panel shows that the neural compensation head exploits the relationship between 2D feature splatting results to improve rendering.
• Figure 2: (a) Pipeline: Our proposed method warms up a 3D Gaussian field firstly, followed by a Graph-based pruning strategy to down-sample the Gaussian primitives, and a convolutional neural network to compensate the losses caused by pruning. (b) Graph-based Pruning: A graph based on the spatial relationship between the Gaussian primitives, is utilized for pruning post warm-up. Employing a band-limited graph filter, this process facilitates the extraction of fine details from high-frequency components, alongside capturing general features from low-frequency parts, thereby enabling a comprehensive and efficient representation of the entire scene.
• Figure 3: The pruned Gaussian primitive centers with different pruning strategy.
• Figure 4: Qualitative results of our novel view synthesis. The scenes are, from the top down: Bicycle, Counter, Garden, Kitchen, Room and Stump from the Mip-NeRF360 dataset; DrJohnson, Playroom from the Deep Blending dataset and Train from Tanks&Temples. Non-obvious differences in quality highlighted by arrows/insets.
• Figure 5: Visualization with and without neural compensation.