Optimized 3D Gaussian Splatting using Coarse-to-Fine Image Frequency Modulation

TL;DR

3D Gaussian Splatting enables real-time rendering but suffers from high GPU memory and storage demands. Opti3DGS introduces a coarse-to-fine optimization driven by image frequency modulation, starting with large coarse Gaussians and progressively refining through filtered training images. It reduces Gaussian primitives by up to 62%, lowers training memory by up to 40%, and speeds up optimization by up to 20% while maintaining or improving visual quality, and acts as a plug-in for many 3DGS methods. The approach naturally yields a level-of-detail representation and can improve accessibility of 3DGS on consumer hardware. It integrates with existing compression or ADC schemes and provides a simple, parameter-light improvement path.

Abstract

The field of Novel View Synthesis has been revolutionized by 3D Gaussian Splatting (3DGS), which enables high-quality scene reconstruction that can be rendered in real-time. 3DGS-based techniques typically suffer from high GPU memory and disk storage requirements which limits their practical application on consumer-grade devices. We propose Opti3DGS, a novel frequency-modulated coarse-to-fine optimization framework that aims to minimize the number of Gaussian primitives used to represent a scene, thus reducing memory and storage demands. Opti3DGS leverages image frequency modulation, initially enforcing a coarse scene representation and progressively refining it by modulating frequency details in the training images. On the baseline 3DGS, we demonstrate an average reduction of 62% in Gaussians, a 40% reduction in the training GPU memory requirements and a 20% reduction in optimization time without sacrificing the visual quality. Furthermore, we show that our method integrates seamlessly with many 3DGS-based techniques, consistently reducing the number of Gaussian primitives while maintaining, and often improving, visual quality. Additionally, Opti3DGS inherently produces a level-of-detail scene representation at no extra cost, a natural byproduct of the optimization pipeline. Results and code will be made publicly available.

Figures (8)

• Figure 1: We start with uniform distribution of larger Gaussians as compared to 3DGS which starts heavy densification from the very start of optimization thus complicating the loss landscape. The zoomed views show efficient use of the initial sparse pointcloud by our approach.
• Figure 2: Method Overview: We start the optimization with very large and coarse Gaussians which are then densified based on the differentiable rendering signal. Specifically we control the frequency spectrum of the training images, gradually increasing the threshold for the low pass filtering.
• Figure 3: Visualization of different decay rates reported in our work. The vertical axis shows the current kernel size for the filtering algorithm for any given iteration. Note we stop all frequency modulation after 12,000 iterations and pass full sized images for the remaining 18,000 iterations.
• Figure 4: We reduce the GPU memory required during optimization due to a reduced Gaussian primitive load, which also reduces the total optimization time. These results are for 3DGSkerbl20233d across the bicycle scenes and are represntive of all scenes tested.
• Figure 5: Pixel level comparison of error maps against the ground truth image. Our approach maintains similar quantitative scores while using fewer Gaussians and results in a decreased pixel wise error. Note the absence of noise like bright points in our results, due to the removal of tiny redundant Gaussians.