FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization

TL;DR

FreGS tackles over-reconstruction in real-time 3D Gaussian splatting by introducing progressive frequency regularization that aligns rendered and ground-truth spectra in Fourier space. It enforces amplitude and phase consistency and employs frequency annealing to progressively incorporate low- to high-frequency information during Gaussian densification, enabling coarse-to-fine improvements. The method demonstrates consistent gains over 3D-GS and competitive state-of-the-art results across multiple benchmarks (Mip-NeRF360, Tanks&Temples, Deep Blending), with ablations confirming the effectiveness of both spectral regularization and the annealing schedule. Overall, FreGS provides a spectral perspective on Gaussian splatting that improves view synthesis quality while preserving real-time rendering capabilities and a controlled number of Gaussians. The approach has practical implications for scalable, high-fidelity neural rendering in diverse indoor and outdoor scenes, where spectral guidance can mitigate over-reconstruction artifacts.

Abstract

3D Gaussian splatting has achieved very impressive performance in real-time novel view synthesis. However, it often suffers from over-reconstruction during Gaussian densification where high-variance image regions are covered by a few large Gaussians only, leading to blur and artifacts in the rendered images. We design a progressive frequency regularization (FreGS) technique to tackle the over-reconstruction issue within the frequency space. Specifically, FreGS performs coarse-to-fine Gaussian densification by exploiting low-to-high frequency components that can be easily extracted with low-pass and high-pass filters in the Fourier space. By minimizing the discrepancy between the frequency spectrum of the rendered image and the corresponding ground truth, it achieves high-quality Gaussian densification and alleviates the over-reconstruction of Gaussian splatting effectively. Experiments over multiple widely adopted benchmarks (e.g., Mip-NeRF360, Tanks-and-Temples and Deep Blending) show that FreGS achieves superior novel view synthesis and outperforms the state-of-the-art consistently.

Figures (6)

• Figure 1: The proposed FreGS mitigates the over-reconstruction of Gaussian densification and renders images with much less blur and artifact as compared with the 3D Gaussian splatting (3D-GS). For the two sample images from Mip-NeRF360 barron2022mip, (a) and (b) show the Rendered Image and the Gaussian Visualization of the highlighted regions, as well as the Spectra of over-reconstructed areas in the rendered image by 3D-GS and corresponding areas in FreGS. The Gaussian Visualization shows how the learnt rasterized 3D Gaussians compose images (all Gaussians are rasterized with full opacity). The Spectra are generated via image Fourier transformation, where the colour changes from blue to green as the spectrum amplitude changes from small to large.
• Figure 2: Overview of the proposed FreGS. 3D Gaussians are initialized by structure-from-motion. After splatting 3D Gaussians, we can obtain 2D Gaussians and then leverage standard $\alpha$-blending for rendering. Frequency spectra $\hat{F}$ and $F$ are generated by applying Fourier transform to rendered image $\hat{I}$ and ground truth $I$, respectively. Frequency regularization is achieved by regularizing discrepancies of amplitude $| F(u, v) |$ and phase$\angle F(u, v)$ in Fourier space. A novel frequency annealing technique is designed to achieve progressive frequency regularization. With low-pass filter $H_l$ and dynamic high-pass filter $H_h$, low-to-high frequency components are progressively leveraged to perform coarse-to-fine Gaussian densification. Note, the progressive frequency regularization is complementary to the pixel-wise loss between $\hat{I}$ and $I$. The red dashed line highlights the regularization process for Gaussian densification.
• Figure 3: Average pixel gradients within over-reconstruction regions and well-reconstruction regions in scene 'Bicycle’. The curve with circle (w/o frequency regularization (FR)) represents the method equivalent to 3D-GS kerbl20233d, which utilizes pixel-wise L1 loss in the spatial domain only. As the Gaussian densification is terminated after the 15000$th$ iteration as in 3D-GS, we only show comparisons before the 15000$th$ iteration. It can be observed that the frequency regularization can increase the pixel gradient within over-reconstruction regions significantly. Thus, compared with L1 loss, the frequency regularization shows superior capability in revealing the over-reconstruction region.
• Figure 4: The comparison of different frequency regularizations. The naive frequency regularization directly employs amplitude and phase discrepancies without distinguishing between low and high frequency. The proposed progressive frequency regularization introduces frequency annealing technique to achieve low-to-high frequency regularization for coarse-to-fine Gaussian densification. It can be observed that the proposed progressive frequency regularization can achieve finer Gaussian densification and superior novel view synthesis. Zoom in for best view.
• Figure 5: Qualitative comparisons of FreGS with three state-of-the-art methods in novel view synthesis. Note that for fair comparison as well as trade-off balance between synthesis quality and memory consumption, we train FreGS with similar number of Gaussians as 3D-GS for these datasets (details in Sec.4.2). The comparisons are conducted over multiple indoor and outdoor scenes including 'Garden' and 'Room' from Mip-NeRF360, 'Train' and 'Truck' from Tank&Temple, and 'Drjohnson' from Deep Blending. 'GT' denotes the ground-truth images. FreGS achieves superior image rendering with much less artifacts but more fine details.